<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Kienlt on Medium]]></title>
        <description><![CDATA[Stories by Kienlt on Medium]]></description>
        <link>https://medium.com/@kienlt.qn?source=rss-b77f4e43d676------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/0*6n2u671AlCI2sY2v</url>
            <title>Stories by Kienlt on Medium</title>
            <link>https://medium.com/@kienlt.qn?source=rss-b77f4e43d676------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Tue, 17 Nov 2026 12:51:10 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@kienlt.qn/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[Prepare for NVIDIA Certified Associate: AI Infrastructure and Operations (NCA-AIIO) Exam]]></title>
            <link>https://medium.com/@kienlt.qn/prepare-for-nvidia-certified-associate-ai-infrastructure-and-operations-nca-aiio-exam-a7365e11bb8d?source=rss-b77f4e43d676------2</link>
            <guid isPermaLink="false">https://medium.com/p/a7365e11bb8d</guid>
            <category><![CDATA[nvidia-certification]]></category>
            <category><![CDATA[nvidia]]></category>
            <dc:creator><![CDATA[Kienlt]]></dc:creator>
            <pubDate>Mon, 28 Sep 2026 14:19:37 GMT</pubDate>
            <atom:updated>2026-09-28T14:19:37.989Z</atom:updated>
            <content:encoded><![CDATA[<p>TLDR: Study guide for the <strong>NVIDIA Certified Associate: AI Infrastructure and Operations (NCA-AIIO)</strong> exam, written from an operations background — if you already run Kubernetes or Slurm, this maps the AI infrastructure world onto what you already know.</p><h3>Introduction: Why This Fuckin’ Certification</h3><p>I’m about to start working with NVIDIA GPUs at work, so I’d rather learn this properly instead of pick it up for fun purpose, because it won’t going anywhere or further and the <a href="https://www.nvidia.com/en-us/learn/certification/ai-infrastructure-operations-associate/"><strong>NCA-AIIO</strong></a> is a reasonable way to force that.</p><p>Everything here is written from an operations background. If you already run Kubernetes clusters, a lot of this world has a familiar shape — and where it doesn’t. This is article have no hands-on, every commands output in this section is just I copied it somewhere xD</p><h3>Where NCA-AIIO Sits</h3><p>NCA-AIIO is the entry point, not the destination. Same idea as passing AWS Cloud Practitioner before Solutions Architect Associate, like everyone:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/901/1*P1tGVI0rOTDL0iEC5Caj4Q.png" /></figure><p>Worth knowing before you pick a path: <strong>NCP-AIO is the only one of these with hands-on labs.</strong> The lab environment is provisioned when the exam starts and you drive real Slurm, Kubernetes and Base Command Manager from a Linux CLI. Everything else on this list, including the one you are reading about, is multiple choice — you can pass those by studying. AIO you cannot.</p><p>Which makes AIO the friendlier next step if you already run clusters for a living, and the harder one if you don’t. Verify the numbers on the official pages before you book anything, they move.</p><h3>Exam Details</h3><p>Full details on the <a href="https://www.nvidia.com/en-us/learn/certification/ai-infrastructure-operations-associate/">official exam page</a> — 50 questions, 1 hour, online proctored, registered through <a href="https://www.certiverse.com/#/checkout/nvidia/store-exam/NCA-AIIO">Certiverse</a>.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*CEDmMZN-sjJ6M0WE.png" /></figure><p>Three weighted domains:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/904/1*xrz1ZKFSgfJ5TViubqtK4Q.png" /></figure><h3>How to Read This</h3><p>Everything here maps to a blueprint bullet <strong>unless the heading says otherwise</strong>. Two markers appear:</p><ul><li><strong><em>(optional — NCP-AIO level)</em></strong> — real ops knowledge that the Associate exam does not test. Skip it on the first pass; come back when you run a cluster.</li><li><strong><em>(optional — reference)</em></strong> — numbers to look up, never to memorize.</li></ul><p>If you are short on time, read everything unmarked, do the <a href="https://blackmetalz.github.io/prepare-for-nvidia-certified-associate-ai-infrastructure-and-operations-nca-aiio-exam.html#exam-checklist">Exam Checklist</a>, skim the <a href="https://blackmetalz.github.io/prepare-for-nvidia-certified-associate-ai-infrastructure-and-operations-nca-aiio-exam.html#quick-recall-exam-day-cheat-sheet">Quick Recall cheat sheet</a>, and ignore the rest.</p><h3>Learning Resources</h3><ul><li><a href="https://www.coursera.org/learn/ai-infrastructure-operations-fundamentals"><strong>AI Infrastructure and Operations Fundamentals</strong></a> on Coursera, the official course. Paid, worth it if your company sponsors training.</li><li><a href="https://www.youtube.com/watch?v=0WjfKQdfeMU"><strong>YouTube full course</strong></a>, free alternative covering the same ground, no chapters though.</li></ul><h3>Mapping: What You Already Operate → AI Infrastructure</h3><p>If you come from Kubernetes, most of this world has a familiar shape. Start here:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/909/1*SQIXNLuMuzEkRZ1Kb8olfg.png" /></figure><p>And the one thing that has <strong>no Kubernetes equivalent at all</strong>:</p><p><strong>The compute fabric.</strong> In K8s you have one network and you mostly don’t think about it (Like me for example xD). In AI infrastructure there are 4 physically separate networks, and the fastest of them exists purely so GPUs in different chassis can behave like one giant GPU. This is the biggest conceptual gap coming from the cloud-native world, and it’s covered in Phase 3.</p><h3>Phase 1: The AI Workload — What Are We Actually Running?</h3><p>You can’t size (capacity planning) a cluster for a workload if you can’t describe it, right?</p><h4>1.1 Training vs. Inference</h4><p>These are two completely different workloads that happen to use the same hardware:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/912/1*7lHB9KONO6Ql1mRxfxdJKQ.png" /></figure><h4>1.2 AI vs. ML vs. Deep Learning vs. GenAI</h4><p>Nested circles, broadest to narrowest:</p><ul><li><strong>AI</strong> — any system doing something that normally needs human intelligence, including plain rule-based systems that never learn.</li><li><strong>Machine Learning</strong> — learns patterns from data instead of being programmed. A human still picks which input features matter.</li><li><strong>Deep Learning</strong> — ML with multi-layer neural networks that work out the features themselves. Needs far more data and compute, which is what made GPUs mandatory.</li><li><strong>Generative AI</strong> — a deep learning application that produces new content (text, images, code) rather than a label or a number.</li></ul><p>Rule of thumb: <strong>tabular data with thousands of rows → classical ML. Unstructured data (images, audio, text) at scale → deep learning.</strong></p><h4>1.3 Why Now — What Actually Changed</h4><p>Neural networks are from the 1980s. The blueprint asks why they only took over recently. Four things landed at once:</p><ul><li><strong>Data (Major)</strong> — <em>self-supervised</em> learning (Phase 1.5) removed the human-labeling bottleneck, so the raw internet became training data.</li><li><strong>Compute (Major)</strong> — Tensor Cores plus ever-lower precision (FP32 → BF16 → FP8 → FP4): same watt, far more math.</li><li><strong>Architecture</strong> — the transformer, a model can finally <em>use</em> ten thousand GPUs. Without this, the other three don’t matter.</li><li><strong>Ecosystem</strong> — CUDA, PyTorch, pretrained models, GPU by the hour. Most teams fine-tune someone else’s model rather than train one.</li></ul><p>The consequence for infrastructure is scaling laws: more parameters + data + compute gives predictably better models, so the cluster keeps getting bigger.</p><h4>1.4 Training Fundamentals</h4><p>Example: Model 70B params -&gt; gradient is 70B numbers, each number is direction for a weight (<strong>weight = weight — learning_rate × gradient</strong>). AllReduce sync gradient, not the fuckin’ data!</p><p>Training is a loop: feed data in + let the model guess (forward pass), measure how wrong it was (loss), adjust slightly (optimizer step), repeat millions of times. The “adjust” step is matrix math, which is why it needs a GPU.</p><p>Terms worth recognizing: it is a wall of text but worth for a noob like me, that is why I put them below</p><ul><li><strong>Batch size</strong> — how many samples the GPU processes before the model updates. Bigger batch = better GPU utilization but more GPU memory.</li><li><strong>Iteration</strong> (step) — one trip around the loop above: one batch in, one weight update out.</li><li><strong>Epoch</strong> — one complete pass over the whole dataset. It is a counter, not an extra step.</li><li><strong>Overfitting</strong> — the model memorized the training data instead of learning the general pattern. The opposite is <strong>underfitting</strong> — too simple, bad at both.</li><li><strong>Checkpoint</strong> — a periodic save of model state to shared storage, so a job that dies on day three resumes instead of restarting.</li></ul><p>Batch, iteration and epoch are the same work counted three ways. Think of revising a question bank:</p><pre><br>Question bank of 1,000 questions      &lt;- the dataset<br>Answer 20, check them, learn          &lt;- batch size 20 = one iteration<br>50 iterations = all 1,000 questions   &lt;- one epoch</pre><p>Epoch 2 is the same bank again, not a new one — each pass only nudges the weights a little, so you go around several times.</p><p>Overfitting is revising that bank until you have the answer key memorized: 10/10 on it, and lost on anything unseen. That is the trap, because the score you are watching looks great. Underfitting is not having studied enough — bad on both. It is the same failure as tuning a config against one recorded traffic pattern: the benchmark looks perfect right up until production shifts.</p><p>You catch it with a <em>validation set</em> the model never trains on, scored after each epoch. Training loss still dropping while validation loss starts climbing means the memorizing has begun, and <em>early stopping</em> is the usual answer.</p><p>Fixing it splits into three groups — past what the Associate exam asks, but I still like to write it down here xD:</p><ol><li><strong>Data, reach here first.</strong> More real data, or <strong>augmentation</strong> (crop, rotate, paraphrase — synthetic variety from what you already have). Nothing beats showing the model more of the world.</li><li><strong>Architecture.</strong> Fewer layers or parameters, because a smaller model has less room to memorize. <strong>Dropout</strong> randomly disables neurons during training so they can’t lean on each other.</li><li><strong>Regularization &amp; optimization.</strong> <strong>Weight decay / L1 / L2</strong> penalize oversized weights. <strong>Early stopping</strong> ends the run when validation loss turns upward, instead of guessing an epoch count.</li></ol><blockquote><em>Exam-style question: </em>a model scores 99% on training data but 60% on test data — what is this?<em> → overfitting.</em></blockquote><p>Two of these land directly on infrastructure: <strong>batch size drives GPU memory</strong>, and <strong>checkpointing drives storage bandwidth</strong> — every rank writes at the same instant, so the filesystem has to absorb the burst. Checkpoint sizing is a real capacity input, not an afterthought.</p><h4>1.5 How Models Learn</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/903/1*xMRJpb0ZarfX7WIKEGldVQ.png" /></figure><p>Two output types under supervised learning: <strong>classification</strong> (discrete — spam or not spam) and <strong>regression</strong> (continuous — predicted price).</p><h4>1.6 Evaluation Metrics</h4><p>This section original is way too fuckin’ hard for me, it is not fuckin’ possible for me to understand how thing work! So I will try to memorize instead, it sucks but better than nothing:</p><ul><li><strong>False positive</strong> — flagged it, nothing was wrong.</li><li><strong>False negative</strong> — stayed silent, something was wrong.</li><li><strong>Precision</strong> — of everything I called out, how much was right?</li><li><strong>Recall</strong> — of everything that actually happened, how much did I catch?</li><li><strong>Low precision = noise.</strong> Calling out a lot, wrong a lot.</li><li><strong>Low recall = misses.</strong> Staying silent while something was going on.</li><li><strong>Accuracy</strong> — correct overall. <strong>Useless when the data is imbalanced</strong>: a check that never fires is still 99.9% accurate, simply because almost nothing is ever wrong.</li><li><strong>F1</strong> — precision and recall rolled into one number. Use it instead of accuracy when the classes are imbalanced.</li><li><strong>Picking one</strong>: which mistake is more expensive? Misses cost more =&gt; recall. False alarms cost more =&gt; precision.</li></ul><p>Questions describe a business scenario and ask which metric matters. The shapes repeat:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/902/1*meRdKZ-YbYlBn-hjozpsnQ.png" /></figure><h4>1.7 Generative AI &amp; LLMs</h4><p>This is the part worth actually understanding, because everything NVIDIA sells right now is aimed at it.</p><ul><li><strong>Transformer</strong> — the architecture behind every modern LLM (<a href="https://arxiv.org/abs/1706.03762"><em>Attention Is All You Need</em></a>, 2017). Its core trick, <strong>self-attention</strong>, lets every word weigh its relevance against every other word — and does it <em>in parallel</em>, which is exactly why GPUs suit it and older sequential architectures didn’t.</li><li><strong>Token</strong> — the unit an LLM processes.</li><li><strong>Context window</strong> — how many tokens it can consider at once.</li><li><strong>Embedding</strong> — text turned into a vector, where similar meaning means nearby in space. Stored in a <strong>vector database</strong> (FAISS, Milvus, pgvector) so you can search by meaning instead of keyword.</li><li><strong>Hallucination</strong>: confident, fluent and fuckin’ wrong. The fix is grounding the model in real documents, not more training.</li></ul><p><strong>Inference has two phases</strong>, and they stress the GPU differently:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/902/1*6GGMRKziET4Bnt1E8V4_0w.png" /></figure><p>That second row is why inference GPUs are chosen for High Bandwidth Memory and capacity. The <strong>KV cache</strong>, which stores the attention state of tokens already generated so each new token doesn’t recompute the whole conversation, grows with batch size × context length and is frequently the real memory ceiling in production serving.</p><p><strong>Customizing a model</strong> — guaranteed exam material:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/902/1*G6smy9qtzMhi9V8zFhiyDg.png" /></figure><p><strong>RAG in one line:</strong> chunk your documents → embed them → store in a vector DB → at query time, retrieve the most relevant chunks and paste them into the prompt as context.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*j3vWtdmdXyUuUsDa.png" /></figure><p><em>Ingest runs offline, query runs per request — and notice the LLM itself is never retrained. Source: </em><a href="https://developer.nvidia.com/blog/rag-101-demystifying-retrieval-augmented-generation-pipelines/"><em>NVIDIA, RAG 101</em></a></p><p>If a question says <em>“must reflect data that changes daily”</em> or <em>“must cite internal documents”</em>, the answer is <strong>RAG</strong>, not fine-tuning. If it says <em>“must always respond in our house style / output strict JSON”</em>, that’s <strong>fine-tuning</strong>.</p><h4>1.8 Who Uses What</h4><p>Scenario questions like <em>“a hospital wants X — which NVIDIA offering?”</em> are common:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/911/1*DaVNMT0WXt--ZS54DDXZ3Q.png" /></figure><p>And for those content above, I feel like they are almost reflection of On-prem vs AWS (for example of Cloud) xDD</p><h3>Phase 2: The Compute Layer</h3><h4>2.1 CPU vs. GPU vs. DPU</h4><ul><li><strong>CPU — latency optimized.</strong> Few powerful cores, big caches, branch prediction. Good at sequential logic and decisions.</li><li><strong>GPU — throughput optimized.</strong> Thousands of small cores doing the same operation on different data (SIMT). Good at matrix math, useless at branching logic.</li><li><strong>DPU (</strong><a href="https://www.nvidia.com/en-us/networking/products/data-processing-unit/"><strong>BlueField</strong></a><strong>) — infrastructure offload.</strong> Takes networking, storage and security work off the host CPU so the CPU can stay busy feeding the GPUs. Programmed with the <strong>DOCA(Data Center on a Chip Architecture)</strong> SDK. If we need one thing for reflection, we can think about AWS Nitro I believe!</li></ul><p><strong>Amdahl’s law is the practical takeaway.</strong> The part you <em>can’t</em> accelerate sets the ceiling:</p><pre># Example!<br>A job takes 100 min: [==== GPU: 80 min ====][ CPU: 20 min ]<br>More and faster GPUs:                       [ CPU: 20 min ]  -&gt; 20 min still here, not going anywhere!</pre><p>Those 20 minutes are a floor no amount of GPUs can touch. In real clusters the leftover is almost always data loading and preprocessing, which is exactly why <a href="https://developer.nvidia.com/dali"><strong>DALI</strong></a> and <a href="https://rapids.ai/"><strong>RAPIDS</strong></a> exist: move that work onto the GPU and the ceiling moves up with it.</p><h4>2.2 GPU Generations — What Actually Changed</h4><p>Skip the transistor counts. Remember what each generation introduced, because that’s what gets asked (I have no fuckin’ idea about <strong>TF32, BF16, FP8….</strong> so it is fuckin’ fine if you don’t understand it xD):</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/906/1*kimzEfjklSMtzwn7kfN82A.png" /></figure><p>Three relationships worth memorizing:</p><ul><li><strong>H100 vs A100</strong> — same 80GB capacity, but faster HBM3, 1.5x NVLink bandwidth, and FP8 via Transformer Engine.</li><li><strong>H200 = H100 compute + better memory</strong> — 141GB HBM3e, ~1.4x memory bandwidth. Drop-in compatible with HGX H100 systems. <strong>Best for large-model inference and long context</strong>, and now you know why (Phase 1.7: decode is memory-bandwidth bound).</li><li><strong>GB200 Superchip</strong> = 1 Grace CPU + 2 Blackwell GPUs over <strong>NVLink-C2C</strong> (900 GB/s, ~7x PCIe Gen5). <strong>GB200 NVL72</strong> = 36 Grace + 72 Blackwell in one liquid-cooled rack acting as a single NVLink domain.</li></ul><p>Also worth recognizing: <strong>L40S</strong> (Ada, 48GB, no NVLink — inference and graphics), <strong>L4</strong> (Ada, 24GB, 72W — low-power inference/video), <strong>GH200</strong> (1 Grace + 1 Hopper).</p><p>Detailed specs are in <a href="https://blackmetalz.github.io/prepare-for-nvidia-certified-associate-ai-infrastructure-and-operations-nca-aiio-exam.html#appendix-a-hardware-spec-reference-optional-reference">Appendix A</a> — reference material, not memorization material.</p><h4>2.3 DGX vs. HGX vs. MGX</h4><ul><li><strong>DGX</strong> — NVIDIA’s own complete turnkey system, hardware plus software stack. Scales into a <strong>DGX SuperPOD</strong> reference architecture.</li><li><strong>HGX</strong> — just the GPU baseboard (8 GPUs + NVSwitch), sold to OEMs like Dell and Supermicro to build their own servers around.</li><li><strong>MGX</strong> — a modular <em>reference design</em> letting OEMs assemble many different CPU/GPU/DPU server configurations.</li></ul><p>One-liner: <strong>DGX you buy finished, HGX is the engine block, MGX is the chassis blueprint.</strong></p><h4>2.4 Tensor Cores &amp; Precision Formats</h4><p><strong>Tensor Cores</strong> are dedicated units that do fused matrix multiply-and-add (D = A × B + C), the single operation that dominates deep learning. Regular CUDA cores can do this too, just far slower.</p><p>Lower precision means fewer bytes per value: less memory, more throughput, some accuracy risk. <strong>That byte count is the number you actually use</strong>, it’s what the sizing math in Phase 2.6 runs on.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/905/1*iEyWCjaam-216Khhw8C4Rw.png" /></figure><p>A 70B model is 70 × 2 = 140 GB at BF16, half that at FP8. Which generation introduced which format is the other thing worth remembering — that&#39;s the shape the exam question takes.</p><blockquote>Optional — none of this is on the exam. (This is the fuckin’ AI Generated content xD)<em> Each format splits its bits between </em><strong><em>exponent</em></strong><em> (buys range) and </em><strong><em>mantissa</em></strong><em> (buys precision). BF16 keeps FP32’s full exponent, so it needs no loss scaling, which is why it displaced FP16 for training. </em><strong><em>TF32</em></strong><em> is an Ampere-era internal compute format the math libraries apply transparently. FP8 ships in two flavors, </em><strong><em>E4M3</em></strong><em> forward and </em><strong><em>E5M2</em></strong><em> backward, because gradients span a wider range. NVFP4 stays accurate enough only because of per-block scaling.</em></blockquote><h4>2.5 Sharing a GPU: MIG vs. vGPU vs. Time-Slicing</h4><p>A full H100 is overkill for most inference workloads, so there are three ways to split one. Ordered from strongest isolation to weakest:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/905/1*sfQPPM8T33Txstc0zsKmAg.png" /></figure><p>MIG is the only one giving <strong>true QoS and fault isolation</strong>!</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/546/0*Pq1GptqI_rPdkgms.png" /></figure><p><em>The same physical GPU, four legal ways to cut it. The crossed-out boxes are the capacity you lose to partitioning. Source: </em><a href="https://docs.nvidia.com/datacenter/tesla/mig-user-guide/concepts.html"><em>NVIDIA MIG User Guide</em></a></p><h4>2.6 Sizing: How Much Hardware?</h4><p>Two blueprint bullets ask you to pick hardware for a described use case. Both come down to memory. Why? this can be daily job if I’m not fuckin’ wrong!.</p><p>Small example: a customer says they want to run a 70B model. The first question back is <strong>train it, or serve it?</strong> Serving it at BF16 takes ~170 GB once the KV cache is counted, so a single DGX H100 (8 × 80 GB = 640 GB) is already more than enough — three GPUs would do — and selling them a second box is wasted money. Training that same model needs over 1 TB before activations, so one DGX is <em>not</em> enough and you are looking at two nodes minimum. Same model, completely different bill.</p><p><strong>Inference — memory holds the weights:</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/901/1*aN5HoeMF2iUeN5aJG_8UKw.png" /></figure><p>Add the KV cache and runtime overhead on top — budget roughly 20%. So a 70B model at BF16 does <strong>not</strong> fit one 80GB H100, but does fit one 141GB H200. That’s the H200 story from 2.2 in concrete form.</p><p><strong>Training — memory holds far more than the weights:</strong></p><p>Weights + gradients + Adam optimizer state runs about <strong>16 bytes per parameter</strong>, before activations. The same 70B model therefore needs ~1.1 TB of state — at least 14 GPUs’ worth of HBM, so 16 in practice, and more once activations are counted.</p><blockquote><strong><em>Training needs roughly 10x the memory of inference.</em></strong><em> If you remember one number from this section, that’s the one.</em></blockquote><p><strong>Scaling up vs. scaling out</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/906/1*V0wTzPFP0nxWujiL-dKRJA.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/908/1*h6NM0mIWQqNPJ_phT9jEIQ.png" /></figure><p><em>Scale-up: every one of the 72 GPUs in a GB200 NVL72 wired into 18 NVLink switch chips. Any GPU reaches any other in a single hop at full NVLink speed — and it stops at 72. Source: </em><a href="https://docs.nvidia.com/dgx-superpod/reference-architecture-scalable-infrastructure-gb200/latest/network-fabrics.html"><em>NVIDIA DGX SuperPOD — Network Fabrics</em></a></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*Eq7BqJAUo502OU5X.png" /></figure><p><em>Scale-out: leaf → spine → core InfiniBand joining 8 racks into a scalable unit, then 16 of those together. The C-UFM boxes are the fabric manager from Phase 3.4, and every one of those lines is an optic with a reach limit — which is Phase 8.3’s problem. Source: </em><a href="https://docs.nvidia.com/dgx-superpod/reference-architecture-scalable-infrastructure-gb200/latest/network-fabrics.html"><em>NVIDIA DGX SuperPOD — Network Fabrics</em></a></p><p>Scale up first, because NVLink is far faster than anything crossing a chassis boundary; scale out when you run out of chassis. <strong>DGX SuperPOD</strong> is NVIDIA’s reference architecture for doing the second properly.</p><p><strong>Picking the part:</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/903/1*CiSVVyLQarIceVHmvZa0vQ.png" /></figure><h3>Phase 3: The Fabric Layer</h3><p>This is the part with no Kubernetes equivalent, and the part most worth understanding.</p><h4>3.1 Why Not Just Use PCIe?</h4><p>Distributed training means every GPU must exchange gradients with every other GPU, every single step. That collective operation is <strong>AllReduce</strong>: each GPU finishes its batch holding its own gradients, AllReduce sums them across every GPU and hands back the identical result to all of them, so all the model copies stay in lockstep. It happens thousands of times per epoch, and nobody computes anything until it finishes.</p><p>PCIe Gen5 x16 gives ~128 GB/s bidirectional. NVLink 4 gives 900 GB/s bidirectional — <strong>roughly 7x</strong>. Run AllReduce over PCIe and your expensive GPUs spend most of their life waiting.</p><p><strong>Quick note on units.</strong> These links are full duplex — separate lanes each way — so the same cable gets quoted two ways, and the two conventions differ by exactly 2x:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/900/1*a5BRcu5PR0heBlDK_j9U2A.png" /></figure><p>NVIDIA prints NVLink bidirectional, PCIe specs are per direction. Compare them as-printed and you find a 14x that isn’t there. Networking flips it again: <strong>400 Gb/s NDR InfiniBand means 400 Gb/s each direction.</strong></p><ul><li><a href="https://www.nvidia.com/en-us/data-center/nvlink/"><strong>NVLink</strong></a> — direct GPU-to-GPU interconnect bypassing PCIe entirely (H100: 18 links × 50 GB/s bidirectional = 900 GB/s).</li><li><strong>NVSwitch</strong> — a non-blocking switch chip so all 8 GPUs in a chassis talk to each other simultaneously at full speed, not just in pairs.</li><li><strong>NVLink Switch</strong> (Blackwell) — extends NVLink <em>outside</em> the chassis, making 72 GPUs in a GB200 NVL72 behave as one NVLink domain.</li></ul><h4>3.2 The Four Fabrics</h4><p>An AI datacenter runs four physically separate networks. Know which does what:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/902/1*hy5cLePPhZhMaLHXmR8V3A.png" /></figure><p>The rule of thumb for a DGX: <strong>one compute NIC per GPU</strong> (8 GPUs -&gt; 8 NICs, a rail-optimized design), plus separate NICs or DPUs for storage and in-band, plus one RJ45 BMC port.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*wYFaj0u9u1DNbGHN.png" /></figure><p><em>The storage and in-band fabrics share Ethernet switching but stay logically separate — and note the link legend: storage runs at 800 Gbps while in-band runs at 100. Source: </em><a href="https://docs.nvidia.com/dgx-superpod/reference-architecture-scalable-infrastructure-gb200/latest/network-fabrics.html"><em>NVIDIA DGX SuperPOD — Network Fabrics</em></a></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*dCfE_oIDxKekvZCf.png" /></figure><p><em>The OOB fabric: 1 Gbps to everything with a management port — PDUs, BMCs, the IB switches themselves, the UFM appliances. Slow, dull, and the one network that must never be reachable from user space. Source: </em><a href="https://docs.nvidia.com/dgx-superpod/reference-architecture-scalable-infrastructure-gb200/latest/network-fabrics.html"><em>NVIDIA DGX SuperPOD — Network Fabrics</em></a></p><h4>3.3 RDMA and Its Friends</h4><ul><li><strong>RDMA</strong> — a NIC writes directly into a remote machine’s <strong>host memory</strong>, bypassing the OS kernel and the remote CPU. Zero-copy, very low latency.</li><li><a href="https://docs.nvidia.com/cuda/gpudirect-rdma/"><strong>GPUDirect RDMA</strong></a> — the same, but straight into <strong>GPU memory</strong>, never touching host RAM. This is what NCCL uses for inter-node traffic, and the distinction between these two is exam material.</li></ul><h4>3.4 InfiniBand vs. RoCE vs. Spectrum-X</h4><p><strong>InfiniBand isn’t Ethernet with different cables.</strong> It’s a separate stack, its own switches, its own adapters (HCAs), no TCP/IP, built for RDMA from day one. It never drops packets under congestion, because a sender may only transmit once the receiver has granted it buffer credits, and routes are computed centrally by a <strong>Subnet Manager</strong> rather than by each switch on its own. Ethernet drops and retransmits, which is fine for web traffic and fatal for AllReduce. <strong>RoCE</strong> is the compromise: RDMA bolted onto Ethernet, which then has to be configured into behaving losslessly. <strong>Spectrum-X</strong> is NVIDIA selling that configuration pre-done.</p><p>If it helps: InfiniBand is to Ethernet what Fibre Channel is to iSCSI.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/900/1*L5LveAyqRtZqWfHkS_xxAw.png" /></figure><p>Two clarifications the exam likes:</p><ul><li><strong>SHARP</strong> (Scalable Hierarchical Aggregation and Reduction Protocol) does part of the AllReduce <em>inside the switch</em>, so less data crosses the wire.</li><li><strong>Subnet Manager</strong>: <strong>OpenSM</strong> is the open-source subnet manager that configures the fabric. <strong>UFM</strong> is NVIDIA’s fabric management <em>platform</em> built around it (monitoring, telemetry, health). Don’t mix them up.</li></ul><h3>Phase 4: The Storage Layer</h3><h4>4.1 Why Storage Is a GPU Problem</h4><p>Traditional I/O wastes CPU and PCIe bandwidth on pure copying:</p><pre>Traditional Path:<br>Storage/NIC ──► Host RAM (page cache) ──► Host RAM (bounce buffer) ──► GPU Memory<br><br>GPUDirect Storage (GDS) Path:<br>Storage/NIC ─────────────────(Direct PCIe P2P / RDMA)───────────────► GPU Memory</pre><p><a href="https://docs.nvidia.com/gpudirect-storage/index.html"><strong>GPUDirect Storage</strong></a> opens a direct DMA path from storage into GPU memory, skipping the CPU bounce buffer. Local NVMe uses PCIe peer-to-peer; remote storage uses RDMA. Applications reach it through the <strong>cuFile API</strong> (kernel module nvidia-fs). Result: lower latency, higher throughput, <strong>and lower CPU utilization</strong> — that last one matters because the CPU is usually busy preprocessing data.</p><h4>4.2 Storage Tiers</h4><pre>Local NVMe          →  Shared parallel FS        →  Object storage<br>(scratch, cache)       (datasets, checkpoints)      (data lake, archive)<br>fastest, smallest      fast, shared by all nodes    cheap, slow, huge</pre><p><strong>Parallel filesystems</strong> (Lustre / DDN EXAScaler, WEKA, VAST Data, IBM Storage Scale, BeeGFS) exist because a normal NFS server cannot feed hundreds of GPUs at once. They spread both data <em>and</em> metadata across many servers.</p><p>Remember the checkpoint problem from Phase 1.4: hundreds of GPUs writing simultaneously, all idle until the write finishes. Checkpoint burst bandwidth is usually what sizes the filesystem, not average throughput.</p><h3>Phase 5: The Software Stack</h3><h4>5.1 The Layers</h4><pre>[ Applications &amp; Models ]   NeMo, BioNeMo, Clara, Riva, Omniverse<br>            ▲<br>[ Serving &amp; Optimization ]  NIM, Triton Inference Server, TensorRT, TensorRT-LLM<br>            ▲<br>[ Frameworks ]              PyTorch, JAX, TensorFlow  │  RAPIDS (data science)<br>            ▲<br>[ Accelerated Libraries ]   CUDA  │  CUDA-X: cuDNN, NCCL, cuBLAS, DALI  │  DOCA<br>            ▲<br>[ System Layer ]            GPU Driver, NVIDIA Container Toolkit<br>            ◄── DCGM monitors every layer above</pre><ul><li><a href="https://developer.nvidia.com/cuda-toolkit"><strong>CUDA</strong></a> — the parallel computing platform and programming model. Everything sits on this.</li><li><strong>cuDNN</strong> — tuned primitives for neural networks (convolution, pooling, normalization).</li><li><a href="https://developer.nvidia.com/nccl"><strong>NCCL</strong></a> — the collective communication library for multi-GPU and multi-node (AllReduce, AllGather, Broadcast, ReduceScatter), optimized for NVLink and InfiniBand. <strong>Any question about optimizing inter-GPU communication is an NCCL question.</strong></li><li><strong>CUDA-X</strong> — the umbrella name for the accelerated libraries sitting on top of CUDA (cuDNN, NCCL, cuBLAS, RAPIDS, DALI). NVIDIA uses the term constantly, so recognize it.</li><li><strong>DOCA</strong> — SDK for BlueField DPUs and SuperNICs.</li></ul><h4>5.2 Serving: TensorRT vs. Triton vs. NIM</h4><p>These three get confused constantly, and the exam knows it: <strong>TensorRT optimizes the model; Triton serves it; NIM ships both as a supported container.</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/904/1*HJQpVQgFq8JWsc-ycSdTug.png" /></figure><p>And three ways to shrink a model <em>before</em> you serve it — classic answer options, so don’t mix them up:</p><ul><li><strong>quantization</strong> (fewer bits per weight, what TensorRT does)</li><li><strong>pruning</strong> (drop weights that barely contribute)</li><li><strong>distillation</strong> (train a small <em>student</em> model to imitate a large <em>teacher</em>).</li></ul><h4>5.3 Distribution &amp; Support</h4><ul><li><a href="https://catalog.ngc.nvidia.com/"><strong>NGC</strong></a> — NVIDIA’s registry of GPU-optimized containers, pretrained models and Helm charts. Think like Docker Hub or container registry.</li><li><strong>NVIDIA AI Enterprise (NVAIE)</strong> — the paid layer: validated images, security patches, enterprise support, and NIM. This is what a company buys so someone else is accountable for CVEs.</li><li><strong>Apptainer (formerly Singularity)</strong> — containerization for HPC, not orchestration. Runs unprivileged and mounts the host filesystem naturally, which is why Slurm sites use it over Docker.</li></ul><h3>Phase 6: Operations — Scheduling &amp; Orchestration</h3><h4>6.1 Slurm vs. Kubernetes vs. Run:ai</h4><p><a href="https://slurm.schedmd.com/">Slurm</a> is still the standard for large training jobs, mainly because it allocates <strong>all or nothing</strong> — vanilla kube-scheduler will happily bind 60 of 64 pods and leave them idling.</p><p><a href="https://www.nvidia.com/en-us/software/run-ai/">Run:ai</a> is worth understanding because it’s essentially “Slurm’s scheduling brain, but for Kubernetes” — teams get quotas, idle quota gets borrowed and reclaimed, and distributed jobs start all-or-nothing instead of deadlocking half-scheduled.</p><p><em>(Note: Slurm uses “gang scheduling” to mean time-slicing oversubscribed jobs — see </em><a href="https://slurm.schedmd.com/gang_scheduling.html"><em>PreemptMode=GANG</em></a><em>. Everyone else uses it to mean all-or-nothing co-allocation)</em></p><h4>6.2 NVIDIA GPU Operator</h4><p>The <a href="https://docs.nvidia.com/datacenter/cloud-native/gpu-operator/latest/index.html"><strong>GPU Operator</strong></a> is the reason you don’t hand-install drivers on 200 nodes. It deploys and manages:</p><ul><li>NVIDIA driver + Container Toolkit</li><li><strong>Device plugin</strong> — exposes nvidia.com/gpu as a schedulable resource</li><li><strong>GPU Feature Discovery</strong> + Node Feature Discovery — labels nodes with GPU model, MIG profile, etc.</li><li><strong>MIG Manager</strong> — applies MIG partitioning declaratively</li><li><strong>DCGM + DCGM Exporter</strong> — monitoring</li><li>Operator validator</li></ul><p>The companion <strong>Network Operator</strong> handles the networking side: MOFED/DOCA drivers, RDMA shared device plugin, GPUDirect RDMA.</p><h4>6.3 Base Command Manager</h4><p><strong>BCM</strong> (formerly Bright Cluster Manager) is bare-metal cluster provisioning for DGX and non-DGX clusters: OS deployment, InfiniBand fabric health checks, monitoring, and integration with both Slurm and Kubernetes. Closest thing to Cluster API in this world.</p><p>Do not confuse it with <strong>Base Command Platform</strong>, which is a SaaS for running training workloads on <strong>DGX Cloud</strong>.</p><h4>6.4 MLOps, Briefly</h4><ul><li><strong>Lifecycle:</strong> data versioning → experiment tracking → training → model registry → validation → deploy → monitor → retrain.</li><li><strong>Model registry</strong> — versioned model artifacts with lineage and stage promotion. NGC plays this role for NVIDIA-published models.</li><li><strong>Deployment</strong> — canary and blue/green apply exactly as they do for services; Triton can serve multiple model versions concurrently on one GPU.</li><li><strong>Drift</strong> — <em>data drift</em> is the inputs changing; <em>concept drift</em> is the input→output relationship changing. Either one is the signal to retrain.</li><li><strong>Reproducibility</strong> — pin the container image, framework version, driver version, dataset version and random seed. Containers are the mechanism.</li></ul><p>The blueprint asks you to <em>“describe software components related to the AI development and deployment life cycle”</em> — and it files that under <strong>Essential AI Knowledge</strong>, the 38% domain. So know which tool sits at which stage:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/904/1*XpXbqVt3jFQnMV_z_Z8K8w.png" /></figure><p>The pattern is the same CI/CD loop you already run, with a model artifact instead of a binary — and one extra failure mode (<strong>drift</strong>) that has no equivalent in ordinary software.</p><h3>Phase 7: Operations — Monitoring &amp; War Stories</h3><h4>7.1 The Tools</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/906/1*hEfb-rGpxJWCRXVsL7xTbw.png" /></figure><p>I don’t have GPU hardware yet, so I’m not going to paste output I didn’t generate. When I do, the hands-on version of this table gets its own article.</p><h4>7.2 Xid Errors</h4><p><a href="https://docs.nvidia.com/deploy/xid-errors/index.html"><strong>Xid</strong></a> errors are codes the NVIDIA driver writes into the kernel log when something goes wrong — hardware fault, bad memory, driver bug, or an application doing something illegal. The number identifies the class of problem.</p><p>Operationally the reflex is the same one you already have for a flaky K8s node: <strong>detect → drain → diagnose → RMA or return to service.</strong> DCGM can watch for them across the fleet so you’re not grepping dmesg on 200 machines.</p><p>For NCA-AIIO, knowing what an Xid is and where it surfaces is enough — matching specific codes to specific faults is NCP-AIO work.</p><h4>7.3 Reading a Bottleneck</h4><blockquote><strong><em>Where the Associate line is.</em></strong><em> The official NCA-AIIO study guide lists Run:ai’s </em>6 Reasons for Low GPU Utilization and How to Improve It<em> as suggested reading under AI Operations, and blueprint bullet 3.3 is </em>“articulate the key measures and criteria related to monitoring GPUs”<em>. So </em><strong><em>“GPU util is low, what is starving it?”</em></strong><em> is exam material. Reading raw telemetry, matching Xid codes and running </em><em>dcgmi yourself is </em><strong><em>NCP-AIO</em></strong><em>, which carries a whole 23% </em>Troubleshooting and Optimization<em> domain plus hands-on labs.</em></blockquote><p><strong>GPU utilization alone tells you nothing</strong>; read it alongside CPU, disk and network. These are the shapes the exam describes in words and asks you to name:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/904/1*UuzHEzegBW_oPvN9fR8bHQ.png" /></figure><p>The pattern behind every row: <strong>the GPU is the most expensive part, so everything else exists to keep it fed.</strong> If it is idle, something upstream is the problem, never the GPU itself.</p><p><strong>The rest is NCP-AIO</strong> <em>(optional — NCP-AIO level)</em>. Same table, but now you are expected to know the memory math and the tooling, not just the name of the bottleneck:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/900/1*MuTiUJpl0KieoYG--t2T3Q.png" /></figure><h4>7.4 Multi-Tenancy &amp; Security</h4><p>This one <em>is</em> on the NCA-AIIO blueprint: <em>“identify key considerations for virtualizing accelerated infrastructure.”</em></p><ul><li><strong>Isolation ladder</strong>, weakest to strongest: time-slicing (none) → MPS (shared fault domain) → vGPU (hypervisor) → <strong>MIG</strong> (hardware) → dedicated GPU.</li><li><strong>Confidential computing</strong> (Hopper and Blackwell) encrypts data <em>in use</em> so even the hypervisor or cloud operator can’t read GPU memory. For regulated workloads on shared infrastructure.</li><li><strong>The OOB fabric must never be routable from user networks.</strong> Same rule as any BMC network, higher stakes.</li></ul><h3>Phase 8: Data Center Realities</h3><p>Three blueprint bullets live here — power and cooling, facility requirements, and on-prem vs. cloud — and all three sit inside the <strong>40% infrastructure domain</strong>, not operations. More exam weight than the length of this phase suggests.</p><h4>8.1 Power</h4><ul><li><strong>Density</strong> — a traditional enterprise rack is 5–10 kW. A single DGX H100 draws ~10.2 kW, and a GB200 NVL72 rack ~120 kW. You cannot drop these into an existing hall and hope.</li><li><strong>Delivery</strong> — that density means three-phase distribution and busways rather than ordinary PDUs, and it resizes the whips, breakers, UPS and generator behind them.</li><li><strong>Redundancy</strong> — <strong>N+1</strong> is one spare unit; <strong>2N</strong> is a complete second path at roughly double the cost. 2N is what you buy when a three-week training run must not die.</li><li><strong>Transients</strong> — GPUs are not a steady load. Thousands of them entering the same AllReduce swing power hard and fast, which stresses upstream gear in a way a rack of web servers never does.</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*Xp-4sgqJwtK9ulKQ.png" /></figure><p><em>One DGX SuperPOD scalable unit: 32 DGX H100 across 8 racks. Four systems per rack at ~10.2 kW each is roughly 40 kW in a rack sized for 10. Source: </em><a href="https://docs.nvidia.com/dgx-superpod/reference-architecture-scalable-infrastructure-h100/latest/dgx-superpod-architecture.html"><em>NVIDIA DGX SuperPOD Reference Architecture</em></a></p><h4>8.2 Cooling</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/908/1*zRI0Fa9DznVaOy7S9Vhf3g.png" /></figure><p>Direct-to-chip needs a <strong>CDU</strong> (coolant distribution unit) to separate the facility water loop from the clean loop touching the cold plates, plus pipework into the row. That is a building project, not a rack purchase.</p><p><strong>PUE (Power Usage Effectiveness)</strong> = total facility power ÷ IT equipment power. <strong>Closer to 1.0 is better.</strong> A PUE of 1.5 means every 2 kW of compute costs another 1 kW in cooling and losses. Liquid cooling is one of the main levers for pushing it down.</p><h4>8.3 Facility Requirements</h4><p>The unglamorous list that stops projects:</p><ul><li><strong>Floor loading</strong> — a fully populated NVL72 rack weighs over a tonne. Raised floors have limits, and so do freight elevators and the route from the loading dock.</li><li><strong>Water</strong> — direct-to-chip means supply, return, leak detection and a CDU. Most enterprise halls have none of it today.</li><li><strong>Cable runs</strong> — one compute NIC per GPU means eight high-speed links per node plus storage and management. Optics have reach limits, which constrains where a rack can physically sit relative to the spine.</li><li><strong>Space and service access</strong> — containment, ceiling height, and enough clearance to actually slide out a 10U chassis that takes two people to lift.</li></ul><h4>8.4 On-prem vs. Cloud vs. Hybrid</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/907/1*tCQZErEMCWKCsQY4cbwqNA.png" /></figure><p><strong>Utilization is the deciding number.</strong> A cluster busy 20% of the time should have been rented; one pinned at 90% for three years should have been bought. Hybrid keeps regulated data local and bursts the overflow out — <strong>DGX Cloud</strong> is NVIDIA’s own version of that.</p><h3>Exam Checklist</h3><p>Study time should follow the blueprint weights, not your comfort zone.</p><p><strong>Essential AI Knowledge (38%)</strong></p><ul><li>Contrast training and inference across duration, bottleneck, failure behavior and scheduler.</li><li>Place AI, ML, DL and GenAI as nested sets; say when classical ML beats deep learning.</li><li>Name the four factors behind AI’s recent takeoff, and why the transformer was the unlock.</li><li>Explain batch size, epoch, overfitting, checkpoint — and which two hit infrastructure.</li><li>Name the four learning paradigms with one use case each.</li><li>Pick between accuracy, precision, recall and F1 given a described business cost.</li><li>Explain self-attention in one sentence, and why prefill is compute-bound but decode is memory-bandwidth-bound.</li><li>Choose between prompt engineering, RAG, LoRA and full fine-tuning for a requirement.</li><li>Describe the RAG pipeline and where embeddings and the vector DB fit.</li><li>Match an industry scenario to Clara / Merlin / Riva / Omniverse / Metropolis / Morpheus / NIM.</li><li>Name the layers of the NVIDIA stack from driver to application, and what CUDA-X covers.</li><li>Walk the AI lifecycle and name the tool at each stage, from RAPIDS to Triton to DCGM.</li></ul><p><strong>AI Infrastructure (40%)</strong></p><ul><li>Explain why GPUs beat CPUs at matrix math, and what a DPU offloads.</li><li>Say what Ampere, Hopper and Blackwell each introduced.</li><li>Explain why H200 is the inference choice despite H100-identical compute.</li><li>Estimate model memory at a given precision, and say why training needs ~10x inference.</li><li>Distinguish scale-up (NVLink) from scale-out (InfiniBand), and say which you exhaust first.</li><li>Differentiate DGX, HGX and MGX.</li><li>Explain NVLink, NVSwitch and NVLink Switch, and why PCIe isn’t enough.</li><li>Differentiate FP32, BF16/FP16, FP8 and FP4 by bytes per value, and say which generation introduced FP8 and FP4.</li><li>Describe MIG’s hardware QoS and how it differs from vGPU and time-slicing.</li><li>List the four fabrics and what each carries.</li><li>Explain RDMA vs. GPUDirect RDMA.</li><li>Compare InfiniBand, RoCEv2 and Spectrum-X; say what SHARP does.</li><li>Explain how GDS removes the CPU bounce buffer, and name three parallel filesystems.</li><li>Distinguish TensorRT, Triton and NIM.</li><li>Distinguish quantization, pruning and distillation as ways to shrink a model.</li><li>Explain what NCCL does and name its main collectives.</li><li>Size power and cooling for a 120 kW rack; explain PUE and when liquid cooling stops being optional.</li><li>List the facility requirements that block a deployment: floor loading, water, cable reach, service access.</li><li>Compare on-prem, cloud and hybrid on cost, control, elasticity — and name utilization as the deciding number.</li></ul><p><strong>AI Operations (22%)</strong></p><ul><li>Choose between Slurm, Kubernetes + GPU Operator, and Run:ai for a workload.</li><li>List what the GPU Operator deploys and what the Network Operator adds.</li><li>Describe what DCGM monitors and how it reaches Grafana.</li><li>Distinguish Base Command Manager from Base Command Platform.</li><li>Given utilization figures, name the bottleneck and the fix. <em>(NCP-AIO level — lowest priority here.)</em></li><li>Explain what an Xid error is and where it appears.</li><li>Outline the MLOps lifecycle; distinguish data drift from concept drift.</li></ul><h3>Quick Recall — Exam-Day Cheat Sheet</h3><p>Keyword → answer, nothing new.</p><h4>Keyword → Product</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/908/1*1Xaa73t5G9kX6H_vRAObDg.png" /></figure><h4>One-Liners</h4><p><strong>Three core types</strong> — <strong>CUDA</strong> = general parallel, <strong>Tensor</strong> = matrix math / AI, <strong>RT</strong> = ray tracing.</p><p><strong>GPUDirect</strong> — one idea (straight to GPU memory, no CPU/host RAM bounce), three routes: <strong>P2P</strong> = GPU↔GPU same box, <strong>RDMA</strong> = GPU↔GPU across the network, <strong>Storage (GDS)</strong> = GPU↔disk. The question names the route, which names the member.</p><p><strong>Utilization ≠ memory usage</strong> — util is <em>time busy</em>, memory is <em>capacity occupied</em>. 80/80 GB at 15% util = starved data pipeline, not a busy GPU.</p><p><strong>PUE</strong> = total facility power ÷ IT power. 1.2 = modern hyperscale, 1.5 = decent enterprise, 2.0 = overhead equals the compute, 3.5/5.0 = distractors. GPU racks are heat bombs, so <strong>liquid cooling is the lever</strong> — every PUE question with a density number is a cooling question.</p><h4>Software: Name → Job</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/900/1*O4JJ9CgC8HuWUvBjg6Xn_w.png" /></figure><p><strong>TensorRT makes the model fast, Triton puts it in production.</strong></p><h4>Monitoring: Symptom → Metric</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/903/1*cMLPaxoyiSd0gVZtB4ECBQ.png" /></figure><h4>Networking</h4><ul><li><strong>IB beats Ethernet on latency</strong> — not on “it has RDMA”, Ethernet does RDMA too via RoCE.</li><li><strong>Three IB traits:</strong> low latency, high bandwidth, CPU offload. Traps: <em>GPU</em> offload, <em>high</em> latency.</li><li><strong>RDMA offloads the CPU</strong> through hardware on the NIC.</li><li>IB needs a <strong>Subnet Manager</strong> to come up — OpenSM is the usual one (or embedded on a switch / UFM).</li><li><strong>Four fabrics:</strong> compute, storage, in-band mgmt, out-of-band mgmt. OOB = via BMC, alive when the OS is dead, off the production network.</li><li>Training network = high throughput + low latency. <strong>Always eliminate Wi-Fi, 1 GbE, and Fibre Channel</strong> (that’s storage).</li></ul><h4>The Rest</h4><ul><li><strong>Slurm</strong> = job scheduler (HPC). <strong>Apptainer</strong> = container runtime. <strong>MLOps</strong> = a methodology, not a tool.</li><li><strong>High density blocked by three things:</strong> power, cooling, physical space.</li><li><strong>On-prem</strong> wins on security + data sovereignty, <strong>cloud</strong> on low upfront cost, <strong>hybrid</strong> on mixed or bursty workloads.</li><li><strong>Predictive maintenance → manufacturing.</strong></li></ul><h3>Appendix A: Hardware Spec Reference (optional — reference)</h3><blockquote><strong><em>Reference only — do not memorize this.</em></strong><em> None of these numbers are worth exam study time at Associate level, and most aren’t worth it at Professional level either. They’re here because I keep needing to look them up. What matters is the </em>why<em> in Phase 2, not the digits.</em></blockquote><h4>GPU Specs</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/907/1*FWnPCsH9Nbr3ELTAwgxpPQ.png" /></figure><p><em>B100 also exists — a 700W Blackwell that drops into HGX H100 chassis — but NVIDIA ships mainly B200 and GB200.</em></p><h4>DGX Systems</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/907/1*xY1fYjd9h2upieECyLpFKA.png" /></figure><p>CPUs, for completeness: DGX A100 = 2× AMD EPYC 7742 (128 cores); DGX H100/H200 = 2× Xeon Platinum 8480C (112 cores); DGX B200 = 2× Xeon Platinum 8570 (112 cores). Storage on all of them is roughly “2× 1.92TB M.2 for the OS, 8× 3.84TB U.2 for cache”.</p><h4>Marketing Numbers</h4><p>NVIDIA quotes these constantly; they occasionally show up as answer options:</p><ul><li><strong>DGX B200 vs. DGX H100</strong> — up to 3x training, 15x inference.</li><li><strong>GB200 NVL72 vs. H100</strong> — up to 30x trillion-parameter inference, 4x training, 25x better energy efficiency.</li></ul><p><em>Specs cross-checked September 2026 against the sources below. NVIDIA edits these pages, so re-verify before your exam.</em></p><h3>Secret Weapons</h3><p>I made a quiz set that mirrors this article. It is probably harder than the real exam, but it should test your knowledge as deep as it can xD</p><p><a href="https://www.udemy.com/course/nvidia-nca-aiio-practice-tests-harder-than-the-real-exam/?referralCode=944B3915CCAB812DE378">Udemy link</a></p><h3>Summary &amp; Next Steps</h3><p>NCA-AIIO is a foundation credential, and the honest way to treat it is as the vocabulary exam before the real ones. Coming from a Kubernetes or Slurm background, most of the operations domain is already familiar — the genuinely new material is the fabric layer and the ML fundamentals.</p><p>Preparation order that made sense to me:</p><ol><li>Watch the <a href="https://www.youtube.com/watch?v=0WjfKQdfeMU"><strong>YouTube course</strong></a> (free) or take the <a href="https://www.coursera.org/learn/ai-infrastructure-operations-fundamentals"><strong>Coursera course</strong></a> (paid).</li><li>Work through Phase 1 here until the training-vs-inference table is automatic — everything else hangs off it.</li><li>Check the <a href="https://www.nvidia.com/en-us/learn/certification/ai-infrastructure-operations-associate/">official blueprint</a> for anything that shifted.</li><li>Book it on <a href="https://www.certiverse.com/#/checkout/nvidia/store-exam/NCA-AIIO">Certiverse</a>.</li></ol><p>Good luck!</p><p>This article is a great start for me and I passed the exam as well xD</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*k_isOyprfTTBOFK0.png" /></figure><blockquote><strong><em>Image credit:</em></strong><em> every diagram and photo in this article belongs to NVIDIA, taken from the documentation and blog posts linked beneath each one. None of it is mine.</em></blockquote><h3>References</h3><ul><li><a href="https://www.nvidia.com/en-us/learn/certification/">NVIDIA Certification catalog</a></li><li><a href="https://www.nvidia.com/en-us/learn/certification/ai-infrastructure-operations-associate/">NCA-AIIO official exam page</a></li><li><a href="https://docs.nvidia.com/datacenter/tesla/mig-user-guide/">NVIDIA MIG User Guide</a></li><li><a href="https://docs.nvidia.com/datacenter/cloud-native/gpu-operator/latest/index.html">NVIDIA GPU Operator Documentation</a></li><li><a href="https://docs.nvidia.com/gpudirect-storage/index.html">NVIDIA GPUDirect Storage Documentation</a></li><li><a href="https://docs.nvidia.com/deploy/xid-errors/index.html">NVIDIA Xid Errors</a></li><li><a href="https://developer.nvidia.com/dcgm">NVIDIA DCGM</a></li><li><a href="https://www.nvidia.com/en-us/software/run-ai/">NVIDIA Run:ai</a></li><li><a href="https://developer.nvidia.com/nccl">NVIDIA NCCL</a></li><li><a href="https://www.nvidia.com/content/dam/en-zz/Solutions/Data-Center/nvidia-dgx-a100-datasheet.pdf">DGX A100 Datasheet</a></li><li><a href="https://docs.nvidia.com/dgx/dgxh100-user-guide/introduction-to-dgxh100.html">DGX H100/H200 User Guide</a></li><li><a href="https://docs.nvidia.com/dgx/dgxb200-user-guide/introduction-to-dgxb200.html">DGX B200 User Guide</a></li><li><a href="https://www.nvidia.com/en-us/data-center/h200/">NVIDIA H200 product page</a></li><li><a href="https://www.nvidia.com/en-us/data-center/gb200-nvl72/">GB200 NVL72 product page</a></li><li><a href="https://developer.nvidia.com/blog/inside-nvidia-blackwell-ultra-the-chip-powering-the-ai-factory-era/">Inside NVIDIA Blackwell Ultra</a></li><li><a href="https://arxiv.org/abs/1706.03762">Attention Is All You Need</a></li></ul><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=a7365e11bb8d" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Prepare for GCP — Associate Cloud Engineer Certification]]></title>
            <link>https://medium.com/@kienlt.qn/prepare-for-gcp-associate-cloud-engineer-certification-b8b82a7529fe?source=rss-b77f4e43d676------2</link>
            <guid isPermaLink="false">https://medium.com/p/b8b82a7529fe</guid>
            <category><![CDATA[gcp]]></category>
            <category><![CDATA[google-cloud-platform]]></category>
            <dc:creator><![CDATA[Kienlt]]></dc:creator>
            <pubDate>Mon, 14 Sep 2026 17:05:34 GMT</pubDate>
            <atom:updated>2026-09-14T17:05:34.057Z</atom:updated>
            <content:encoded><![CDATA[<h3>1.Intro</h3><p>Ah shit, here we go again xD. This time is Google Cloud, my first time. Please prepare new trial account with <strong>300$</strong>, we will use it for 3 months in a row. Not sure I can do what I’m aiming for (Cloud Engineer Associate + DevOps Pro). In this article, I’m mainly using resource from <a href="https://www.udemy.com/course/gcp-associate-cloud-engineer-certification/">Udemy course</a> and visit <a href="https://cloud.google.com/learn/certification/cloud-engineer/">Associate Cloud Engineer</a> link for me information!</p><p>Useful links:</p><ul><li><a href="https://www.skills.google/paths/11">Train for the exam</a></li><li><a href="https://services.google.com/fh/files/misc/associate_cloud_engineer_exam_guide_english.pdf">Exam Guide</a></li></ul><p>Section of the exam:</p><ul><li>Section 1: Setting up a cloud solution environment (~20% of the exam)</li><li>Section 2: Planning and configuring a cloud solution (~10% of the exam)</li><li>Section 3: Deploying and implementing a cloud solution (~30% of the exam)</li><li>Section 4: Ensuring successful operation of a cloud solution (~30% of the exam)</li><li>Section 5: Configuring access and security (~10% of the exam)</li></ul><p><strong>Update</strong>: My goal changed, so no more cloud certifications for this year.</p><h3>2.Resource Hierarchy</h3><h4>Concept of project in GCP</h4><p>A project is the main container for all your resources. (Ref: <strong>ResourceGroup</strong> at Azure)</p><p>Key project identifiers (Classic ACE exam traps!):</p><ul><li><strong>Project Name</strong>: User-friendly, mutable (change anytime), doesn’t need to be globally unique.</li><li><strong>Project ID</strong>: User-chosen (or auto-gen), globally unique, <strong>IMMUTABLE</strong> (can NEVER be changed once created). Used heavily in gcloud and APIs.</li><li><strong>Project Number</strong>: Google-assigned digits, immutable, globally unique. You will often see it prefixing default service accounts (e.g. [project-number]-compute@developer.gserviceaccount.com).</li></ul><p>Accidental deletion protection:</p><ul><li>Deleting a project enters a <strong>30-day pending deletion</strong> window (soft delete). Can be restored within 30 days via gcloud projects undelete &lt;PROJECT_ID&gt;.</li><li>To completely prevent accidental deletion (even from project owners), place a <strong>Project Lien</strong> on it (resourcemanager.projects.updateLiens). You must remove the lien before anyone can delete the project.</li></ul><h4>Resource Hierarchy</h4><p>Same like Azure, pretty familiar xD. Permissions management in GCP work with mechanism inheritance. GCP Resource Hierarchy has 4 levels: Organization, Folders, Projects and Resources.</p><p><a href="https://docs.cloud.google.com/static/resource-manager/img/cloud-hierarchy.svg">https://docs.cloud.google.com/static/resource-manager/img/cloud-hierarchy.svg</a></p><p>Important rules for the exam:</p><ul><li><strong>Inheritance is additive (Union, NOT intersection)</strong>: Flow goes downward (Org -&gt; Folder -&gt; Project -&gt; Resource). If a user has Viewer at Folder level, you <strong>CANNOT</strong> restrict or revoke that permission at the Project level. Child nodes can only grant additional permissions, never restrict inherited ones.</li><li><strong>Folders require an Organization node</strong>: You cannot create folders without an Org (e.g. personal trial accounts without Cloud Identity / Google Workspace cannot use folders).</li></ul><h4>Manage Projects</h4><p>We can use this for tracking costs of specific project like this</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*PLevYEsYzx7Pe197.png" /></figure><h4>Organization Policies VS IAM</h4><p>Nothing new for me, no need to explain detailed:</p><ul><li>Org Policies: WHAT can be done (Example: Edit/Create a policy for resource location lock, only allow to create instance in specific location like es-east1/europe-west1.)</li><li>IAM: WHO can do something.</li></ul><h4>API</h4><p>For some services, we need to enable before use it!</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*15Q6fS4GpsvO6C0o.png" /></figure><p>Enable via CLI: gcloud services enable compute.googleapis.com</p><h4>Cloud Identity</h4><p><strong>WHO</strong> is the people that allow to do <strong>WHAT</strong></p><ul><li>Cloud Identity: Manages the “WHO”</li><li>IAM: Manages the “WHAT”</li></ul><p>Best practice is manage people with group for sure, not only in GCP. For example in Elasticsearch, we create a role called data-reader that has permission to read indices kienlt-data-*, then we add right user to that role, so we don&#39;t need to create multiple role for each user or remove user from that role when they are no longer needed, pretty common practice I believe!</p><h4>Asset Inventory</h4><p>Inventory of everything in your GCP. Is the central catalog for every single resource in your GCP Org.</p><p>I remember when my boss ask me to create inventory of whole system on-premises, it would take shit load of time… But in GCP, there are already service for this shit!</p><h4>Cloud Billing Account</h4><p>A single billing account can pay for thoudsands of projects, centralizing financial control. And need to remember that billing accounts are “Separate” from projects in GCP (to use paid service, a project must be linked to an active cloud billing account so Google know where to send the fuckin’ bill!)</p><p>Exam traps to remember:</p><ul><li><strong>Roles to link Billing to a Project</strong>: Requires <strong>Billing Account User</strong> on the billing account + <strong>Project Billing Manager</strong> (roles/billing.projectManager, or Project Owner) on the target project. (Project Creator only creates new projects under Org/Folders; it cannot link billing to an existing project!).</li><li><strong>Budget alerts DO NOT stop resources</strong>: Setting a budget with 50%/90%/100% alerts only sends notifications (Email or Pub/Sub). It will <strong>never</strong> automatically shut down VMs or stop spending by itself. If you want automated cost capping, export budget alerts to a Pub/Sub topic and trigger a Cloud Function/Run to disable billing or delete resources.</li></ul><h4>Cost Optimization: CUD vs SUD (Exam Staple!)</h4><ul><li><strong>Committed Use Discounts (CUD)</strong>: 1-year or 3-year contractual commitment for steady-state workloads (up to 57% — 70% discount). Resource-based (vCPU/RAM) or Spend-based ($/hr commitment).</li><li><strong>Sustained Use Discounts (SUD)</strong>: Automatic discount (up to 30%) applied when running certain Compute Engine VM types (e.g. N1, N2) for more than 25% of a billing month. <strong>No upfront commitment needed</strong>!</li></ul><h4>Quota Management</h4><ul><li><strong>Rate Quotas</strong>: Resets after a time interval (e.g. 60 requests/minute to an API).</li><li><strong>Allocation Quotas</strong>: Maximum concurrent resources in a project/region (e.g. max 24 vCPUs or 5 static external IPs).</li><li><strong>Requesting Increase</strong>: Go to <strong>IAM &amp; Admin -&gt; Quotas</strong>, select the quota, and click <strong>Edit Quotas</strong> (requires business justification; approved automatically or reviewed by Google).</li></ul><p>Ofc, like other public cloud providers, you are able to view the cost with labels…</p><h4>Other shit</h4><ul><li>We can setting billing exports to BigQuery to analyze the costs!</li><li>Managing multiple gcloud environments / projects:</li><li>gcloud config set project &lt;PROJECT_ID&gt;</li><li>gcloud config configurations create &lt;NAME&gt; &amp; gcloud config configurations activate &lt;NAME&gt; (favorite exam topic for switching between dev, staging, and prod profiles without re-authenticating each time).</li></ul><h3>3. IAM</h3><p>Just little more detail of Cloud Identity I told above!</p><h4>Role Types (Classic Exam Topic)</h4><ul><li><a href="https://docs.cloud.google.com/iam/docs/roles-overview#role-types"><strong>Primitive / Basic Roles</strong></a>: Viewer, Editor, Owner. Broad and legacy. Never use in production!</li><li><strong>Predefined Roles</strong>: Fine-grained, managed and maintained by Google (e.g. roles/compute.instanceAdmin.v1, roles/storage.objectViewer). Recommended default!</li><li><strong>Custom Roles</strong>: User-tailored set of granular permissions when predefined roles are too broad.</li><li><strong>Exam Trap</strong>: Custom roles can ONLY be created at <strong>Organization</strong> or <strong>Project</strong> levels (CANNOT be created at Folder level, nor defined on individual resources!).</li><li>They do NOT auto-update when Google introduces new service permissions.</li></ul><h4>Service Accounts (Both Identity and Resource!)</h4><p>A Service Account (SA) has a dual nature:</p><ol><li><strong>As an Identity</strong>: You grant the SA permissions to talk to other GCP resources (e.g. read GCS bucket).</li><li><strong>As a Resource</strong>: You grant human users permissions <em>on the SA itself</em>!</li></ol><p><strong>The </strong><strong>serviceAccountUser vs </strong><strong>serviceAccountTokenCreator Trap (Exam favorite)</strong>:</p><ul><li><strong>Attaching to resources (</strong><strong>actAs)</strong>: If a developer needs to deploy a Compute Engine VM or Cloud Run service running under a Service Account, they need <strong>roles/iam.serviceAccountUser</strong> (iam.serviceAccounts.actAs) granted ON that Service Account. Without it, GCP throws a permission denied error when launching the VM.</li><li><strong>Direct Impersonation</strong>: If a developer needs to directly impersonate an SA (e.g. gcloud ... --impersonate-service-account or generate short-lived tokens via API), they need <strong>roles/iam.serviceAccountTokenCreator</strong>, NOT serviceAccountUser!</li></ul><p><strong>Default Service Account Risks</strong>:</p><ul><li>Compute Engine creates a default SA: &lt;project-number&gt;-compute@developer.gserviceaccount.com.</li><li><strong>Legacy behavior vs Modern Org Policy</strong>: Historically, it was automatically granted the primitive <strong>Editor</strong> role (a massive security hazard). Note that for organizations created after <strong>May 3, 2024</strong>, Google enforces the Org Policy constraint iam.automaticIamGrantsForDefaultServiceAccounts by default, disabling this automatic grant. However, in exam questions and older environments, this automatic Editor grant is still treated as a classic security trap!</li><li>Best practice / Exam answer: Disable or avoid using default service accounts; create a dedicated custom SA with least privilege.</li></ul><h4>Impersonate</h4><p>GCP IAM Impersonation example, impersonate GCP Service Account to test permission read storage. So you don’t need to use serviceAccount Key (More fuckin’ secure!)</p><pre>gcloud storage ls gs://my-company-bucket \<br>  --impersonate-service-account=app-reader@my-project.iam.gserviceaccount.com</pre><p>You feel like familiar with something in K8S? right? Yes, exactly! It is fuckin’ serviceAccount in K8S. Example test serviceAccount <strong>developer-sa</strong> in namespace <strong>dev</strong> if it is able to get pods!</p><pre>kubectl get pods -n dev --as=system:serviceaccount:dev:developer-sa</pre><h4>Service Account Keys vs Short-Lived Credentials</h4><ul><li>Avoid user-managed keys (.json key files). They can be leaked, committed to Git, and are painful to rotate.</li><li>VM / Cloud Run / GCF: Attach SA directly to resource instance metadata.</li><li>GKE: Workload Identity Federation.</li><li>Local dev / testing: gcloud auth application-default login or IAM Impersonation (roles/iam.serviceAccountTokenCreator).</li></ul><h4>Fine-grained accounts</h4><p>Format:</p><pre>[app-name]-[env]-[purpose]-sa</pre><p>And yes, fuckin’ <strong>Least privileges</strong> please!</p><h4>IAM Conditions</h4><p>Allow conditional role bindings based on attributes (e.g. resource name prefix, current time, or IP address):</p><ul><li>Example: Grant access to Cloud Storage only if bucket name starts with gs://dev-*, or temporary access that expires automatically at a specific date/time.</li></ul><h4>Workload vs Workforce Identity Federation</h4><ul><li>Workforce: for human (e.g. login via Microsoft Entra ID / Okta for employees).</li><li>Workload: for machine (e.g. GitHub Actions, AWS/Azure workloads calling GCP APIs without service account keys).</li></ul><h4>Useful gcloud IAM Commands</h4><pre># Add IAM binding to a project<br>gcloud projects add-iam-policy-binding &lt;PROJECT_ID&gt; \<br>  --member=&quot;user:dev@example.com&quot; \<br>  --role=&quot;roles/storage.objectViewer&quot;<br><br># Create a custom service account<br>gcloud iam service-accounts create my-app-sa \<br>  --display-name=&quot;My App SA&quot;<br><br># Grant a developer permission to use/attach that SA to VMs<br>gcloud iam service-accounts add-iam-policy-binding my-app-sa@&lt;PROJECT_ID&gt;.iam.gserviceaccount.com \<br>  --member=&quot;user:dev@example.com&quot; \<br>  --role=&quot;roles/iam.serviceAccountUser&quot;</pre><h3>4. Compute &amp; Scaling</h3><h4>VM Access</h4><p>We can login with IAM instead of SSH or User/Password. Actually behind the scene, GCP did many many thing it in:</p><ul><li><a href="https://docs.cloud.google.com/compute/docs/oslogin?authuser=1">OS Login</a></li><li><a href="https://docs.cloud.google.com/compute/docs/images/guest-agent-functions?authuser=1">Guest agent functionality</a></li><li><a href="https://github.com/GoogleCloudPlatform/guest-oslogin">Github Guest Os Login</a></li><li><a href="https://docs.cloud.google.com/iap/docs/using-tcp-forwarding"><strong>Identity-Aware Proxy (IAP) TCP forwarding</strong></a>: Enables secure SSH/RDP to VM instances over private internal IPs without needing public IP addresses or bastion/jump hosts.</li><li><strong>Short-Lived SSH Certificates</strong>: When using OS Login with 2FA or certificate authentication, Google Cloud issues ephemeral SSH certificates valid for only <strong>5 minutes</strong> (300 seconds), eliminating long-lived SSH key sprawl.</li></ul><p>Default Google-provided OS images already included the Guest Agent and OS Login packages come pre-installed and pre-configured. So we don’t need to install them manually xD.</p><p>The OS Login Package provides the actual binaries and Linux modules that communicate with GCP (detail in Github above):</p><ul><li>Authorized Keys Command</li><li>NSS Modules</li><li>PAM(Pluggable Authentication Modules) Modules: You will like me feel like the last word “Modules” is little duplicate, but actually they are widely used. For example: ATM machine -&gt; Automated Teller Machine machine, PDF format -&gt; Portable Document Format format, SQL language -&gt; Structured Query Language language. That is how we talk in common of software engineering I guess xD</li></ul><h4>Managed Instance Group (MIG)</h4><p>Almost same as AWS Auto Scaling Group (ASG), but keep in mind these GCP-specific gotchas:</p><ul><li><strong>Instance Templates are IMMUTABLE</strong>: You cannot edit them in-place; you must create a new version or copy from an existing one.</li><li><strong>Auto-healing</strong>: Requires a dedicated application health check (separate from the LB health check) to detect and recreate unhealthy VMs.</li><li><strong>Stateful MIG</strong>: Preserves attached persistent disks and static IPs across VM recreations (useful for stateful workloads like Kafka/DB replicas).</li></ul><h4>GCP Compute Quirks &amp; Exam Traps</h4><ul><li><strong>Spot vs Preemptible</strong>: Spot VMs have <strong>NO 24-hour limit</strong> (legacy Preemptible strictly died after 24h). Both have a 30-second termination notice.</li><li><strong>Live Migration</strong>: Default behavior during host maintenance is live migration with zero downtime. Exception: GPU instances cannot live-migrate (OnHostMaintenance=TERMINATE).</li><li><strong>Persistent Disk (PD) Types</strong>:</li><li>pd-standard (HDD): Low cost, large sequential I/O, backup targets.</li><li>pd-balanced (SSD): General-purpose balanced price/performance (default for many VMs).</li><li>pd-ssd (SSD): High performance, high IOPS/low latency for production databases.</li><li>pd-extreme / Hyperdisk: Ultra-high throughput and IOPS (dynamically configurable independent of disk size) for mission-critical enterprise workloads (SAP HANA, high-end Oracle).</li><li><strong>Persistent Disk multi-attach</strong>: Standard and SSD PDs can be attached to <strong>multiple VMs</strong> simultaneously in <strong>ReadOnly</strong> mode.</li><li><strong>Online disk expansion</strong>: Disks can be grown online without downtime, but can <strong>never be shrunk</strong>.</li><li><strong>Scopes vs IAM</strong>: Modern best practice is --scopes=cloud-platform on the VM, controlling actual permissions strictly via the assigned Service Account.</li></ul><h4>Other shit</h4><ul><li>Create Patch Deployment for patch management of fleet of computer engine, that will generate patch jobs</li></ul><h3>5. Google Kubernetes Engine (GKE)</h3><p>This one worth to mention since this is new for me.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*2B3W7wtcAZVxh99z.png" /></figure><p>And yes, we have same thing in Azure + AWS:</p><ul><li>Azure: Azure Container Apps or AKS Automatic</li><li>AWS: AWS Fargate (cho EKS / ECS)</li></ul><h4>Cluster Types: Zonal vs Regional (Exam Trap)</h4><ul><li><strong>Zonal</strong>: Single control plane in 1 zone. Control plane upgrades cause temporary API downtime (running workloads unaffected).</li><li><strong>Regional</strong>: Control plane replicated across 3 zones; node pools can span across multiple zones in the region (default is 3 nodes/zone = 9 nodes in default standard pool, but can be customized). Zero downtime during control plane upgrades. Higher SLA.</li><li><strong>Exam trap</strong>: Cluster location type is <strong>strictly immutable in BOTH directions</strong> — you <strong>CANNOT</strong> convert Zonal to Regional, nor Regional to Zonal (must spin up a new cluster and migrate workloads).</li></ul><h4>Standard vs Autopilot</h4><ul><li><strong>Standard</strong>: Full control over node pools, OS, and machine types. Pay per VM instance.</li><li><strong>Autopilot</strong>: Fully managed nodes. Pay strictly per Pod resource requests (vCPU/RAM). SSH to nodes and privileged containers are disabled by default.</li></ul><h4>Node Pool Machine Type Upgrades</h4><ul><li>You <strong>cannot modify the machine type</strong> of an existing node pool.</li><li>Migration pattern: Create a new node pool with target machine type -&gt; Cordon &amp; Drain old node pool (kubectl drain) -&gt; Delete the old node pool.</li></ul><h4>Security &amp; Connectivity</h4><ul><li><strong>Private Cluster</strong>: Nodes have NO public IPs (requires Cloud NAT for outbound internet).</li><li><strong>Master Authorized Networks</strong>: Restrict which specific CIDRs can access the K8s API server endpoint.</li><li><strong>Fetch credentials</strong>: gcloud container clusters get-credentials &lt;NAME&gt; --region=&lt;REGION&gt;</li></ul><h4>Cluster Autoscaling</h4><ul><li>Scales node pools based on pending (unschedulable) pod resource requests, NOT node CPU/memory utilization.</li></ul><h4>Workload Identity</h4><p>The recommended, secure way for GKE pods to access GCP services without keys (Equivalent to AWS IRSA).</p><ul><li>Bind a Kubernetes Service Account (KSA) to a Google Cloud Service Account (GSA) with roles/iam.workloadIdentityUser.</li><li>The GKE metadata server automatically exchanges the KSA token for short-lived Google Cloud credentials.</li></ul><h4>Backup for GKE</h4><ul><li>Fully managed service to protect, backup, and restore application workloads (K8s manifests/configurations) and persistent volume data in GKE clusters.</li></ul><h3>6. Serverless with Cloud Run</h3><h4>Cloud run services (Serverless containers)</h4><ul><li>GCP: Cloud Run</li><li>AWS: Fargate or App Runner</li><li>Azure: Azure Container Apps / App Service Container</li></ul><h4>Cloud run function (Serverless functions)</h4><ul><li>GCP: Cloud Run Functions</li><li>AWS: Lambda</li><li>Azure: Azure Functions</li></ul><h4>Event Trigger for function</h4><ul><li>GCP: Eventarc or Pub/Sub</li><li>AWS: EventBridge</li><li>Azure: Azure Event Grid</li></ul><h4>Example with Eventarc</h4><p>This is fuckin’ popular example that would appear in every tutorial, LOL. When user upload an image to Cloud Storage Bucket (S3), Eventarc will catch this event and trigger Cloud Run Function to handle (resize image, extract metadata of image and write to somewhere?)</p><p>Another example would to mention: Pub/Sub will related to decoupling services (using Eventarc). Also remember, there is no fuckin’ persistant data in Cloud Run Service (But mount NFS or publish data to Cloud Storage are fine xD)</p><h4>Concurrency (Differs from AWS Lambda)</h4><ul><li>In Lambda: 1 invocation = 1 container.</li><li>In Cloud Run: 1 container handles <strong>multiple concurrent requests</strong> (up to 1,000, default 80). Set --concurrency=1 only if your app cannot handle multi-threading/concurrency.</li></ul><h4>App Engine: Standard vs Flexible (Legacy Exam Favorite)</h4><ul><li><strong>Standard</strong>: Pre-configured sandboxes, scales to zero instantly, fast startup (ms), no Docker customizations, cannot write to local disk.</li><li><strong>Flexible</strong>: Runs on GCE VMs inside custom Docker containers. Supports any runtime, can write to disk, can SSH, but <strong>cannot scale to zero</strong> (min 1 VM, slower boot, more expensive).</li></ul><h3>7. Cloud Storage</h3><h4>Storage Classes</h4><p>Bucket name must be globally unique. We have some storage classes like others but almost just name different, LOL:</p><ul><li>Standard (Hot)</li><li>Nearline (Warm)</li><li>Coldline (Cool)</li><li>Archive (Cold)</li></ul><p>And for early deletion fee. The author explained pretty well and easy to understand in this picture:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*cEf0I2upO9HedTqv.png" /></figure><p>And there is little different in retrieval time of GCP vs AWS/Azure:</p><ul><li>Archive in GCP allow you able to read data from archive class</li><li>AWS (S3 Glacier) / Azure (Archive Tier): ask you to wait 3 to 12 hours to retrieve data after “request restore”.</li></ul><h4>Policies &amp; Permissions</h4><ul><li>Signed URL in GCP (GCS) = SAS Token in Azure (Blob Storage) = Presigned URL in AWS (S3)</li><li>Delete encryption key = permanent data loss, because there is no way to decrypt files storage in bucket without key (But we have Pending deletion for key and soft delete for files in bucket. So don’t worry if accident happens xD).</li><li>Lifecycle policy management: same as other, move files from current classes to other classes or delete it for costs saving. But wait, early deletion fees STILL apply if objects are deleted or overwritten before their minimum duration (30d Nearline, 90d Coldline, 365d Archive)! Though transitions themselves are free, and Autoclass buckets are exempt from early deletion fees.</li><li>And use lifecycle for know, autoclass for unknown. Age condition then set storage class actions.</li><li>Autoclass ignores file size &lt; 128Kb by default, it only transitions to Nearline (unless you configured it for Archive).</li><li>And even with Automation, remember minimum durations: Nearline(30 days), Coldline(90 days), Archive (365 days)</li><li>Uniform Bucket-Level access is the standard, no need legacy ACL and required to use feature like Managed Folders. And enabling Uniform bucket-level access will disable existing object-level ACLs.</li></ul><h4>Storage Transfer Service (STS)</h4><ul><li>Use when needed for large transfers into GCS: support Cloud to Cloud or On-premises to Cloud</li><li>Able to run directly or via scheduling, run synchronization and have data integrity for validation.</li></ul><h4>Bucket Lock (Retention Policy Lock — Compliance)</h4><ul><li>Set retention policy for WORM (Write Once, Read Many) compliance.</li><li><strong>Exam trap</strong>: Once you <strong>Lock</strong> the retention policy, it is <strong>IRREVERSIBLE</strong>. Neither the Project Owner nor Google Cloud Support can unlock it, delete the bucket, or reduce the retention duration until it expires.</li></ul><h4>Data Migration: STS vs Transfer Appliance</h4><ul><li><strong>Storage Transfer Service (STS)</strong>: Online over-the-network transfer (from S3/Azure/on-prem).</li><li><strong>Transfer Appliance</strong>: High-capacity physical hardware appliance shipped to your datacenter. Choose this when data volume is massive (&gt;10–20 TB up to PB) and network bandwidth is too slow/constrained.</li></ul><h4>Dual-Region &amp; Turbo Replication</h4><ul><li><strong>Dual-Region</strong>: Replicated across 2 specific regions asynchronously by default (no strict replication time SLA).</li><li><strong>Turbo Replication</strong>: An <strong>optional, paid add-on</strong> that provides an SLA-backed guarantee: 100% of newly written objects are replicated to the paired region within <strong>15 minutes</strong> (critical for strict DR/RPO requirements).</li></ul><h3>8. Managed Databases in GCP</h3><h4>Relational (SQL) options</h4><ul><li>Cloud SQL: Mysql, PostgreSQL, SQL Server…</li><li>AlloyDB: PostgreSQL by GCP, not fuckin’ open source xD, like Aurora in AWS.</li><li>Cloud Spanner: horizontally scalable Relational (SQL) database (AWS doesn’t have a direct equivalent, but think of it as managed CockroachDB/YugabyteDB). Offers a signature SLA of 99.999% for multi-region (and 99.99% for single-region).</li></ul><h4>NoSQL options</h4><ul><li>Firestore: Document database, like MongoDB. Perfect for mobile/web apps</li><li>Bigtable: Wide-column NoSQL database, same as Apache HBase or Apache Cassandra. (Clickhouse/TiFlash are OLAP column stores, their actual GCP equivalent is BigQuery, not Bigtable).</li><li><strong>Use cases</strong>: IoT data, clickstreams, time-series, financial data. Massive scale, high-throughput, and low latency (TB to PB).</li><li><strong>Single index</strong>: Data is only indexed by the row key.</li><li><strong>Avoid Hotspotting</strong>: Sequential row keys (like raw timestamps or alphabetical ID sequences) cause traffic to hit a single node (hotspotting). Prevent this by salting (pre-pending a hash), reversing IDs, or promoting fields.</li><li><strong>Dynamic scale</strong>: Storage is decoupled from compute. You can dynamically add/remove nodes to scale throughput/IOPS without downtime or data migration.</li><li><strong>Storage Types</strong>: SSD (default, recommended for low-latency queries) vs HDD (recommended for large historical/archival datasets where latency isn’t critical).</li><li><strong>Tooling</strong>: Use the cbt CLI tool to interact with and query Bigtable.</li><li>MemoryStore: Redis/Memcached</li></ul><h4>Backup &amp; Recovery</h4><ul><li><strong>PITR (Point-in-Time Recovery)</strong>: Always restores to a <strong>NEW database instance with a NEW IP address</strong>; the original instance remains untouched. And this required enable binary logigng option!</li><li><strong>Automated backups</strong>: Bound to the instance lifecycle (deleted when the instance is deleted). Export to GCS for permanent retention.</li></ul><h4>High availability (HA) vs Failover Trap</h4><ul><li><strong>HA Failover</strong>: Uses a standby instance in a secondary zone and regional PD. <strong>IP address DOES NOT CHANGE</strong> during failover (no connection string reconfig needed for apps!).</li><li><strong>Cloud SQL Auth Proxy</strong>: Google-recommended best practice to connect from apps/GKE/VMs to Cloud SQL over IAM &amp; TLS without managing SSL certs or whitelisting public IPs.</li></ul><h4>Exam Decision Matrix: Which Database to Pick?</h4><ul><li><strong>Cloud SQL</strong>: Traditional RDBMS (PostgreSQL/MySQL/SQL Server), single-region, OLTP.</li><li><strong>Cloud Spanner</strong>: Global/Multi-region distributed SQL, horizontal write scaling, 99.999% SLA.</li><li><strong>Firestore</strong>: Mobile/web document DB, real-time client sync, offline support.</li><li><strong>Bigtable</strong>: Heavy write throughput, low latency (&lt;10ms), IoT/time-series/financial data (TB to PB).</li><li><strong>BigQuery</strong>: OLAP / SQL queries on petabytes (never use Bigtable for analytical ad-hoc aggregations).</li></ul><h4>Other shit</h4><ul><li>Want a centralized view -&gt; go for Database Center</li><li><a href="https://cloud.google.com/products/alloydb?hl=en">AlloyDB</a> is a fully managed PostgreSQL-compatible database service for your most demanding enterprise database workloads</li></ul><h4>Cloud Managed Services vs. Open-Source Equivalents</h4><p>Not related but I want to put here, everytime I see services in AWS/GCP/Azure, I often search if there is same solution but <strong>open-source</strong>. So here is my little research:</p><ul><li>GCP Cloud Spanner = CockroachDB/YugabyteDB (Distributed SQL).</li><li>AWS DynamoDB = ScyllaDB (Distributed NoSQL, Key-Value/Documents.</li><li>Cloud-Native RDBMS: Aurora (AWS) &amp; AlloyDB (GCP). Not multi-write, still using single master (primary node for write).</li></ul><h3>9. Big Data &amp; Analytics Platforms</h3><h4>BigQuery Optimization (Exam Favorites)</h4><ul><li><strong>Partitioning</strong>: Segments table by a Date/Timestamp column. Massive cost &amp; performance booster because queries only scan the relevant partition (e.g. WHERE date = &#39;2026-01-01&#39;).</li><li><strong>Clustering</strong>: Colocates and sorts data based on up to 4 columns. Best for high-cardinality filters (WHERE customer_id = 123).</li><li><strong>Best Practice</strong>: Partition by date first, then cluster by frequently filtered dimensions.</li><li><a href="https://docs.cloud.google.com/bigquery/docs/running-queries"><strong>BigQuery dry run</strong></a>: Estimates the number of bytes a query will scan and validates syntax for free, without using query slots or processing actual data.</li></ul><h4>Dataproc vs Dataflow (Cheat Sheet)</h4><ul><li><strong>Dataproc</strong>: Managed <strong>Hadoop / Spark / Hive</strong> (AWS EMR equivalent). Pick this when migrating existing on-prem Spark/Hadoop jobs with <strong>minimal code changes</strong>.</li><li><strong>Dataflow</strong>: Fully managed, serverless stream/batch processing based on <strong>Apache Beam</strong>. No clusters to size or manage. For example: Raw Data (CSV, logs…) → Dataflow (Transform/aggregate) → Clean Data in BigQuery.</li><li><strong>Pub/Sub</strong>: Ingestion buffer for streaming pipelines (Global, decoupled, at-least-once delivery).</li></ul><h3>10. Managed file storage</h3><ul><li><strong>Filestore</strong>: Managed <strong>NFSv3</strong> (POSIX-compliant) file share. Equivalent to AWS EFS / Azure Files.</li><li><strong>ReadWriteMany (RWX)</strong>: Pick this when multiple VMs or GKE pods must mount and <strong>concurrently write</strong> to the same shared filesystem (since Persistent Disks only support multi-attach in ReadOnly mode).</li><li>Use cases: Legacy apps lift-and-shift, shared CMS (WordPress media), GKE shared PVCs.</li><li>Minimum provisioned size starts at <strong>1 TiB</strong> (applies to legacy Basic HDD/SSD; newer Zonal and Enterprise tiers have different minimum capacity thresholds).</li><li><strong>Google Cloud NetApp Volumes</strong>: Enterprise-grade NFS/SMB file services (primarily for Windows SMB workloads and high-performance enterprise storage).</li></ul><h3>11. Virtual Private Cloud (VPC) Networking</h3><p>Unlike AWS or Azure, a Google VPC is <strong>Global</strong> by default, remember this!</p><h4>Auto vs Custom Mode VPC (One-way street!)</h4><ul><li><strong>Auto Mode</strong>: Creates a /20 subnet in every region automatically. Risk of IP overlapping with on-prem networks.</li><li><strong>Custom Mode</strong>: Production standard. Zero subnets by default; define your own CIDRs.</li><li><strong>Exam trap</strong>: You can convert Auto Mode to Custom Mode (switch-mode --mode=custom), but <strong>CANNOT convert Custom Mode back to Auto Mode</strong>.</li></ul><h4>Subnet</h4><ul><li>Subnets are regional resources. They are Cross-zone (meaning a single subnet spans all zones within its region!)</li><li>When IPs in subnet run out of IP, zero downtime expansion, I meaning it is able to change the subnet mask (ex: from /24 to /20) without fuckin’ downtime! And yeah expand is easy but not able to shrink xD, like volumes.</li></ul><h4>Cloud NGFW</h4><p>It is fuckin’ stateful, meaning you only need to allow ingress and it will allow egress for you! No need a seprate egress rule.</p><p><a href="https://docs.cloud.google.com/firewall/docs/firewall-policies"><strong>Hierarchical firewall policies</strong></a>: Enforce consistent firewall policies across the entire Organization or Folder levels (evaluated before VPC firewall rules).</p><h4>Shared VPC</h4><p>No need to talk about VPC Peering but shared VPC, one network, many projects:</p><ul><li>Multiple projects share the same VPC</li><li>Centralized control by networking team</li><li>One set of firewall rules for all</li></ul><h4>Private access to google services private</h4><p>How to enable:</p><ul><li>Per subnet setting (you can turn it on in subnet setting)</li><li>Internal traffic only</li><li>Google services only.</li></ul><p>What we can access via private services:</p><ul><li>Cloud Storage</li><li>BigQuery</li><li>Pub/Sub</li><li>Cloud Logging &amp; Monitoring</li><li>Container Registry</li><li>Artifact Registry</li></ul><h4>PGA vs PSA (Don’t confuse these two!)</h4><ul><li><strong>Private Google Access (PGA)</strong>: Subnet-level setting. Allows VMs with only internal IPs to reach <strong>Google APIs</strong> (Cloud Storage, BigQuery, Pub/Sub).</li><li><strong>Private Services Access (PSA)</strong>: Internal peering connection to reach managed services inside Google’s tenant VPC (specifically <strong>Cloud SQL &amp; Memorystore</strong>).</li></ul><p>If you need internet access, you will need cloud NAT, hmm what about Cloud NAT:</p><ul><li>One Cloud Nat gateway for the fuckin’ entire subnet or region.</li><li>Auto scales to handle traffic from thousands of VMs.</li><li>Outbound only.</li></ul><h4>Cloud DNS &amp; Hybrid Resolution</h4><ul><li><a href="https://oneuptime.com/blog/post/2026-02-17-how-to-set-up-inbound-dns-forwarding-to-allow-on-premises-queries-to-cloud-dns-in-gcp/"><strong>Inbound DNS forwarding</strong></a>: Allows on-premises networks to resolve private DNS zones hosted in Cloud DNS by configuring a DNS server policy.</li></ul><h3>12. Load Balancer</h3><p>Nothing to talk much about this section, remember the different between 2 tiers; Standard vs Premium Tier:</p><ul><li>Standard Tier: cheaper, more hops, can not use global LB.</li><li>Premium Tier: cost more, less hops (which mean faster, lower latency).</li></ul><h4>GCP Load Balancing Quirks vs AWS</h4><ul><li><strong>Global Anycast IP</strong>: Unlike AWS (which provides regional DNS names), GCP Global External LBs provide a <strong>single Anycast IP globally</strong>. Traffic enters Google’s private backbone at the nearest edge PoP(Point of Presence).</li><li><strong>Proxy vs Passthrough (Client IP Trap)</strong>:</li><li><strong>Proxy LBs (L7 HTTP/S &amp; SSL Proxy)</strong>: Terminates connection at LB. Backend sees LB proxy IP; original IP is in X-Forwarded-For.</li><li><strong>Passthrough LBs (L4 Network LB)</strong>: Does NOT terminate TCP. <strong>Preserves original client source IP natively</strong> at packet level (Direct Server Return).</li><li><strong>Health Check Firewall Trap</strong>: If all backend VMs report unhealthy, verify firewall allows health check probes from Google’s fixed IP ranges: 35.191.0.0/16 and 130.211.0.0/22.</li><li><strong>Cloud Armor (WAF/DDoS) &amp; Cloud CDN</strong>: Can ONLY attach to <strong>External Application Load Balancers</strong> (HTTP/HTTPS).</li></ul><h4>Other shit</h4><ul><li>When autoscaling based on Load Balancing capacity, you must first define the capacity (Maximum RPS) in the Backend Service. Once defined, configure the autoscaler to maintain a target utilization percentage (commonly 80%) of that capacity.</li></ul><h3>13. Infrastructure as Code (IaC) &amp; CI/CD</h3><p>LOL, first time I heard about <strong>Config Connector</strong>, you have k8s cluster running in GCP, manifest with kind Storage Bucket, kubectl apply that fuckin’ file, config connector calls GCP APIs.</p><p>And <strong>Fabric FAST</strong>, ready-made landing zone for new org with pre-built terraform modules, GCP best practices, secure by default, production-ready.</p><h4>Terraform on GCP Best Practice</h4><ul><li><strong>GCS Remote Backend</strong>: Natively supports state locking (no separate locking database required like legacy AWS S3/DynamoDB setup).</li><li><strong>State Security</strong>: Always enable <strong>Object Versioning</strong> on the state GCS bucket to protect against state file corruption or accidental overwrites.</li></ul><h4>Cloud Deployment Manager (GCP Native IaC)</h4><ul><li>Google’s native declarative service (equivalent to AWS CloudFormation / Azure ARM). Uses YAML configs with Jinja2 or Python templates. (Though Google officially recommends Terraform now).</li></ul><h4>CI/CD: Source Repositories, Cloud Build &amp; Cloud Deploy</h4><ul><li><strong>Cloud Source Repositories (CSR)</strong>: Fully managed private Git repositories hosted on Google Cloud (AWS CodeCommit equivalent). Integrates natively with Cloud Build triggers and uses Cloud IAM for access control.</li><li><strong>Cloud Build</strong>: Fully managed serverless build service (equivalent to AWS CodeBuild / GitHub Actions).</li><li>Defined via cloudbuild.yaml (or Dockerfile) using containerized steps (name: &#39;gcr.io/cloud-builders/docker&#39;, name: &#39;gcr.io/google.com/cloudsdktool/cloud-sdk&#39;).</li><li><strong>Triggers</strong>: Automatically runs builds on code pushes or PRs from GitHub, GitLab, or Cloud Source Repositories.</li><li><strong>Service Account Permissions Trap</strong>: Cloud Build executes under its default service account ([project-number]@cloudbuild.gserviceaccount.com). To deploy to GKE or Cloud Run, you <strong>MUST explicitly grant</strong> that service account the required target roles (e.g. roles/run.admin, roles/container.developer, roles/iam.serviceAccountUser).</li><li><strong>Cloud Deploy</strong>: Managed, opinionated continuous delivery pipeline for GKE, Cloud Run, and Anthos. Provides multi-target promotion (dev -&gt; staging -&gt; prod), release approvals, and automatic rollbacks.</li></ul><h4>AI Tooling</h4><ul><li>Application Design Center: is a console tool that converts plain language descriptions into visual architecture diagrams and deployable Terraform code, using pre-approved, best-practice templates.</li><li>Cloud Hub: is your unified operational dashboard, giving a single view of application health, alerts, costs, security findings, and quota usage across all your deployed applications.</li></ul><h3>14.Monitoring</h3><p>Nothing much worht to mention here, except Gemini Cloud Assist, since we sent logs, metrics to GCP and simply we only need ask Gemini what is the fuckin’ wrong, it’s accuracy about 70–80%. Pretty sure we need to validate it before apply xD</p><h4>Multi-Project Monitoring (Metrics Scope)</h4><ul><li>Host multiple projects under a single <strong>Scoping Project</strong>.</li><li>Allows a central SRE team to view dashboards, charts, and alerts across multiple projects in a single pane of glass.</li></ul><h4>Log-based Metrics (Bridge between Logs &amp; Alerts)</h4><ul><li>Convert log occurrences (e.g. counting ERROR 500 strings or regex matches) into measurable metrics in Cloud Monitoring to trigger alert policies.</li></ul><h4>Uptime Checks &amp; Muting Rules</h4><ul><li><strong>Uptime Checks</strong>: Probes public endpoints from multiple global locations to verify external availability and latency.</li><li><strong>Muting Rules</strong>: Temporarily suppress notification channels during planned maintenance windows.</li></ul><h4>Ops Agent (Guest-level Metrics)</h4><ul><li>Default Compute Engine monitoring only captures hypervisor-level metrics (CPU, disk throughput, network).</li><li>To monitor <strong>Memory (RAM) utilization, available disk space, and process-level metrics</strong>, you MUST install the <strong>Google Cloud Ops Agent</strong> inside the VM.</li></ul><h3>15. Logging</h3><p>GCP Cloud logging = AWS CloudWatch Logs.</p><ul><li>Log Buckets are container within cloud logging. <strong>_Required</strong> bucket stores for 400 days and can not be disabled. <strong>_Default</strong> bucket stores for 30 days by default,retention can be optimized.</li><li>Buckets are regional, a specialized, database-like engine built exclusively inside Cloud Logging.</li><li>Log Analytics allows you to run SQL queries directly against your log data without exporting it to BigQuery.</li><li>You can upgrade existing bucket to use Log Analytics.</li><li>Cloud Audit Log GCP = AWS CloudTrail</li><li>Need more metrics (Ram, disk space used, app metrics…), required to install Agent Metrics (Ops Agent).</li><li>Cloud Trace is for tracing, find latency bottlenecks, why this shit takes 5 seconds to complete!</li><li>Cloud Profiler is for code performance: Finding cpu/memory leaks in specific functions.</li><li>Query Insights helps you identify slow SQL queries in Cloud SQL based on CPU and latency.</li></ul><h4>Log Router Sinks &amp; Writer Identity Trap</h4><ul><li><strong>4 Valid Destinations</strong>:</li></ul><ol><li><strong>Cloud Storage</strong>: Cheap long-term compliance archival.</li><li><strong>BigQuery</strong>: Analytical querying &amp; BI dashboards.</li><li><strong>Pub/Sub</strong>: Stream logs to external SIEMs (Splunk, Datadog, Elastic) or trigger automation.</li><li><strong>Another Cloud Logging Bucket</strong>: Centralize logs in a dedicated security/compliance project.</li></ol><ul><li><strong>Writer Identity Trap</strong>: Creating a sink generates a unique service account (Writer Identity). You <strong>MUST explicitly grant</strong> that identity write permission on the destination resource (e.g. Storage Object Creator on the target GCS bucket).</li></ul><h4>Cloud Audit Logs: Admin Activity vs Data Access</h4><ul><li><strong>Admin Activity</strong>: Modifying metadata/configs (create VM, update IAM). <strong>Always ON, FREE</strong>, retained 400 days in _Required.</li><li><strong>Data Access</strong>: Reading or writing user data (reading GCS objects, querying tables). <strong>DISABLED by default</strong> (except BigQuery) to prevent massive volume &amp; ingestion costs. Must be manually enabled.</li></ul><h4>Exclusion Filters (Cost Optimization)</h4><ul><li>Drop noisy, high-volume logs (e.g., LB health check pings or verbose DEBUG logs) before ingestion to avoid storage charges.</li></ul><h3>16. Other shit</h3><ul><li><strong>Vertex AI Workbench</strong> (rebranded to <strong>Gemini Enterprise Agent Platform Workbench</strong> since Cloud Next ‘26): Managed Jupyter notebook-based development environments on Google Cloud without managing infrastructure. (Exam questions still refer to it as <strong>Vertex AI Workbench</strong>).</li><li><a href="https://cloud.google.com/blog/products/management-tools/active-assist-comes-to-google-cloud"><strong>Active Assist</strong></a> refers to the portfolio of tools used in Google Cloud to generate recommendations and insights to help you optimize your Google Cloud projects (cost, security, performance).</li><li><strong>Data Encryption Tiers (KMS Trap)</strong>:</li><li><strong>Google-default</strong>: Encrypted at rest automatically with Google-managed keys (zero config, free).</li><li><strong>CMEK</strong>: Keys generated and managed in <strong>Google Cloud KMS</strong> (customer controls rotation and IAM).</li><li><strong>CSEK</strong>: Keys generated and kept on-premises by customer, provided with API calls. <strong>Google NEVER stores CSEK keys</strong> (if you lose the key, data is permanently unrecoverable).</li><li><strong>Artifact Registry</strong>: The official and sole container/package registry on GCP following the shutdown of Container Registry (GCR) in early 2025. Supports Docker/OCI images, Helm charts, and language packages (npm/Maven/Python) with fine-grained per-repository IAM (legacy gcr.io hostnames automatically route to Artifact Registry repositories).</li><li><strong>Secret Manager</strong>: Securely stores API keys and credentials with versioning. Applications only need <strong>roles/secretmanager.secretAccessor</strong> to retrieve secret payloads.</li><li><strong>Cloud Shell</strong>: Ephemeral VM with a <strong>persistent 5 GB </strong><strong>$HOME directory</strong>. Auto-terminates after 20 minutes of inactivity (anything outside $HOME is lost).</li><li>Google Cloud’s <a href="https://cloud.google.com/tpu?hl=en"><strong>Tensor Processing Units</strong></a> (TPUs) are custom-built to help speed up machine learning workloads</li></ul><h3>17. Conclusion</h3><p>A quick wrap-up for anyone preparing for the exam:</p><ul><li><strong>Hands-on labs are not strictly mandatory</strong>: If you already have a solid foundation in Linux, networking, containers, and another major cloud (AWS/Azure), you don’t necessarily need to spend hours doing hands-on labs to pass this exam (like me xD). Grasping the architectural concepts, GCP-specific terminologies, and exam traps is more than enough.</li><li><strong>Practice Exams</strong>: Highly recommend checking out this course to get familiar with question patterns and real exam scenarios: <a href="https://www.udemy.com/course/google-certified-associate-cloud-engineer-practice-exams-gcp">Google Certified Associate Cloud Engineer Practice Exams on Udemy</a>.</li><li><strong>Final thoughts</strong>: This is definitely my last cloud cert for now because I don’t really have many opportunities to work extensively with cloud infrastructure in my current daily work. I will only aim for higher-level certs (like Professional DevOps / Architect) when I actually have the chance to work directly with production cloud infra.</li></ul><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=b8b82a7529fe" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Python standard json vs orjson]]></title>
            <link>https://medium.com/@kienlt.qn/python-standard-json-vs-orjson-c4c2cc00954f?source=rss-b77f4e43d676------2</link>
            <guid isPermaLink="false">https://medium.com/p/c4c2cc00954f</guid>
            <category><![CDATA[python]]></category>
            <category><![CDATA[orjson]]></category>
            <dc:creator><![CDATA[Kienlt]]></dc:creator>
            <pubDate>Sun, 30 Aug 2026 17:00:21 GMT</pubDate>
            <atom:updated>2026-08-31T14:24:12.914Z</atom:updated>
            <content:encoded><![CDATA[<p>TLDR: Use <strong>orjson</strong> when you need performance when dumping (serializing) json data, where every millisecond counts.</p><h3>Introduction</h3><p>In our current project at work, I noticed that my team uses orjson instead of Python&#39;s standard json module. At first, I was like: <em>&quot;Why do we need to </em><em>pip install an external library when Python already has built-in </em><em>json support?&quot;</em>. I&#39;m a simple guy who likes to keep things simple xD</p><p>My boss told me that <strong>orjson</strong> is much faster than standard <strong>json</strong>, and I was like: “Ohh, this shit is definitely something worth digging into!”</p><p>To satisfy my curiosity, I used AI to generate a small benchmark to test both on 500,000 records. And the result is good as I expected for: orjson <strong>is faster (about ~9.3x) on serialization (dumps)!!!</strong></p><h3>Section 1: Quick Comparison</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/903/1*J16Ak9rrOFKTadaGVDGUig.png" /></figure><h3>Section 2: Benchmark Results</h3><p>I benchmarked 3 different scenarios with <strong>500,000 records</strong> on my local machine:</p><ol><li><strong>bench_orjson.py</strong>: <strong>orjson</strong> with raw datetime &amp; UUID.</li><li><strong>bench_stdlib_json_no_str.py</strong>: <strong>Standard </strong><strong>json (no pre-str)</strong>, with raw datetime &amp; UUID + default=str.</li><li><strong>bench_stdlib_json.py</strong>: <strong>Standard</strong><strong>json</strong>, pre-wrapped str() during data creation.</li></ol><h4>The Numbers</h4><p>Environment:</p><ul><li>Python: 3.10</li><li>orjson: 3.10.0</li><li>CPU: AMD Ryzen 5900X</li></ul><pre>((venv) ) kienlt@kienlt-pc:/data/work$ python bench_orjson.py <br>Generating 500,000 records with raw datetime &amp; UUID...<br>[ORJSON] Dumps time : 0.1439 seconds<br>[ORJSON] Payload size: 71045.02 KB<br>[ORJSON] Loads time : 0.2917 seconds<br><br>((venv) ) kienlt@kienlt-pc:/data/work$ python bench_stdlib_json_no_str.py <br>Generating 500,000 records with raw datetime &amp; UUID...<br>[STANDARD JSON] Dumps time : 1.3381 seconds<br>[STANDARD JSON] Payload size: 75927.84 KB<br>[STANDARD JSON] Loads time : 0.3811 seconds<br><br>((venv) ) kienlt@kienlt-pc:/data/work$ python bench_stdlib_json.py <br>Generating 500,000 records...<br>[STANDARD JSON] Dumps time : 0.3866 seconds<br>[STANDARD JSON] Payload size: 75927.84 KB<br>[STANDARD JSON] Loads time : 0.3766 seconds</pre><h4>Better view</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/902/1*-9Z_vUOaUjAeCoC3RTGkmQ.png" /></figure><h3>Section 3: Why orjson is faster?</h3><h4>The “Python Callback” Bottleneck</h4><p>In real-world applications (FastAPI, ORMs, Data pipelines), data usually contains real objects like datetime.datetime and uuid.UUID.</p><ul><li>Standard <strong>json</strong>: Doesn’t know what datetime is. For every record, it has to jump back <a href="https://github.com/python/cpython/blob/main/Modules/_json.c#L1637-L1640">to Python to execute default=str</a>. With 500k records (1 UUID + 1 datetime each), Python runs 1M callbacks. This kills serialization performance.</li><li><strong>orjson</strong>: Written in Rust. It reads the CPython object memory pointers directly and formats them into ISO-8601 strings in machine code without touching Python runtime.</li></ul><h4>Outputting bytes directly</h4><p>Here is a quick example:</p><pre>import json<br>import orjson<br><br>data = {&quot;username&quot;: &quot;kienlt&quot;, &quot;role&quot;: &quot;admin&quot;}<br><br>res_json = json.dumps(data) # &lt;--- json.dumps() returns str, not bytes<br>print(type(res_json))  # &lt;class &#39;str&#39;&gt;<br>print(res_json)<br><br>res_orjson = orjson.dumps(data)  # &lt;--- orjson.dumps() returns bytes, not str<br>print(type(res_orjson))  # &lt;class &#39;bytes&#39;&gt;<br>print(res_orjson)</pre><p>Output:</p><pre>&lt;class &#39;str&#39;&gt;<br>{&quot;username&quot;: &quot;kienlt&quot;, &quot;role&quot;: &quot;admin&quot;}<br>&lt;class &#39;bytes&#39;&gt;<br>b&#39;{&quot;username&quot;:&quot;kienlt&quot;,&quot;role&quot;:&quot;admin&quot;}&#39;</pre><h4>Why is the payload ~4.77 MB smaller</h4><ul><li><a href="https://docs.python.org/3/library/json.html#basic-usage">Standard <strong>json</strong></a>: Uses <strong>separators=(‘, ‘, ‘: ‘)</strong>, adding a space after every comma and colon.</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/819/0*ZabFMJyT0CjSinYH.png" /></figure><ul><li><a href="https://github.com/ijl/orjson#serialize"><strong>orjson</strong></a>: Uses compact format <strong>separators=(‘,’, ‘:’)</strong> by default, eliminating all unnecessary whitespace. Every eliminated whitespace adds up to saved bandwidth and memory! Take another look at the output above:</li></ul><pre>&lt;class &#39;str&#39;&gt;<br>{&quot;username&quot;: &quot;kienlt&quot;, &quot;role&quot;: &quot;admin&quot;} ==&gt; spaces<br>&lt;class &#39;bytes&#39;&gt;<br>b&#39;{&quot;username&quot;:&quot;kienlt&quot;,&quot;role&quot;:&quot;admin&quot;}&#39; ==&gt; No fuckin&#39; space</pre><p>You can see there is no fuckin’ space when we print!</p><h3>Section 4: When should you use which?</h3><h4>Use Standard json when</h4><ul><li>Writing simple automation scripts or CLI tools where you want zero external dependencies (pip install).</li><li>Performance is not a bottleneck (who cares if an automation script runs 1–2ms faster?).</li></ul><h4>Use orjson when</h4><ul><li>Building high-throughput Web APIs where performance is a must (e.g., FastAPI handling thousands of requests or IoT devices!!!)</li><li>Processing large JSON files or stream data (Kafka consumers, log pipelines).</li></ul><h4>Trade-offs of orjson (things to know before switching)</h4><ul><li><strong>Indent is limited to 2 spaces</strong> — option=orjson.OPT_INDENT_2 is the only indentation level supported, unlike stdlib&#39;s, indent=N which accepts any integer. (<a href="https://github.com/ijl/orjson#opt_indent_2">orjson README</a>) Since <strong>orjson</strong> is designed with performance as the top priority, hardcoding the indent logic lets the Rust compiler optimize it better at compile time, instead of handling an arbitrary N at runtime.</li><li><strong>sort_keys isn&#39;t a kwarg</strong> — it&#39;s replaced byoption=orjson.OPT_SORT_KEYS, so any code doing json.dumps(data, sort_keys=True) needs a small rewrite. (<a href="https://github.com/ijl/orjson#opt_sort_keys">orjson README</a>) Standard json parses sort_keys as a keyword argument and checks it on every call. orjson replaces this with a single bitwise flag, so the check becomes one cheap bit operation instead of parsing/validating a Python kwarg each time — saving a bit of CPU overhead.</li><li>And more, but I don’t fully understand them yet, so I’ll leave them out of this article xD</li></ul><h3>References</h3><ul><li><a href="https://gitlab.com/blackmetalz/compare_orjson_vs_json_default_python">GitLab Repo: compare_orjson_vs_json_default_python</a></li><li><a href="https://docs.python.org/3/library/json.html">Python json Documentation</a></li><li><a href="https://github.com/ijl/orjson">orjson GitHub Repository</a></li></ul><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=c4c2cc00954f" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Understand ArgoCD architecture with theory and hands-on]]></title>
            <link>https://medium.com/@kienlt.qn/understand-argocd-architecture-with-theory-and-hands-on-34fcc76c04e7?source=rss-b77f4e43d676------2</link>
            <guid isPermaLink="false">https://medium.com/p/34fcc76c04e7</guid>
            <category><![CDATA[gitops-with-argocd]]></category>
            <category><![CDATA[argo-cd]]></category>
            <category><![CDATA[kubernetes]]></category>
            <dc:creator><![CDATA[Kienlt]]></dc:creator>
            <pubDate>Sun, 26 Jul 2026 05:23:11 GMT</pubDate>
            <atom:updated>2026-07-26T05:23:11.385Z</atom:updated>
            <content:encoded><![CDATA[<h3>Intro</h3><p>I thought I understood ArgoCD well enough. In the past, I ran ArgoCD with multiple K8s clusters inside it (&gt;10 clusters), and had around 500–600 ArgoCD applications in multiple k8s clusters. But I was a contributor there, not the one who built it from scratch, and it turns out I had 100% misconceptions that I often ran into. Even though I had the <strong>Certified Argo Project Associate (CAPA)</strong>, it was a fuckin’ joke to me, no fuckin’ value gained! So that is the reason this article was born to help me understand the “basic” knowledge of ArgoCD!</p><p>Ok, we will go through the sections below in this article:</p><ul><li>Architecture &amp; Reconciliation loop</li><li>Application — atomic unit</li><li>Cluster registration</li><li>AppProject — governance layer</li><li>Sync mechanics deep dive</li><li>RBAC 3 layers</li><li>How Sealed Secrets works</li><li>Hands-on Lab</li><li>Conclusion</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/800/0*KiotEu3_BnIhgyQD.png" /></figure><h3><a href="https://argo-cd.readthedocs.io/en/stable/operator-manual/architecture/">Architecture</a> &amp; Reconciliation loop</h3><p>ArgoCD is a workload that commonly runs in the <strong>argocd</strong> namespace. I thought ArgoCD’s watch mechanism was the same as K8s’s watch mechanism, but it turns out there’s a little difference, let’s figure it out!</p><h4>Difference between ArgoCD and Kubernetes</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*s8H9U_ceDcFLkXVoHC8qDQ.png" /></figure><h4>Static View</h4><ul><li><strong>argocd-server</strong>: API Gateway, takes requests from UI/CLI, checks auth/RBAC of user only.</li><li><strong>argocd-repo-server</strong>: clones Git, runs <strong>helm template</strong> or <strong>kustomize build</strong> to render raw yaml.</li><li><strong>argocd-application-controller</strong>: where we run reconciliation loop to see the drift between the desired state (rendered YAML from repo-server) versus live state in k8s. This is the only component that calls the API to apply manifests.</li><li><strong>argocd-redis</strong>: Caches the rendered manifests. Without Redis, each sync check, repo-server will need to clone Git and render helm again! That is why we need fuckin’ cache!</li></ul><h4>Dynamic View</h4><p>Here is the flow (combining <a href="https://argo-cd.readthedocs.io/en/stable/operator-manual/architecture/">Architecture</a> and <a href="https://argo-cd.readthedocs.io/en/stable/operator-manual/webhook/">Webhook</a> docs):</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*JjuMF4sPbq2khwQ4.png" /></figure><h4>Kubernetes Controller Manager: Real-time Watch (Event-Driven)</h4><p>In K8s, Controllers (like the ReplicaSet Controller, Deployment Controller) set up an <a href="https://kubernetes.io/docs/reference/using-api/api-concepts/#efficient-detection-of-changes">HTTP Streaming Watch</a> mechanism to the kube-apiserver:</p><ul><li><strong>Desired State</strong>: state stored in etcd (for example “replicas: 1”)</li><li><strong>Actual State</strong>: actual state of the pod in the cluster</li><li>When something changes: the kube-apiserver sends an event to let the Controller Manager know. The Controller Manager sees actual pod replicas = 0, but desired = 1, so it makes a request to the kube-apiserver to create pod replicas that match the desired state. That is why when we delete a pod in a deployment with 1 replica, it automatically brings up a new pod!</li></ul><p><strong>Conclusion</strong>: K8s works in an event-driven way, if an event appears, it reacts and makes the change.</p><h4>Overview when combining ArgoCD + K8s</h4><p>Actually, when we use GitOps (ArgoCD + K8s), we’re using a double <strong>reconciliation</strong> loop:</p><ul><li><strong>First loop</strong> (ArgoCD): compare Git (desired) versus K8s object spec read via kube-apiserver (this is actual from ArgoCD&#39;s view, but it&#39;s also exactly what feeds into loop 2 below as desired). If the GitOps repo changes, or someone drifts the live spec away from Git (and selfHeal: true is set), ArgoCD re-applies via kube-apiserver to overwrite the drift back to match Git.</li><li><strong>Second loop</strong> (Kubernetes, independent of ArgoCD): compare K8s object spec in etcd (desired at this layer) versus actual runtime state (is the pod really running as spec says?). If the pod dies, K8s controllers recreate it to match spec.</li><li>When someone uses kubectl delete pod, K8s controllers will recreate that pod (ofc, I&#39;m talking about Deployment/ReplicaSet, not a fuckin&#39; standalone pod)</li><li>If someone uses kubectl edit deployment to change a resource/spec, ArgoCD will notice and change it back to match the data in Git (if <strong>selfHeal: true</strong>)</li></ul><p>And one more thing worth mentioning: <a href="https://kubernetes.io/docs/concepts/extend-kubernetes/operator/">Kube-operators</a> use the same mechanism as the <strong>second loop</strong></p><h4>Reconciliation Fight (War between 2 reconciliation loops)</h4><p>We have an easy example to understand: HPA (Horizontal Pod Autoscaler)</p><ul><li>Desired state in Git: Deployment with <strong>replicas:2</strong></li><li>ArgoCD loop: reads Git, sees <strong>replicas:2</strong>, apply to K8S.</li><li>K8S HPA Controller Loop: sees pod resource reached condition to scale, updates Deployment spec in K8S cluster to <strong>replicas:5</strong> for example.</li><li>ArgoCD loop: (if enable <strong>selfHeal: true</strong>): Scan cluster, see actual <strong>replicas:5</strong> while Git defined <strong>replicas:2</strong> (Drift). ArgoCD updates the deployment spec to match what Git defined.</li><li>K8S HPA Controller Loop: scale up again because pod reached condition to scale.</li><li>Result: It creates an infinite loop causing unnecessary churn (constant rollouts, wasted resources) and potential service disruption.</li></ul><p>This is actually a very common real-world issue unless you configure ArgoCD to ignore the replicas field using <a href="https://argo-cd.readthedocs.io/en/stable/user-guide/diffing/#application-level-configuration"><strong>ignoreDifferences</strong></a>, or <strong>omit the replicas</strong> field from Git completely xD</p><h3>Application — <a href="https://octopus.com/blog/argo-cd-application-dependencies">Atomic Unit</a></h3><h3>Application vs ApplicationSet</h3><ul><li><a href="https://argo-cd.readthedocs.io/en/stable/user-guide/application-specification/">Application</a>: a CRD (Custom Resource Definition) that describes 1 source (repo + path/chart + revision) mapped to a destination (cluster + namespace). It’s handled by application-controller’s reconcile and sync loop.</li></ul><pre>apiVersion: argoproj.io/v1alpha1<br>kind: Application<br>metadata:<br>  name: guestbook-app<br>  namespace: argocd     # Must match the namespace where Argo CD is installed<br>  finalizers:<br>    - resources-finalizer.argocd.argoproj.io # Enables cascading delete (deletes K8s resources when the app is deleted)<br>spec:<br>  project: default       # The Argo CD project this app belongs to<br>  # 1. Source: Where your Kubernetes manifests live<br>  source:<br>    repoURL: &#39;https://github.com/argoproj/argocd-example-apps.git&#39; # Git or Helm repo URL<br>    targetRevision: HEAD # Git branch, tag, or commit hash<br>    path: guestbook      # Directory inside the repository containing the YAMLs<br><br>  # 2. Destination: Where to deploy the resources<br>  destination:<br>    server: &#39;https://kubernetes.default.svc&#39; # Cluster API URL (this refers to the local cluster). We could point to another cluster if we have one.<br>    namespace: guestbook                     # Target namespace in your cluster<br><br>  # 3. Sync Policy: How Argo CD handles configuration drift<br>  syncPolicy:<br>    automated:<br>      prune: true     # Automatically delete resources no longer present in Git<br>      selfHeal: true  # Automatically fix deviations if someone manually edits live cluster resources<br>    syncOptions:<br>      - CreateNamespace=true # Create the target namespace automatically if it doesn&#39;t exist<br>      - ApplyOutOfSyncOnly=true # Only sync components that have drifted to optimize performance</pre><ul><li><a href="https://argo-cd.readthedocs.io/en/stable/operator-manual/applicationset/">ApplicationSet</a>: a controller that runs separately (known as the <strong>argocd-applicationset-controller</strong> pod in the argocd namespace). Its job is to generate multiple Applications from 1 template + generator(s), it doesn’t do the sync job itself, it just creates/updates/deletes the Application resource, then <strong>application-controller</strong> does that part.</li></ul><pre>apiVersion: argoproj.io/v1alpha1<br>kind: ApplicationSet<br>metadata:<br>  name: guestbook<br>  namespace: argocd        # Must match the namespace where Argo CD is installed<br>spec:<br>  goTemplate: true          # Enables Go template syntax ({{.path.basename}}) used in `template:` below<br>  goTemplateOptions: [&quot;missingkey=error&quot;] # Fail render instead of silently rendering empty if a field is missing<br>  # 1. Generators: how ApplicationSet discovers &quot;what Applications should exist&quot;<br>  generators:<br>    - git:<br>        repoURL: https://github.com/argoproj/argocd-example-apps.git # Same repo concept as a normal Application source<br>        revision: HEAD<br>        directories:<br>          - path: apps/*    # Glob pattern — each matching folder becomes 1 generated Application<br><br>  # 2. Template: how each matched folder gets turned into an actual Application<br>  template:<br>    metadata:<br>      name: &#39;{{.path.basename}}&#39;   # e.g. folder &quot;apps/guestbook&quot; -&gt; Application name &quot;guestbook&quot;<br>    spec:<br>      project: default<br>      source:<br>        repoURL: https://github.com/argoproj/argocd-example-apps.git<br>        targetRevision: HEAD<br>        path: &#39;{{.path.path}}&#39;     # Full matched path, e.g. &quot;apps/guestbook&quot;<br>      destination:<br>        server: https://kubernetes.default.svc<br>        namespace: &#39;{{.path.basename}}&#39;  # Deploy each generated app into its own namespace or in same namespace, your choice.<br>      syncPolicy:<br>        automated:<br>          prune: true<br>          selfHeal: true</pre><p>An ApplicationSet relies on <a href="https://argo-cd.readthedocs.io/en/stable/operator-manual/applicationset/Generators/"><strong>Generators</strong></a> to discover targets and parameters. Common generators include:</p><ul><li><a href="https://argo-cd.readthedocs.io/en/stable/operator-manual/applicationset/Generators-Git/">Git Generator</a>: scans a Git repo for directories/files (used in our initial lab).</li><li><a href="https://argo-cd.readthedocs.io/en/stable/operator-manual/applicationset/Generators-Cluster/">Cluster Generator</a>: automatically discovers registered clusters.</li><li><a href="https://argo-cd.readthedocs.io/en/stable/operator-manual/applicationset/Generators-Matrix/">Matrix Generator</a>: combines multiple generators (e.g., deploying all Git apps to all registered clusters).</li></ul><h4>Flow between ApplicationSet and Application in ArgoCD</h4><p>So basically, the flow in order from ApplicationSet to Application:</p><ul><li>Update the value file of <strong>applicationSet</strong>, or add a new folder in the Gitops repo (generator source changes), <strong>applicationset-controller</strong> learns about the changes via periodic requeue (polling interval) or webhook.</li><li><strong>ApplicationSet</strong> runs a generator (for example git-dir), asking <strong>argocd-repo-server</strong> for help to checkout the repo and list folders that match the pattern.</li><li><strong>argocd-repo-server</strong> will checkout git, list the folders/files that match the pattern, then return a raw parameter list to <strong>applicationset-controller</strong>. After that, <strong>applicationset-controller</strong> renders the templates with those parameters.</li><li>Then <strong>applicationset-controller</strong> compares the expected Application vs the current Application; if there’s a diff, it modifies/updates the current Application.</li><li>After that, the current Application gets compared against live-state, the result is the Sync/OutOfSync status you often see.</li></ul><h3><a href="https://argo-cd.readthedocs.io/en/stable/operator-manual/declarative-setup/#clusters">Cluster Registration</a></h3><p>So basically, for cluster registration in ArgoCD, <strong>application-controller</strong> calls the endpoint of the <strong>kube-apiserver</strong> of the destination cluster (the same idea as kubectl --context X).</p><p>The destination cluster credential is saved as a Secret in the argocd namespace of the cluster hosting ArgoCD. It must have the label argocd.argoproj.io/secret-type: cluster.</p><p>Fields:</p><ul><li>name: display name, used as destination.name</li><li>server: endpoint</li><li>config: JSON of bearerToken, tlsClientConfig/exec-based</li></ul><p><strong>application-controller</strong> reads the secret via a separate informer/watch, it doesn’t go through the manifest-render pipeline (<strong>repo-server</strong>) like a normal Application does.</p><p>Mostly we use <strong>argocd-vault-plugin</strong> for the secret, we only store the path that defines where the secret is saved in Vault.</p><p>The Application delivering the Secret must target destination.name: in-cluster, which is the built-in cluster, meaning the cluster hosting ArgoCD. This Secret needs to be stored in the local argocd namespace, not the destination cluster!</p><p>Provisioning the actual credential (ServiceAccount + ClusterRole + Binding + mint token) on the destination cluster is a separate concern, ArgoCD itself doesn’t do this:</p><ul><li>CLI flow (argocd cluster add): the CLI (using your own local kubeconfig with admin access) bootstraps SA + ClusterRole + ClusterRoleBinding on the destination cluster, as a one-time setup. The default ClusterRole (argocd-manager-role) is near cluster-admin (*/*/*) unless scoped down manually.</li><li>Declarative/GitOps flow (what we do): this provisioning has to happen outside ArgoCD (Terraform/other pipeline) against the destination cluster; only the resulting token gets pushed to Vault for AVP to inject. The cluster only actually shows up in ArgoCD (Settings &gt; Clusters, usable as a destination) AFTER the Secret is applied via a real sync, not the moment the Application object gets created. If the sync policy is manual and hasn’t run yet, the Application just sits OutOfSync and the cluster stays invisible.</li></ul><h3><a href="https://argo-cd.readthedocs.io/en/stable/user-guide/projects/">AppProject</a></h3><p>AppProject is a guardrail of ArgoCD that limits 3 independent axes:</p><ul><li>sourceRepos (which git repo)</li><li>destinations (which cluster + namespace)</li><li>clusterResourceWhitelist/namespaceResourceWhitelist (which K8s resource kind, cluster-scoped vs namespace-scoped)</li></ul><p>clusterResourceWhitelist defaults to empty (deny all cluster-scoped resources like ClusterRole/ClusterRoleBinding/CRD/Namespace) unless explicitly whitelisted, this default-deny is what actually gives AppProject teeth.</p><p>AppProject is a secondary defense, running concurrently with (not nested inside) the destination cluster’s own RBAC:</p><ul><li>First (Real, K8s-enforced): RBAC of the SA/token on the destination cluster. An absolute limit!</li><li>Second (Software, defined and enforced by ArgoCD): only allows requests permitted by clusterResourceWhitelist/namespaceResourceWhitelist, sourceRepos, and destinations, anything outside that gets blocked.</li></ul><p>An Application’s spec.project must reference an AppProject that already exists, validated right at Application creation time, before it even gets to sync. No AppProject means the Application object fails to even get created, full stop.</p><p>AppProject whitelists <strong>namespace-scoped</strong> resources by default, but cluster-scoped resources (like ClusterRole, CustomResourceDefinition, and Namespace itself) are blocked by default. If your app uses <strong>CreateNamespace=true</strong>, ArgoCD will fail to sync unless you explicitly whitelist the Namespace kind under clusterResourceWhitelist.</p><h3>Sync mechanics deep dive</h3><ul><li>Sync Phases: ArgoCD supports <a href="https://argo-cd.readthedocs.io/en/stable/user-guide/resource_hooks/">Hooks</a> (<a href="https://argo-cd.readthedocs.io/en/stable/user-guide/sync-waves/">PreSync for DB migration, Sync, PostSync</a>).</li><li><a href="https://argo-cd.readthedocs.io/en/stable/user-guide/auto_sync/">Automated Sync</a>: Automatically updates resources when Git changes, but enters a cooling-down period if sync fails.</li><li><a href="https://argo-cd.readthedocs.io/en/stable/user-guide/sync-options/#server-side-apply">Server-Side Apply</a>: Crucial for deploying huge resources (like CRDs) that exceed the annotation size limit.</li></ul><h3><a href="https://argo-cd.readthedocs.io/en/stable/operator-manual/rbac/">RBAC</a> 3 Layers</h3><ul><li><a href="https://kubernetes.io/docs/reference/access-authn-authz/rbac/">K8s RBAC</a>: What ArgoCD is allowed to do in target clusters (enforced by the cluster API).</li><li>AppProject: Logical governance within ArgoCD (restricting repos, namespaces, and resource types).</li><li>SSO/User RBAC: Restricting what UI/CLI users can do (e.g., who can click “Sync” or “Delete”).</li></ul><h3>How Sealed Secrets works</h3><p>To keep it secure, <a href="https://github.com/bitnami-labs/sealed-secrets">Bitnami Sealed Secrets</a> uses public-key cryptography:</p><ul><li><strong>Encryption (Local):</strong> When you run kubeseal, it uses your kubeconfig to fetch only the <strong>Public Key</strong> from the controller in your cluster (or you can download the public cert and run it completely offline). The CLI encrypts your secret locally using this public key.</li><li><strong>Decryption (On-cluster):</strong> Once committed to Git, ArgoCD applies the SealedSecret custom resource to K8s. The <strong>Sealed Secrets Controller</strong> running inside the cluster is the only one holding the <strong>Private Key</strong>. It automatically watches the resource, decrypts it using the private key, and generates a standard unencrypted K8s Secret in the target namespace.</li></ul><p><strong>Note on Kubernetes Security:</strong> Please remember once decrypted inside the cluster, the output is a standard Kubernetes Secret (which is only base64 encoded, unless you&#39;ve explicitly enabled <a href="https://kubernetes.io/docs/tasks/administer-cluster/encrypt-data/">etcd encryption-at-rest</a>). Sealed Secrets (and Vault) only solve &quot;Secret-in-Git&quot; security, not &quot;Secret-at-Rest&quot; inside etcd. Back in 2024, it took me a whole week to realize this....</p><h3>Hands-on Lab</h3><p>We have 2 fuckin’ repos:</p><ul><li>First, <a href="https://gitlab.com/kienlt-gitlab-ci-demo/gitops-bootstrap">gitops-bootstrap</a> which is the Platform/SRE control plane repo storing bootstrap manifests (we only need to apply the root-app manually once to kickstart the cluster)</li><li>Second, <a href="https://gitlab.com/kienlt-gitlab-ci-demo/gitops-manifests">gitops-manifests</a> where we store application configs (which are scanned and applied automatically by the ApplicationSet)</li></ul><pre>kienlt@Luongs-MacBook-Pro gitops-bootstrap % k get nodes<br>NAME                                                            STATUS   ROLES    AGE    VERSION<br>fke-kienlt51-cluster1-hojhilgi-worker-rlqssho7-z1-bf598-qjcg2   Ready    &lt;none&gt;   5d3h   v1.31.4<br>kienlt@Luongs-MacBook-Pro gitops-bootstrap % k apply -f root-app.yaml<br>application.argoproj.io/bootstrap-app created<br># Remember to get init password and port-forward to access ArgoCD UI locally.</pre><p>After that, we have our first connected repo:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*xALkb1aPEe9kxqyR.png" /></figure><p>And our applications:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*4I8HPjoZIX8yVdAD.png" /></figure><p>Oh, what went wrong to prevent ArgoCD from syncing the application?</p><pre>resource :Namespace is not permitted in project lab-project</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*OmGoVBhtZKdKngsH.png" /></figure><p>I’m pretty fuckin’ sure that’s caused by the destination we defined here that allows only namespaces with this pattern: <strong>lab-</strong>*</p><pre>apiVersion: argoproj.io/v1alpha1<br>kind: AppProject<br>metadata:<br>  name: lab-project<br>  namespace: argocd<br>spec:<br>  description: &quot;Project for ArgoCD Hands-on Lab&quot;<br>  sourceRepos:<br>  - &quot;https://gitlab.com/kienlt-gitlab-ci-demo/gitops-manifests.git&quot;<br>  destinations:<br>  - namespace: &quot;lab-*&quot;<br>    server: &quot;https://kubernetes.default.svc&quot;<br>  namespaceResourceWhitelist:<br>  - group: &#39;*&#39;<br>    kind: &#39;*&#39;</pre><p>No, I was wrong, LOL. Here is <a href="https://gitlab.com/kienlt-gitlab-ci-demo/gitops-bootstrap/-/commit/b0f7a5d58cebcadadfc5a4b7a367ff365f03f631">the fix</a>:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*LYVpBcgK7Ksaz2UI.png" /></figure><p>Click sync again, and the expected version will look like this!</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*OSvrKttY2FTZOoIT.png" /></figure><p>And for testing, I added 3 more repos and waited for <a href="https://argo-cd.readthedocs.io/en/stable/operator-manual/applicationset/Controlling-Resource-Modification/#prevent-an-applicationsets-child-apps-from-being-deleted-or-modified">3 minutes</a> (default interval that <strong>applicationSet-controller</strong> will scan repo <strong>gitops-manifest</strong>)</p><pre>kienlt@Luongs-MacBook-Pro gitops-manifests % ll apps<br>total 0<br>drwxr-xr-x@ 6 kienlt  staff  192 25 Jul 17:32 .<br>drwxr-xr-x@ 5 kienlt  staff  160 20 Jul 14:42 ..<br>drwxr-xr-x@ 5 kienlt  staff  160 25 Jul 17:28 lab-new-nginx<br>drwxr-xr-x@ 5 kienlt  staff  160 25 Jul 17:32 lab-new-nginx-1<br>drwxr-xr-x@ 5 kienlt  staff  160 25 Jul 17:32 lab-new-nginx-2<br>drwxr-xr-x@ 5 kienlt  staff  160 20 Jul 14:43 lab-nginx</pre><p>They will appear in <strong>bootstrap-app</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*8QcWyRnvvPMe1fFv.png" /></figure><p>Ok, we have set up a connected repo, we have set up an <strong>applicationSet</strong> that generates ArgoCD Applications. Next, we will move to cluster registration. We commonly register clusters via <strong>argocd-cli</strong>, but I want to show you another approach that uses the GitOps way!</p><p>Before that, we need to setup/install Bitnami Sealed Secret because we are gonna push secret to gitops…</p><pre># Run this in current cluster, where ArgoCD installed!<br>kubectl apply -f https://github.com/bitnami-labs/sealed-secrets/releases/download/v0.22.0/controller.yaml<br><br># I&#39;m working in Mac, so brew for me...<br>brew install kubeseal</pre><p>We create a raw cluster secret locally named: <strong>raw-cluster-register.yaml</strong>. Please notice that <strong>bearerToken</strong> is actually field <strong>token</strong> under users section. If you are using managed k8s for testing like me, remove <strong>tlsClientConfig</strong>.</p><pre>apiVersion: v1<br>kind: Secret<br>metadata:<br>  name: kienlt-cluster2-secret<br>  namespace: argocd<br>  labels:<br>    argocd.argoproj.io/secret-type: cluster<br>type: Opaque<br>stringData:<br>  name: kienlt-cluster2<br>  server: &quot;https://&lt;IP_API_SERVER_CLUSTER_2&gt;&quot;<br>  config: |<br>    {<br>      &quot;bearerToken&quot;: &quot;TOKEN_OF_CLUSTER_2&quot;,<br>      &quot;tlsClientConfig&quot;: {<br>        &quot;insecure&quot;: true<br>      }<br>    }</pre><p>Then run the fuckin command to encrypt it and store it in the bootstrap repo.</p><pre>kubeseal --format=yaml &lt; raw-cluster-register.yaml &gt; bootstrap/cluster-registration.yaml</pre><p>Commit it to the <strong>gitops-bootstrap</strong> repo. Then refresh <strong>bootstrap-app</strong>, and we will have a new SealedSecret resource</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*3Ey_og1i0Z9S5kDM.png" /></figure><p>But cluster is “Unknown”?</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*hOmMG9XvpOnRi5uJ.png" /></figure><p>Don’t panic! ArgoCD handles cluster <strong>connections lazily</strong>. It won’t actively probe the cluster API unless there is at least one Application targeting it. Once we deploy a test application to <strong>kienlt-cluster2</strong>, the status will immediately transition to Successful (or show auth/network errors if we fucked up the token/API endpoint).</p><p>We commit <a href="https://gitlab.com/kienlt-gitlab-ci-demo/gitops-bootstrap/-/commit/4965f0745ad75f1bc4314245a597c81b821fb4c5">this change</a> to the <strong>gitops-bootstrap</strong> repo, then hit the refresh button on the <strong>bootstrap-app</strong> in ArgoCD.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*dBv3dqTksR5fn_yb.png" /></figure><p><strong>Why switch from Git generator to Matrix generator?</strong> The original Git generator only scans folders in apps/* and generates one Application per folder, but the destination.server in the template is hardcoded to a single cluster. It has no awareness of other registered clusters. By switching to a Matrix generator (clusters: {} × git: {directories: apps/*}), we create a <strong>cross-product</strong>: every folder × every registered cluster = one Application. So 4 folders × 2 clusters = 8 Applications instead of just 4. And when we add Cluster 3 in the future, clusters: {} auto-discovers it, no ApplicationSet changes needed!</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*CPo4ZyO0MwINBtLS.png" /></figure><p>But we run into an issue on the new Cluster 2 xD:</p><pre>&gt; application destination server &#39;https://hidden.com&#39; and namespace &#39;lab-new-nginx&#39; do not match any of the allowed destinations in project &#39;lab-project&#39;</pre><p>This is because our AppProject didn’t permit it yet. Here is the <a href="https://gitlab.com/kienlt-gitlab-ci-demo/gitops-bootstrap/-/commit/3066c35480ca6df5f01ef2b5f1ebff7d666e2e0c">commit</a> that fixes it.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*Ts54PBjuEtFD9nUY.png" /></figure><p>After this change, we finally see both clusters showing “Successful” in their Connection Status:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*fR8-7eerMXRqg24D.png" /></figure><p>This pattern is commonly used for Multi-Cluster GitOps to architect High Availability (HA) and Disaster Recovery (DR) in production environments. But everything comes with a trade-off at a higher level, nothing is perfect and there is no fuckin’ free lunch!</p><h4>Hub-and-Spoke Topology vs Decentralized</h4><ul><li>Hub (ArgoCD Control Plane): Installed in Cluster 1, acting as the central management plane.</li><li>Spokes (Target Clusters): Cluster 2, Cluster 3… only receive deployment instructions.</li></ul><pre>                   ┌────────────────┐<br>                   │    Git Repo    │<br>                   └───────┬────────┘<br>                           │<br>                           ▼<br>               ┌───────────────────────┐<br>               │ Cluster 1 (Hub/Argo)  │<br>               └──────┬─────────┬──────┘<br>                      │         │<br>            (Deploy)  │         │  (Deploy)<br>                      ▼         ▼<br>        ┌───────────────┐     ┌───────────────┐<br>        │  Cluster 2    │     │  Cluster 3    │<br>        │    (Spoke)    │     │    (Spoke)    │<br>        └───────────────┘     └───────────────┘</pre><ul><li>Advantage: Centralized management — you only need a single UI dashboard to monitor the entire system.</li><li>Disadvantage: Single Point of Failure. If Cluster 1 goes down, we cannot deploy or update workloads on Clusters 2 and 3. However, existing workloads will continue running fine — this only affects new deployments.</li></ul><p>Alternative pattern: Decentralized. Each cluster runs its own ArgoCD instance separately, watching the same GitOps repo! The catch is we have to manage multiple UI dashboards and it consumes slightly more cluster resources.</p><p>Furthermore, we also need to handle traffic redirection (failover) when Cluster 1 goes down and tackle data replication challenges for stateful apps. But I won’t cover those here due to complexity. Maybe in the future once I get the fuckin’ chance xD.</p><h3>Conclusion</h3><ul><li>ArgoCD is harder than you ever imagine if you really want to understand it.</li><li>Reconciliation Fight: Avoid conflicts with other controllers (like HPA replicas) by using <strong>ignoreDifferences</strong>.</li><li>Repo Server CPU/RAM spikes: Rendering heavy Helm charts can OOM-kill the repo-server; configure parallel limits or cache properly.</li></ul><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=34fcc76c04e7" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Prepare for AWS Certified CloudOps Engineer Associate SOA-C03 2026 exam]]></title>
            <link>https://medium.com/@kienlt.qn/prepare-for-aws-certified-cloudops-engineer-associate-soa-c03-2026-exam-b33f6a8d598c?source=rss-b77f4e43d676------2</link>
            <guid isPermaLink="false">https://medium.com/p/b33f6a8d598c</guid>
            <category><![CDATA[aws-certification]]></category>
            <category><![CDATA[aws]]></category>
            <dc:creator><![CDATA[Kienlt]]></dc:creator>
            <pubDate>Sun, 19 Jul 2026 17:00:51 GMT</pubDate>
            <atom:updated>2026-07-21T15:06:11.719Z</atom:updated>
            <content:encoded><![CDATA[<p>I start the progress by using <a href="https://blackmetalz.github.io/udemy.com/course/aws-certified-cloudops-associate/">Stephane Maarek’s course</a></p><p>Honestly, if I keep studying the same way I always have in the past which is memorization keyword, I’m going to fail this exam for fuckin’ sure. The real problem is I don’t have hands-on AWS experience. I work with Kubernetes and on-premises stuff day to day, so a lot of this is purely theoretical for me. Watching videos and taking notes clearly isn’t enough when the exam throws scenario-based questions at you that only make sense if you’ve actually touched the console.</p><h3>Learning Sections</h3><p>I will follow Udemy course, no idea if this article will be more than 1k5 lines long….</p><ul><li>Section 1: EC2 for CloudOps</li><li>Section 2: AMI — Amazon Machine Image</li><li>Section 3: Managing EC2 at Scale — Systems Manager (SSM)</li><li>Section 4: EC2 High Availability and Scalability</li><li>Section 5: CloudFormation for CloudOps</li><li>Section 6: Lambda for CloudOps</li><li>Section 7: EC2 Storage and Data Management — EBS and EFS</li><li>Section 8: Amazon S3</li><li>Section 9: Advanced Amazon S3 &amp; Athena</li><li>Section 10: Amazon S3 Security</li><li>Section 11: Advanced Storage Section</li><li>Section 12: CloudFront</li><li>Section 13: Databases for CloudOps</li><li>Section 14: Monitoring, Auditing and Performance</li><li>Section 15: AWS Account Management</li><li>Section 16: Disaster Recovery</li><li>Section 17: Security and Compliance for CloudOps</li><li>Section 18: Identity</li><li>Section 19: Networking — Route 53</li><li>Section 20: Networking — VPC</li><li>Section 21: Other Services</li></ul><h3>Some shortcuts I will use in this article</h3><ul><li>Load Balancer (LB)</li><li>Placement Groups (PG)</li><li>CloudWatch (CW)</li><li>Auto Scaling Group (ASG)</li><li>Relational Database Service (RDS)</li><li>Amazon Machine Image (AMI)</li><li>Systems Manager (SSM)</li><li>Resource Groups (RGs)</li><li>Elastic Load Balancer (ELB)</li><li>Application Load Balancer (ALB)</li><li>Security Group (SG)</li><li>Network Load Balancer (NLB)</li><li>Gateway Load Balancer (GWLB)</li><li>Server Name Indication (SNI)</li><li>Function as a Service (FAAS)</li><li>Cross-region replication (CRR)</li><li>Same-region replication (SRR)</li><li>Internet Gateway (IGW)</li><li>Network Access Control List (NACL)</li><li>Dead Letter Queue (DLQ)</li></ul><h3>Section 1: EC2 for CloudOps</h3><h4>Placement Groups</h4><p>When I see this, I associate it with HDFS default replication (factor=3).</p><ul><li>Replica 1: Place in node is writing.</li><li>Replica 2: Place in node that have different rack than replica 1.</li><li>Replica 3: Place in same rack with replica 2 but different node.</li></ul><p>So let’s make a simple compare with AWS PG</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/904/1*qSDMlFuDsHy9mSBEDsOe_A.png" /></figure><p>Ok, why PG?</p><ul><li>Pros: Great network with high throughput, low latency (10Gbps between instances with Enhanced Networking enable).</li><li>Cons: If the AZ fails, all instances fails at the same time. PG spread have limited to 7 instances per AZ per PG. For PG paritions, we can have up to 7 partitions per AZ and can be span across multiple AZs in the same region.</li></ul><h4>CloudWatch for EC2</h4><p>Remind: LGTM (Loki-Grafana-Tempo-Mimir) Stack = CloudWatch.</p><p>AWS provided metrics (AWS push them for us) includes CPU,Network,Disk and Status check metric every 5 minutes with no agent needed. (paid version or Detailed Monitoring will be 1 minute interval). Good thing to know that defaults metrics are collected by AWS hypervisor from outside of instance?</p><p>Custom metric: Hmm, RAM is a custom metric? (I thought it should be default…), make sure IAM permissions on the EC2 instance role are correct.</p><p>Yeah, RAM is not fucking included into the AWS EC2 metrics.</p><p>If you have 3 or fewer dashboards using only standard EC2 metrics, it’s completely free. If you have a 4th dashboard, it’s $3/month.</p><p>How CW collect metrics and logs from EC2 instance? → Unified CW Agent (with right permissions).</p><p>Unified CW Agent — procstat Plugin will come to exam:</p><ul><li>Collect metrics and monitor system utilization of individual process (support both Linux/Window)</li><li>Select which process to monitor by pid_file/process name/pattern… This shit is free in <strong>CheckMK</strong></li><li>Metrics collected by procstat plugin begins with “procstat” prefix.</li></ul><p>Conclusion: CW agent not only push metrics, it is also able to push logs to CW. Detailed Monitoring won’t help you get RAM, it just gives you faster intervals on the same limited set of metrics.</p><h4>EC2 Instance Connect Endpoint</h4><p>Useful when no fucking bastion, public IP on the instance or EC2 instances running in a private subnet in a VPC without Internet Gateway or NAT Gateway.</p><h4>Status Check</h4><p>So for common we have 3 status check: failed system (host hardware), failed instance (our instance) and attached EBS.</p><p>With CW Metrics and Recovery</p><ul><li>Option 1: CW Alarm, recover/restart EC2 instance on metric trigger (if <strong>StatusCheckFailed_System</strong> == 1 for example). Then send notification using SNS (Remember, SNS is <strong>push-based</strong> service, not a pull/polling service. You do not need to build a consumer that constantly checks the topic at intervals)</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/684/0*SVIqgz-N_uEdVWfO.png" /></figure><ul><li>Option 2: By default, an ASG uses EC2 status checks (<strong>StatusCheckFailed_System</strong> and <strong>StatusCheckFailed_Instance</strong>) to monitor the health of its instances and yeah, ASG launch new instance replaced failed instance.</li></ul><p>So after we config CW Alarm, we can test it by using CLI:</p><pre>aws cloudwatch set-alarm-state --alarm-name name-here-bro --state-value ALARM --state-reason &quot;testing how recover in action after alarm triggered&quot;</pre><p>Also worth to mention action in CW Alarm:</p><ul><li>Recover: moves the instance to new physical host hardware (AWS migrates it), then starts it up.</li><li>Reboot: restarts the OS on the same physical host (For example: <strong>StatusCheckFailed_Instance</strong>, action will be reboot)</li></ul><h4>EC2 Hibernate</h4><p>Short: keep in-memory state in EBS volume, next time you start EC2 instance, you will load exactly state of your instance before hibernate and shutdown.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/723/0*S7yCAf_k0lD0sPZ9.png" /></figure><p>We need to enable it when launch ec2 instance, root volume must have enough size to store hibernate data and root volume must be set to encrypted.</p><p>These 3 limits above are commonly tested in SOA-C03!</p><h4>Instance Scheduler on AWS</h4><p>Yeah, that is how we save every bucks when running in Cloud. Features:</p><ul><li>Supports EC2 instances and RDS instances. (Do NOT use this for EC2 ASG, because if it stops an instance in ASG, ASG thinks that instance is dead/unhealthy then terminates and recreates it. For ASG scheduling, use ASG Scheduled Actions/Scaling instead!)</li><li>Schedules are managed in a DynamoDB table.</li><li>Stop/Start instance by using resource’s tag and Lambda.</li><li>Works in cross-account and cross-region resources.</li></ul><p>We have to create instance scheduler via CloudFormation → Stack → create stack or use exists template to create this fuckin’ stack, after that we have a lot of created resources!</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*Wo1lfSewehKa3wE2.png" /></figure><h3>Section 2: AMI — Amazon Machine Image</h3><h4>Overview</h4><p>We are gonna use public AMI, most well-known is Amazon Linux 2, very popular AMI for AWS.</p><p>We can create AMI from running instance without shutdown AMI with AMI No-reboot option (It was disable by default).</p><p>Hmm, so much services in AWS. We can create AMI via AWS Backup also (Doesn’t reboot the instances while taking EBS snapshot)</p><p>Migrate? yes we can migrate EC2 instance between AZ via AMI</p><p>EC2 Image Builder: Hmm, auto build AMI manually or with schedule and run shit load of test cases to validate AMI (free service btw). Oh indetail of EC2 Image Builder, there is shit load of predefined components and OS select to create AMI:</p><ul><li>amazon-cloudwatch-agent-linux</li><li>amazon-corretto-11-headless</li><li>aws-cli-version-2-linux … and more</li></ul><p>And in window of new Image Builder, we need to add a role for it also (Create a role with multiple required policies) and last one was Distributed settings where you can define where AMI will be distributed.</p><p>Cross-Account AMI sharing: you can share AMI with another AWS account that AMI can have unencrypted or encrypted volumes via customer managed key (You need to share managed key as well, so they can decrypt the encrypted AMI).</p><p>AMI in Production: add tag to AMI and use policy to approve only AMI have tags.</p><p>Holy fuck, remember AMI is region locked. You can copy it to other regions, but cannot share it directly. Sharing with other AWS accounts is region-wide (available across all AZs in that region, not locked to single AZ).</p><h3>Section 3: Managing EC2 at Scale — Systems Manager (SSM)</h3><h4>Overview</h4><ul><li>Manage your EC2 and on-premises systems at scale</li><li>Patching automation for enhanced compliance</li><li>Works for both Window and Linux OS</li><li>Stephane Maarek recommend learn keyword that in blue xD</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*VIBnaEGRinJ5oKC1.png" /></figure><ul><li>How it works: basically, it is server-agent model, it needs to install SSM agent into EC2/On-premise Server/VM we control, installed by default on Amazon Linux 2 and some Ubuntu AMI. And make sure EC2 instances have a proper IAM role to allow SSM actions. We manage EC2 instances that installed SSM agent in “Node tools — Fleet manager”</li></ul><h4>Resource Groups</h4><p>Holy shiet, this make me remember about Azure since Resource Group is one of important knowledge in Azure. So RGs create,view or manage logical group of resources by using <strong>tags</strong>. For example EC2 instances have tags <strong>Team = Platform</strong>, we can create a RG based that tag.</p><p>This is regional service, works with EC2, S3,DynamoDB, Lambda, etc…</p><p>And we can create RG via UI with filter like resource type (EC2 instances…), tags, then group name, details…</p><p>Conclusion: It is all about manage resources, Azure when you create a resource you need to put it on a RG, in AWS only when works with SSM xD</p><h4>SSM — Documents</h4><p>Hmm, this is more like Saltstack run command or Ansible shell module? In AWS it’s called documents, LOL. It also sync with State Manager, Patch Manager, Automation. We can write our own document also.</p><p>We can create Command or Session and Automation.</p><p>Let’s take a look with <strong>Run Command</strong>:</p><ul><li>Execute a document (= script) or just run a simple command across multiple instances (using RG, hmm this make sense now for RG usage.) with rate control/error control with targets are RG/Instance tags/Instance selected manually and able to config rate control like concurrency and error threshold to stop the task fail.</li><li>Integrated with IAM &amp; CloudTrail (You are able to see who run that commands) and yeah no need for SSH because it is Server-Agent architecture (Saltstack like).</li><li>Command output will be show via console but can be send to S3 bucket or CW logs. Send notification to SNS about command status (In progress, success, failed…) and can be invoked using EventBridge for automation. (AWS really know how to make services… )</li><li>If you know about Saltstack/Ansible, this is pretty easy to understand! But no idea if you need to remember structure of run command for exam!</li></ul><h4>SSM — Automation</h4><p>Simplifies common maintenance and deployment task for EC2 instances and other AWS resources, for example: restart instances, create AMI, EBS snapshot.</p><p>Hmm, it mostly choose which module (AWS provided) to run.</p><p>Patch AMIs &amp; Update ASG.</p><h4>SSM — Parameter Store</h4><p>Secure storage for configuration and secrets, best for App config + lightweight secrets, no fucking auto-rotation (need auto-rotations/DB Credential/API Keys/Tokens safe location? → Secrets Manager)</p><p>Used with Parameter Store (not just SSM):</p><ul><li>Lambda: reads DB URLs, API keys at runtime</li><li>ECS/EKS: inject secrets as env vars into container</li><li>Code Deploy/Code Build: pull credentials during CI/CD</li><li>Any EC2 app — via SDK/CLI with proper IAM role (ssm:GetParameter)</li></ul><p>Hierarchy uses <em>/</em> delimited paths, like a file system:</p><pre><br>/                                    ← root<br>├── myapp/<br>│   ├── prod/<br>│   │   ├── db/<br>│   │   │   ├── url                  → &quot;postgres://prod-db.rds.amazonaws.com:5432/mydb&quot;<br>│   │   │   ├── username             → &quot;admin&quot;<br>│   │   │   └── password             → (SecureString, KMS encrypted)<br>│   │   ├── redis/<br>│   │   │   └── endpoint             → &quot;redis://prod-cache.abc123.ng.0001.use1.cache.amazonaws.com&quot;<br>│   │   └── feature_flags/<br>│   │       └── new_dashboard        → &quot;true&quot;<br>│   └── dev/<br>│       ├── db/<br>│       │   ├── url                  → &quot;postgres://dev-db.rds.amazonaws.com:5432/mydb&quot;<br>│       │   └── password             → (SecureString)<br>│       └── feature_flags/<br>│           └── new_dashboard        → &quot;false&quot;<br>└── shared/<br>    ├── datadog_api_key              → (SecureString)<br>    └── slack_webhook_url            → &quot;https://hooks.slack.com/...&quot;</pre><p>Fetch by path in CLI:</p><pre># single param<br>aws ssm get-parameter --name &quot;/myapp/prod/db/password&quot; <br># Multi param<br>aws ssm get-parameter --names &quot;/myapp/prod/db/password&quot; &quot;/myapp/prod/db/username&quot;<br># With decryption <br>aws ssm get-parameter --name &quot;/myapp/prod/db/password&quot; --with-decryption<br># entire subtree at once (recursive)<br>aws ssm get-parameters-by-path --path &quot;/myapp/prod&quot; --recursive</pre><p>Free parameters tier used up to 10000 number of parameters while paid up to 100k with maximum size of parameter value is 4 KB (advanced 8Kb) and no fucking parameter policies available, parameters policies only available if you upgrade free tier to advanced tier (you can upgrade to advanced but not able to downgrade to free tier, remember it.)</p><p>So what is the fucking parameters policies:</p><ul><li>Allow to assign a TTL to a parameter to force updating or deleting sensitive data such as passwords, tokens. Can assign multiple polices at a time.</li><li>ExpirationNotification Policy: fires an event to EventBridge when a parameter is nearing expiration or has just expired.</li><li>NoChangeNotification Policy: Fires an event to EventBridge if a parameter has not been modified/updated within a certain number of days.</li></ul><h4>SSM — Default Host Management Configuration (DHMC)</h4><p>Short explain: turn on one setting per region -&gt; all EC2 instances in that region get SSM managed automatically, no per-instance IAM role needed.</p><p>Required IMDSv2 to be active for enhanced security and to successfully identify instances.</p><h4>SSM — Inventory &amp; State Manager</h4><p>Short explain: what’s installed/running inside my instances → It is fucking SSM Inventory</p><p>How it works:</p><ul><li>EC2 (SSM Agent)</li><li>Collects metadata then sends to SSM Inventory</li><li>S3 bucket (optional sync)</li><li>Athena / QuickSight (query/visualize)</li></ul><p>Key points for fucking exam:</p><ul><li>Data synced to S3 via Resource Data Sync — aggregate inventory across multiple accounts/regions into one S3 bucket.</li><li>Query with Athena on top of that S3 data</li><li>If instance doesn’t show inventory → check SSM Agent running + IAM role has <em>ssm:PutInventory</em> permission</li><li>Works for both EC2 and on-premises servers (any SSM managed node)</li></ul><h4>SSM — Patch Manager</h4><p>Automate the process of patching managed instances, patch on demand or on a schedule using Maintenance Windows. Scan instances and generate patch compliance report (missing patches)</p><p>Patch baseline:</p><ul><li>Defines which patches should and shouldn’t be installed on your instances</li><li>Ability to create custom patch baseline (specify approved/rejected patches)</li></ul><p>Patch Group:</p><ul><li>Associate a set of instances with specific patch baseline</li><li>Example: create patch groups for different environments (dev,test,prod)</li><li>An instance can only be in one patch group, patch group can be registered with only one patch baseline</li></ul><p>For exam perspective: patch manager is used to patch your instances</p><h4>SSM — Session Manager</h4><p>Allow you to start a secure shell on your EC2 and on-premises servers. Does not need SSH access, bastion hosts or SSH keys. This is possible because we have fucking SSM Agent installed.</p><p>Required for SSM Manager works:</p><ul><li>SSM Agent need to be installed and running in the instance.</li><li>Instance must have IAM role with related policy to SSM Managed</li><li>Instance must have internet access or VPC endpoint to SSM connect to AWS endpoint?</li></ul><p>And commands in this shell session can be logged and sent to S3, and control user in which group can access shell. Easier to audit I think.</p><h4>SSM — Distributor</h4><p>Packet and deploy software to your managed instances, you can create Distributor package (SSM Document)</p><p>Can be install via one-time (Run Command — yes, fucking like saltstack xD) or on a schedule (using state manager)</p><h4>SSM — OpsCenter</h4><p>Allow you to view, investigate and remediate issues in one place (no need to navigate across different AWS services)</p><p>Issues could be security, performance issues, failures which help to reduce meantime to resolve issues.</p><h4>Conclusion</h4><p>For this section, I think it has a lot of logic,bussiness and behavior like Ansible(agent-less)/Saltstack(server-agent) model. If you understand Ansible/Saltstack’s essence, you should able to understand this section without any issues!</p><h3>Section 4: EC2 High Availability and Scalability</h3><p>I skip section and jump direct to quiz. My wrong answer for this question in this quiz:</p><p>Question: You would like to expose a fixed static IP to your end-users for compliance purposes, so they can write firewall rules that will be stable and approved by regulators. Which Load Balancer should you use?</p><p>Answer: I picked ALB with EIP attached to it because it sounds right LOL. But in fact the ALB itself dynamically manages its IP addresses, so if you need fixed static IP, you need NLB</p><p>Second question: Which of the following is NOT a valid target while you create a Target Group for your Application Load Balancer?</p><p>Answer: I picked wrong answer again. <a href="https://docs.aws.amazon.com/elasticloadbalancing/latest/application/load-balancer-target-groups.html">It is fucking Public IP</a></p><p>And I failed 6 in total of 16 questions, I think i need to watch this section again.</p><h4>Application Load Balancer (ALB)</h4><ul><li><strong>ALBs have Security Groups!</strong> (A common misconception is that SGs are EC2-only). Because ALBs are public-facing, they need an attached Security Group to control inbound traffic from the internet (e.g., allowing ports 80/443 from 0.0.0.0/0).</li><li><strong>Which AWS resources use Security Groups?</strong> Basically any resource with an Elastic Network Interface (ENI). This includes: EC2 instances, RDS databases, Lambda (inside VPC), ECS Tasks, VPC Endpoints, and ALBs/NLBs.</li></ul><p>Routing tables to different target groups:</p><ul><li>Routing based path in URL</li><li>Routing based on hostname in URL (It means you can have multiple domain point to ALB)</li><li>Routing based on Query String, Headers (domain.com/users?id=123&amp;order=true)</li></ul><p>Target Groups:</p><ul><li>EC2 instances — HTTP</li><li>ECS tasks — HTTP</li><li>Lambda functions — HTTP request is translated into a JSON event</li><li>IP Address (private IPs)</li></ul><p>ALB can route to mulitple target groups and health check are at the target group level. There is something great to know from this section (Which I never know/understand before). Header X-Forwarded-For can be spoofed but can be prevented by using last client in ALB (ALB auto inject header)</p><pre>  Client sends:    X-Forwarded-For: fake-ip<br>  ALB appends:     X-Forwarded-For: fake-ip, real-client-ip  ← ALB controls this<br>                                              ^^^^^^^^^^^^<br>                                              always trustworthy<br><br><br># Unsafe, can get fake-able ip<br>client_ip = request.headers[&quot;X-Forwarded-For&quot;].split(&quot;,&quot;)[0].strip()<br># Safe, getting real-client-ip which control by ALB, it can not goes wrong!<br>client_ip = request.headers[&quot;X-Forwarded-For&quot;].split(&quot;,&quot;)[-1].strip()</pre><p>Allow only requests coming from ALB (Security Group Referencing / Chaining): In the Security Group of the EC2 instance, add an inbound rule:</p><ul><li><strong>Type</strong>: HTTP (Port 80), HTTPS (Port 443), or Custom TCP (your app port, e.g., 8080)</li><li><strong>Source</strong>: Select <strong>Custom</strong> -&gt; then select/type the <strong>Security Group ID of the ALB</strong> (e.g., sg-xxxxxxxx).</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*LDytjVTVyVu-iJjp.png" /></figure><p><strong>Why?</strong> Because the ALB’s IP addresses are dynamic and change as AWS scales it. By referencing the ALB’s Security Group as the source, you ensure only traffic routed through the ALB is allowed, without needing to maintain IP address lists.</p><p>We can have listener rule to do routing according conditions/rules. (like add custom path with custom fixed response code, body, content type.)</p><h4>Network Load Balancer (NLB)</h4><p>NLB (Layer 4), has one static ip per AZ and supports assigning Elastic IP (helpful for whitelisting specific IP). And the rest is almost the same with ALB.</p><p>NLB does not support cookie-based session affinity, but uses source IP hash for connection stickiness.</p><h4>Gateway Load Balancer (GWLB)</h4><p>Gateway for network virtual appliances (NVA) in AWS (Ex: Firewalls, Intrusion Detection, Deep packet inspection and so on..).</p><p>Flow: <strong>User/Internet Gateway</strong> → <strong>GWLB</strong> → <strong>3rd-party NVA</strong> (If passed, traffic will be sent to GWLB otherwise drop) → <strong>GWLB</strong> → <strong>Backend</strong></p><p>Operates at Layer 3 — Ip Packets. Exam perspective: Use GENEVE protocol on port 6081 (remember 6081, not 6801 or other shit!)</p><h4>ELB — Server Name Indication (SNI)</h4><p>Used in scenario where you have multiple domain that point to same ALB/NLB, each domain serving specific target group.</p><h4>Connection Draining</h4><p>Feature naming: <strong>Deregistration Delay</strong> — For ALB &amp; NLB. It is time to complete “in-flight requests” while the instance is de-registering or unhealthy.</p><p>Think like graceful shutdown:</p><ul><li>Instance marked as Draining(like k8s drain node xDD)/Deregistration.</li><li>ALB stops sending NEW requests to it.</li><li>But existing in-flight requests get time to finish so users will see no errors.</li><li>After timeout (default 300s), instance is fully removed.</li></ul><h4>ELB — monitoring, troubleshooting, logging and tracing</h4><p>All LB metrics are directly pushed to CW metrics:</p><ul><li><strong>SurgeQueueLength</strong>: The total of request (in HTTP listener) or connection (TCP listener) that are pending routing to a healthy instance.</li><li><strong>SpilloverCount</strong>: the total number of requests that were rejected because the surge queue is full.</li></ul><p>For request tracing, each HTTP request has an added custom header “X-Amzn-Trace-Id”. This is useful in logs / distributed tracing platform to track a single request.</p><h4>Auto Scaling — Instance Refresh</h4><p>When we update launch template and then re-creating all EC2 instances, instance refresh is like rolling update in k8s deployment xD, same idea, pattern. Just little different start a VM is slower than start a pod, make sure you have right setting for warm-up time.</p><h4>ASG — Warm Pools</h4><p>To resolve latency issue, we have warm pool, pre-initialized VM but costs more money since they are up and ready to join auto scaling group?</p><h4>CloudOps — ASG</h4><p>There is some lifecycle hooks: instance launching, instance terminating.</p><p>This is kind great feature for me when use public cloud, lifecycle event can be trigger to EventBridge,SNS,SQS and with EventBridge it can invoke Lambda function to do your custom logic.</p><p>Example, SQS with ASG</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/995/0*xGgRPSe4As89RzOl.png" /></figure><h4>CW for ASG</h4><p>Metrics are collected every 1 minutes</p><h3>Section 5: CloudFormation for CloudOps</h3><p>ah shit, here we go again. I <strong>forfeited</strong> this fucking section 2 times when I’m learning this section for AWS DOP-C02 in 2024 and 2025…. YAML developer, a lot of fucking hooks =.=</p><p>Short: IAC but for AWS only. You are able to visualization with Infrastructure Composer to see how components are related to each other in CloudFormation.</p><p>How it works?:</p><ul><li>You should create many stacks separately, each stack for specific resource (VPC/Network/App stacks).</li><li>You need to upload template to S3 then refers it in the CloudFormation.</li><li>To update template, you will have to upload new version of template, can not edit previous, so you will be able to rollback to previous revision in case new version is messed up!</li><li>Stacks are defined by name. Delete stack will delete any resource/artifact created via that.</li><li>You can deploy using manual way or via CLI / CD tools of AWS.</li><li>Templates could be in YAML or JSON format, YAML is prefered</li></ul><p>Building Blocks, we will deep dive in each section below soon:</p><ul><li>AWSTemplateFormatVersion: Identifies the capabilities of template</li><li>Description: description, comment LOL</li><li>Resource: Your resources declared in the fucking template.</li><li>Parameters: it is fucking parameters (dynamic inputs) for your template.</li><li>Mappings: the static variables for your template</li><li>Outputs: yeah, like terraform</li><li>Conditionals: list of conditions to perform resource creation</li><li>Template helpers, we have references and functions.</li></ul><h4>Resources</h4><p>Resources are the core of CloudFormation template (this is the only one mandatory in whole template of CloudFormation.)</p><h4>Parameters</h4><p>This section I tell the AI create the content I want from what I understand xD.</p><p>Dynamic inputs for your template. The main benefit: define a value once in Parameters, reference it in multiple places — when you need to change it, update only the parameter instead of hunting down every hardcoded value across the template.</p><pre>Parameters:<br>  EnvName:<br>    Description: Environment name (e.g. dev, prod)<br>    Type: String<br>    Default: dev<br><br>Resources:<br>  WebServerSG:<br>    Type: AWS::EC2::SecurityGroup<br>    Properties:<br>      GroupDescription: !Sub &quot;SG for web server - ${EnvName}&quot;<br>      Tags:<br>        - Key: Name<br>          Value: !Sub &quot;${EnvName}-web-sg&quot;<br>        - Key: Environment<br>          Value: !Ref EnvName<br><br>  WebInstance:<br>    Type: AWS::EC2::Instance<br>    Properties:<br>      InstanceType: t2.micro<br>      ImageId: ami-0a3c3a20c09d6f377<br>      Tags:<br>        - Key: Name<br>          Value: !Sub &quot;${EnvName}-web-server&quot;<br>        - Key: Environment<br>          Value: !Ref EnvName<br>      SecurityGroups:<br>        - !Ref WebServerSG<br><br>  MyBucket:<br>    Type: AWS::S3::Bucket<br>    Properties:<br>      BucketName: !Sub &quot;my-app-${EnvName}-bucket&quot;<br>      Tags:<br>        - Key: Environment<br>          Value: !Ref EnvName</pre><p>EnvName is used in 7 places, deploy with EnvName=prod and everything updates automatically.</p><p>Two ways to reference a parameter:</p><ul><li>!Ref EnvName - returns the raw value, use when assigning directly to a field</li><li>!Sub &quot;${EnvName}-web-server&quot; - use when you need string interpolation (prefix/suffix)</li></ul><p>Exam information:</p><ul><li><strong>AllowedValues</strong>: allow user select only allowed values in a parameters.</li><li><strong>NoEcho</strong>: to not display, echo the sensitive data like password.</li></ul><p>Pre-defined / Pseudo parameters:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*uQTfO5uCQJ991dO1.png" /></figure><h4>Mappings</h4><p>This section I tell the AI create the content I want from what I understand xD.</p><p>Static variables hardcoded in the template, no user input and no dynamic lookup. Use when you know the values in advance and they vary by some fixed dimension (region, environment, arch).</p><p>Classic use case: AMI IDs. AMIs are region-specific with ami-0ff8a91507f77f867 in us-east-1 is a completely different image than ami-0bdb828fd58c52235 in us-west-1. Mappings let you encode that lookup table once and resolve it automatically based on the current region.</p><pre>Mappings:<br>  RegionMap:<br>    us-east-1:<br>      HVM64: ami-0ff8a91507f77f867<br>      HVMG2: ami-0a584ac55a7631c0c<br>    us-west-1:<br>      HVM64: ami-0bdb828fd58c52235<br>      HVMG2: ami-066ee5fd4a9ef77f1<br><br>Resources:<br>  MyEC2Instance:<br>    Type: AWS::EC2::Instance<br>    Properties:<br>      ImageId: !FindInMap [RegionMap, !Ref &quot;AWS::Region&quot;, HVM64]<br>      InstanceType: t2.micro</pre><p>!FindInMap [MapName, TopLevelKey, SecondLevelKey] are three-level lookup, always this structure.</p><p>!Ref &quot;AWS::Region&quot; is a pseudo parameter. AWS injects the current region automatically, no user input needed. That&#39;s what makes this pattern useful: deploy the same template to any region and it picks the right AMI ID.</p><p>Mappings vs Parameters: if the value is truly static and predictable (like AMI IDs per region), use Mappings. If it needs to vary per deployment based on user input, use Parameters.</p><h4>Outputs and Exports</h4><p>Holy shiet, If I use AI too much, this fucking article will be more than 2k lines. So for outputs, think like output in terraform ok? For exports, export in 1 stack, import in another stack:</p><p>Stack 1: export</p><pre>Outputs:<br>  StackSSHSecurityGroup:<br>    Value: !Ref MyCompanyWideSSHSecurityGroup<br>    Export:<br>      Name: SSHSecurityGroup   # this is the export name</pre><p>Stack 2: import</p><pre>Resources:<br>  MyInstance:<br>    Type: AWS::EC2::Instance<br>    Properties:<br>      SecurityGroups:<br>        - !ImportValue SSHSecurityGroup   # must match the fucking export name</pre><h4>Conditions</h4><p>Holy shit, I just don’t want to throw fucking code here but without it, everything else is useless</p><pre>Parameters:<br>  EnvType:<br>    Type: String<br>    AllowedValues: [dev, prod]<br><br>Conditions:<br>  IsProd: !Equals [!Ref EnvType, prod]<br><br>Resources:<br>  MyInstance:<br>    Type: AWS::EC2::Instance<br>    Properties:<br>      ImageId: ami-0ff8a91507f77f867<br>      InstanceType: t2.micro<br>      Volumes:<br>        - !If<br>          - IsProd<br>          - Device: /dev/xvdb<br>            VolumeId: !Ref ProdDataVolume<br>          - !Ref AWS::NoValue   # remove this property entirely if not prod<br><br>  ProdDataVolume:<br>    Type: AWS::EC2::Volume<br>    Condition: IsProd           # resource itself only created if IsProd=true<br>    Properties:<br>      AvailabilityZone: us-east-1a<br>      Size: 100<br>      VolumeType: gp3</pre><p>With this you will understand how conditions works (resource create by conditions)</p><h4>Rollbacks</h4><p>We have some common scenario for rolls back:</p><ul><li>If stack creation failed: Default, everything rolls back (deleted).</li><li>If stack update failed: the stack auto rolls back to previous working state.</li><li>If rolls back failure: fix it manually then ContinueUpdateRollback API from console.</li></ul><h4>Other useful sections</h4><ul><li>Delete policy: default will delete resource on delete CloudFormation stack, but we don’t want to delete S3 bucket, we set <strong>DeletionPolicy=Retain</strong>, and last one is snapshot (create one final snapshot before delete the resource, works for some supported resource).</li><li>Update replace policy: same idea with deletion policy but apply only during stack updates.</li><li>Stack Policy: protect your resource from unintended update, default all resources are protected, specify an explicit ALLOW for the resources you want to be allowed to be updated.</li><li>Termination Protection: to prevent accidental deletes on CloudFormation, enable this.</li><li>Custom resources: Running lambda function to empty s3 bucket before delete bucket. Define by “Type: Custom::MyCustomResource” and required ServiceToken which can be SNS/Lambda ARN.</li><li>Dynamic references: refer to external values stored in SSM parameter store or Secrets Manager. Format: <strong>{{resolve:service-name:reference-key}}</strong>. Service name value supports: ssm, ssm-secure, secretsmanager.</li><li>Propertie <strong>ManageMasterUsersPassword</strong> = true, CloudFormation will create a secret in Secrets Manager and it’s rotation</li><li>For UserData in CloudFormation, but it will always show success in CloudFormation even script in user data failed.</li><li>And yeah issue in user data scripts could be fail and will give us hardtime to debug, that is why CloudFormation provide a better way, it is <strong>cfn-init</strong>, let’s take a look at this picture below. Instead of writing 100+ lines of raw bash, you declare packages/files/commands/services in the <strong>AWS::CloudFormation::Init</strong> metadata, then just call <strong>cfn-init</strong> from UserData to run it. Main benefit is declarative and maintainable, not some magic bash script anymore. It also logs to <strong>/var/log/cfn-init.log</strong> on the instance so debugging is a nice side bonus xD.</li></ul><pre>Resources:<br>  MyInstance:<br>    Type: AWS::EC2::Instance<br>    Properties:<br>      ImageId: ami-123456<br>      InstanceType: t2.micro<br>      UserData:<br>        Fn::Base64: |<br>          #!/bin/bash<br>          yum update -y<br>          yum install -y httpd<br><br></pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*yuNz4qwAUTY6a5iD.png" /></figure><ul><li><strong>cfn-signal</strong> is not really a “debug” tool, it is a synchronization mechanism. Think of it like healthchecks.io, where your script sends a “success” ping to a URL, and if the ping fails or times out, an alert/rollback is triggered).</li><li>for <strong>cfn-signal</strong> normally CloudFormation marks an EC2 instance as CREATE_COMPLETE as soon as it&#39;s &quot;running&quot;, it has no idea whether your UserData/cfn-init actually succeeded inside. With a CreationPolicy attached, CloudFormation will pause and wait for a signal from the instance. At the end of your bootstrap script you call cfn-signal -e $? ..., exit code 0 means success, non-zero (or timeout with no signal at all) means failure, and then stack rolls back (deleted), like we talked about some lines above xD. This is also what makes ASG rolling update wait for each instance to be healthy before moving to the next one.</li><li>Nested stack: DRY LOL, mindset is same with GitLab template repo.</li><li>Depends On: depend on other resource before create xD. For example we have 2 fucking resources are EC2instances and S3 Bucket, we set S3bucket depend on EC2instance, that mean s3bucket will not create until ec2instance is created.</li><li>StackSets: holy fucking shit, this is just deploy 1 CloudFormation template to multiple AWS account and multiple fucking region at the same time.</li><li>Drift: It is same like state drift in terraform, little different drift detection in CloudFormation need to trigger manually or schedule, not like terraform plan.</li><li>Troubleshooting:</li></ul><pre>- DELETE_FAILED: some resource must be emptied before deleting such as S3 or SecGroup can not be deleted until all EC2 in the fucking group are gone. With DeletionPolicy = Retain --&gt; skip fucking deletion.<br>- UPDATE_ROLLBACK_FAILED: resource changed outside of CloudFormation.<br>- And more, I&#39;m fucking lazy right now!</pre><ul><li>StackFailures: Onfailure = rollback (default) or Do nothing (resource in state CREATE_FAILED) for debug purpose I think or event DELETE for fucking clean.</li></ul><h4>Conclusion</h4><p>We need to understand and remember this section more because it will related to AWS Devops Pro certification path.</p><h3>Section 6: Lambda for CloudOps</h3><p>Lambda is <strong>FAAS</strong>, run on-demand, limited by time (short executions). Lambda support container image but they are not prefer, ECS/Fargate are more preferred for running containers. And for the most important, it is integrations with a lot of AWS services by built-in.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/996/0*oUbShsG4qp22HEII.png" /></figure><h4>Some note for Lambda</h4><ul><li>Most example for Lambda is serverless cronjob I guess.</li><li>We can view lambda logs from CW logs.</li><li>Max memory supports for Lambda is 10Gb. <strong>/tmp storage</strong>: max is also 10GB.</li><li>Integration with CW Events / EventBridge: Create trigger every 1 hour that trigger Lambda function to perform a specific task.</li></ul><h3>Section 7: EC2 Storage and Data Management — EBS and EFS</h3><h4>EBS Volume</h4><p>It is like common volume in OpenStack, can be mounted to only 1 VM. Locked to AZ for sure, LOL.</p><p>There are some type for this volume type:</p><ul><li>gp2/gp3 (SSD): general purpose</li><li>io1/io2: highest performance ssd (this is exception for multi-attach in EBS)</li><li>st1: low cost hdd for requently access</li><li>sc1: low cost hdd but less frequently access</li></ul><p>Some notes:</p><ul><li>Remember only ssd (gp/io series) can be used as boot volume for high performance.</li><li>We can make EBS snapshots in AWS like snapshot in volume of OpenStack, same logic. AWS create a service called <strong>Amazon Data Lifecycle Manager</strong> to manage <strong>only</strong> creation,retention and deletion of EBS snapshots &amp; EBS-backed AMIs.</li><li>Locked at the AZ level. If you need more iops and is frequently running out of iops, increase volume size of EBS.</li></ul><h4>EFS — Elastic File System</h4><p>It is fuckin’ <strong>managed NFS</strong>. Available at region level, it means you can mount NFS from any EC2 in any AZs in that fucking region. We have lifecycle management feature — move file after N days, as a related storage service in AWS, it will have standard storage tiers:</p><p><strong>In case you forget: IA and Archive will reduce cost but also increase latency</strong></p><h3>Section 8: Amazon S3</h3><h4>S3 — Bucket</h4><p>Same meaning with Azure Blob Storage Container. There are 2 bucket type:</p><ul><li>General purpose</li><li>Directory: low-latency use cases, use only S3 Express One Zone storage class.</li></ul><p>Hmm, let’s figure why it is so fast in directory type: <a href="https://docs.aws.amazon.com/AmazonS3/latest/userguide/directory-bucket-high-performance.html#s3-express-one-zone">High performance workloads</a></p><ul><li>Single AZ + colocate storage with compute to reduce latency, network hops…</li><li>Session-based authentication: It uses CreateSession to get temporary credentials 1 time, next requests will use session token, reduce overhead in auth a lot.</li><li>Zonal endpoint: object operations (GET,PUT) use Zonal endpoint instead of Regional endpoint, traffic go directly into AZ, by passing regional layer xD</li><li>No replicate cross-AZ: data only stored in 1 AZ (multiple devices for sure), not overhead for sync cross-AZ</li><li>Custom or Purpose-built hardware: only storage data in high-performance hardware, doesn’t use same infrastructure with standard S3.</li></ul><h4>Pre-signed URL</h4><p>Use case: let user download file from private S3 bucket without create IAM user or make public bucket (Like SAS from Azure Storage!)</p><p>How it works:</p><ul><li>Credential and Signature embeded directly into URL.</li><li>Object still private, but anyone have URL can access.</li></ul><h4>S3 security: Bucket Policy</h4><p>User-based: IAM policies</p><p>Resource-based:</p><ul><li>Bucket polices — bucket wide rules from the s3 console, allow across account.</li><li>Object ACL — fine grain</li><li>Bucket ACL — less common</li></ul><p>S3 Bucket Policies is Json based policies that define :</p><ul><li>Effect (Allow/Deny)</li><li>Action (Action to Allow/Deny)</li><li>Principal (scope, account or user to apply policy to, * mean public)</li><li>Resource (s3 path arn). Example: “arn:aws:s3:::example-bucket/*”</li></ul><p>Use cases:</p><ul><li>Grant public access to the bucket</li><li>Force objects to be encrypted at upload</li><li>Grant access to another account (Cross Account)</li></ul><h4>AWS S3 Access Methods from an EC2 Instance</h4><p>Not related to the exam, but I want to put here to improve my understand. There are two primary ways to grant an Amazon EC2 instance access to an AWS S3 bucket.</p><p><strong>Method 1: Using IAM Roles (Recommended / Best Practice)</strong></p><ul><li>Mechanism: Attach an IAM Role (via an IAM Instance Profile) directly to the EC2 instance.</li><li>Authentication: The AWS CLI/SDK inside the VM automatically retrieves temporary, rotating credentials from the Instance Metadata Service (IMDS) at <a href="http://169.254.169.254/">http://169.254.169.254</a>.</li><li>Configuration: No credentials configuration required inside the VM. You only need to install the AWS CLI and run commands immediately.</li><li>Security: High security. It eliminates the risk of hardcoded keys and avoids credential leakage since AWS manages key rotation automatically.</li></ul><p><strong>Method 2: Using IAM User Credentials (Not Recommended for EC2)</strong></p><ul><li>Mechanism: Create a dedicated IAM User and generate a pair of long-term Access Key ID and Secret Access Key.</li><li>Authentication: The VM authenticates using these static, long-term credentials.</li><li>Configuration: Requires manual setup inside the VM by either running aws configure (saved in ~/.aws/credentials) or injecting them as environment variables (<strong>AWS_ACCESS_KEY_ID</strong> &amp; <strong>AWS_SECRET_ACCESS_KEY</strong>).</li><li>Security: Low security for EC2. If the VM is compromised or the image is shared, the static credentials could be exposed, leading to unauthorized access from anywhere.</li></ul><p>Key Takeaway:</p><ul><li>Use Method 1 (IAM Role) for resources inside AWS (like EC2 instances).</li><li>Use Method 2 (IAM User) only for resources outside AWS (like local development machines or external servers).</li></ul><p>Not related but for IAM User, without checking the <strong>Provide user access to the AWS Management Console</strong> option, we are essentially creating a <strong>Machine-to-Machine (M2M) / Robot account</strong> meant solely for programmatic access via APIs, CLI, or SDKs.</p><h4>Troubleshooting</h4><p>Don’t mistake S3 Versioning propagation with S3 Replication Time Control (RTC) which has a 15-minute SLA. Versioning is active immediately once enabled, so no fucking propagation delay of 15 minutes here! If you get a 404 NoSuchKey, it’s just because the key doesn’t exist, not because versioning is lagging.</p><h4>S3 Replication (CRR &amp; SRR)</h4><p>Replication using asynchronous mostly all the time. Same for Azure storage blob container, we need to enable versioning for replication! Use cases:</p><ul><li><strong>CRR</strong>: Compliance (store data in required region), low latency access from user region, replication across account</li><li><strong>SRR</strong>: log aggreration (Like Mysql master-slave, we did analytic from slave node, not master node). Live replication between prod and test account</li></ul><p>Only new objects are gonna replicate after enable replication, if you want replicate exists object, use S3 Batch Replication. Version ID also going to get replicated in destination bucket xD</p><p>We can replicate delete markers from source to target, Deletions with version ID are not replicated, prevent malicious deletes.</p><p>AWS S3 Replication Time Control (RTC) guarantees that 99.99% of new objects are replicated within 15 minutes. This feature behaves similarly to MySQL Slave Replication Lag. We can set up a CloudWatch Alarm to alert us if the replication delay exceeds xxx seconds, akin to monitoring Seconds_Behind_Master in a MySQL replica. Ofcourse, there is no fucking free lunch, extra cost per GB for the replication!</p><p>Another use case we can do with Batch Replication is add tag to object using job with manifest that contains bucket and path for objects we want to.</p><h3>Section 9: Advanced Amazon S3 &amp; Athena</h3><h4>Comparison vs Azure storage classes</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/906/1*6BbAGmjV6FUbOwlCK3OXsQ.png" /></figure><h4>S3 Analytics — Storage Class Analysis</h4><p>Great feature, that help you decide when to transition objects to right storage class for cost saving. Recommendations for Standard and Standard IA (Infrequently Access). Output file is CSV report</p><p>Report is updated daily, 24–48 hours to start seeing data analysis.</p><h4>Lifecycle policies</h4><p>Moving between storage classes, alright? We could have 2 separated transition, one for <strong>current version</strong> and one for <strong>non-current version</strong>.</p><h4>S3 Event Notification</h4><p>Handle event happens in S3 bucket, then trigger to SNS/SQS/Lambda for handle event. And yeah, everything required permission, SNS/SQS/Lambda are not an exception.</p><p>So let’s take an example for SNS topic, we create IAM Policy that attach to SNS Topic to allow S3 bucket send message directly to SNS Topic, same for SQS queue and Lambda function.</p><p>Basically, you attach this <strong>Resource-based Policy</strong> to the <strong>destination</strong> (SNS Topic, SQS Queue, or Lambda Function). <strong>NOT</strong> the S3 bucket:</p><ul><li><strong>SNS Topic Policy</strong>: Allows S3 to perform sns:Publish on the topic.</li><li><strong>SQS Queue Policy</strong>: Allows S3 to perform sqs:SendMessage on the queue.</li><li><strong>Lambda Function Policy</strong>: Allows S3 to perform lambda:InvokeFunction on the function.</li></ul><p>Generic template:</p><pre>{<br>  &quot;Effect&quot;: &quot;Allow&quot;,<br>  &quot;Principal&quot;: { &quot;Service&quot;: &quot;s3.amazonaws.com&quot; },<br>  &quot;Action&quot;: &quot;sns:Publish&quot;, // OR sqs:SendMessage OR lambda:InvokeFunction<br>  &quot;Resource&quot;: &quot;arn:aws:...&quot;, // ARN of SNS/SQS/Lambda<br>  &quot;Condition&quot;: {<br>    &quot;ArnLike&quot;: { &quot;aws:SourceArn&quot;: &quot;arn:aws:s3:::my-bucket&quot; }<br>  }<br>}</pre><p>And one more thing that all events from S3 will be send to AWS EventBridge, so from EventBridge can we have rules sent to more than 18++ AWS services as destination. EventBridge provide more options like Advanced Filter(metadata, object size…), Multiple Destination, EventBridge Capabilities… and so on. Worth to mention, AWS EventBridge = event-driven.</p><h4>S3 Inventory</h4><p>List objects and their corresponding metadata. Usage:</p><ul><li>Audit and report on the replication and encryption status of objects</li><li>Get the number of objects in S3 Bucket.</li><li>Identify the total storage of previous object versions.</li></ul><h4>Athena</h4><p>Serverless query service to analyze data stored in S3. Hmm, this shit is powerful!</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*LEz9HBkKQJ_21eFW.png" /></figure><p>Result can be stored in S3 Bucket for later analysis.</p><p>And again, there is no fucking free lunch. <a href="https://aws.amazon.com/athena/pricing/">Core Pricing Metric</a>: $5 per 1 TB of data scanned</p><ul><li>Athena does not charge based on query execution time.</li><li>Athena does not charge based on the amount of data returned (result set).</li><li>It charges you strictly based on the total volume of data it has to read (scan) from S3 to fulfill your query.</li></ul><p>Crucial Real-World Example:</p><ul><li>You have a 2 TB uncompressed text log file (CSV/JSON) stored on S3. You run a simple query: SELECT * FROM table LIMIT 10;.</li><li>Because raw text files lack an intelligent columnar layout or metadata index, Athena is forced to scan the entire 2 TB file from top to bottom just to find those 10 rows.</li><li>The result: That single query will cost you $10 in just a few seconds. If a BI Dashboard or an automated script triggers this query 50 times a day, you will burn $500/day instantly.</li></ul><p>That is enough I guess, there are other information related to convert Columnar (Parquet / ORC) to reduce data scan, partition data, gzip data and more….</p><h3>Section 10: Amazon S3 Security</h3><ul><li>VPC Gateway Endpoint for Amazon S3: No idea, only remember use VPC Gateway Endpoint to access S3 privately from EC2 for exam…</li><li>IAM Access Analyzer for S3: Ensure that only intended people have access to ur S3 buckets, we can review it is normal, expected or un-expected so you have right action to do.</li></ul><h3>Section 11: Advanced Storage Section</h3><h4>Amazon FSx</h4><p>Use cases:</p><ul><li>FSx for Windows: sharing disk that support Windows (SMB/NTFS), auto scale up/down. Managed service</li><li>FSx for OpenZFS: High performance, high speed for Linux.</li><li>FSx for Lustre: super performance for HPC/AI for Linux</li><li>FSx for NetApp OnTap: Support both of Window/Linux, for lift and shift to cloud.</li></ul><p>Summary: Higher performance than EBS/EFS but cost more money, right?</p><h4>AWS Storage Gateway</h4><p>Bridge between on-premises data and cloud data. Use cases:</p><ul><li>DR</li><li>Backup &amp; Restore… and more</li></ul><p>Types of Storage Gateway:</p><ul><li>S3 File Gateway: POSIX compliant (Linux filesystem)</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*PBJ1JjHsWMXwU6X9.png" /></figure><ul><li>Volume Gateway</li><li>Tape Gateway</li></ul><p>Better to take a picture for summary….</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*AaDeYy86o1Ns4J1d.png" /></figure><p>Storage Gateway VM is an on-premises virtual machine that bridges local servers to AWS cloud storage using standard file protocols.</p><h4>Little shit for exam:</h4><ul><li>Hybrid, Extend on-premises storage (Extend local disk), pick Storage Gateway</li><li>Migrate Windows File Server to Cloud, pick Amazon FSx for windows</li><li>Shared storage for thousands of EC2/GPUs for HPC/AI training -&gt; Pick FSx for Lustre (EFS/EBS can’t handle the bandwidth).</li><li>Enterprise needs NetApp ONTAP features (SnapMirror / Deduplication / Multi-protocol) -&gt; Auto-pick FSx for NetApp ONTAP.</li><li>Replace physical tape drives / Backup tape to cloud -&gt; Auto-pick Tape Gateway (AWS’s classic legacy backup trap).</li></ul><h3>Section 12: CloudFront</h3><p>CloudFront = CDN xD</p><h4>CloudFront — Origins</h4><ul><li>S3 bucket / Custom Origin (HTTP/S3 website)</li><li>VPC Origin: Private ALB/NLB/EC2 instances</li></ul><p>So basically we can make website from S3 bucket without making public bucket by assign right policy for CloudFront (via OAC — Origin Access Control), CloudFront will serve request from user then use it’s permission to access S3 data and return to user.</p><p>Origin Shield: a cache for your cache.</p><p>Every request made to CloudFront into a logging S3 bucket, that is where access logs stored.</p><h4>CloudFront reports</h4><p>We are able to generate reports on the data from Access Logs</p><ul><li>Cache Statistics Report</li><li>Popular Objects Report</li><li>Top Referrers Report</li><li>Usage Reports</li><li>Viewers Reports</li></ul><h4>CloudFront with ALB sticky sessions</h4><ul><li>Client must send Cookie within request to CloudFront</li><li>Then CloudFront must forward/whitelist the cookie that controls the session affinity to the origin to allow the session affinity to work.</li><li>You must set TTL to a value lesser than when the authentication cookie expires</li></ul><h4>AWS Global Accelerator</h4><ul><li>Unicast IP: one server holds one IP address</li><li>Anycast IP: all servers hold the same IP address and the client is routed to the nearest one. Holy shiet, for real?</li></ul><p>Yes, AWS Global Accelerator use Anycast IP concept to works.</p><p>Compare vs CloudFront</p><ul><li>CloudFront: Improve performance for both cacheable content, dynamic content and content is served at the edge location.</li><li>Global Accelerator: Improves performance for a wide range of application over TCP or UDP, proxying packets at the edge to applications running in one or more AWS region.</li></ul><h3>Section 13: Databases for CloudOps</h3><h4>RDS Read Replicas and Multi AZ</h4><p>I write some notes because this will appear in exam xD.</p><p>RDS read replicas for read scalability:</p><p>RDS Multi AZ (DR):</p><ul><li>Auto failover/promote standby to active.</li><li>The read replicas can be setup as Multi AZ for DR</li></ul><p>From Single-AZ to Multi-AZ: new DB is restored from snapshot in new AZ then synchronization is established between the two databases.</p><h4>Extra section not related to AWS</h4><p>Ok, there are 3 replication formats in MySQL, but worth, really worth to understand how Mysql Replication works!</p><ul><li><strong>Statement-based replication</strong>: master write exactly query we did, then slave replay it. Cons: will be issue if related to function like NOW(). Because NOW() from master at 12:00 will be something like 12:03 in slave, diff issue bro!</li><li><strong>Row-based replication</strong>: instead of write query SQL, master will record every change of “rows” (data changed). So Slave will replay exactly what is binlog, no diff issue. Cons: if we run update that target 1M rows, Binlog will increase way too big instead write simple 1 sql query.</li><li><strong>Mixed-based replication</strong>: This is default method of modern MySQL and is exactly what AWS RDS use in common I guess xD. It will make priority for Statement-based, but if any query contain something dangerous like <strong>NOW(), UUID(), RAND()</strong>, it will convert to <strong>Row-based</strong> to make data consistency!</li></ul><p>How sync work in Threads? So to understand why it called Async, we need to imagin that in replica side, there are 2 Threads work separately:</p><h4>Performance Insights</h4><p>This feature expensive AF, I think most company will use self-hosted stack instead using AWS if there are giant log/metric (high throughput/high cardinality)!</p><h4>Other shit</h4><p>I don’t mention them here because personally I think I understand them enough, I can reflection self-hosted in on-premises versus managed service in AWS…</p><p>I spent 15–20 minutes to watch whole section xD</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/812/0*KybQsYqVGIGLRr6b.png" /></figure><p>I fail with RDS backup, it only able to take backup from standby, even Read replicas still is a standby with read traffic, so it can not make sure you have the fucking latest data to promote standby to active. And 2 fail question because I did skip them xDD</p><h3>Section 14: Monitoring, Auditing and Performance</h3><p>Holy fucking shit, pay more to gain more metric or faster interval when we have fucking Prometheus/victoriaMetrics! And custom metric for what when we have fucking node exporter that include 99% what we need. For anomaly detection instead of using CloudWatch we can use following in Prometheus for example:</p><ul><li><a href="https://prometheus.io/docs/prometheus/latest/querying/functions/#predict_linear">predict_linear</a></li><li>Compare current week with last week:</li></ul><pre># Alert if current traffic is higher 50% than last week at this time!<br>abs(<br>  (rate(http_requests_total[5m]) - rate(http_requests_total[5m] offset 1w))<br>  / <br>  rate(http_requests_total[5m] offset 1w)<br>) &gt; 0.5</pre><p>Custom dashboard in CloudWatch (3$/month after free dashboard)? Hmm, we have metrics, we can draw any dashboard we want in Grafana.</p><p>CW Synthetics Canary = <strong>Blackbox Monitoring</strong> like zabbix/prometheus blackbox exporter. But Synthetics Canary can do more with headless google chrome browser, not only monitoring….</p><h4>EventBridge Content Filtering</h4><p>Worth money feature I guess.</p><p><strong>Cost optimization</strong>:</p><ul><li>Without filtering: lambda trigger 1M times → holy fucking AWS billing</li><li>With filtering: EventBridge stop right front door, only event have right condition go through to trigger Lambda.</li></ul><p><strong>Decoupling</strong>: No need to handle if/else for event in Lambda</p><p><strong>Powerful Matching Operators</strong>: Check exists, filtering numberic range, filtering with suffix,prefix, ip address….</p><p><strong>With Input Transformation</strong>: zero cost and cost saving because we don’t need to parse data and it is fucking free!</p><h4>AWS CloudTrail</h4><ul><li>Get history of events / API calls made within your AWS account.</li><li>Audit for your account, enable by default.</li><li>If a resource is deleted in AWS, investigate CloudTrail first!</li><li>Stored for 90 days.</li></ul><h4>AWS Config Overview</h4><p>AWS Config = Azure Policy + Azure Resource Graph</p><p>Record configuration Changes and evaluate resources against compliance rules, remediation action…</p><p><strong>Configuration timeline / time travel</strong>: Like 2h AM last sunday, how EC2 instance is configured with which spec, secGroup open which port???</p><p><strong>Resource Relationship Tracking</strong>: visualize relationship between resources. If EBS config changed, AWS Config tell exactly where EBS disk is attached to which EC2, which VPC!</p><h3>Section 15: AWS Account Management</h3><h4>SCP — Service Control Policy</h4><p>So for example, we can restrict specific AWS account in our Org that not able list s3 buckets by use Deny action on s3:ListAllMyBuckets, even that account has Admin or an IAM user has policy that allow s3:ListAllMyBuckets, why?</p><ul><li><strong>Explicit Deny wins</strong>: In AWS IAM policy evaluation, an explicit deny always overrides any allow, no matter where it is defined (in IAM policies, Resource policies, or SCPs).</li><li><strong>SCPs set the limit</strong>: Service Control Policies (SCPs) define the maximum available permissions for the accounts in an organization. If an SCP denies an action (like s3:ListAllMyBuckets), no user or role in that account can perform it, even if their local IAM policies explicitly allow it.</li></ul><p>Example SCP code:</p><pre>{<br>  &quot;Version&quot;: &quot;2012-10-17&quot;,<br>  &quot;Statement&quot;: [<br>    {<br>      &quot;Sid&quot;: &quot;DenyAllFuckingListBuckets&quot;,<br>      &quot;Effect&quot;: &quot;Deny&quot;,<br>      &quot;Action&quot;: [<br>        &quot;s3:ListAllMyBuckets&quot;<br>      ],<br>      &quot;Resource&quot;: [<br>        &quot;*&quot;<br>      ]<br>    }<br>  ]<br>}</pre><p>SCP doesn’t work with Management Account, it is able to deny root user aswell.</p><p>In fact, SCP is just a filter. It does <strong>not</strong> grant permissions; it only sets the maximum permission boundaries (the “guardrail”).</p><ul><li>If the SCP <strong>allows</strong> it, but the user’s IAM Policy <strong>does not</strong>, the user <strong>cannot</strong> do it.</li><li>If the IAM Policy <strong>allows</strong> it, but the SCP <strong>denies</strong> it (or doesn’t allow it), the user <strong>cannot</strong> do it.</li></ul><p>Think of it exactly like a <strong>Network ACL (NACL) and a Security Group (SG)</strong>:</p><ul><li><strong>NACL (SCP)</strong>: Subnet-level firewall (defines what can/cannot enter the account).</li><li><strong>Security Group (IAM Policy)</strong>: Instance-level firewall (defines what the specific identity is actually permitted to do).</li><li><strong>Rule</strong>: Both must allow the action. If either blocks it (or doesn’t allow it), the action is denied.</li></ul><h3>Other shit</h3><ul><li>AWS Control Tower = Azure Landing Zone. AWS Control Tower is service that handle multiple aws account or orchestrator.</li><li>AWS Service Catalog: self-service portal with tag option(group them by tag)… Seem like <strong>Backstage</strong>! And able to share service catalog to other account in same ORG. User can use service catalog to deploy safely, no need admin aws permission!</li><li>Create Billing alarm with CW metric and SNS topic:</li></ul><pre>- Important constraint: The billing metric (EstimatedCharges) is only available in us-east-1 (N. Virginia). You must create the alarm there.<br>- Prerequisite: You must enable &quot;Receive Billing Alerts&quot; in the Billing Preferences console first.<br>- Frequency: The metric is updated only every 4-6 hours (not real-time).<br>- Basically when the metric reaches the threshold, CloudWatch Alarm pushes a message containing alarm info (alarm name, reason, cost details) to the SNS topic, which notifies subscribers (emails, webhooks).</pre><ul><li>AWS Billing Conductor for customizing and presenting AWS billing data, common use cases: enterprise, finance team, managed service provider (MSP)…</li></ul><h3>Section 16: Disaster Recovery</h3><h4>AWS Datasync</h4><ul><li>Move large amount data to and from: on-prem/other cloud to AWS, need to install aws agent. And AWS to AWS no need to install agent</li><li>Synchronize: Amazon S3 to any storage class for example (Glacier)</li><li>Replication tasks can be scheduled hourly, daily, weekly.</li></ul><h4>Other shits</h4><ul><li>When enable backup vault lock policy, we can not delete backups even we are using root user.</li></ul><h3>Section 17: Security and Compliance for CloudOps</h3><p>Quick remember:</p><ul><li>WAF = Layer 7</li><li>Firewall Manager = Centralized Security Management</li><li>Shield = DDOS Protection</li><li>Amazon Inspector: analyze issue in EC2, image in ECR, Lambda functions only!</li><li>Amazon GuardDuty: analyze your log to detect threats. CloudTrail, VPC Flow Logs, DNS logs….</li><li>Amazon Macie: Protect sensitive data in AWS…</li><li>AWS Security Hub: Central security tools….</li><li>AWS Secret Manager = Hashicorp Vault for an easy reflection!</li></ul><h3>Section 18: Identity</h3><ul><li>AWS Identity Federation: no idea… Example: provide access to write to s3 bucket using facebook login… (Like Entra ID B2B/B2C)</li><li>AWS STS — Security Token Service.</li><li>IAM Policy Simulator: testing policies….</li></ul><h3>Section 19: Networking — Route 53</h3><p>Even thought there are a lot of basics, but some basics I don’t understand enough so I will write down here!</p><p>Let’s compare Simple route vs Weighted route first. I didn’t understand this back in 2024 when I use self-hosted DNS.</p><p>Other routing policies:</p><ul><li><strong>Geolocation</strong>: based user location or where user want to be from (VPN xD)</li><li><strong>Geoproximity</strong>: pretty new for me, route traffic to your resources based on the geographic location of users and resources. For example we need to remember this keyword for shift traffic by increasing bias….</li><li><strong>IP-based routing</strong>: routing based on client’s IP address. Use case: optimize performance, network costs. Example: route end users from a particular ISP to a specific endpoint..</li><li><strong>Alias Record</strong>: This is fuckin’ unique feature of AWS only that allow map domain to AWS services directly, use-able on root domain also and free fuckin’ traffic DNS fee!</li><li><strong>Inbound resolver</strong>: On-premise ask AWS.</li><li><strong>Outbound resolver</strong>: AWS ask On-premise.</li></ul><p>Other shits:</p><ul><li>A <strong>Public Hosted Zone</strong> is a container that holds DNS records for a domain that you want to be accessible on the public internet.</li><li>A <strong>Private Hosted Zone</strong> is a container that holds DNS records for a domain that you want to resolve only within one or more of your Amazon VPCs.</li><li>Both <strong>Public/Private Hosted Zone</strong> available inside Amazon <strong>Route 53</strong>. Nothing to say more about Public Hosted Zone but for Private Hosted Zone, you will be able to select the VPCs you want to associate it with!</li><li><strong>Route 53 Query Logging for Public Hosted Zones</strong>: Enabled directly on the hosted zone. Logs are sent <strong>only</strong> to <strong>CloudWatch Logs</strong> (the log group <strong>must</strong> be created in us-east-1).</li><li><strong>Route 53 Resolver Query Logging for Private Hosted Zones / VPC queries</strong>: You cannot enable query logging directly on private hosted zones. Instead, you must use <strong>Route 53 Resolver Query Logging</strong> at the VPC level. These logs can be sent to <strong>CloudWatch Logs, S3, or Kinesis Data Firehose</strong>.</li><li><strong>Resolver DNS Firewall</strong>: filter outbound DNS queries going out through Route53 resolver.</li><li>If we want setup APEX Domain (It is fuckin’ Alias Record) for S3 bucket with static hosting enabled, the bucket name must the same as APEX Domain (Root domain, zone Apex….)</li></ul><h3>Section 20: Networking — VPC</h3><p>When you create a subnet remember that you are only able to use subnet range — <strong>5 IPs</strong> that is reversed by AWS, pretty same like how VPC in Azure works! The smallest subnet you can use is <strong>/28</strong> which has 16 IPs total — 5 reversed IPs = 11 useable IPs!</p><p>Actually, we’re gonna make a lot of fun together. <a href="https://www.youtube.com/watch?v=njtfNPB-YKg"><strong>Insert Tuco sentence from Breaking Bad here 0:51</strong></a></p><h4>Subnets</h4><p>You may see a lot of tutorial related to create public/private subnet in VPC in AWS. But you may don’t know why like me in the past xD, let’s talk about it:</p><ul><li><strong>Private Subnet</strong>: we often use this to place our applications, for example EC2 instances running Web/Application workloads.</li><li><strong>Public Subnet</strong>: Bastion, ALB. For example, we would never put DB instances in a Public Subnet, for interface facing.</li></ul><p>Without route table, can we use EC2 instance connect? No, it still need access to establish SSH to IP of EC2 instance, because without routing properly (missing route <strong>0.0.0.0/0</strong> that point to IGW), so packet return from EC2 don’t know which way for packet return back to AWS/Internet. This is easy to understand if you have working in the past with VMWare VDC.</p><p>With EC2 Instance Connect Endpoint (EIC Endpoint), released in 2023, no need route to IGW, no need public IP either!</p><p>So for 3 tiers application as theory:</p><ul><li><strong>Presentation tier</strong>: ALB -&gt; Public Subnet (require atleast 2 AZ for HA)</li><li><strong>Application tier</strong>: EC2 instances/ECS/EKS Worker group…. -&gt; Private Subnet</li><li><strong>Data tier</strong>: RDS -&gt; Private subnet, different with subnet of application tier</li></ul><h4>NAT Gateway</h4><p>Pretty sure it is fuckin’ easier than NAT Instance, because it is fuckin’ managed service. You only need to add route from your instance to NAT Gateway, that’s all! But we still need to remember litte things:</p><ul><li>Public NAT Gateway (Standard Internet Access) -&gt; Must be in a Public Subnet.</li><li>Private NAT Gateway (Internal Routing Only) -&gt; Can be in a Private Subnet</li></ul><p>Conclusion:</p><ul><li>Need to download packages from the internet? Public subnet only.</li><li>Need to connect to an overlapping private VPC/on-premises network without internet access? Private subnet.</li></ul><h4>NACL</h4><ul><li>NACL works at subnet level, not instance level like SecGroup.</li><li>Not like SecGroup which is stateful, NACL is stateless, need define rule for both inbound/outbound!</li><li>Rule based number, lower number, higher priority. Support both Allow/Deny while SecGroup has only Allow. If you have rule number 100 is Deny and rule number 120 is Allow, then it will deny and stop processing for that target!</li><li>Each VPC has a default NACL, auto allow all traffic in/out and attach by default for all subnet in that VPC.</li><li>1 VPC can have more than 1 NACL (Default NACL + custom NACL).</li><li>Please don’t modify default NACL, instead create fuckin’ custom NACLs.</li><li>1 subnet can have only 1 NACL attached at the same time, 1 NACL can attach to multiple subnets.</li></ul><h4>VPC Endpoints</h4><p>Private endpoints connection, we have 2 type of endpoints:</p><p><strong>Interface Endpoint</strong>: Creates an ENI (Elastic Network Interface) with a private IP in your subnet.</p><ul><li><strong>Security</strong>: You can attach a Security Group directly to it to control inbound traffic (e.g., allow port 443 from your App instances).</li><li><strong>Resolution</strong>: Uses Private DNS to map the AWS service endpoint directly to this private IP.</li><li><strong>Coverage</strong>: Supports most AWS services.</li><li><strong>Cost</strong>: Costs money per hour and per GB data processed.</li></ul><p><strong>Gateway Endpoints</strong>: Free!. Muse be used as target in route table, it means you need to add route for specific ip range via Gateway Endpoints. Only support <strong>S3 and DynamoDB</strong></p><h4>Other shits:</h4><ul><li>You need to know how to calculate CIDR in your head without CIDR tool during exam xD. For example <strong>10.0.4.0/28</strong>. So 32–28=4, 2⁴=16 (16 IPs). Then you can see what is right answer here. Note: I wish I understand this earlier during AZ-104 exam….</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/834/0*PLfpPgBoruOH3LHr.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/834/0*JFlk_iKKqyFFzEAQ.png" /></figure><ul><li>Ok, why this is wrong answer for question above??</li><li>32–27 = 5. 2⁵ = 32, seem fit right?. But it required at least capacity for 28 EC2 instances and I forgot there are always 5 IP reversed by AWS!. So 32–5 = 27 useable IPs!. So the damn correct fuckin’ answer should be /26!</li><li><a href="https://docs.aws.amazon.com/vpc/latest/userguide/vpc-cidr-blocks.html">max CIDR size in AWS</a> is /16. Smallest is /28 ( 16 IPs)</li><li><a href="https://aws.amazon.com/network-firewall/">AWS Network Firewall</a> is a managed, stateful Layer 3 to Layer 7 service for Amazon VPCs. It offers deep packet inspection, intrusion detection and prevention (IPS), and web filtering…</li><li><a href="https://docs.aws.amazon.com/vpc/latest/userguide/working-with-managed-prefix-lists.html">Customer-managed prefix lists</a> allow you to define and maintain your own sets of IP address ranges, known as prefixes, within AWS.</li><li><strong>VPC Block Public Access (BPA)</strong>: A global “kill switch” at the Account/Org level to block all Internet Gateway (IGW) traffic to prevent accidental public exposure. You can exclude/whitelist specific subnets (like ALB/Bastion) that need internet.</li><li><strong>VPC Reachability Analyzer tool</strong>: analyze connection between 2 VPC, show which component not allow/block connection. Cost money for each analyze!</li><li><strong>VPC peering</strong>, this is easy to understand, right? But even you setup already in dashboard, you still need to modify both route table on both side the VPC you are connecting, so each subnet knows how to route traffic through the peering connection. It is pretty easy to understand if you are familiar with setup routing in Linux.</li><li>Like other <strong>VPC Peering</strong>, for example Azure, VPC Peering is not transitive!</li><li>So you know limit with VPC Peering, we have Transit Gateway that able to resolve pain of VPC Peering in scenario you have thousands of VPC….</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/409/0*Mtn3nubgkNRUAmM3.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*Nv9q5L1dVX8vjVcf.png" /></figure><ul><li>The picture above is for 1 region, how about multiple region? we have inter-region communication above!</li><li><strong>AWS PrivateLink</strong>: connect to your service in other VPC without exposing to public.</li><li><strong>Traffic Mirroring</strong>: yes, mirroring, you know what it means. Use case route the traffic to security appliances that you manage to analyze traffic without affect production workload!</li><li><strong>Egress-only internet gateway</strong>: Used for IPv6 only. You must update the route tables like IGW or NAT GW. Oh IPv6 have no fuckin’ private address, they are all public/route able in internet! So your EC2 instance may have IPv6 but it may get un-wanted access. That is why we have fuckin’ Egress-only internet gateway! Nice thing to know about IPv6! This is what I really love when learning!</li><li>Egress-only IGW is fuckin’ free! It doesn’t need to translate private ip to public ip like NAT Gateway (IPv4)!</li><li><a href="https://aws.amazon.com/blogs/networking-and-content-delivery/vpc-block-public-access/">VPC Block Public Access</a> is a simple, declarative control that authoritatively blocks incoming (ingress) and outgoing (egress) VPC traffic through AWS provided internet paths!</li><li>Again, SecGroup is stateful and NACL is stateless. Remember it! On-prem reflection: Openstack vs Vmware VDC!</li></ul><h3>Section 21: Other Services</h3><p>I will review this section after I took exam, nothing much to write here right now!</p><h4>SQS</h4><ul><li>Send msg to DLQ after x times process fail!</li><li>Visibility Timeout: when a consumer receive msg from SQS, that msg will be hidden from queue during processing time. If consumer process more than Visibility Timeout without haven’t call deleteMsg action, SQS think that consumer dead, return msg to queue, other consumer or “fake dead” consumer process it again → duplicate process.</li><li>SNS Filter Policy: Imagine we have multiple subscribers, for example: Billing, Shipping, and Fraud Detection systems. When we publish a message, by default all subscribers receive every message, even if it’s not relevant to their logic. That’s why we use an SNS Filter Policy: we attach a filter policy to each subscription, based on message attributes (e.g., messageType: billing), so each subscriber only receives the messages that match its filter — instead of processing every message and discarding irrelevant ones.</li><li>X-Ray = distributed tracing service, sidecar container is 1 way to deploy daemon for EKS/ECS. Other shits are EC2/Lambda.</li></ul><h3>Tip trick</h3><h4>Read the damn question carefully!!!</h4><p>LOL, I don’t read questions careful enough. For example:</p><pre>You have attached an Internet Gateway to your VPC, but your EC2 instances still don&#39;t have access to the Internet. What is NOT a possible issue?</pre><ul><li>I pick route table are missing, wrong!</li><li>I pick NACL doesn’t allow network traffic out, wrong again!</li><li>I pick the EC2 instance doesn’t have a fuckin’ public IP, still wrong! WTF!!!</li></ul><p>Wait a minute…. What is “NOT” a possible issue. Then I realized….</p><p>NOTE: As of March 28, 2023, the AWS Certified SysOps Administrator — Associate exam will not include exam labs until further notice….. They should make exam hard like Azure (AZ-104) hard AF!!!!</p><h3>Secret Weapons</h3><p>I used this course to practice and test my knowledge:</p><p><a href="https://www.udemy.com/course/prepare-for-aws-certified-cloudops-engineer-associate-exam/">Prepare for AWS Certified CloudOps Engineer — Associate exam</a></p><ul><li>Not too bad, but yeah, this article is lack of domain 4 and 5 I guess or I didn’t remember them well..</li></ul><h3>Conclusion</h3><ul><li>Not too bad, but yeah, this article is lack of domain 4 and 5 I guess or I didn’t remember them well..</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/780/0*0amw7BPFeIi3Gmd-.png" /></figure><ul><li>I didn’t remember correct what <strong>GWLB</strong> do exactly during exam or not sure <strong>Amazon Macie</strong> can do scan sensitive data on S3 only or able to scan on other datasource.</li><li>The amount of service you need to memorize is a lot.</li><li>It still easier a lot than Azure AZ-104 LOL. Possible done within a week if you worked with AWS before (I haven’t work in AWS single day, just doing some labs….).</li></ul><p><em>Originally published at </em><a href="https://blackmetalz.github.io/prepare-for-aws-certified-cloudops-engineer-associate-soa-c03-2026-exam.html"><em>https://blackmetalz.github.io</em></a><em> on July 19, 2026.</em></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=b33f6a8d598c" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Prepare for the Microsoft Certified: Azure Administrator Associate (AZ-104) exam]]></title>
            <link>https://medium.com/@kienlt.qn/prepare-for-the-microsoft-certified-azure-administrator-associate-az-104-exam-094d89f0806f?source=rss-b77f4e43d676------2</link>
            <guid isPermaLink="false">https://medium.com/p/094d89f0806f</guid>
            <category><![CDATA[azure]]></category>
            <category><![CDATA[azure-certification]]></category>
            <category><![CDATA[az-104-certification]]></category>
            <dc:creator><![CDATA[Kienlt]]></dc:creator>
            <pubDate>Sun, 28 Jun 2026 14:35:57 GMT</pubDate>
            <atom:updated>2026-06-28T14:35:57.069Z</atom:updated>
            <content:encoded><![CDATA[<p>New challenge, acquire all DevOps certifications in 3 popular public cloud providers, Azure is the first target of mine. Let’s fucking go!!!</p><p>I will try to refer to AWS as much as I can because AWS is the first cloud I learned.</p><p>This article could contain some wrong information since I’m new to this, correct me if I’m wrong, thanks xD</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*d2icnUNBIS1Vt_qb.png" /></figure><h3>Prepare for the exam</h3><h4>Useful resources:</h4><p>Many images are taken from <a href="https://learn.microsoft.com/">https://learn.microsoft.com/</a>. I think next article better use link instead copy to this repo and include to article xD</p><p>Resources must watch (Yeah, it is a lot if you never been Azure portal before like me!):</p><ul><li><a href="https://learn.microsoft.com/en-us/training/paths/az-104-administrator-prerequisites/">AZ-104: Prerequisites for Azure administrators</a> — Provided by Microsoft</li><li><a href="https://www.youtube.com/watch?v=uBqp3ivFC_o&amp;list=PLahhVEj9XNTcj4dwEwRHozO3xcxI_UHYG">AZ-104: Azure Administrator</a> — Provided by Microsoft</li><li><a href="https://www.udemy.com/course/70533-azure/">AZ-104 Microsoft Azure Administrator — Complete Exam Prep — Scott Duffy</a> — Not recommend unless you are working daily with Azure.</li><li><a href="https://www.youtube.com/watch?v=KD1cEc1-vOE">AZ-104 Preparation Done Wrong | What Top Scorers Do Differently</a></li><li><a href="https://viblo.asia/p/kinh-nghiem-thi-chung-chi-az-104-microsoft-azure-administrator-5pPLkxgyVRZ">Vietnamese — Kinh nghiệm thi chứng chỉ AZ-104: Microsoft Azure Administrator</a></li><li><a href="https://www.youtube.com/watch?v=w_G1Rufop-s">Prepare for Microsoft Certification Exam AZ-104: Microsoft Azure Administrator</a> — Provided by Microsoft</li></ul><h4>Some sections need to be understood before taking the exam.</h4><ul><li>Domain 1 — Identity &amp; Governance (20–25%)</li><li>Domain 2 — Compute (20–25%)</li><li>Domain 3 — Storage (15–20%)</li><li>Domain 4 — Networking (15–20%)</li><li>Domain 5 — Monitor &amp; Backup (10–15%)</li><li>Exam Tip/Tricks</li><li>Secret Weapons</li><li>Taking the Exam!</li></ul><p>Ok, let’s take them one by one!</p><h4>Azure physical infrastructure</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*41iW1HBKojhBRZre.png" /></figure><h4>Availability Zones</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*E4nt2EaUtq1cYkAG.png" /></figure><h4>Use availability zones for your workloads</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*hNIiEh1kDQ-nHmSB.png" /></figure><h3>Domain 1 — Identity &amp; Governance</h3><h4>Hierarchy</h4><p>We can simply understand it by following (Generated by Sonnet 4.6 xD):</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*QvxyKtZ4k-jtFHNt.png" /></figure><h4>RBAC</h4><p>I like this fucking picture that explain everything!</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*wwlf5Idj7YxaEYL5.png" /></figure><p>And even more with this!</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*may0FVxtxw8afI9M.png" /></figure><p>RBAC = IAM = Who → What → Where</p><p>Here are common built-in roles of Azure RBAC:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/902/1*lAoEc8xbj1U4AcO67pSItA.png" /></figure><p>Resource reference:</p><p><a href="https://learn.microsoft.com/en-us/azure/role-based-access-control/overview">https://learn.microsoft.com/en-us/azure/role-based-access-control/overview</a></p><p>Inheritance: very very very important, <strong>assigned 1 time in Management Group, all sub resource inherit.</strong></p><h4>Azure Policy vs IAM Policy</h4><p>Totally different versus AWS IAM Policy. Azure Policy enforce rules on resources, not permissions.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/902/1*3GhrWRxBHP1DWKE_z6h5HA.png" /></figure><p>Example about Azure Policy:</p><ul><li>VM only created in Southeast Asia.</li><li>Storage Account must enable encryption.</li><li>All resources must have tag Environment</li><li>VM can not create with SKU 700 core and 12 Tb ram xD</li></ul><h4>Management Groups</h4><p>Like AWS Organization, group Subscription for apply policies/RBAC</p><pre>Root Management Group<br>  ├── Production<br>  │     ├── Subscription: prod-app1<br>  │     └── Subscription: prod-app2<br>  └── Development<br>        └── Subscription: dev-sandbox</pre><p>Policy/RBAC assigned in Management Group, inherit to all subscription. Easy to understand, right?</p><p>Resource reference:</p><p><a href="https://learn.microsoft.com/en-us/azure/governance/management-groups/overview">https://learn.microsoft.com/en-us/azure/governance/management-groups/overview</a></p><h4>Little concept comparison</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/901/1*BnkkbYcWHGyLhBS6ugIEbg.png" /></figure><h4>Entra ID (Azure AD) B2B/ B2C</h4><p>B2B: <a href="https://learn.microsoft.com/en-us/entra/external-id/what-is-b2b">Business to Business</a></p><p>B2C: <a href="https://learn.microsoft.com/en-us/azure/active-directory-b2c/overview">Business to Consumer</a></p><ul><li>B2B: invite user from external tenant (Different Entra ID or Microsoft Account), that user still uses their own credentials from their ORG to login, our Entra ID only creates a Guest Account in our tenant. Use case for partner companies, contractors, vendors that need access to our resources</li><li>B2C: User can login via third party provider like facebook, google, github via Azure AD B2C tenant (not normal Entra ID), used for application that have customer-facing. Use case for our application allow end-user sign-up/sign-in</li></ul><h4>System-Assigned Managed Identity (SAMI) vs User-Assigned Managed Identity (UAMI)</h4><p>Resource reference:</p><p><a href="https://learn.microsoft.com/en-us/entra/identity/managed-identities-azure-resources/overview">https://learn.microsoft.com/en-us/entra/identity/managed-identities-azure-resources/overview</a></p><p><a href="https://learn.microsoft.com/en-us/entra/identity/managed-identities-azure-resources/managed-identity-best-practice-recommendations">https://learn.microsoft.com/en-us/entra/identity/managed-identities-azure-resources/managed-identity-best-practice-recommendations</a></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/906/1*sp62bo9dMKYRH5nw8kvwHg.png" /></figure><h4>Control Plane vs Data Plane</h4><pre>Subscription Owner<br>├── Control plane: create/delete/manage resource (Microsoft.Storage/*, Microsoft.KeyVault/*)<br>└── Data plane: No — must assign role explicitly</pre><p>For example:</p><ul><li>Owner can create Storage Account</li><li>Owner access blob (Binary Large Object) -&gt; need role “Storage Blob Data Reader”</li><li>Owner can create Key Vault</li><li>Owner access secret -&gt; need role “Key Vault Secrets User”</li></ul><p>Resource Reference:</p><ul><li><a href="https://learn.microsoft.com/en-us/azure/azure-resource-manager/management/control-plane-and-data-plane">Azure Control Plane and Data Plane</a></li><li><a href="https://learn.microsoft.com/en-us/azure/storage/blobs/assign-azure-role-data-access?tabs=portal">RBAC for Storage</a></li><li><a href="https://learn.microsoft.com/en-us/azure/key-vault/general/rbac-guide?tabs=azure-cli">RBAC for Key Vault</a></li></ul><h4>Managed Identity + Key Vault</h4><p>Pattern:</p><pre>App Service → [SAMI] → Key Vault Reference → Secret</pre><p>Setup step:</p><ul><li>Enable SAMI in App Service</li><li>Assign role Key Vault Secrets User for managed identity in Key Vault</li><li>App setting uses @Microsoft.KeyVault(SecretUri=...)</li></ul><p>No need to store credentials anywhere, managed identity auto get token from Azure AD</p><p><strong>K8s analogy:</strong> This pattern is identical to <strong>Workload Identity</strong> (AKS/GKE) or <strong>IRSA</strong> (EKS — IAM Roles for Service Accounts):</p><pre>K8s:    Pod → [ServiceAccount] → IAM Role / Managed Identity → AWS/Azure resource<br>Azure:  App Service → [SAMI] → RBAC role → Key Vault secret</pre><p>Same idea: workload is bound to an identity, the identity is granted permissions, and the runtime auto-exchanges tokens (via OIDC / IMDS endpoint). Code only calls the SDK — no secrets or keys needed anywhere. When Pod/App restarts, the identity persists — no manual credential rotation.</p><p>It also applies to in-cluster K8s RBAC: a Pod runs as a ServiceAccount, that SA is bound via Role/RoleBinding (or ClusterRole/ClusterRoleBinding), and calls to the K8s API server (default endpoint <a href="https://kubernetes.default.svc/"><strong>https://kubernetes.default.svc</strong></a> inside the Pod) are authenticated using the projected token at <strong>/var/run/secrets/kubernetes.io/serviceaccount/token</strong>. Same identity-binds-to-workload pattern, just scoped to the cluster instead of cloud resources.</p><p>Resource Reference:</p><ul><li><a href="https://learn.microsoft.com/en-us/azure/app-service/app-service-key-vault-references?tabs=azure-cli">Use Key Vault references for App Service and Azure Functions</a></li><li><a href="https://learn.microsoft.com/en-us/azure/app-service/overview-managed-identity?tabs=portal%2Chttp">How to use managed identities for App Service and Azure Functions</a></li><li><a href="https://learn.microsoft.com/en-us/azure/aks/workload-identity-overview">Azure AD Workload Identity for AKS</a></li></ul><h4>Conclusion and Gotchas</h4><ul><li>RBAC = IAM Roles.</li><li>Azure Policy = Service Control Policies (but have more flexibility)</li><li>SAMI is often preferred because it will not create orphaned identities xD</li><li>Subscription owner can not read KV secret by default. Owner only has control plane permissions — need to self-grant access. For Azure RBAC model, assign data plane role (e.g. Key Vault Secrets User). For Vault Access Policy model, configure Key Vault access policies to grant permissions.</li><li>Key Vault Contributor = manage vault configuration (control plane), NOT read secret</li><li>Key Vault Administrator = full data plane access (read/write/delete secrets, keys, certs)</li><li>For vault access model best practice: pick RBAC!</li><li>Microsoft Identity Manager (MIM) = bridging/sync identity store on-premises (like AD FS) with Microsoft Entra ID</li><li>When need to assign permission to access Azure resource, use security group.</li><li>When need to create team collaboration (email, teams, sharepoint), use M365 group (M365 group could be used for assign RBAC but primary use still collaboration!)</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*3Wap5DX2VH6ioJ9z.png" /></figure><p>Resource Reference:</p><ul><li><a href="https://learn.microsoft.com/en-us/azure/key-vault/general/rbac-guide?tabs=azure-cli#azure-built-in-roles-for-key-vault-data-plane-operations">Built-in roles for Key Vault data plane operations</a></li></ul><h4>Enhanced after fail first exam</h4><ul><li>3 different type of group: cloud, hybrid and guest group?</li><li>Microsoft entra ID has different licenses: Free, P1, P2 and Governance. Like PIM for only P2 license, SSPR required P1.</li><li>No fucking break inheritance. Initiatives are groups of policy definitions, RG can’t be nested</li><li>User type can be: Member or Guest. Dynamic membership is a rule if user have some specific attribute that matched with dynamic membership, they will automatically added to that group, that is why it called Dynamic Group I guess! Group with membership assigned is common group that Admin user select member manually.</li><li>Administratives users can work which some parts of your organization. Administrative units allow you to segregate your directory such that certain administrators only have permissions to work on certain areas.</li><li>Devices in Azure AD represent and manage physical devices used by users that access organization’s resources. You can allow/block devices of specific users.</li><li>Bulk Operations: Allow you create a lot of users from csv file.</li><li>External Users: Can be understood as Guest Users (Need to invite from user creation page) who are granted access to your organization’s resources (Partners, vendors, contractors… who you need to collaborate with) and you don’t want them to have an account in your Azure AD.</li><li>Self service password reset: Allow member/group able to reset their password, default only Administrators have that feature.</li><li>RBAC: not thing much new, default comes with over 270++ built-in roles.</li><li>We can swap Storage Account from using Access Key to RBAC as well. And we can grant role/permission on various levels from the specific container, storage account, resource group.</li><li>Custom role defined with AssignableScopes, specify range(Management group, subscription or resource group) that role can be assigned.</li><li>Interpret Access Assignments: additive union permissions of assigned role. Because RBAC have no fucking deny in role, so Contributor role will eat custom role that have some permission like in Contributor role</li><li>Azure Account/User: a person(username, password, multi-factor) or a program (managed identity), basis for authentication.</li><li>Tenant: represent of an org, usually with public domain name. Will be assigned a domain if not specified (example.onmicrosoft.com). It is a fucking dedicated instance of Azure Active Directory, every Azure Account is part of at least one tenant! More than one account can be the owner in a tenant!</li><li>Subscription: An agreement with Microsoft to use Azure services and how you are fucking to pay for that. All Azure resource usage gets billed to the payment method of subscription (Free, PAYG, Enterprise agreements…). Not every tenant needs to have a subscription. Tenants can have more than one subscription.</li><li>Resources: Nothing much to say about this… Account can be given read,update, owner rights to resources.</li><li>Resource Groups: a way to organizing resources in a subscription. A Folder struct, all resources must belong to only one resource group.</li><li>Subscription have option for Resource locks, that prevent you from delete any resource in subscription scope or resource scope or even more, for example specific Function App. It means no one would able to delete if the lock still exists! (Or control people have permission that able to delete). Lock type Read only, you are not able to change setting. So 2 types, remember this.</li><li>We can restrict location access for specific resource or deny unencrypted HTTPS storage accounts as example of Policy. Policy definition is single policy, but Initiative is set of policies.</li><li>Custom policy, you want may want to copy syntax from template first before create custom policy, because they are all ARM templates. And after create custom policy, assign to specific scope like resource group.</li><li>Tags: Is one of important thing to manage resource on any public cloud provider, not only Azure.</li><li>Move Resources: It can be move to another resource group/subscription in same region. In facts it moves only metadata if RG/Sub in same region, that is why we have no fucking downtime during moving xD.</li><li>Manage policy by Powershell, mostly you can do everything like manage resource/policy in Portal via Powershell or CLI.</li><li>Subscription and Management Group: We can create new subscription and put it under a management group</li><li>Remember about SSPR:</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*RL3agUXBn02LOFPd.png" /></figure><ul><li>Remember about this, even the tutor told you not to memory but I recommend to remember for time saving during exam!</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*M2F47YDRHjyJXZ_X.png" /></figure><h3>Domain 2 — Compute</h3><p>Manage VM Sizes:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*ZEw35OzzHm0EB8At.png" /></figure><h4>Virtual Machine (VM) Availability</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/907/1*FLTlb8einCIO4TG9WosOBg.png" /></figure><p>What is <a href="https://learn.microsoft.com/en-us/training/modules/configure-virtual-machine-availability/4-review-update-fault-domains">update domains</a>: An update domain is a group of nodes that are upgraded together during the process of a service upgrade (or roll out). An update domain allows Azure to perform incremental or rolling upgrades across a deployment.</p><p>What is <a href="https://learn.microsoft.com/en-us/training/modules/configure-virtual-machine-availability/4-review-update-fault-domains">fault domains</a>: A fault domain is a group of nodes that represent a physical unit of failure. Think of a fault domain as nodes that belong to the same physical rack. Yes, it is acting as rack!</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/578/0*JSn7OUPBMyRjnfry.png" /></figure><p>Exam Gotchas</p><ul><li>SLA 99.99% -&gt; must pick Zones, not Set</li><li>Can not combine Availability Set and Availability Zones</li><li>Availability Set: <strong>Fault Domain (FD):</strong> different physical rack (power, switch) — default 2, max 3 (some regions support 3)</li><li>Availability Set: <strong>Update Domain (UD):</strong> group VM not going to restart together when Azure patch — max 20</li></ul><p>Resource Reference:</p><ul><li><a href="https://learn.microsoft.com/en-us/azure/virtual-machines/availability">Availability options for Azure Virtual Machines</a></li><li><a href="https://www.microsoft.com/licensing/docs/view/Service-Level-Agreements-SLA-for-Online-Services">SLA for Online Services</a></li></ul><h4>VM Encryption</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/903/1*hkb1lcND2OlHDlGc0APKqA.png" /></figure><p>Picture I took from <a href="https://www.udemy.com/course/70533-azure/">Azure Course</a></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1016/0*0nzAqoUeHU3aA6bE.png" /></figure><p><strong>Verdict:</strong> SSE almost ready for everything. ADE only when compliance is required.</p><p>Resource Reference:</p><ul><li><a href="https://learn.microsoft.com/en-us/azure/virtual-machines/disk-encryption-overview">Overview of managed disk encryption options</a></li><li><a href="https://learn.microsoft.com/en-us/azure/virtual-machines/disk-encryption">Server-side encryption of Azure Disk Storage</a></li><li><a href="https://learn.microsoft.com/en-us/azure/security/fundamentals/azure-disk-encryption-vms-vmss">Azure Disk Encryption for VMs and VMSS</a></li></ul><pre>Note: ADE is scheduled for retirement on September 15, 2028. <br>Microsoft recommends migrating to encryption at host for new VMs.</pre><h4>Deployment slot</h4><p>Azure App Service deployment slots are live, distinct environments for your web apps. They allow you to test changes, warm up instances, and achieve zero-downtime deployments. By managing configurations independently, you can deploy updates to a staging slot, verify them, and instantly swap them into production. (from AI Overview when google search xD).</p><p>Reference: It is same like AWS Beanstalk</p><ul><li>Staging/Production Slot = AWS — Elastic Beanstalk Environments / AWS — Lambda Versions</li><li>Swap Slots = AWS Swap Environment URLs</li><li>Traffic % (Routing) = Weighted Aliases (Lambda) or Weighted Routing (Route 53 / ALB)</li></ul><p>Some note for exam:</p><ul><li>Deployment slot is not available in Free/Basic plan. If question related to staging environment, pick Standard.</li><li>After slot swap, both slot still running, for rolling back, just simple by swap back.</li><li>App Service Plan = compute resource.</li></ul><p>Resource Reference:</p><ul><li><a href="https://learn.microsoft.com/en-us/azure/app-service/deploy-staging-slots">Set up staging environments in Azure App Service</a></li></ul><h4>ARM (Azure Resource Manager) and Bicep</h4><ul><li>ARM Template = Cloudformation Template of AWS</li><li>Bicep still is ARM but better syntax (Azure only)</li><li>Still prefer to use Terraform instead of this fucking shit when it comes to production!</li><li>From Bicep to ARM (build/compile):</li></ul><pre>az bicep build --file main.bicep<br># Output: main.json (ARM template)<br># Directly<br>bicep build main.bicep<br>bicep decompile main.json</pre><ul><li>From ARM to Bicep (decompile):</li></ul><pre>az bicep decompile --file main.json<br># Output: main.bicep</pre><h4>Containers</h4><p>Comparison between 3 services</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/904/1*3oFU9GnDPoVOUft9wTv2ag.png" /></figure><p>Resource Reference:</p><ul><li><a href="https://learn.microsoft.com/en-us/azure/container-instances/container-instances-overview">What is Azure Container Instances?</a></li><li><a href="https://learn.microsoft.com/en-us/azure/container-apps/overview">What is Azure Container Apps?</a></li></ul><p>Gotchas: Remember this fucking shit!</p><ul><li>strong / complete / security isolation / security boundaries = Virtual Machine</li><li>light weight / fast / microservices = Containers</li></ul><h4>Notes for exam:</h4><ul><li>Quota is calculated based on the total number of cores in use both allocated AND deallocated.</li></ul><h3>Domain 2.1 — ARM + Bicep</h3><h4>ARM template basic</h4><p>Whenever we create VM or VNET, there is alway a section called “View automation template” in review + create resource.</p><p>ARM template section:</p><ul><li>$schema</li><li>contentVersion</li><li>variables</li><li>resources</li><li>parameters</li><li>outputs</li></ul><p>Good shit:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*nGnbYq6RM-GdyU1-.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*QJxUHbuMFypOXOe0.png" /></figure><p>Minimum required to create virtual network:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/972/0*IVZ2xCillB2turVI.png" /></figure><p>And yeah, we have <em>dependsOn</em> inside resources section which depend on other specific resource to create resource</p><p>We can create template spec then use it to deploy resources via Portal or cmd (Powershell)</p><p>ARM Templates Capture a Point in Time: This is basic, same like snapshot but without data, just spec/blueprint.</p><h4>Azure Blueprint</h4><p>It is a package governance: ARM templates + Policy + RBAC assignments.</p><p>Update: Azure Blueprints will be retired on January 31, 2027, with a phased retirement beginning July 31, 2026</p><h4>Azure Custom Script Extension</h4><ul><li>Run script on VM <strong>after deploy</strong>, can trigger <strong>anytime</strong> (not just first boot)</li><li>Use case: install software (NGINX, etc.), configure OS</li><li>Works with VMSS → applies to ALL instances</li><li>Can be embedded in ARM template</li></ul><p>vs User Data (cloud-init): only runs at <strong>first boot</strong>, can’t re-trigger</p><h4>Bicep overview</h4><p>We could deploy bicep to Azure via some cmd:</p><ul><li>Deploy to RG:</li></ul><pre>az deployment group create \<br>  --resource-group &lt;name-of-rg&gt; \<br>  --template-file main.bicep \<br>  --parameters param1=value1 param2=value2</pre><h3>Domain 2.2 — Azure App Server</h3><h4>Web App/App Service</h4><p>Nothing much to write, you upload your code, Azure runs your code and helps you to manage the server.</p><p>It also have feature backup so you can restore from backup instead of upload code again for speed faster?</p><p>App Service Networking:</p><ul><li>Control inbound traffic (into app): Access restrictions, private endpoints, service endpoints, app-assigned address</li><li>Control outbound traffic (out from app): VNet integration, Gateway-required Vnet integration….</li><li>Custom domain: point DNS into app, not networking feature.</li></ul><p>We also able to config cert and TLS for app service. They are auto renew at 45 days before expires. Limit: Not support wildcard, available with Basic plan and above.</p><h4>Extra note for VM</h4><ul><li>VM disk stored as? → Page Blob (VHD of VM — Unmanaged Disk)</li></ul><h3>Domain 2.3 — Containers</h3><h4>Azure Container Instances</h4><p>Quick to deploy but has limitations, should be used for testing/demonstration. Cannot scale.</p><h4>ACI container groups</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*1ZUogc_kB1L4dUAu.png" /></figure><h4>Azure container apps</h4><p>Almost same like container instance but have more options like ingress settings, revision and replicas (like deployment slot in Azure App), scale based HTTP Traffic, Keda, manual.</p><h3>Domain 3 — Storage</h3><h4>Azure Storage Account vs AWS S3 — Mental Model</h4><pre>AWS                              Azure<br>───────────────────────────────────────────────────<br>(No Need!)                   →   Storage Account (wrap layer)<br>S3 Bucket                    →   Blob Container<br>S3 Object                    →   Blob<br>S3 Bucket Policy             →   Container Access Policy / RBAC<br>S3 Lifecycle Rules           →   Lifecycle Management<br>S3 Versioning                →   Blob Versioning<br>S3 Storage Classes           →   Access Tiers (Hot/Cool/Archive)<br>S3 Presigned URL             →   SAS Token<br>S3 Static Website            →   Static Website (in Blob)<br>───────────────────────────────────────────────────<br>EFS                          →   Azure Files<br>SQS                          →   Queue Storage<br>DynamoDB (simple use)        →   Table Storage</pre><p>Key Differences:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/909/1*g3YQUtn2TTnupnC1GUV56w.png" /></figure><p>Example:</p><pre>AWS:<br>  - s3://my-images-bucket        (Create)<br>  - s3://my-backup-bucket        (Create)  <br>  - EFS: my-team-share           (Other service)<br><br>Azure:<br>  - mystorageaccount<br>      ├── container: images      (Like S3 bucket)<br>      ├── container: backups     (Like S3 bucket)<br>      └── fileshare: team-share  (Like EFS)</pre><h4>Immutable vs Mutable Settings</h4><p>Can not change after created:</p><ul><li>Storage Account name</li><li>Location/Region</li><li>Performance tier (Standard ←-&gt; Premium)</li><li>Hierarchical namespace (One-way upgradeable: Cannot be disabled once enabled, but can be migrated from Flat namespace via Azure tools)</li></ul><p>Can be change after created:</p><ul><li>Redundancy (LRS/ZRS/GRS) — but limit</li><li>Access tier (Hot/Cool)</li><li>Network/Firewall rules</li><li>Data protection (soft delete, versioning)</li><li>Encryption settings</li></ul><p>We need to pick Name, Region, Performance tier right for the first damn time!</p><h4>Replication Types</h4><p>In single region:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/904/1*u3b5QehKZ0g-3pUOlf8pQA.png" /></figure><p>Cross-region:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/906/1*wqExwiScE8eQ53R7DNdxYA.png" /></figure><p>Pick replication by keyword:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/909/1*JlYnVW3b0wPgskyJH_YVeQ.png" /></figure><p>Resource Reference:</p><ul><li><a href="https://learn.microsoft.com/en-us/azure/storage/common/storage-redundancy">Azure Storage redundancy</a></li><li><a href="https://learn.microsoft.com/en-us/azure/storage/common/redundancy-migration">Change how a storage account is replicated</a></li><li><a href="https://learn.microsoft.com/en-us/azure/storage/blobs/access-tiers-overview">Hot, cool, cold, and archive access tiers for blob data</a></li><li><a href="https://learn.microsoft.com/en-us/azure/storage/common/storage-sas-overview">Grant limited access to data with shared access signatures (SAS)</a></li></ul><h4>AzCopy</h4><p>Standalone CLI tool to copy data to/from Azure Storage. aws s3 cp / aws s3 sync equivalent.</p><p>Auth: azcopy login (Azure AD, RBAC) or SAS Token embedded in URL.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/905/1*oQWdruXebgSwx22ktJKNZw.png" /></figure><p>Exam Gotchas:</p><ul><li>azcopy make only creates containers/file shares, NOT the Storage Account itself</li><li>azcopy sync does <strong>NOT</strong> delete by default — need --delete-destination=true</li><li>Cross-account copy is <strong>server-side</strong> — no local bandwidth used</li><li>Standalone binary, <strong>no Azure CLI</strong> dependency</li></ul><p>Resource Reference:</p><ul><li><a href="https://learn.microsoft.com/en-us/azure/storage/common/storage-use-azcopy-v10">Get started with AzCopy</a></li></ul><h4>Exam Tricks for Storage Features (non-replication)</h4><blockquote><em>For replication keywords (LRS/ZRS/GRS/RA-GRS/GZRS/RA-GZRS), see the </em><strong><em>Replication Types</em></strong><em> section above.</em></blockquote><figure><img alt="" src="https://cdn-images-1.medium.com/max/909/1*xUkldJfSQmLo1--7Go6UOg.png" /></figure><h4>Hierarchical Namespace</h4><p><strong>Hierarchical Namespace</strong> converts Azure Blob Storage into <strong>Azure Data Lake Storage Gen2 (ADLS Gen2)</strong>:</p><ul><li>Directories are <strong>real</strong> (rename/delete folder without touching individual files)</li><li>Supports <strong>POSIX ACL</strong> permissions at file/folder level</li><li>Designed for <strong>Big Data / Analytics</strong> workloads (Databricks, Synapse, HDInsight)</li><li><strong>Cannot be disabled</strong> once enabled</li><li>Object Replication requires Hierarchical Namespace to be <strong>Disabled</strong></li></ul><h4>Enhanced after fail first exam</h4><ul><li>Not related to exam but some region required to store data in their region, for example, when you are working with EU clients, you need to create storage account in EU region I guess.</li><li>Redundancy type: Not all region have option for GZRS.</li><li>Storage account advanced setting: Required secure transfer for REST API, it means you can only use HTTPS for interacting. For authentication we have 2 option storage account key access(enable by default) and microsoft entra authorization.</li><li>Disabling “Allow storage account key access” will block both Shared Key Auth and Service/Account SAS, but User Delegation SAS (via Entra ID) will still work</li><li>Storage account access tier: So for more cheaper from store the data(Cool/Cold), the more expensive for operation(read/write) for example. And you can have a mix of access tier file, some files are hot, some files are cool/cold/archive.</li><li>Storage account networking: Public access from all networks are not worth to mention. But public access from selected virtual network and IP address are (You have an option to add IP Address for public access in this case like test from your home?). And finally disable public access and use private access. Network routing, lets use Microsoft network routing by default. (Travel over a private network or Azure backbone network I guess).</li><li>Storage account data protection: soft delete can be enabled for blobs, containers, file shares as an option in this tab. And enable versioning for blobs, the more money you are going to pays xD</li><li>Storage account encryption: This section is easy to understand xD</li><li>We can generate SAS with assigned access policy. (Control how we access on file share).</li><li>If enable (Entra ID/Azure AD), we can give specific user to have access read/write on specific containers.</li><li>Object replication is copied “asynchronously” from a source storage account to a destination account.</li><li>For quick example with Azcopy, create a folder as destination, generate SAS token and get SAS URL, then:</li></ul><pre>azcopy copy &quot;C:\my\file.txt&quot; &quot;https://blob-sas-url-here&quot;</pre><ul><li>And we have talking a lot about containers, one relevant to this fucking exam called file shares, we can understand it as fucking NFS (we have a tool to keep file sync between on-premises and Azure called Azure File Sync). Also Azure Files can be backed up as Snapshot.</li><li>Which backup type does Azure Blob backup use? <a href="https://learn.microsoft.com/en-us/azure/backup/blob-backup-overview?tabs=operational-backup">operational backup: Continuous backups</a></li><li>Block Blob: Save common file (image, video, document..). Can be overwrite data.</li><li>Append Blob: Append without modify/delete old data, useful for log data, monitoring logs, audit trails.</li><li>Page Blob: Random read/write, common use case: Virtual Disk (VHD) for VM, database files.</li><li>This is interesting: <strong>..core.windows.net//…..</strong></li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*XqmD0zP4AjUFxJMp.png" /></figure><ul><li>Remember about what required to create SAS:</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*NvzOubhKdQ7jMdBl.png" /></figure><ul><li>Recommend practice for the exam:</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*Kxn3gZ3-dhZdEkIZ.png" /></figure><ul><li>Condition for Object Replication works</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/909/1*2sj29JFIB4n-SCVtuyiDIQ.png" /></figure><h3>Domain 4 — Networking</h3><p>Resource Reference:</p><ul><li><a href="https://learn.microsoft.com/en-us/azure/architecture/aws-professional/networking">AWS to Azure services comparison — Networking</a></li><li><a href="https://learn.microsoft.com/en-us/azure/virtual-network/virtual-networks-overview">Azure Virtual Network (VNet) Overview</a></li></ul><h4>Quick Mapping from AWS VPC xD</h4><pre>AWS                              Azure<br>───────────────────────────────────────────────────<br>VPC                          →   VNet<br>Subnet                       →   Subnet<br>Security Group               →   NSG (Network Security Group)<br>Elastic IP                   →   Public IP (Static)<br>ENI                          →   NIC (Network Interface Card)<br>VPC Peering                  →   VNet Peering<br>NAT Gateway                  →   NAT Gateway<br>Route Table                  →   Route Table</pre><p>But there is still some little different. And we will go one by one</p><p>Subnet and Availability Zones (AZs)</p><ul><li>AWS: each Subnet locked into 1 AZ. To achieve HA, we need to create multiple subnet in different AZ.</li><li>Azure: Subnet expanded to all AZ in single region. We can place VM in AZ 1, AZ 2, AZ 3 into same subnet.</li></ul><p>NSG (Network Security Group) vs SG (Security Group) / NACL</p><ul><li>AWS: Use SG (applied at ENI/Instance) and Network ACL (NACL) (applied at Subnet)</li><li>Azure: Use only NSG. NSG can be applied both to NIC or Subnet. This is core for our exam.</li><li>NSG is stateful, NACL is stateless</li></ul><p><strong>Stateful vs Stateless — what does it mean?</strong></p><ul><li><strong>Stateful (NSG, AWS SG):</strong> firewall <em>remembers</em> connections. When outbound traffic is allowed, the return traffic is <strong>automatically allowed</strong> — the firewall tracks the session in a connection table.</li><li><strong>Stateless (AWS NACL):</strong> firewall has <em>no memory</em>. Every packet is evaluated independently → you must define rules for <strong>both directions</strong> (inbound AND outbound).</li></ul><p>Quick example — VM calling GitHub API (HTTPS port 443):</p><pre>VM (random port 54321) --&gt; [request] --&gt; GitHub (port 443)<br>VM (random port 54321) &lt;-- [response] &lt;-- GitHub (port 443)</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/901/1*FJ60eTAMOo145GuSO8HDyw.png" /></figure><p>That’s why Azure only needs <strong>one NSG</strong> instead of AWS’s two-layer SG + NACL model — simpler, no need to open wide ephemeral port ranges, less misconfiguration risk.</p><p>Resource Reference:</p><ul><li><a href="https://learn.microsoft.com/en-us/azure/virtual-network/network-security-groups-overview">Azure network security groups overview</a></li></ul><h4>VNet Peering &amp; Gateway</h4><pre>AWS                              Azure<br>───────────────────────────────────────────────────<br>VPC Peering (same region)    →   VNet Peering<br>VPC Peering (cross-region)   →   Global VNet Peering<br>VPN Gateway                  →   Virtual Network Gateway<br>Transit Gateway              →   Virtual WAN / VNet Gateway<br>Site-to-Site VPN             →   VPN Gateway Connection<br>Direct Connect               →   ExpressRoute</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/900/1*Owu82BHtTEv2z84Mj3x2Sw.png" /></figure><p>Resource Reference:</p><ul><li><a href="https://learn.microsoft.com/en-us/azure/virtual-network/virtual-network-peering-overview">Azure Virtual Network peering</a></li><li><a href="https://learn.microsoft.com/en-us/azure/vpn-gateway/vpn-gateway-about-vpngateways">About Azure VPN Gateway</a></li><li><a href="https://learn.microsoft.com/en-us/azure/expressroute/expressroute-introduction">Azure ExpressRoute overview</a></li></ul><h4>Azure DNS</h4><pre>AWS                              Azure<br>───────────────────────────────────────────────────<br>Route53                      →   Azure DNS<br>Route53 Hosted Zone (public) →   Public DNS Zone<br>Route53 Private Hosted Zone  →   Private DNS Zone<br>Route53 Domain Registration  →   App Service Domain<br>Route53 Health Checks        →   Traffic Manager<br>Route53 Routing Policies     →   Traffic Manager</pre><p>Azure separates health checks + traffic routings via Traffic Manager, not built-in like Route53.</p><h4>Azure Load Balancing</h4><pre>AWS                              Azure<br>───────────────────────────────────────────────────<br>NLB (Layer 4)                →   Azure Load Balancer<br>ALB (Layer 7)                →   Application Gateway<br>CloudFront + ALB (global)    →   Azure Front Door<br>Route53 routing policies     →   Traffic Manager (DNS-based)</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/902/1*lg9AfuABQX3IpxPVIEHPfQ.png" /></figure><p>Resource Reference:</p><ul><li><a href="https://learn.microsoft.com/en-us/azure/architecture/guide/technology-choices/load-balancing-overview">Load-balancing options</a> — Official Microsoft comparison</li></ul><h4>Azure Network Watcher (Debug tools)</h4><pre>AWS                              Azure<br>───────────────────────────────────────────────────<br>VPC Flow Logs                →   NSG Flow Logs<br>VPC Reachability Analyzer    →   IP Flow Verify / Connection Troubleshoot<br>VPC Traffic Mirroring        →   Packet Capture</pre><p>Mostly we can checking connection between Azure resource together. And it does costs money btw.</p><p><strong>Verdict:</strong> Almost 1:1 with AWS VPC/Route53/ELB, just a fucking name different.</p><p>Resource Reference:</p><ul><li><a href="https://learn.microsoft.com/en-us/azure/network-watcher/network-watcher-overview">Azure Network Watcher overview</a></li></ul><h4>Hub and Spoke</h4><p>Reference Resource: <a href="https://learn.microsoft.com/en-us/training/modules/configure-virtual-networks/9-simulation-create-networks">Exercise: Create and configure virtual networks</a></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/846/0*8GDZs86FPa5e8pu_.png" /></figure><p>Flow request (write by Claude):</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*x7vHnXe1_vzTfFdK.png" /></figure><p>Explain by Claude (I don’t think I can explain it better LOL)</p><p>What is hub-and-spoke used for?</p><pre>Hub-and-spoke is a network architecture model that helps organize multiple <br>VNets in a clean and cost-effective way. Instead of having each VNet <br>provision its own shared services such as firewall, gateway, or DNS <br>(which is both expensive and hard to manage), you consolidate them into a <br>central VNet called the hub. The application VNets (called spokes) only <br>hold their own workloads and &quot;borrow&quot; the shared services from the hub. <br>In this architecture, hub-vnet (10.0.0.0/16) acts as the hub hosting the <br>shared Azure Firewall, while app-vnet (10.1.0.0/16) is the spoke containing <br>the application (the frontend and backend VMs). The main benefits are <br>centralized security control, lower cost by avoiding duplication of <br>expensive services, and easy scalability — adding a new spoke simply <br>requires peering it to the hub.</pre><p>Why do we need Virtual Network Peering?</p><pre>By default, two different VNets are fully isolated and cannot communicate <br>with each other, even when they&#39;re in the same region or subscription. <br>VNet Peering is the mechanism that connects two VNets so they can route <br>traffic to each other over private IPs, traveling across Microsoft&#39;s <br>backbone infrastructure instead of going out to the internet. <br>In a hub-and-spoke model, peering is mandatory: without peering between<br> the hub and the spoke, the spoke cannot send its traffic through the <br>Azure Firewall located in the hub. In other words, peering is what turns <br>separate, isolated VNets into a unified hub-and-spoke architecture.</pre><h4>Exam Gotchas</h4><ul><li>ExpressRoute <strong>does not go via internet</strong> → question “secure private connection” = ExpressRoute.</li><li>VPN Policy-based <strong>not support</strong> Point-to-Site.</li><li><strong>Active-Active</strong> VPN Gateway = 2 tunnel, high availability.</li><li>ExpressRoute <strong>Global Reach:</strong> connect 2 on-prem sites via Azure backbone (no need internet).</li><li>Question “consistent latency, high bandwidth” = ExpressRoute.</li><li>Question “quick setup, cost-effective” = VPN Gateway.</li><li>Specific subnets allowed into PAAS (Ex: Azure SQL Database) = <strong>Service Endpoint</strong></li><li>Private IP for service in VNet / Disable public access = <strong>Private Endpoint</strong></li><li>Filter IP/Port in VNet/Subnet = <strong>NSG</strong></li><li>Again, <strong>NSG</strong> = firewall rules (allow/deny by IP, port, protocol). <strong>ASG</strong> (Application Security Group) = a logical group tag for NICs/VMs, so rules can reference a group by name instead of listing every IP.</li><li><strong>DMZ</strong> = buffer / barrier between external and internal.</li><li>A virtual network can have only one VPN gateway.</li><li>Spoke uses the hub’s shared VPN/ExpressRoute gateway to return to on-premises = Gateway Transit.</li><li>You can assign public IP to LB/NIC</li></ul><h4>Enhanced after fail first exam</h4><p>Virtual Network Gateway or VPN gateway, to connect between 2 Vnet using VPN gateway:</p><pre>1. add new subnet for vpn gateway in each VNET<br>2. create virtual network gateway with subnet you just created. Create/use <br>exists First and Second Public IP for it (second IP for HA i guess?)<br>3. create virtual network gateway again for second vnet<br>4. adding network gateway connection between 2 vpn gateway we just created, <br>remember to change Autonomous system number (ASN) if not we will fail!<br>5. Create Connection between 2 vpn gateway, connection type: VNet-to-VNet, <br>remember to choice correction for 2 of your VNet<br>6. Review at connections panel.</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*-DSX50oeL028RKPU.png" /></figure><ul><li>We are able to auto register domain for VM when linking the VNet to the Private DNS Zone with the “Enable Auto Registration” option checked (not an option during VM creation).</li><li>We can group multiple rule to Application Security Group(ASG), and allow traffic to ASG is much better than hardcode IP in NSG. For ASG we have to create new ASG as a resource. And go to VM → Networking → Config the ASG → Link VM to specific ASG, 1 VM can be attach more than 1 ASG. After that we can create new Inbound Security Rule in NSG with destination is ASG. Picture for example:</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*2RuyBgooWZKGIgBG.png" /></figure><p>Again ASG really is just an abstraction of the destination instead of specifying a list of IP addresses.</p><ul><li>Connection troubleshooter: Enter via VM → Help → Connection Troubleshoot, select source which is current VM and destination which is VM we can’t connect from current VM, we are able to use TCP/ICMP for probe settings. When we click run diagnostic tests, it deployed an agent to the VM and take couples minute. Example:</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*_qWDlg5pAgxudMGZ.png" /></figure><ul><li>Standard LB supports: Zonal (in one zone) + Zone-redundant (across zones)</li><li>Azure Front Door = Global LB, SSL offloading, CDN. So for SSL offloading, we can call TLS Termination, it means Front Door/LB will handle encrypt/decrypt HTTPS for backend server, so backend server we only need to use HTTP, no need double encrypt while LB/FrontDoor already help you!</li><li>Traffic Manager = DNS-based traffic redirection. Like you come from USA it will redirect you to USA region, you come from Vietnam, it will redirect you to Southeast Asia region</li><li>Holy shiet. Azure provide some good shit. For example <a href="https://github.com/Azure/azure-quickstart-templates/tree/master/quickstarts/microsoft.network/load-balancer-standard-create">this link</a> have a button that “Deploy to Azure” and after we logged into portal of azure:</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*uPTmBpdzJle5n3f-.png" /></figure><ul><li>Monitor -&gt; Logs: you can query for diagnostics issue and there are some template for those query right in portal UI</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*8yqxW4SNT68--a3A.png" /></figure><ul><li>Capture actual packets, TCP handshake, payload details → <a href="https://learn.microsoft.com/en-us/azure/network-watcher/packet-capture-overview">Packet capture of Network Watcher</a></li><li>Inbound internet → private IP/port, <a href="https://learn.microsoft.com/en-us/azure/firewall/tutorial-firewall-dnat-policy">always use DNAT</a> rule in Azure Firewall, bro!</li><li>Private endpoint alway create record when Private DNS zone linked, no need tick option “auto-registration” like in VM.</li><li>VNet peering not auto share DNS zone link. Hub link zone != spoke link zone, <a href="https://learn.microsoft.com/en-us/azure/dns/private-dns-virtual-network-links">need to link VNet seprated</a></li><li>Subnet delegation = subnet used for only 1 service, <a href="https://learn.microsoft.com/en-us/azure/virtual-network/subnet-delegation-overview">not share with VM or other resource</a>.</li><li>Allow gateway transit (VNet with Gateway) + Use remote gateways (VNet have no gateway) = VNet spoke useable VPN/ExpressRoute of Hub, no need any extra configuration or UDR.</li><li>Route/Advertise 0.0.0.0/0 via ExpressRoute, all traffic (internet + Azure Paas public endpoints) go through on-premises. <a href="https://learn.microsoft.com/en-us/azure/expressroute/expressroute-faqs">VM totally lost direct internet access</a></li></ul><h3>Domain 5 — Monitor &amp; Backup</h3><h4>Quick Mapping from AWS for Backup xD</h4><pre>AWS                              Azure<br>───────────────────────────────────────────────────<br>AWS Backup                   →   Azure Backup<br>Backup Vault                 →   Recovery Services Vault<br>AWS DRS / CloudEndure        →   Azure Site Recovery (ASR)<br>EBS Snapshots                →   VM Backup (disk snapshots)</pre><h4>Backup Services</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/902/1*Y02x-_As2MTsierNtKWbyw.png" /></figure><p><strong>Verdict:</strong> Azure Backup = AWS Backup, ASR = DRS (AWS Elastic Disaster Recovery).</p><p>Resource Reference:</p><ul><li><a href="https://learn.microsoft.com/en-us/azure/backup/backup-overview">What is Azure Backup?</a></li><li><a href="https://learn.microsoft.com/en-us/azure/site-recovery/site-recovery-overview">About Azure Site Recovery</a></li></ul><h4>Azure Backup</h4><p>So there is something need to be understand first:</p><ul><li>Azure Backup = Service, like EC2 xD</li><li>Vault = Location where contain and manage backup data + policies</li></ul><p>There are 2 type of vault in Azure backup:</p><ul><li>Recovery Services Vault: Older, used for Azure VM, SQL in VM, Azure Files, on-prem (MARS agent)</li><li>Backup Vault: Newer, used for Azure disk, Azure Blobs, PostgreSQL,AKS</li></ul><p>Basic flow generated by AI xD:</p><pre>Azure Backup service<br>└── Recovery Services Vault  ← We create this!<br>    ├── Backup Policy (schedule, retention)<br>    ├── Backup Items (VM1, VM2, SQL DB...)<br>    └── Backup Data (actual data stored here)</pre><h4>Monitoring</h4><p>An Action Group defines <strong>what actions to take</strong> when an alert is triggered (e.g., send email, call webhook, run automation). For example: an Alert Rule monitors CPU usage &gt; 90% on a specific VM → when triggered, it invokes an Action Group that sends an email notification.</p><h4>Quick Mapping from AWS CloudWatch</h4><pre>AWS                              Azure<br>───────────────────────────────────────────────────<br>CloudWatch                   →   Azure Monitor<br>CloudWatch Metrics           →   Metrics<br>CloudWatch Logs              →   Log Analytics Workspace<br>CloudWatch Logs Insights     →   Kusto Query Language (KQL)<br>CloudWatch Alarms            →   Alerts<br>CloudWatch Agent             →   Azure Monitor Agent<br>X-Ray                        →   Application Insights</pre><p>Resource Reference:</p><ul><li><a href="https://learn.microsoft.com/en-us/azure/azure-monitor/overview">Azure Monitor overview</a></li><li><a href="https://learn.microsoft.com/en-us/kusto/query/">Kusto Query Language (KQL) overview</a></li></ul><h4>Enhanced after fail first exam</h4><ul><li>Monitor needs an agent installed on Windows or Linux VMs to gather data and send to Azure to display.</li><li>We can enable diagnostic for specific resource or resource group.</li><li>For VM, we can turn on diagnostics in order to get the various operating system level metrics outside of the virtual machine. And the logs we get from “Insight” (need to enable it) which will collect the data and send to Azure for view, which is basically the agent will read/extract data from “Event Viewer” (VM itself) and send to monitor for centralized monitoring that we can run queries on.</li><li>We can create dashboard in Monitor -&gt; Metrics with specific metrics we want to visualization.</li><li>For query, we need to select scope resource that we gonna do query log in there (Monitor -&gt; Logs)</li><li>For Azure Backup, we need to create a policy and inside that fucking policy we have backup schedule, instant restore (retain instant recovery for snapshots for x days), retention range (default 30 days).</li><li>For manually, we need to create recovery services vault (is a storage account that has certain rules on it because it is specially for backup).</li><li>File recovery from backup, choose restore point + download script to browse and recover files (it basically mount a restored disk for you in window for example) then in step 3 you have to click on button unmount disk to remove backup disk, hmm this is pretty good feature without remember syntax.</li><li>For full restore of VM, we use option restore VM which is next to “File Recovery” with restore type like create new virtual machine with some information need for restore VM</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*qJHpiDzxkOS1g8CU.png" /></figure><ul><li>In Backup Jobs, we can find information related about that backup…. Also you can query logs for backup jobs…</li><li>Azure backup have soft delete enable by default, free for 14 days</li><li>Azure Disaster Recovery, we can start VM replication for another Azure region in VM → Disaster Recovery and yeah data transfer will be charge ( will required brand new RG,VNet..)</li><li>We also have test failover for site recovery job which will help us to validate it, what a great feature if we do this on-premises, will take shit load of time for automation.</li><li>Identity-based access = authenticate with Kerberos/AD, like traditional file server. A fucking table to remember:</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/902/1*y6H2z8ioI2WA1VI_MM5PnA.png" /></figure><h3>Exam Tip/Tricks</h3><p>Summary of common pitfalls, this is not study guide. Only pattern to drop wrong answer xD.</p><p>Never, never, never ever skip free practice assessment available, if somethings are important, repeat 3 times!</p><h4>1. “X replaces NSG/Firewall”</h4><p>This is always wrong. Bastion, Private Endpoint, VPN Gateway, Front Door, App Gateway… do <strong>not</strong> replace NSG. NSG is a layer filter, always applied together.</p><h4>2. VNet Peering</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/902/1*rdQFO805EMIF6SjamQc1_w.png" /></figure><h4>3. Storage Replication</h4><p>See <strong>Domain 3 — Replication Types</strong> for the full picker table.</p><p><strong>Pitfalls:</strong> “highest availability within a region” -&gt; ZRS (not GRS). GRS is <strong>cross-region</strong>. G stands for GEO</p><h4>4. Backup vs Site Recovery</h4><ul><li><strong>Azure Backup</strong> = data recovery (file, VM disk, DB) — RPO calculate by hour/day</li><li><strong>Azure Site Recovery (ASR)</strong> = disaster recovery (replicate VM into another region) — RPO/RTO low</li></ul><p>Question “recover deleted file” -&gt; Azure Backup. “Region down, switch to another region” -&gt; ASR.</p><h4>5. Recovery Services vault vs Backup vault (naming pitfall )</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/906/1*jIWrk93Y_Vr8hvHjbOjPaA.png" /></figure><p>Keyword <strong>“files, folders, system state”</strong> -&gt; always are <strong>Recovery Services vault</strong> (Because attach with MARS agent). Don’t be wrong with Azure Backup vault</p><p>Resource Reference:</p><ul><li><a href="https://learn.microsoft.com/en-us/azure/backup/backup-vault-overview">Overview of Backup vaults</a></li><li><a href="https://learn.microsoft.com/en-us/azure/backup/backup-azure-recovery-services-vault-overview">Recovery Services vaults overview</a></li></ul><h4>6. RBAC vs Azure Policy</h4><ul><li><strong>RBAC</strong> = who <strong>can</strong> do what (access control)</li><li><strong>Policy</strong> = resource allows to exist or not (compliance, deny tag missing, deny SKU …)</li></ul><p>If questions are “enforce tag”, “block region” -&gt; Policy. Otherwise “grant read access” -&gt; RBAC.</p><h4>7. Four built-in roles</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/903/1*2kUTf4YC8NttietQvNayEQ.png" /></figure><p><strong>Common pitfalls:</strong> exam question full manage resources <strong>but no</strong> assign roles -&gt; answer is <strong>Contributor</strong>, NOT User Access Administrator (UAA). UAA is reversed role. Only assign roles, not going to have permissions to touch resources.</p><p><strong>Trick:</strong></p><ul><li>Owner = Contributor + UAA</li><li>Contributor = Everything but not able to give permissions</li><li>UAA = Only able to give permissions</li></ul><h4>8. RBAC scope — 4 levels</h4><pre>Management Group  (top, contains multiple subscription)<br>    ↓ inherit<br>Subscription<br>    ↓ inherit<br>Resource Group<br>    ↓ inherit<br>Resource          ← VNet, VM, Storage, Key Vault... individual resource.</pre><p><strong>Rules</strong>: Anything that contains Resource ID can be scope. Role assigned at higher scope, inherit to lower scope.</p><p><strong>Pitfalls</strong>: List specific like Virtual Network, Storage Account make distractor, it still is valid scope (Level Resource).</p><p><strong>Least privilege pattern</strong>: Instead assign Contributor at Subscription, assign role lower scope, like Network Contributor at that damn Virtual Network.</p><h4>9. Locks</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/904/1*Jm2ZO9vSxHQun1D3Bdga7Q.png" /></figure><p>Gotcha: ReadOnly lock can break some operations though it was read.</p><p>Example: list keys of Storage Account actually is post request.</p><p>Resource Reference:</p><ul><li><a href="https://learn.microsoft.com/en-us/azure/azure-resource-manager/management/lock-resources?tabs=json#considerations-before-applying-your-locks">Considerations before applying your locks</a></li></ul><h4>10. Cost Management Pitfalls</h4><ul><li><strong>Reserved Instance</strong> = commit 1/3 year, reduced up to 72%</li><li><strong>Spot VM</strong> = cheap but can be evicted</li><li><strong>Hybrid Benefit</strong> = bring license Windows/SQL on-prem to Azure</li></ul><p>If question predictable workload, lowest cost -&gt; Reserved Instance. Batch job, can be restart -&gt; Spot.</p><p>Resource Reference:</p><ul><li><a href="https://learn.microsoft.com/en-us/azure/cost-management-billing/reservations/save-compute-costs-reservations">What are Azure Reservations?</a></li><li><a href="https://learn.microsoft.com/en-us/azure/virtual-machines/spot-vms">Use Azure Spot Virtual Machines</a></li></ul><h4>11. An overly absolute answer → 90% chance of being a distractor</h4><p>Red flags in answer xD:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/903/1*Ynqx-BlUScMBsdv548ViJA.png" /></figure><h4>12. Network Watcher — choose the right tool</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/905/1*O4eM9sWEBBFZRT0ch4AYrQ.png" /></figure><p>Need to remember:</p><ul><li>If question between two VMs + network -&gt; reflex Network Watcher.</li><li>If question continuous/monitor over time -&gt; Connection Monitor.</li><li>If question one-time test -&gt; Connection Troubleshoot.</li></ul><h4>13. Azure Monitor — Metrics vs Logs vs Insights</h4><p>Pitfalls: exam ask diagnostic between 2 specific VM -&gt; <strong>Hell No</strong>, is NOT Insights. Insights is overview, not deep-dive.</p><p><strong>Differentiation Insights vs Log Analytics when exam have analyze:</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/899/1*sYqkDBhUrmYQJn_4gYWnHg.png" /></figure><h4>14. Private DNS + Private Endpoint + Hybrid resolution</h4><p>Questions around can VM/on-prem resolve name of private or not?. Need to remember scope of each components.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/900/1*ItYtFF6wY27BBvE_JKEb_Q.png" /></figure><p>Pitfall 1: Peering not share DNS</p><ul><li>VNet-A have Private Endpoint + zone already link -&gt; VM-A resolve OK. VM-B in VNet-B (peered) <strong>fail</strong> because zone haven’t link to VNet-B.</li><li>Fix: Link the Private DNS zone to both VNets. (Note: A Private DNS zone is global and can be linked to up to 1,000 VNets across different regions/subscriptions).</li></ul><p>Pitfall 2: Hybrid DNS need Private Resolver</p><ul><li>On-prem need to resolve mystorage.privatelink.blob... →</li></ul><pre># Generate with AI xD<br>On-prem DNS  ──&gt; conditional fwd --&gt;   DNS Private Resolver<br>                                   (inbound endpoint, region X)<br>                                          ↓<br>                                   linked Private DNS Zones<br>                                          ↓<br>                                   return IP private endpoint</pre><ul><li><strong>Inbound endpoint</strong> = on-prem DNS forward query into Azure (resolve privatelink zones)</li><li><strong>Outbound endpoint</strong> = Azure VM resolve DNS on-prem via conditional forwarding rules.</li><li>HA cross-region -&gt; deploy resolver in 2 region</li></ul><p>Common Scenario:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/905/1*r0p4IK2intUwRBvPl2Z4EA.png" /></figure><p>Resource Reference:</p><ul><li><a href="https://learn.microsoft.com/en-us/azure/dns/dns-private-resolver-overview">Azure DNS Private Resolver overview</a></li><li><a href="https://learn.microsoft.com/en-us/azure/dns/private-dns-overview">What is an Azure Private DNS zone?</a></li></ul><h4>15. Azure Files vs Azure Blob Storage</h4><p>Azure files when: file share / shared files / mount/ SMB (Access via mount, smb…).</p><p>Azure blob storage when: object / blob / images / unstructured data (Access via Rest API/URl).</p><h3>Taking the Exam</h3><p>From my experience preparing for cloud certs, AZ-104 feels harder than AWS SAA / GCP ACE at the associate level, broader scope and tons of edge cases xD</p><p>Even though you can access <a href="https://learn.microsoft.com/">https://learn.microsoft.com/</a> but should use only for verifying information like plans, very slow and time consuming, like watch document during CKAD/CKS/CKA exam!</p><p>Schedule <a href="https://learn.microsoft.com/en-us/credentials/certifications/azure-administrator/?practice-assessment-type=certification">exam here</a></p><p>Random useful comment from <a href="https://www.youtube.com/watch?v=3mxjUtHHVKI">Youtube</a> I think, 9 months ago, so it maybe like September-2025 this comment appears.</p><pre>Just Passed Today. Most of question were around File/Blob Storage Access <br>Tier and LifeCycle. RBAC Role assingment, AzCopy, Recovery service vault, <br>LoadBalancer,VNET . Thanks TechBlackBoard for providing compiled question <br>and pattern. 15-20% in actual exams questions were almost same from the <br>video. There will be 50 questions which include 40 normal question and 2 <br>not skipped question set of 3 and 3 which we cant review at end and 1 case <br>study having 4 questions. Total Time is 100 minutes. Now these days microsoft <br>allows the open notebook so that we can use the micosoft.learn portal <br>in exam and that is too helpful. you can review the price tier and static <br>content for various azure services.</pre><p>Update 1st try in June-04–2026: failed with score 622/1000. It is just hard because I haven’t prepared seriously enough, still need more practice. The domain Compute and Storage I felt pretty confident, but domain Entra ID is really really bad for me =.= But I didn’t watch serious <strong>AZ-104 Microsoft Azure Administrator — Complete Exam Prep</strong>, I skip a lot and spam only on practice test!</p><p>Start learning again in morning saturday June-06–2026! Plan for retake in June-14–2026!</p><p>Update 2nd (June-14–2026): Fuckkkkkkk. 672/1000!</p><p>Update 3rd (June-28–2026): 800/1000 LOL.</p><h3>Secret Weapons</h3><p>Honestly, if I keep studying the same way I always have, I’m going to fail this exam for the third fucking time. The real problem is I don’t have hands-on Azure experience — I work with Kubernetes and on-premises stuff day to day, so a lot of this is purely theoretical for me. Watching videos and taking notes clearly isn’t enough when the exam throws scenario-based questions at you that only make sense if you’ve actually touched the console. So this time I need some secret weapons: very high quality practice questions that actually reflect the real exam, and a way to build intuition for Azure-specific behavior without spending months running real workloads. Let’s see if this approach finally works xD</p><p>Good shit for exam prep: <a href="https://learn.microsoft.com/en-us/shows/exam-readiness-zone/preparing-for-az-104-manage-azure-identities-and-governance-1-of-5">very very quality</a></p><p>For example we have this fucking question (taken from secret weapon xD)</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*dYdzwyIvn1N_iWeo.png" /></figure><p>We can’t remember everything for fucking sure. Let’s me told you what I reliazed after practice for exam. It is how fucking we use <a href="https://learn.microsoft.com/">https://learn.microsoft.com/</a> during exam.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*R4VEgJE5rUE8PRwt.png" /></figure><p>Enter first result found. Read as fast as you can, you will see Consumption plan here with key word “Pay for compute resources only when your functions are running (pay-as-you-go) with automatic scale on Windows.”</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/798/0*OYxwYfDtfRIYLuFO.png" /></figure><p>It is possible but remember use it fast, like under 120 seconds if you need to use document, because we have only fucking 100 minutes to resolve more than 50 questions!!!</p><p>If I recall correctly, there is no question related to AKS service during exam, so you could save a little time to prepare.</p><h4>Tutorial Dojo</h4><p>Pretty good to familiar with exam: I definitely <a href="https://portal.tutorialsdojo.com/product/az-104-microsoft-azure-administrator-practice-exams/">recommend it</a></p><p>Why? You will be overwhelmed with how hard AZ-104 is, you need to familiar with practice test in TD first before taking real exam.</p><h4>Udemy practice tests</h4><p><a href="https://www.udemy.com/course/prepare-for-microsoft-azure-administrator-az-104-exam">Prepare for Microsoft Azure Administrator AZ-104 exam</a> realistically, this practice set will help you cover about 30% of the exam readiness journey.</p><p>Understand why you choose the solution is better a lot than memorization, this exam is not like OTCA,Backstage, Cilium, Kyverno….</p><h3>Conclusion</h3><ul><li>Hard AF for people like me, who never working in single day with Azure Cloud!</li><li>Need to learn skill how to use <a href="https://learn.microsoft.com/">https://learn.microsoft.com/</a>, we can not remember memory all stuff, we need to read the fucking document. But reading document during exam is fucking hard, that is why you need to practice it hard!</li><li>Trust me, if you don’t work in Azure but still want to try exam. Learn for it for non-stop 30 days before taking exam!</li><li>This exam is fucking ridiculous, even you are able to read document, but time is fucking limited!!! holy shiet!!!</li><li>Don’t use microsoft during exam, it is fucking waste time that will lead to pressure! I was thought if I have 20 minutes to think, I would not fail :((</li><li>Questions are tricky, please understand correctly…. I failed a lot during the reading phase which led to choosing wrong answers. Fucking tricky!!!!</li></ul><p>I would spam this word in every article related to certifications: Passing an exam, getting certifications is not an achievement for me, no fucking salary increase, I’m not proud if I have achieved certifications because I’m fucking lowkey, introverted person. I just have no idea what to do at sometime in every year, so instead of wasting time, I think learning is better for fucking sure. The exam is just 1 fucking gate or guardrail I created to make sure I understand enough what is fucking I’m learning. Wasted time and money (buy an exam to get cert) or be fucking serious about learning? If I need to compare, I simply compare my current year with myself from last year; that’s all. LOL</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=094d89f0806f" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[My experience on learning Azure DevOps — Part 2: Azure DevOps]]></title>
            <link>https://medium.com/@kienlt.qn/my-experience-on-learning-azure-devops-part-2-azure-devops-10623e5e4248?source=rss-b77f4e43d676------2</link>
            <guid isPermaLink="false">https://medium.com/p/10623e5e4248</guid>
            <category><![CDATA[azure-devops]]></category>
            <category><![CDATA[azure-pipelines]]></category>
            <dc:creator><![CDATA[Kienlt]]></dc:creator>
            <pubDate>Sat, 09 May 2026 05:26:00 GMT</pubDate>
            <atom:updated>2026-05-16T04:07:37.991Z</atom:updated>
            <content:encoded><![CDATA[<h3>Introduction</h3><p>This article is part 2, part 1 you can find it here: <a href="https://medium.com/@kienlt.qn/my-experience-on-learning-azure-devops-part-1-acr-aks-96fe1c721b05">My experience on learning Azure Devops — Part 1: ACR + AK</a>S</p><p>In this article, I will show you some little knowledge about Azure DevOps that I learned and try to implement in my lab.</p><h3>Overview</h3><p>So I’m gonna use Azure DevOps for this lab. <strong>My idea is: Push code → trigger pipeline in Azure DevOps → pipeline will build image -&gt; push ACR -&gt; update gitops repo -&gt; ArgoCD auto sync.</strong></p><p>So in this setup, ArgoCD is the only external component, the rest is managed by Azure!</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*3cNfBVa6k9ZmdYdn.png" /></figure><p>I think this is a better diagram xD</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*UZqMdjRE4BXIp6kD.png" /></figure><h3>Step 1: Install ArgoCD</h3><p>For this fucking lab, no HA — just standalone install.</p><pre># Create namespace<br>kubectl create namespace argocd<br># Install ArgoCD<br>kubectl apply -n argocd -f https://raw.githubusercontent.com/argoproj/argo-cd/stable/manifests/install.yaml</pre><p>Get admin password:</p><pre>kubectl -n argocd get secret argocd-initial-admin-secret \<br>    -o jsonpath=&quot;{.data.password}&quot; | base64 -d ; echo</pre><p>Get public IP:</p><pre>NODE_RG=$(az aks show -g rg-aks-lab -n aks-lab --query nodeResourceGroup -o tsv)<br>INGRESS_IP=$(az network public-ip show -g $NODE_RG -n pip-ingress --query ipAddress -o tsv)<br>echo $INGRESS_IP</pre><p>Manifest for argocd-ingress.yaml:</p><pre>apiVersion: networking.k8s.io/v1<br>kind: Ingress<br>metadata:<br>  name: argocd-server<br>  namespace: argocd<br>  annotations:<br>    nginx.ingress.kubernetes.io/backend-protocol: &quot;HTTPS&quot;<br>spec:<br>  ingressClassName: webapprouting.kubernetes.azure.com<br>  rules:<br>  - host: argocd.kienlt.com<br>    http:<br>      paths:<br>      - path: /<br>        pathType: Prefix<br>        backend:<br>          service:<br>            name: argocd-server<br>            port:<br>              number: 443</pre><p>Apply manifest for argocd ingress: k apply -f argocd-ingress.yaml</p><h3>Step 2: Azure Repos + Helm chart + ArgoCD Application</h3><h4>Create Azure DevOps org + project</h4><p>Create a new Org with your current subscription</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*gPO_eYOqeN_CEgk5.png" /></figure><p>New project aks-lab</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*Zg_3NUzzU_dcVbEz.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*7XzG0fKcYFTW8iBB.png" /></figure><p>Create new Repos for app + gitops:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*D-ER32v2OPIU-TT2.png" /></figure><p>Oke, we have 2 new ones plus 1 default xD</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*EEmIplGxrWVA29fB.png" /></figure><p>Next, we need to setup ssh key for cloning without credentials xD. Local dev = git clone via SSH. But the pipeline needs PAT because the self-hosted agent / ArgoCD needs HTTPS auth!</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*8yZKpIXcVGEdr47q.png" /></figure><p>Then clone it, pretty much the same as other git providers xD</p><pre>kienlt@Luongs-MacBook-Pro dev.azure.com % git clone git@ssh.dev.azure.com:v3/kienltqn/aks-lab/aks-lab-app<br>Cloning into &#39;aks-lab-app&#39;...<br>Warning: Permanently added &#39;ssh.dev.azure.com&#39; (RSA) to the list of known hosts.<br>remote: Azure Repos<br>remote: Found 3 objects to send. (51 ms)<br>Receiving objects: 100% (3/3), done.</pre><p>And our repo will have the following files from the previous project</p><pre>kienlt@Luongs-MacBook-Pro aks-lab-app % ll<br>total 32<br>drwxr-xr-x@  8 kienlt  staff  256  3 May 10:05 .<br>drwxr-xr-x@  3 kienlt  staff   96  3 May 10:02 ..<br>drwxr-xr-x@ 12 kienlt  staff  384  3 May 10:04 .git<br>-rw-r--r--@  1 kienlt  staff  199  3 May 10:03 default.conf<br>-rw-r--r--@  1 kienlt  staff  125  3 May 10:03 Dockerfile<br>-rw-r--r--@  1 kienlt  staff  461  3 May 10:03 index.html<br>drwxr-xr-x@  4 kienlt  staff  128  3 May 10:05 manifests<br>-rw-r--r--@  1 kienlt  staff  985  3 May 10:02 README.md<br>kienlt@Luongs-MacBook-Pro aks-lab-app % ll manifests<br>total 16<br>drwxr-xr-x@ 4 kienlt  staff  128  3 May 10:05 .<br>drwxr-xr-x@ 8 kienlt  staff  256  3 May 10:05 ..<br>-rw-r--r--@ 1 kienlt  staff  707  3 May 10:03 01.deploy.yaml<br>-rw-r--r--@ 1 kienlt  staff  516  3 May 10:03 02.ingress.yaml</pre><h4>Helm chart and GitOps</h4><p>Next, clone the GitOps repo:</p><pre>git clone git@ssh.dev.azure.com:v3/kienltqn/aks-lab/aks-lab-gitops<br>Cloning into &#39;aks-lab-gitops&#39;...<br>remote: Azure Repos<br>remote: Found 3 objects to send. (72 ms)<br>Receiving objects: 100% (3/3), done.</pre><p>And then we have very, very basic….</p><pre>aks-lab-gitops/<br>└── charts/<br>    └── hello-aks/<br>        ├── Chart.yaml<br>        ├── values.yaml<br>        └── templates/<br>            ├── deployment.yaml<br>            ├── service.yaml<br>            └── ingress.yaml</pre><p>Validate it by using helm template. You can install it via brew install helm</p><pre>helm template hello-aks ./charts/hello-aks/<br>---<br># Source: hello-aks/templates/service.yaml<br>apiVersion: v1<br>kind: Service<br>metadata:<br>  name: hello-aks<br>spec:<br>  type: ClusterIP<br>  selector:<br>    app: hello-aks<br>  ports:<br>  - port: 80<br>    targetPort: 80<br>---<br># Source: hello-aks/templates/deployment.yaml<br>apiVersion: apps/v1<br>kind: Deployment<br>metadata:<br>  name: hello-aks<br>  labels:<br>    app: hello-aks<br>spec:<br>  replicas: 2<br>  selector:<br>    matchLabels:<br>      app: hello-aks<br>  template:<br>    metadata:<br>      labels:<br>        app: hello-aks<br>    spec:<br>      containers:<br>      - name: app<br>        image: &quot;acrakslab20546.azurecr.io/hello-aks:v1&quot;<br>        imagePullPolicy: IfNotPresent<br>        ports:<br>        - containerPort: 80<br>---<br># Source: hello-aks/templates/ingress.yaml<br>apiVersion: networking.k8s.io/v1<br>kind: Ingress<br>metadata:<br>  name: hello-aks<br>spec:<br>  ingressClassName: webapprouting.kubernetes.azure.com<br>  rules:<br>  - host: hello.kienlt.com<br>    http:<br>      paths:<br>      - path: /<br>        pathType: Prefix<br>        backend:<br>          service:<br>            name: hello-aks<br>            port:<br>              number: 80</pre><p>Ok, remember to clean from the previous section</p><pre>kienlt@Luongs-MacBook-Pro 10-ecr-aks % k delete -f 01.deploy.yaml<br>deployment.apps &quot;hello-aks&quot; deleted from default namespace<br>service &quot;hello-aks&quot; deleted from default namespace<br>kienlt@Luongs-MacBook-Pro 10-ecr-aks % k delete -f 02.ingress.yaml<br>ingress.networking.k8s.io &quot;hello-aks&quot; deleted from default namespace</pre><h4>ArgoCD</h4><p>We need to create a PAT (Personal Access Token) to give Argocd permission to sync…</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*U80dVcov7LhefAWZ.png" /></figure><p>Yeah, full access for this fucking demo xD or serious with Scopes: Custom defined → tick Code → Read</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*_kW2qoYRvIyLTuQS.png" /></figure><p>Test token time:</p><pre>PAT=&quot;&lt;your-pat&gt;&quot;<br>curl -sI -u &quot;kienltqn:$PAT&quot; \<br>&quot;https://dev.azure.com/kienltqn/aks-lab/_git/aks-lab-gitops/info/refs?service=git-upload-pack&quot; \<br>| head -3</pre><p>Response ok:</p><pre>HTTP/2 200<br>cache-control: no-cache, no-store, must-revalidate<br>pragma: no-cache</pre><p>Connect Repo in ArgoCD: remember, no fucking username (@username) in repo URL.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*3c1A_0vESi18Gagx.png" /></figure><p>Expected result</p><figure><img alt="" src="https://cdn-images-1.medium.com/proxy/1*b31hiO4ynbDLRrXWEFF4aQ.png" /></figure><p>Expected output in applications</p><figure><img alt="" src="https://cdn-images-1.medium.com/proxy/1*b31hiO4ynbDLRrXWEFF4aQ.png" /></figure><p>My manifest for that application:</p><pre>apiVersion: argoproj.io/v1alpha1<br>kind: Application<br>metadata:<br>  name: hello-aks<br>  namespace: argocd<br>spec:<br>  project: default<br>  source:<br>    repoURL: https://dev.azure.com/kienltqn/aks-lab/_git/aks-lab-gitops<br>    targetRevision: HEAD<br>    path: charts/hello-aks<br>    helm:<br>      valueFiles:<br>        - values.yaml<br>  destination:<br>    server: https://kubernetes.default.svc<br>    namespace: hello-aks<br>  syncPolicy:<br>    syncOptions:<br>      - CreateNamespace=true</pre><h3>Step 3: Azure Pipeline build -&gt; push ACR -&gt; bump tag GitOps</h3><h4>Create Service Connection for ACR</h4><p>Pipeline needs to authenticate into ACR to push the image. The recommended way is Service connection using Service Principal.</p><p>Project Settings → Service Connections → New Service Connections → Docker Registry</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*tpk4C4x1t6Clv4o3.png" /></figure><p>After that:</p><ul><li>Registry type: Azure Container Registry</li><li>Authentication Type: Service Principal</li><li>Subscription: pick a subscription that contains ACR</li><li>Azure container registry: your ACR</li><li>Service connection name: acr-aks-lab</li><li>Grant access permission to all pipelines</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/952/0*rmtQh_hwF5eZbm5H.png" /></figure><p>Output:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*90uZb4nAmN7rQclO.png" /></figure><h4>Grant access for the Build Service push into the GitOps repo</h4><p>Project Settings -&gt; Repositories -&gt; aks-lab-gitops -&gt; tab Security -&gt; find user like: aks-lab Build Service (kienltqn). Then set the following permission to “Allow” — Contribute — Create branch</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*NFzXcppK9dl2Ivr_.png" /></figure><h4>Create azure-pipelines.yml (like .gitlab-ci.yml xD)</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*Q84DWalWTYpRxWc0.png" /></figure><pre>trigger:<br>  branches:<br>    include: [main]<br>  paths:<br>    exclude: [&#39;**/*.md&#39;, &#39;azure-pipelines.yml&#39;]<br><br>pr: none<br><br>variables:<br>  acrServiceConnection: &#39;acr-aks-lab&#39;<br>  acrLoginServer: &#39;acrakslab20546.azurecr.io&#39;<br>  imageRepository: &#39;hello-aks&#39;<br>  gitopsRepo: &#39;aks-lab-gitops&#39;<br>  azdoOrg: &#39;kienltqn&#39;<br>  azdoProject: &#39;aks-lab&#39;<br>  valuesFile: &#39;charts/hello-aks/values.yaml&#39;<br><br>pool:<br>  name: local-mac<br><br>stages:<br>- stage: Build<br>  displayName: Build &amp; push image<br>  jobs:<br>  - job: BuildAndPush<br>    steps:<br>    - task: Docker@2<br>      displayName: Login ACR<br>      inputs:<br>        command: login<br>        containerRegistry: $(acrServiceConnection)<br><br>    - bash: |<br>        set -euo pipefail<br><br>        BUILDX=&quot;${HOME}/.docker/cli-plugins/docker-buildx&quot;<br><br>        # Verify<br>        &quot;$BUILDX&quot; version<br><br>        # Create builder (idempotent)<br>        &quot;$BUILDX&quot; create --use --name multiarch 2&gt;/dev/null || &quot;$BUILDX&quot; use multiarch<br>        &quot;$BUILDX&quot; inspect --bootstrap<br><br>        &quot;$BUILDX&quot; build \<br>          --platform linux/amd64 \<br>          --tag $(acrLoginServer)/$(imageRepository):$(Build.BuildId) \<br>          --push \<br>          .<br>      displayName: Build linux/amd64 &amp; push (direct binary)<br><br>    - task: Docker@2<br>      displayName: Logout ACR<br>      inputs:<br>        command: logout<br>        containerRegistry: $(acrServiceConnection)<br><br>- stage: UpdateGitOps<br>  displayName: Bump image tag in gitops<br>  dependsOn: Build<br>  condition: succeeded()<br>  jobs:<br>  - job: BumpTag<br>    steps:<br>    - checkout: none<br>    - bash: |<br>        set -euo pipefail<br><br>        # Diagnostic: verify PAT reached script<br>        echo &quot;PAT length: ${#GITOPS_PAT}&quot;   <br><br>        # URL embed PAT (Azure Pipelines auto-mask secret in log)<br>        REPO_URL=&quot;@dev.azure.com/$(azdoOrg)/$(azdoProject)/_git/$(gitopsRepo)&quot;&gt;https://kienltqn:${GITOPS_PAT}@dev.azure.com/$(azdoOrg)/$(azdoProject)/_git/$(gitopsRepo)&quot;<br><br>        git clone &quot;$REPO_URL&quot; gitops<br>        cd gitops<br><br>        yq -i &quot;.image.tag = \&quot;$(Build.BuildId)\&quot;&quot; $(valuesFile)<br>        echo &quot;----- updated values.yaml -----&quot;<br>        cat $(valuesFile)<br><br>        git config user.email &quot;pipeline@dev.azure.com&quot;<br>        git config user.name  &quot;Azure Pipeline&quot;<br>        git add $(valuesFile)<br><br>        if git diff --cached --quiet; then<br>          echo &quot;No tag change, skip commit.&quot;<br>          exit 0<br>        fi<br><br>        git commit -m &quot;ci: bump hello-aks image to $(Build.BuildId) [skip ci]&quot;<br>        git push origin HEAD:main<br>      env:<br>        GITOPS_PAT: $(GITOPS_PAT)<br>      displayName: Update values.yaml &amp; push</pre><p>Note: This pipeline is designed for a self-hosted Mac agent. We’ll see why below.</p><h4>Create Pipeline definition in Azure DevOps UI</h4><p>Pipelines -&gt; Pipelines -&gt; Create Pipeline -&gt; Azure Repos Git -&gt; chọn aks-lab-app -&gt; Existing Azure Pipelines YAML file -&gt; Branch main, Path /azure-pipelines.yml -&gt; Continue -&gt; Run.</p><p>LOL, Holy fucking shit</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*_zVVa92Bi6shZhyZ.png" /></figure><h4>Time to create a new agent-pool self-host</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*6Ittju3F0OjA_fri.png" /></figure><p>Create new PAT for agent-local-mac</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*IS0cIAj7dqwKFdrf.png" /></figure><p>User Settings (avatar) → Personal access tokens → New Token:</p><ul><li>Name: agent-local-mac</li><li>Organization: kienltqn</li><li>Expiration: 90 days</li><li>Scopes: Custom defined → Agent Pools → tick Read &amp; manage</li></ul><h4>Install and configure the agent on a local Mac</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*IfwbYed81SnhYPzW.png" /></figure><p>Need to run: brew install docker-buildx for Mac Local and symlink</p><pre>mkdir -p ~/.docker/cli-plugins<br>ln -sfn $(brew --prefix)/opt/docker-buildx/bin/docker-buildx \<br>        ~/.docker/cli-plugins/docker-buildx</pre><p>Follow the instructions, and finally, we have the following thing below. Remember PAT is what we just created above.</p><pre>kienlt@Luongs-MacBook-Pro azp-agent % ./config.sh<br><br>  ___                      ______ _            _ _<br> / _ \                     | ___ (_)          | (_)<br>/ /_\ \_____   _ _ __ ___  | |_/ /_ _ __   ___| |_ _ __   ___  ___<br>|  _  |_  / | | | &#39;__/ _ \ |  __/| | &#39;_ \ / _ \ | | &#39;_ \ / _ \/ __|<br>| | | |/ /| |_| | | |  __/ | |   | | |_) |  __/ | | | | |  __/\__ \<br>\_| |_/___|\__,_|_|  \___| \_|   |_| .__/ \___|_|_|_| |_|\___||___/<br>                                   | |<br>        agent v4.272.0             |_|          (commit 491e611)<br><br><br>&gt;&gt; End User License Agreements:<br><br>Building sources from a TFVC repository requires accepting the Team Explorer Everywhere End User License Agreement. This step is not required for building sources from Git repositories.<br><br>A copy of the Team Explorer Everywhere license agreement can be found at:<br>  /Users/kienlt/azp-agent/license.html<br><br>Enter (Y/N) Accept the Team Explorer Everywhere license agreement now? (press enter for N) &gt; Y<br><br>&gt;&gt; Connect:<br><br>Enter server URL &gt; https://dev.azure.com/kienltqn<br>Enter authentication type (press enter for PAT) &gt; PAT<br>Enter personal access token &gt; ************************************************************************************<br>Connecting to server ...<br><br>&gt;&gt; Register Agent:<br><br>Enter agent pool (press enter for default) &gt; local-mac<br>Enter agent name (press enter for Luongs-MacBook-Pro) &gt;<br>Scanning for tool capabilities.<br>Connecting to the server.<br>Successfully added the agent<br>Testing agent connection.<br>Enter work folder (press enter for _work) &gt;<br>2026-05-03 10:38:46Z: Settings Saved.</pre><p>Ok, let’s gooooooo:</p><pre>kienlt@Luongs-MacBook-Pro azp-agent % ./run.sh<br>Scanning for tool capabilities.<br>Connecting to the server.<br>2026-05-03 10:39:06Z: Listening for Jobs</pre><p>Verify agent online:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*x1xcjnsoQa46i0uY.png" /></figure><p>Change from azure-pipelines.yml from</p><pre>pool:<br>  vmImage: ubuntu-latest</pre><p>to</p><pre>pool:<br>  name: local-mac</pre><p>The step change pool, I did it already.</p><p>Delete existing pipeline, run new pipeline and here is output, failed but it run xD</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*BIyjb9FJi5Ew9AOD.png" /></figure><p>Hmm, it still fucked up in part clone gitops repo. Time to create a new PAT for it — or just skip and reuse that fucking FULL ACCESS PAT!!!</p><p>Add PAT for secret variable of pipeline, it is simple, create environment variable, like this!</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*CTPyW-dIIEOICwCm.png" /></figure><p>At least we got build image works</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*NMD1XnFSIICSAnmW.png" /></figure><p>Hmmm, seem like I already fucked up, I create env, not pipeline variable?</p><p>In Azure DevOps: — Pipeline variable = key-value injected as env var in pipeline run (can mark as secret) — Environment = deployment target (k8s/VM/etc.) used for approval gates, not variable scope</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*ovTX9XESxsxuih5w.png" /></figure><p>Yeah, god damn it!!!</p><pre>Starting: Update values.yaml &amp; push<br>==============================================================================<br>Task         : Bash<br>Description  : Run a Bash script on macOS, Linux, or Windows<br>Version      : 3.268.1<br>Author       : Microsoft Corporation<br>Help         : https://docs.microsoft.com/azure/devops/pipelines/tasks/utility/bash<br>==============================================================================<br>Generating script.<br>========================== Starting Command Output ===========================<br>/bin/bash /Users/kienlt/azp-agent/_work/_temp/3651f932-ea84-4340-bf3c-573b2d13a766.sh<br>PAT length: 84<br>Cloning into &#39;gitops&#39;...<br>/Users/kienlt/azp-agent/_work/_temp/3651f932-ea84-4340-bf3c-573b2d13a766.sh: line 12: yq: command not found<br><br>##[error]Bash exited with code &#39;127&#39;.<br>Finishing: Update values.yaml &amp; push</pre><p>Time to install yq in my local Mac brew install yq. Remember to delete the environment we created in the previous step!</p><p>Yayyy, finally</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*1hkDE2aNcWw9fZXz.png" /></figure><p>Verify in Gitops Repo</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*ec1f1t1UeVV-A0bZ.png" /></figure><p>View via domain</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*u_oMzhmUTr8_eW2A.png" /></figure><h3>Conclusion</h3><p>If possible, I’d recommend GitHub Actions or GitLab Runners over Azure DevOps. It feels unnecessarily complex to me!</p><p>But good to know how it works when you have to use it, especially for Azure’s ecosystem!</p><h3>Cleanup</h3><ul><li>Delete resource group:</li></ul><pre>az group delete -n rg-aks-lab --yes --no-wait</pre><ul><li>Stop the self-hosted agent</li><li>Revoke PATs</li></ul><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=10623e5e4248" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[My experience on learning Azure DevOps — Part 1: ACR + AKS]]></title>
            <link>https://medium.com/@kienlt.qn/my-experience-on-learning-azure-devops-part-1-acr-aks-96fe1c721b05?source=rss-b77f4e43d676------2</link>
            <guid isPermaLink="false">https://medium.com/p/96fe1c721b05</guid>
            <category><![CDATA[azure-devops]]></category>
            <dc:creator><![CDATA[Kienlt]]></dc:creator>
            <pubDate>Tue, 05 May 2026 13:33:49 GMT</pubDate>
            <atom:updated>2026-05-05T13:33:49.706Z</atom:updated>
            <content:encoded><![CDATA[<h3>Introduction</h3><p>I tried to learn Azure to prepare for certification AZ-104, but I really got bored while watching videos in the Udemy course. So I decided to learn a real project from basic to a little level higher than basic xD (not really advanced)</p><h3>Prerequisite</h3><p>This is not an introduction to Azure, so I’m assuming you have little knowledge related to Azure with the following sections:</p><ul><li>Azure CLI!</li><li>VNet (Virtual Network): Subnet, NSG (Network Security Group)…</li><li>Azure Container Registry (ACR)</li><li>Kubernetes Service (AKS)</li><li>and so on…</li></ul><p>Also, I’m using a trial account with 200$ credit and a very, very limited resource quota for this lab!</p><p>Standard_D2s_v3 is the only SKU I can use for this lab.</p><h3>Resource Creation</h3><h4>Step 1: Create a resource group</h4><p>Azure Resource Group(RG) is a logical container that holds related resources (VMs, databases, web apps) for an Azure solution. When you delete a RG, all resources in that RG will be gone, remember this xD</p><pre>az group create -n rg-aks-lab -l southeastasia --tags Project=aks-lab Owner=kien Environment=lab</pre><h4>Step 2: Create VNet + subnet for AKS</h4><pre>az network vnet create \<br>    -g rg-aks-lab \<br>    -n vnet-aks \<br>    --location southeastasia \<br>    --address-prefix 10.10.0.0/16 \<br>    --subnet-name snet-aks \<br>    --subnet-prefix 10.10.1.0/24 \<br>    --tags Project=aks-lab Owner=kien Environment=lab</pre><p>subnet snet-aks: 10.10.1.0/24 I&#39;m gonna use it for AKS.</p><p>Verify subnet creation:</p><pre>az network vnet subnet list \<br>    -g rg-aks-lab \<br>    --vnet-name vnet-aks \<br>    --query &quot;[].{Name:name, CIDR:addressPrefix}&quot; -o table<br># Expected output<br>Name      CIDR<br>--------  ------------<br>snet-aks  10.10.1.0/24</pre><p>Save Subnet ID in a variable</p><pre>SUBNET_ID=$(az network vnet subnet show \<br>    -g rg-aks-lab \<br>    --vnet-name vnet-aks \<br>    -n snet-aks \<br>    --query id -o tsv)</pre><h4>Step 3: Create ACR</h4><p>Generate a unique name:</p><pre>ACR_NAME=&quot;acrakslab$RANDOM&quot;<br>echo &quot;ACR name will be: $ACR_NAME&quot;<br># ACR name will be: acrakslab20546</pre><p>Create ACR Basic:</p><pre>az acr create \<br>    -g rg-aks-lab \<br>    -n $ACR_NAME \<br>    --sku Basic \<br>    --location southeastasia \<br>    --tags Project=aks-lab Owner=kien Environment=lab</pre><p>Show ACR Info:</p><pre>az acr show -n $ACR_NAME \<br>    --query &quot;{name:name, loginServer:loginServer, sku:sku.name, status:provisioningState}&quot; \<br>    -o jsonc</pre><p>Expected output of ACR info:</p><pre>{<br>  &quot;loginServer&quot;: &quot;acrakslab20546.azurecr.io&quot;,<br>  &quot;name&quot;: &quot;acrakslab20546&quot;,<br>  &quot;sku&quot;: &quot;Basic&quot;,<br>  &quot;status&quot;: &quot;Succeeded&quot;<br>}</pre><p>For login later to the container registry:</p><pre>ACR_LOGIN_SERVER=$(az acr show -n $ACR_NAME --query loginServer -o tsv)</pre><p>Test login container registry:</p><pre>az acr login -n $ACR_NAME<br># Login Succeeded</pre><h4>Step 4: Create AKS</h4><p>Make sure Microsoft.ContainerService registered for subscription</p><pre>az provider register --namespace Microsoft.ContainerService<br>az provider register --namespace Microsoft.ContainerRegistry</pre><p>Verify by:</p><pre>az provider show -n Microsoft.ContainerService --query registrationState -o tsv</pre><p>Get your public IP for whitelist access to K8s api server:</p><pre>MYIP=$(curl -s https://api.ipify.org)</pre><p>Create AKS command:</p><pre>az aks create \<br>    --resource-group rg-aks-lab \<br>    --name aks-lab \<br>    --location southeastasia \<br>    --tier free \<br>    --kubernetes-version 1.35.0 \<br>    --node-count 1 \<br>    --node-vm-size Standard_D2s_v3 \<br>    --vnet-subnet-id $SUBNET_ID \<br>    --network-plugin azure \<br>    --network-plugin-mode overlay \<br>    --pod-cidr 10.244.0.0/16 \<br>    --service-cidr 10.20.0.0/16 \<br>    --dns-service-ip 10.20.0.10 \<br>    --enable-managed-identity \<br>    --attach-acr $ACR_NAME \<br>    --api-server-authorized-ip-ranges $MYIP/32 \<br>    --generate-ssh-keys \<br>    --tags Project=aks-lab Owner=kien Environment=lab</pre><blockquote><em>Note regarding </em><em>--network-plugin azure --network-plugin-mode overlay: This is </em><strong><em>Azure CNI Overlay</em></strong><em>, different from regular Azure CNI (each pod occupies 1 IP from the VNet subnet). Overlay mode uses its own </em><em>--pod-cidr (10.244.0.0/16) so pods don&#39;t consume IPs from the </em><em>snet-aks subnet → saving IPs, allowing for easy scaling of multiple pods without worrying about running out of subnet IPs. Suitable for small labs and production.</em></blockquote><p>If there something fucked up related to version of k8s: az aks get-versions -l southeastasia try to change to other version (latest for example!). It will tooks about 5-10 minutes</p><p>Verify by:</p><pre>az aks show -g rg-aks-lab -n aks-lab \<br>    --query &quot;{name:name, state:provisioningState, k8s:kubernetesVersion}&quot; \<br>    -o jsonc<br># Expected output<br>{<br>  &quot;k8s&quot;: &quot;1.35.0&quot;,<br>  &quot;name&quot;: &quot;aks-lab&quot;,<br>  &quot;state&quot;: &quot;Succeeded&quot;<br>}</pre><p>K8S cluster Info:</p><pre>az aks show -g rg-aks-lab -n aks-lab \<br>    --query &quot;{name:name, fqdn:fqdn, k8s:kubernetesVersion, nodeRG:nodeResourceGroup, identity:identity.type}&quot; \<br>    -o jsonc</pre><p>Output:</p><pre>{<br>  &quot;fqdn&quot;: &quot;aks-lab-rg-aks-lab-6ffa29-svgtoxzi.hcp.southeastasia.azmk8s.io&quot;,<br>  &quot;identity&quot;: &quot;SystemAssigned&quot;,<br>  &quot;k8s&quot;: &quot;1.35.0&quot;,<br>  &quot;name&quot;: &quot;aks-lab&quot;,<br>  &quot;nodeRG&quot;: &quot;MC_rg-aks-lab_aks-lab_southeastasia&quot;<br>}</pre><blockquote><em>Note: AKS auto created 1 RG named MC_ that contains VM, NSG, LB, Public IP. Don’t touch this fucking RG, AKS will delete this auto when we delete AKS</em></blockquote><p>Check role assignment in ACR:</p><pre>ACR_ID=$(az acr show -n $ACR_NAME --query id -o tsv)<br>az role assignment list --scope $ACR_ID -o table | grep AcrPull<br># Output<br>.....Microsoft.ContainerRegistry/registries/acrakslab20546</pre><p>If in this scenario, <strong>— attach-acr</strong> not working, you can run:</p><pre>az aks update -g rg-aks-lab -n aks-lab --attach-acr $ACR_NAME</pre><p>Verify:</p><pre>az aks get-credentials -g rg-aks-lab -n aks-lab<br># Merged &quot;aks-lab&quot; as current context in /Users/kienlt/.kube/config<br># kctx installed here: https://github.com/BlackMetalz/ChgK8sCtx<br>kctx<br># Using default kubeconfig path: /Users/kienlt/.kube/config<br># ✔ aks-lab(current-context)<br># You already selected this context!<br>k get nodes<br># NAME                                STATUS   ROLES    AGE     VERSION<br># aks-nodepool1-22202873-vmss000000   Ready    &lt;none&gt;   7m23s   v1.35.0</pre><p>Here is overview after created above resources:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*s_kE5-0kIBOEQ2cZ.png" /></figure><p>Architecture diagram (network flow):</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*1cik2hgUTbvzwK7U.png" /></figure><h3>Deploy, Interact with K8S cluster</h3><h4>Step 5: kubectl + ACR build/push</h4><p>Here is demo repo of mine after a lot of modify: <a href="https://gitlab.com/devops-azure/aks-lab-app">https://gitlab.com/devops-azure/aks-lab-app</a> but I updated a lot to final version so you only see 1 init commit.</p><p>Build local because az acr build will not work on trial subscription. And why I used to build multiple platform? because I&#39;m using Mac to build, not Ubuntu (linux/amd64)</p><pre>az acr login -n $ACR_NAME<br># Login Succeeded<br>docker buildx build \<br>    --platform linux/amd64,linux/arm64 \<br>    --tag $ACR_LOGIN_SERVER/hello-aks:v1 \<br>    --tag $ACR_LOGIN_SERVER/hello-aks:latest \<br>    --push \<br>    .</pre><p>Verify by: az acr repository show-tags -n $ACR_NAME --repository hello-aks -o table</p><pre>Result<br>--------<br>latest<br>v1</pre><p>Verify via Portal:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*p9CA4txoTPdriA6G.png" /></figure><h4>Step 6: Deploy APP into AKS</h4><p>Go to folder aks-lab-app and deploy manifests:</p><pre>cd manifests<br>k apply -f 01.deploy.yaml</pre><p>Output:</p><pre>k get pod -owide<br>NAME                        READY   STATUS    RESTARTS   AGE   IP             NODE                                NOMINATED NODE   READINESS GATES<br>hello-aks-58d5fbc6b-8krf4   1/1     Running   0          24s   10.244.0.155   aks-nodepool1-22202873-vmss000000   &lt;none&gt;           &lt;none&gt;<br>hello-aks-58d5fbc6b-gqxrz   1/1     Running   0          26s   10.244.0.167   aks-nodepool1-22202873-vmss000000   &lt;none&gt;           &lt;none&gt;<br>hello-aks-58d5fbc6b-j2qdl   1/1     Running   0          22s   10.244.0.132   aks-nodepool1-22202873-vmss000000   &lt;none&gt;           &lt;none&gt;</pre><h4>Step 7 — Expose app via LoadBalancer Service (We can skip this section)</h4><p>svc.yaml manifest:</p><pre>apiVersion: v1<br>kind: Service<br>metadata:<br>  name: hello-aks<br>spec:<br>  type: LoadBalancer<br>  selector:<br>    app: hello-aks<br>  ports:<br>    - port: 80<br>      targetPort: 80<br>      protocol: TCP</pre><p>Verify Azure resource created:</p><ul><li>Public IP for LoadBalancer in nodeRG:</li></ul><pre>az network public-ip list \<br>    -g MC_rg-aks-lab_aks-lab_southeastasia \<br>    --query &quot;[].{name:name, ip:ipAddress, sku:sku.name}&quot; -o table<br># output (2 rows)<br>Name                                         Ip             Sku<br>-------------------------------------------  -------------  --------</pre><ul><li>Standard LB:</li></ul><pre>az network lb list \<br>    -g MC_rg-aks-lab_aks-lab_southeastasia \<br>    --query &quot;[].{name:name, sku:sku.name, frontends:frontendIpConfigurations[].name}&quot; -o table<br><br>Name        Sku<br>----------  --------<br>kubernetes  Standard</pre><p>Get external IP to test:</p><pre>EXTERNAL_IP=$(kubectl get svc hello-aks -o jsonpath=&#39;{.status.loadBalancer.ingress[0].ip}&#39;)</pre><p>Testing time:</p><pre>for i in {1..6}; do<br>    echo &quot;--- Request $i ---&quot;<br>    curl -s http://$EXTERNAL_IP | grep &quot;pod\&quot;&gt;&quot; | sed &#39;s/&lt;[^&gt;]*&gt;//g&#39;<br>  done<br>--- Request 1 ---<br>hello-aks-7fd4ff6c4f-lgt2w<br>--- Request 2 ---<br>hello-aks-7fd4ff6c4f-lgt2w<br>--- Request 3 ---<br>hello-aks-7fd4ff6c4f-bljl8<br>--- Request 4 ---<br>hello-aks-7fd4ff6c4f-bljl8<br>--- Request 5 ---<br>hello-aks-7fd4ff6c4f-bljl8<br>--- Request 6 ---<br>hello-aks-7fd4ff6c4f-th8tp</pre><h4>Step 8 — Scale AKS worker node</h4><pre>az aks nodepool scale \<br>    --resource-group rg-aks-lab \<br>    --cluster-name aks-lab \<br>    --name nodepool1 \<br>    --node-count 2</pre><p>Verify:</p><pre>kubectl get nodes<br>NAME                                STATUS   ROLES    AGE    VERSION<br>aks-nodepool1-22202873-vmss000000   Ready    &lt;none&gt;   121m   v1.35.0<br>aks-nodepool1-22202873-vmss000001   Ready    &lt;none&gt;   14m    v1.35.0</pre><p>Test again with 10 pods (after I scale up):</p><pre>for i in {1..10}; do<br>    echo &quot;--- Request $i ---&quot;<br>    curl -s http://$EXTERNAL_IP | grep &quot;pod\&quot;&gt;&quot; | sed &#39;s/&lt;[^&gt;]*&gt;//g&#39;<br>  done<br>--- Request 1 ---<br>hello-aks-7fd4ff6c4f-75ppz<br>--- Request 2 ---<br>hello-aks-7fd4ff6c4f-9956v<br>--- Request 3 ---<br>hello-aks-7fd4ff6c4f-v5fmt<br>--- Request 4 ---<br>hello-aks-7fd4ff6c4f-9956v<br>--- Request 5 ---<br>hello-aks-7fd4ff6c4f-v5fmt<br>--- Request 6 ---<br>hello-aks-7fd4ff6c4f-bwcz5<br>--- Request 7 ---<br>hello-aks-7fd4ff6c4f-75ppz<br>--- Request 8 ---<br>hello-aks-7fd4ff6c4f-v5fmt<br>--- Request 9 ---<br>hello-aks-7fd4ff6c4f-7js52<br>--- Request 10 ---<br>hello-aks-7fd4ff6c4f-th8tp</pre><h4>Step 9 — Static IP for Service (Skip this section also, same reason for step 7)</h4><pre>NODE_RG=$(az aks show -g rg-aks-lab -n aks-lab --query nodeResourceGroup -o tsv)<br>echo &quot;Node RG: $NODE_RG&quot;<br>az network public-ip create \<br>    -g $NODE_RG \<br>    -n pip-hello-aks \<br>    --location southeastasia \<br>    --sku Standard \<br>    --allocation-method Static<br><br>STATIC_IP=$(az network public-ip show -g $NODE_RG -n pip-hello-aks --query ipAddress -o tsv)<br>echo &quot;Static IP: $STATIC_IP&quot;</pre><p>Manifest:</p><pre>apiVersion: v1<br>kind: Service<br>metadata:<br>  name: hello-aks<br>  annotations:<br>    service.beta.kubernetes.io/azure-load-balancer-resource-group: MC_rg-aks-lab_aks-lab_southeastasia<br>    service.beta.kubernetes.io/azure-pip-name: pip-hello-aks<br>    # Bonus: provide DNS Azure<br>    service.beta.kubernetes.io/azure-dns-label-name: hello-aks-kien<br>spec:<br>  type: LoadBalancer<br>  selector:<br>    app: hello-aks<br>  ports:<br>    - port: 80<br>      targetPort: 80</pre><p>Apply: k apply -f svc-static.yaml</p><p>This is an example. For a better example, we could use Ingress Nginx</p><h3>AKS App Routing — Nginx Ingress</h3><h4>Step 10 — AKS App Routing (Practical for ingress)</h4><p>Enable it by:</p><pre>az aks approuting enable -g rg-aks-lab -n aks-lab</pre><p>Verify:</p><pre>k get pod -n app-routing-system<br>NAME                     READY   STATUS    RESTARTS   AGE<br>nginx-78ff99b4ff-h6f7t   1/1     Running   0          3m57s<br>nginx-78ff99b4ff-wjq2t   1/1     Running   0          4m11s</pre><blockquote><em>Note: The delete </em><em>pip-hello-aks section below only applies if you perform Step 9 (static IP for the service). If you skip Step 9, you will always skip this cleanup.</em></blockquote><p>Need to delete the previous public IP created for LB:</p><pre># List first<br>az network public-ip list -g $NODE_RG -o table<br>Name                                         ResourceGroup                        Location       Zones    Address        IdleTimeoutInMinutes    ProvisioningState<br>-------------------------------------------  -----------------------------------  -------------  -------  -------------  ----------------------  -------------------<br>e8da99b3-8a95-4685-aa9e-8949a9d7f39b         MC_rg-aks-lab_aks-lab_southeastasia  southeastasia  213      20.10.10.1     4                       Succeeded<br>kubernetes-afdc67b3ca4d3445ba9d6beeaeed1884  mc_rg-aks-lab_aks-lab_southeastasia  southeastasia  213      20.10.10.2     4                       Succeeded<br>pip-hello-aks                                MC_rg-aks-lab_aks-lab_southeastasia  southeastasia           20.10.10.3     4                       Succeeded</pre><p>We should delete pip-hello-aks which we created via svc: k delete svc hello-aks</p><p>Check it attached to anything:</p><pre>az network public-ip show -g $NODE_RG -n pip-hello-aks \<br>    --query &quot;{name:name, ip:ipAddress, attachedTo:ipConfiguration}&quot; \<br>    -o jsonc<br>{<br>  &quot;attachedTo&quot;: null,<br>  &quot;ip&quot;: &quot;20.10.10.3&quot;,<br>  &quot;name&quot;: &quot;pip-hello-aks&quot;<br>}</pre><p>Ok, null. Time to delete: az network public-ip delete -g $NODE_RG -n pip-hello-aks</p><p>Make static public IP in nodeRG.</p><pre>NODE_RG=$(az aks show -g rg-aks-lab -n aks-lab --query nodeResourceGroup -o tsv)<br>az network public-ip create \<br>    -g $NODE_RG \<br>    -n pip-ingress \<br>    --location southeastasia \<br>    --sku Standard \<br>    --allocation-method Static \<br>    --dns-name hello-aks-kienlt<br><br>INGRESS_IP=$(az network public-ip show -g $NODE_RG -n pip-ingress --query ipAddress -o tsv)<br>INGRESS_FQDN=$(az network public-ip show -g $NODE_RG -n pip-ingress --query dnsSettings.fqdn -o tsv)<br><br>echo &quot;Static IP:   $INGRESS_IP&quot;<br>echo &quot;Azure FQDN:  $INGRESS_FQDN&quot;</pre><p>Configure App Routing to use this Static IP by creating nginx-ingress.yaml:</p><pre>apiVersion: approuting.kubernetes.azure.com/v1alpha1<br>kind: NginxIngressController<br>metadata:<br>  name: default<br>spec:<br>  ingressClassName: webapprouting.kubernetes.azure.com<br>  controllerNamePrefix: nginx<br>  loadBalancerAnnotations:<br>    service.beta.kubernetes.io/azure-load-balancer-resource-group: MC_rg-aks-lab_aks-lab_southeastasia<br>    service.beta.kubernetes.io/azure-pip-name: pip-ingress</pre><p>ingress.yaml:</p><pre>  apiVersion: networking.k8s.io/v1<br>  kind: Ingress<br>  metadata:<br>    name: hello-aks<br>    annotations:<br>      nginx.ingress.kubernetes.io/rewrite-target: /<br>  spec:<br>    ingressClassName: webapprouting.kubernetes.azure.com<br>    rules:<br>      - host: kienlt.com    # Your domain here.<br>        http:<br>          paths:<br>            - path: /<br>              pathType: Prefix<br>              backend:<br>                service:<br>                  name: hello-aks<br>                  port:<br>                    number: 80</pre><p>Apply the configuration:</p><pre>kubectl apply -f nginx-ingress.yaml</pre><p>Ok. Make a CNAME via Azure FQDN, then test</p><h4>Step 11 — Whitelist IP access to API Server</h4><p>There is no fucking NSG for managed K8S, check which address is currently allowed</p><pre>az aks show -g rg-aks-lab -n aks-lab \<br>    --query &quot;apiServerAccessProfile&quot; -o jsonc</pre><p>Create a fake rule then test you can not connect to API Server via kubectl</p><pre>az aks update -g rg-aks-lab -n aks-lab \<br>    --api-server-authorized-ip-ranges &quot;1.2.3.4/32&quot;</pre><p>Restore rule:</p><pre>MYIP=$(curl -s https://api.ipify.org)<br>az aks update -g rg-aks-lab -n aks-lab \<br>    --api-server-authorized-ip-ranges $MYIP/32</pre><h3>Conclusion</h3><p>This is not a complete article like have no mistake, this is just what I want to save the steps I did while learning Azure.</p><p>I personally think this step is great for me, who has no fucking idea about Azure before (Never touched or created an account in Azure xD)</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=96fe1c721b05" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Some notes during the Claude code in action course]]></title>
            <link>https://medium.com/@kienlt.qn/some-notes-during-the-claude-code-in-action-course-dd487aa858fc?source=rss-b77f4e43d676------2</link>
            <guid isPermaLink="false">https://medium.com/p/dd487aa858fc</guid>
            <category><![CDATA[claude-code]]></category>
            <category><![CDATA[claude]]></category>
            <dc:creator><![CDATA[Kienlt]]></dc:creator>
            <pubDate>Sat, 18 Apr 2026 06:43:16 GMT</pubDate>
            <atom:updated>2026-04-18T06:43:16.359Z</atom:updated>
            <content:encoded><![CDATA[<h3>Getting hands-on</h3><h4>Some useful commands in Claude Code</h4><ul><li>Double esc: Restore the code and/or conversation to the point before. This could help to remove context not revelant to the current task.</li><li>/compact: Use this when you are feeling you are done with current task and move to next task, distinct task within the same session or your context window is getting full (60%-75% Utilization)</li><li>/clear: When you feeling AI seems confused or become dumb because previous conversation &quot;noise&quot;. Btw it&#39;s actually is fresh restart?</li></ul><p>This image is copied from the course: <a href="https://anthropic.skilljar.com/claude-code-in-action/">Claude Code in Action</a></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*aFRujoJt5Xo0AfL9.png" /></figure><h4>Commands</h4><p>Now it is called skills. So in essence, when we type /command-name in Claude Code, it will load content of markdown file into conversation like an instruction for LLM, it is not shell script it is prompt-based. Claude Code will read and follow that fucking instruction.</p><p>Where we would define it?:</p><ul><li>Project: .claude/skills/&lt;skill-name-here&gt;/SKILL.md</li><li>Personal (for global): ~/.claude/skills/&lt;skill-name-here&gt;/SKILL.md</li></ul><p>You can take a look here: <a href="https://github.com/anthropics/skills">https://github.com/anthropics/skills</a></p><p>Example skill defination that I take from link above:</p><pre>---<br>name: my-skill-name<br>description: A clear description of what this skill does and when to use it<br>---<br><br># My Skill Name<br><br>[Add your instructions here that Claude will follow when this skill is active]<br><br>## Examples<br>- Example usage 1<br>- Example usage 2<br><br>## Guidelines<br>- Guideline 1<br>- Guideline 2</pre><p>It does support argument which already mentioned in course: $ARGUMENTS in markdown file. Example: /holy-fuck 123 will replace $ARGUMENTS to be 123. That is basic, no idea about $0, $1 right now xD</p><p>Important: Project skills override personal skills when skill name is duplicated!</p><p>Conclusion: custom command = skill = shortcut for instruction!</p><h4>MCP servers with Claude Code</h4><p>The most common MCP server you often see with Claude code is: <a href="https://github.com/microsoft/playwright-mcp">playwright</a></p><p>Install it by: claude mcp add playwright npx @playwright/mcp@latest</p><p>Example: while writing this fucking article. I have http://127.0.0.1:8000/ is opening</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*I9YTKBfnodNptxMR.png" /></figure><p>Claude setting for auto allow MCP Playwright</p><pre>{<br>  &quot;permissions&quot;: {<br>    &quot;allow&quot;: [<br>      &quot;mcp__playwright__browser_navigate&quot;,<br>      &quot;mcp__playwright__browser_snapshot&quot;<br>    ]<br>  }<br>}</pre><p>Or even shorter</p><pre>{<br>  &quot;permissions&quot;: {<br>    &quot;allow&quot;: [&quot;mcp__playwright&quot;],<br>    &quot;deny&quot;: []<br>  }<br>}</pre><p>Image conclusion (taken from the course), pretty helpful to understand how it works!</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*QbHWlDs5rTcSBqLD.png" /></figure><h4>MCP Servers explain</h4><p>Hmm, I mentioned MCP Server above but I don’t understand it well enough I guess. So let’s me introduce what I understand about it. MCP Server (Model Context Protocol Server)</p><p>I think it is the best to make a reflection, I want to point to Grafana Data Source.</p><p>What make they are the same?</p><ul><li>They are all standard interface (seems like CNI for K8S? xD), Grafana don’t give a fuck backend is VictoriaMetric or Prometheus, as long as data source plugin follows interface. Same for Claude don’t need to know what is behind MCP Server, only need MCP Server expose right protocol?</li><li>What is right protocol? Read it here: <a href="https://modelcontextprotocol.io/specification/2025-11-25">https://modelcontextprotocol.io/specification/2025-11-25</a>.</li></ul><p>So for MCP Server need expose right format</p><ul><li>Tools — define name of tool, describe what this tool do and JSON schema for input/output. Example Playwright MCP Server define tool <a href="https://playwright.dev/docs/navigations">browser_navigate</a> take param url as string type.</li><li>Message format — All messages in MCP MUST follow the <a href="https://modelcontextprotocol.io/specification/2024-11-05/basic/messages">JSON-RPC 2.0 specification</a>. The protocol defines three types of messages: Request, Responses, Notification</li><li>Transport: <a href="https://modelcontextprotocol.io/specification/2025-06-18/basic/transports">The protocol</a> currently defines two standard transport mechanisms for client-server communication. stdio and Streamable HTTP</li></ul><p>Alright, If you want to deep dive about MCP Server: <a href="https://modelcontextprotocol.io/docs/getting-started/intro">Go here bro</a>. Back to reflection…</p><ul><li>It is all follow plugin architecture. If you want to use new source, install new data source / MCP Server, no need to edit the fucking core.</li><li>Allow query/interact with system ouside via middle interface.</li></ul><p>But here is little different: Grafana data source mostly read (query metric/log/display). While MCP do both way, read (search) and write (click, navigate, create issue, send msg...) So for better reflection, MCP like data source but have ability write-back/ execution action, not read only.</p><p>I hope this reflection will help you to understand essence of MCP Server xD</p><h4>Github Integration</h4><p>For complete detail you can watch here: <a href="https://anthropic.skilljar.com/claude-code-in-action/303240">Github Integration</a></p><p>What I understand, so basically, everytime we call claude in PR/Issue by @claude, it will trigger Github Action workflow (we need to set it up), Claude run with full codebase access --&gt; reply result directly in that fucking issue/pr.</p><p>Playwright MCP is optional, not really required if you don’t need to test UI, no need to setup it then.</p><p>One more thing that worth to mention: auto PR review, everytime we create new fucking PR, Claude automatically review and comment without tag.</p><p>So for complete simple flow:</p><ul><li>Create new issue with description like:</li></ul><pre>@claude Fix this fucking login bug. It doesn&#39;t work</pre><ul><li>Github Action trigger.</li><li>Claude read code, analyze and fix bug.</li><li>Claude reply result or even can create PR fix directly in that issue.</li></ul><p>Note: Permission setup can be little confused or taking time, but only need 1 time setup.</p><h3>Hooks and the SDK</h3><h4>Hooks Introduction</h4><p>Run a command before or after Claude Code does something. Optionally can block Claude’s action.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*k7_uph9Zo0xKBYba.png" /></figure><p>There are some hooks type: PreToolUse (Hooks that run before a tool is called), PostToolUse (Hooks that run after a tool is called)</p><p>Defination, same for skill if can be global, project or project but not gonna commit.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*dYBuwgxn3beMn-Qp.png" /></figure><p>I would give a sample setting for PostToolUse in my Go projects: <a href="https://github.com/BlackMetalz/kienlt-scripts/blob/main/claude-settings/go-project.json#L70">Full in here</a> .</p><pre>&quot;hooks&quot;: {<br>    &quot;PostToolUse&quot;: [<br>      {<br>        &quot;matcher&quot;: &quot;Write|Edit|MultiEdit&quot;,<br>        &quot;hooks&quot;: [<br>          {<br>            &quot;type&quot;: &quot;command&quot;,<br>            &quot;command&quot;: &quot;f=$(jq -r &#39;(.tool_input.file_path // .tool_input.edits[0].file_path // empty)&#39; | grep &#39;\\.go$&#39;); if [ -n \&quot;$f\&quot; ] &amp;&amp; [ -f \&quot;$f\&quot; ]; then (goimports -w \&quot;$f\&quot; 2&gt;/dev/null || gofmt -w \&quot;$f\&quot;) &amp;&amp; (cd \&quot;$(dirname \&quot;$f\&quot;)\&quot; &amp;&amp; go vet ./... 2&gt;&amp;1 | head -10); fi || true&quot;,<br>            &quot;timeout&quot;: 30,<br>            &quot;statusMessage&quot;: &quot;Refining Go code (fmt + vet)...&quot;<br>          }<br>        ]<br>      }<br>    ]<br>  }</pre><p>So it will format (go fmt) and do static analysis tool (go vet). After any of tool in matcher list (Write|Edit|MultiEdit) was used. This can catch issue before Claude tells us it’s finished, giving it a chance to fix them automatically!</p><p>The course did too good to explain, here is why:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*74FVOLSRGyyDH2Ce.png" /></figure><h4>Defining hooks</h4><p>First of all, we need to take a look at <a href="https://code.claude.com/docs/en/tools-reference">tools</a>, you often see this during claude code session. Here is the list of tools in that course mentioned</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*RocNVv7eJRW8KaMG.png" /></figure><p>And here is live action I took from my Claude Code session</p><pre># Fetch tool<br>⏺ Fetch(https://github.com/BlackMetalz/lazy-hole/tree/main/.github/workflows)<br>  ⎿  Received 217.5KB (200 OK)<br>⏺ Fetch(https://raw.githubusercontent.com/BlackMetalz/lazy-hole/main/.github/workflows/release.yaml)<br>  ⎿  Received 1.3KB (200 OK)<br># Bash tool<br>Bash(mkdir -p /Users/kienlt/data/github.com/es-cli/.github/workflows)<br>  ⎿  Done<br>⏺ Bash(swift build -c release 2&gt;&amp;1)<br>  ⎿  Building for production...<br>     [0/4] Write sources<br>     [1/4] Write swift-version--1AB21518FC5DEDBE.txt<br>     … +4 lines (ctrl+o to expand)<br>  ⎿  (timeout 2m)<br># Write tool<br>⏺ Write(.github/workflows/cleanup-releases.yaml)<br>  ⎿  Wrote 29 lines to .github/workflows/cleanup-releases.yaml<br>       1 name: Cleanup Old Releases<br>       2<br>       3 on:<br>       4   workflow_run:<br>       5     workflows: [&quot;Release&quot;]<br>       6     types:<br>       7       - completed<br>       8   workflow_dispatch:<br>       9     inputs:<br>      10       keep_latest:<br>     … +19 lines (ctrl+o to expand)<br># Edit/ MultiEdit tool<br>⏺ Update(.github/workflows/release.yaml)<br>  ⎿  Removed 3 lines<br>      22        with:<br>      23          go-version: &#39;1.26.1&#39;<br>      24<br>      25 -    - name: Run tests<br>      26 -      run: go test -race ./...<br>      27 -<br>      25      - name: Build Linux amd64<br>      26        run: GOOS=linux GOARCH=amd64 go build -ldflags &quot;-s -w -X main.version=${{ github.ref_name }}&quot; -o es-cli-linux-amd64 ./cmd/es-cli</pre><p>And for tool call data which we can think of as payload when working with API. We send it via stdin to our command</p><p>Here is example of tool call data</p><pre>{<br>  &quot;session_id&quot;: &quot;...&quot;,<br>  &quot;transcript_path&quot;: &quot;...&quot;,<br>  &quot;hook_event_name&quot;: &quot;PreToolUse&quot;,<br>  &quot;tool_name&quot;: &quot;Read&quot;,<br>  &quot;tool_input&quot;: {<br>    &quot;file_path&quot;: &quot;~/workspace/code/.env&quot;<br>  }<br>}</pre><h4>Implementing a hook</h4><p>So let’s try to achieve goal which prevent Claude Code read .env file. I watched this course several times and realized that it is only a demo of how PreToolUse works.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*o5ndBM7CqUY1SiPh.png" /></figure><p>But we can make it work by simply setting up in settings.json config. Here is my example to prevent access to file data.ini</p><pre>{<br>  &quot;permissions&quot;: {<br>    &quot;allow&quot;: [],<br>    &quot;deny&quot;: [<br>      &quot;Read(./data.ini)&quot;,<br>      &quot;Read(./data.ini.*)&quot;,<br>      &quot;Read(**/data.ini)&quot;,<br>      &quot;Bash(cat *data.ini*)&quot;,<br>      &quot;Bash(grep *data.ini*)&quot;,<br>      &quot;Bash(head *data.ini*)&quot;,<br>      &quot;Bash(tail *data.ini*)&quot;,<br>      &quot;Bash(less *data.ini*)&quot;,<br>      &quot;Bash(more *data.ini*)&quot;<br>    ]<br>  }<br>}</pre><p>Expected output:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*xp0qWBi0tf1WFOrN.png" /></figure><p>Haha, I was forget the teacher is teaching us how to prevent access to specific file using PreToolUse, not permission.deny via setting.json. Here is an example of use PreToolUse</p><pre>{<br>  &quot;hooks&quot;: {<br>    &quot;PreToolUse&quot;: [<br>      {<br>        &quot;matcher&quot;: &quot;Read&quot;,<br>        &quot;hooks&quot;: [<br>          {<br>            &quot;type&quot;: &quot;command&quot;,<br>            &quot;command&quot;: &quot;f=$(jq -r &#39;.tool_input.file_path&#39;); if echo \&quot;$f\&quot; | grep -qE &#39;(\\.env|data\\.ini)$&#39;; then echo &#39;Blocked: sensitive file&#39; &gt;&amp;2; exit 2; fi&quot;<br>          }<br>        ]<br>      }<br>    ]<br>  }<br>}</pre><p>You might be curious about exit 2. This isn&#39;t just a generic error code — in Claude Code hooks, exit code 2 is a special convention: it <strong>blocks</strong> the action and pipes stderr back to Claude so it knows why it was blocked. Other non-zero codes (1, 3, ...) only show a warning without blocking. So exit 2 is the magic number for &quot;stop, don&#39;t do this&quot;.</p><h4>Useful Hooks</h4><p>Holy fucking shit, in this section I realized I understood it before with my previous Go project.</p><p>Here is an example taken from course</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*ddlc9GWdwWfci6S5.png" /></figure><p>And here is a real example I used for Go project:</p><p><a href="https://github.com/BlackMetalz/kienlt-scripts/blob/main/claude-settings/go-project.json#L71">https://github.com/BlackMetalz/kienlt-scripts/blob/main/claude-settings/go-project.json#L71</a></p><p>And for more useful hooks, please visit the course: <a href="https://anthropic.skilljar.com/claude-code-in-action/312427">https://anthropic.skilljar.com/claude-code-in-action/312427</a></p><h4>Claude Code SDK</h4><p>Ohh, fun section to watch. So as I understand, it is same as using CLI but integrated into your code.</p><ul><li>CLI: claude -p &quot;Why I&#39;m so bad at coding, thinking?&quot;</li><li>With SDK:</li></ul><pre>import anthropic_claude_code as sdk<br><br>response = sdk.query(&quot;Why I&#39;m so bad at coding, thinking?&quot;)</pre><p>So we can make a reflection with k8s or put it in k8s terms:</p><ul><li>CLI = kubectl</li><li>SDK = <a href="https://github.com/kubernetes/client-go">client-go</a> library</li></ul><p>Here is picture taken from the course, really great to explain how SDK works for me xD</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*fMkgMkfGyw1fl0Ac.png" /></figure><h3>Quiz</h3><p>I finished the quiz with 1st try, I personally think I understand 30–40% the course content (I was using claude code a lot before take this course xD)</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*rfACEVg7KEN_NyKa.png" /></figure><h3>Conclusion</h3><p>Learn this course and it won’t waste your time, believe me xD</p><p>Btw, my older brother told me: First, understand what we need AI to do and why. Then choose the right technology and optimize it. Don’t just throw random skills and workflows at AI — instead, open the right doors so AI can easily access context and help us validate whether we’re doing the right thing</p><p>Example: If you need AI to test and fix API endpoints, you could write an MCP Server for that API. The agent can then understand its features and test it easily.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=dd487aa858fc" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Containers Are Just Linux Primitives: Namespaces, Cgroups, OverlayFS, and Podman]]></title>
            <link>https://medium.com/@kienlt.qn/containers-are-just-linux-primitives-namespaces-cgroups-overlayfs-and-podman-4f61d6c78fbc?source=rss-b77f4e43d676------2</link>
            <guid isPermaLink="false">https://medium.com/p/4f61d6c78fbc</guid>
            <category><![CDATA[containers]]></category>
            <category><![CDATA[containerization]]></category>
            <dc:creator><![CDATA[Kienlt]]></dc:creator>
            <pubDate>Wed, 25 Mar 2026 12:38:33 GMT</pubDate>
            <atom:updated>2026-03-25T13:05:11.098Z</atom:updated>
            <content:encoded><![CDATA[<p>Containers Are Just <a href="https://www.youtube.com/watch?v=7CKCWqUkMJ4"><strong>Linux process</strong></a> (or process tree).</p><p>It is a combination of <strong>Linux primitives</strong>. I would call these the <strong>3 common Linux primitives used by containers</strong>:</p><ol><li><strong>Namespaces</strong> for isolation</li><li><strong>Cgroups</strong> for resource control</li><li><strong>A layered root filesystem</strong> for the container’s file view, often implemented with <strong>OverlayFS</strong> (for a simple reflection)</li></ol><p>This article uses <strong>Podman</strong> for the demos, because Podman is close enough to Docker to feel familiar, but transparent enough that we can still see the Linux primitives.</p><h3>0. What is OCI?</h3><p>It has 2 parts: OCI and OCI Runtime</p><h3>OCI</h3><p>From what I understand, <a href="https://opencontainers.org/">Open Container Initiative (OCI)</a> is a <strong>standard</strong> for container. With this standard, an image built with Docker can be run pretty well in Podman or Kubernetes and vice versa.</p><p>Content copy from homepage of OCI:</p><pre>The OCI currently contains three specifications: the Runtime Specification (runtime-spec), the Image Specification (image-spec) and the Distribution Specification (distribution-spec). The Runtime Specification outlines how to run a “filesystem bundle” that is unpacked on disk. At a high-level an OCI implementation would download an OCI Image then unpack that image into an OCI Runtime filesystem bundle. At this point the OCI Runtime Bundle would be run by an OCI Runtime.</pre><h3>OCI Runtime</h3><p>The OCI Runtime is the implementation that communicates with the kernel to expand the image into a container xD</p><h3>1. What exactly is Podman?</h3><p>If you are coming from Docker, you can think of Podman as:</p><ul><li><strong>A daemonless container engine</strong></li><li>It speaks a <strong>Docker-like CLI</strong></li><li>It uses an OCI runtime like <a href="https://github.com/containers/crun"><strong>crun</strong></a> or <a href="https://github.com/opencontainers/runc"><strong>runc</strong></a> underneath</li><li>No need service running in background.</li></ul><p>The word <strong>daemonless</strong> is important!.</p><p>Let’s me show you a quick example but not really related to container, but it’s about daemonless vs daemon xD</p><p>Gitlab CI with Docker:</p><pre>build_docker:<br>  image: docker:latest<br>  services:<br>    - docker:dind          # Required for Docker daemon<br>  script:<br>    - docker build -t image-name .</pre><p>Gitlab CI with Podman:</p><pre>build_podman:<br>  image: quay.io/podman/stable:latest<br>  script:<br>    - podman build -t image-name .</pre><h3>2. The 3 Common Linux Primitives Behind Containers</h3><h3>Primitive 1: Namespaces</h3><p>Isolation is what I understand about namespaces in Linux. It gives a process its <strong>own view</strong> of specific system resources.</p><p>Common namespace types you will meet in containers, detail in <a href="https://www.redhat.com/en/blog/7-linux-namespaces">Redhat blog</a></p><ul><li><strong>PID namespace</strong>: the process gets its own process tree</li><li><strong>NET namespace</strong>: its own network stack, interfaces, routes, ports. For example, a container in a new NET namespace gets its own private network (its own localhost), its own IP address (like 10.88.0.2), and its own virtual network devices (like eth0), completely isolated from the host machine&#39;s networking.</li></ul><p><strong>MNT namespace</strong>: its own mount table</p><ul><li><strong>UTS namespace</strong>: its own hostname</li><li><strong>IPC namespace</strong>: its own shared memory / semaphores</li><li><strong>USER namespace</strong>: user and group ID remapping, especially important for rootless containers</li></ul><p>There are also <strong>cgroup</strong> and <strong>time</strong> namespaces, but the ones above are the types you will encounter most often in container discussions.</p><p>This is why process 12345 on the host can look like <strong>PID 1</strong> inside the container. It is the same underlying kernel, but the process is looking through a different namespace view.</p><h3>Primitive 2: Cgroups</h3><p>While namespaces isolate what a process can <strong>see</strong>. Cgroups control how much of the machine a process can <strong>consume</strong>.</p><p>That includes things like:</p><ul><li>CPU time</li><li>Memory</li><li>Number of processes</li><li>I/O accounting and limits</li></ul><p>Without cgroups, a “container” could still exist as an isolated process, but it would be terrible for multi-tenant systems because one noisy workload could fucked up everything else. That is why resource limit is the fucking important!</p><h3>Primitive 3: A Layered Root Filesystem</h3><p>Every container needs a filesystem view:</p><ul><li>/bin/sh</li><li>/etc</li><li>/usr</li><li>and so on…</li></ul><p>That is the job of the container root filesystem.</p><p>In modern Linux container engines, this filesystem is often built from <strong>image layers</strong> plus a <strong>writable container layer</strong>. A very common implementation is <strong>OverlayFS</strong>.</p><p>That gives us the classic model:</p><pre>lower layers (image) + upper layer (container writes) -&gt; merged view</pre><h3>3. Lab Setup</h3><p>To follow this lab on <strong>Ubuntu 24.04</strong>, install:</p><ol><li><strong>Podman</strong></li><li><strong>jq</strong></li><li><strong>util-linux</strong> for nsenter and related tools</li><li><strong>iproute2</strong> for ip addr</li></ol><pre>apt update<br>apt install -y podman jq util-linux iproute2</pre><p>Quick checks:</p><pre>podman version<br>podman info --format &#39;OCI_Runtime={{.Host.OCIRuntime.Name}} Cgroup_Manager={{.Host.CgroupManager}}  Storage_Driver={{.Store.GraphDriverName}} Rootless={{.Host.Security.Rootless}}&#39;</pre><p>Expected output:</p><pre># Version<br>Client:       Podman Engine<br>Version:      4.9.3<br>API Version:  4.9.3<br>Go Version:   go1.22.2<br>Built:        Thu Jan  1 08:00:00 1970<br>OS/Arch:      linux/amd64<br><br># Info<br>OCI_Runtime=runc <br>Cgroup_Manager=systemd <br>Storage_Driver=overlay <br>Rootless=false</pre><p>For this article, I will use <strong>rootful Podman</strong> in the low-level inspection steps because it makes namespace and filesystem introspection simpler. Podman supports rootless mode too, but we will not talk about it in this article xD</p><h3>4. Step 1: Run One Real Container</h3><p>Let us create a tiny container with very visible limits, it will has fully explained in the next steps:</p><pre>podman run -d --name kienlt-lab \<br>  --hostname kienlt-lab \<br>  --memory 128m \<br>  --cpus 0.5 \<br>  --pids-limit 64 \<br>  docker.io/library/alpine:3.22 \<br>  sleep infinity</pre><p>Check it:</p><pre>podman ps<br># Output<br>CONTAINER ID  IMAGE                          COMMAND         CREATED         STATUS         PORTS       NAMES<br>71d926d3c61c  docker.io/library/alpine:3.22  sleep infinity  13 seconds ago  Up 14 seconds              kienlt-lab<br>podman inspect kienlt-lab --format &#39;{{.State.Status}}&#39;<br># Output<br>running</pre><p>Get the host PID of the container’s main process:</p><pre>PID=$(sudo podman inspect -f &#39;{{.State.Pid}}&#39; kienlt-lab)<br>echo &quot;$PID&quot;<br># Example output:<br># 890812</pre><p>kienlt-lab is <strong>not</strong> a mini-VM. It is just a process (or process tree) on the host. Podman simply created it with a different set of namespaces, cgroups, and mounts. Here is the proof</p><pre># Inside the container<br>root@kienlt-lab-utilities:~# podman exec kienlt-lab ps -ef<br>PID   USER     TIME  COMMAND<br>    1 root      0:00 sleep infinity<br>    6 root      0:00 ps -ef<br>root@kienlt-lab-utilities:~# podman top kienlt-lab<br>USER        PID         PPID        %CPU        ELAPSED          TTY         TIME        COMMAND<br>root        1           0           0.000       6m12.258388147s  ?           0s          sleep infinity <br># On the host<br>root@kienlt-lab-utilities:~# ps -ef|grep sleep<br>root      890812  890800  0 00:50 ?        00:00:00 sleep infinity<br># Or even better grep<br>root@kienlt-lab-utilities:~# ps -fp $(pgrep -f &quot;sleep infinity&quot;)<br>UID          PID    PPID  C STIME TTY          TIME CMD<br>root      890812  890800  0 00:50 ?        00:00:00 sleep infinity</pre><p>Let’s see detail of process 890812</p><pre>root@kienlt-lab-utilities:~# pstree -aps 890812<br>systemd,1 --system --deserialize=81<br>  └─conmon,890800 --api-version 1 -c 71d926d3c61c8c4a9beecc7d1a81a516d247dcf1c541bb71e9a39fb89721b7d5 -u 71d926d3c61c8c4a9beecc7d1a81a516d247dcf1c541bb71e9a39fb89721b7d5 -r /usr/bin/runc -b...<br>      └─sleep,890812 infinity</pre><p>Yes, /usr/bin/runc... the OCI runtime</p><h3>5. Step 2: Prove the Namespace Story</h3><p>List the namespaces associated with that process:</p><pre>lsns -p &quot;$PID&quot;</pre><p>You will typically see namespace with types like time,user,net….</p><pre>root@kienlt-lab-utilities:~# lsns -p &quot;$PID&quot;<br>        NS TYPE   NPROCS    PID USER COMMAND<br>4026531834 time      226      1 root /usr/lib/systemd/systemd --system --deserialize=81<br>4026531837 user      225      1 root /usr/lib/systemd/systemd --system --deserialize=81<br>4026533601 net         1 890812 root sleep infinity<br>4026533660 mnt         1 890812 root sleep infinity<br>4026533661 uts         1 890812 root sleep infinity<br>4026533662 ipc         1 890812 root sleep infinity<br>4026533663 pid         1 890812 root sleep infinity<br>4026533664 cgroup      1 890812 root sleep infinity</pre><p>Enter the container’s namespaces from the host:</p><pre>nsenter -t &quot;$PID&quot; -m -u -i -n -p sh<br># Or shorter way to enter namespace with PID...<br>nsenter -t &quot;$PID&quot; -a sh</pre><p>Now run these commands <strong>inside</strong> that namespace view:</p><pre>hostname<br>ip addr<br>ps -ef<br>mount | head</pre><p>Expected output:</p><pre>root@kienlt-lab-utilities:~# nsenter -t &quot;$PID&quot; -m -u -i -n -p sh<br>/ # ps -ef<br>PID   USER     TIME  COMMAND<br>    1 root      0:00 sleep infinity # This is the main process of container<br>   11 root      0:00 sh # This is the shell I&#39;m using<br>   12 root      0:00 ps -ef # And this is the command I&#39;m running<br>/ # hostname<br>kienlt-lab<br>/ # ip addr<br>1: lo: &lt;LOOPBACK,UP,LOWER_UP&gt; mtu 65536 qdisc noqueue state UNKNOWN qlen 1000<br>    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00<br>    inet 127.0.0.1/8 scope host lo<br>       valid_lft forever preferred_lft forever<br>    inet6 ::1/128 scope host <br>       valid_lft forever preferred_lft forever<br>2: eth0@if38: &lt;BROADCAST,MULTICAST,UP,LOWER_UP,M-DOWN&gt; mtu 1500 qdisc noqueue state UP qlen 1000<br>    link/ether 92:62:ee:30:ce:4d brd ff:ff:ff:ff:ff:ff<br>    inet 10.88.0.2/16 brd 10.88.255.255 scope global eth0<br>       valid_lft forever preferred_lft forever<br>    inet6 fe80::9062:eeff:fe30:ce4d/64 scope link <br>       valid_lft forever preferred_lft forever<br>/ # mount|head<br>overlay on / type overlay (rw,relatime,lowerdir=/var/lib/containers/storage/overlay/l/OZSBEMCB4NRVVXNZV5W5NJS2HM,upperdir=/var/lib/containers/storage/overlay/10fd7e8145ead961bd9f42c8402410974548da353e02474a93e5428d30962c5f/diff,workdir=/var/lib/containers/storage/overlay/10fd7e8145ead961bd9f42c8402410974548da353e02474a93e5428d30962c5f/work,uuid=on,nouserxattr)<br>proc on /proc type proc (rw,nosuid,nodev,noexec,relatime)<br>tmpfs on /dev type tmpfs (rw,nosuid,size=65536k,mode=755,inode64)<br>sysfs on /sys type sysfs (ro,nosuid,nodev,noexec,relatime)<br>devpts on /dev/pts type devpts (rw,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=666)<br>mqueue on /dev/mqueue type mqueue (rw,nosuid,nodev,noexec,relatime)<br>tmpfs on /run/.containerenv type tmpfs (rw,nosuid,nodev,noexec,relatime,size=813152k,mode=755,inode64)<br>tmpfs on /etc/hostname type tmpfs (rw,nosuid,nodev,noexec,relatime,size=813152k,mode=755,inode64)<br>tmpfs on /etc/resolv.conf type tmpfs (rw,nosuid,nodev,noexec,relatime,size=813152k,mode=755,inode64)<br>tmpfs on /etc/hosts type tmpfs (rw,nosuid,nodev,noexec,relatime,size=813152k,mode=755,inode64)<br># xD<br>root@kienlt-lab-utilities:~# ifconfig podman0<br>podman0: flags=4163&lt;UP,BROADCAST,RUNNING,MULTICAST&gt;  mtu 1500<br>        inet 10.88.0.1  netmask 255.255.0.0  broadcast 10.88.255.255<br>        inet6 fe80::50ff:afff:fea1:ed7c  prefixlen 64  scopeid 0x20&lt;link&gt;</pre><p>What you should notice:</p><ul><li>hostname is kienlt-lab, not the host hostname</li><li>the process list is tiny compared with the host</li><li>the network interfaces and routes are container-specific</li><li>the mount table is not the same as the host mount table</li></ul><p>That is namespaces which is able to explain in one line: The process is still on the same kernel, but it sees a different world.</p><h3>6. Step 3: Prove the Cgroup Story</h3><p>We started the container with:</p><ul><li>--memory 128m</li><li>--cpus 0.5</li><li>--pids-limit 64</li></ul><p>Now let us see where those limits live.</p><p>First, inspect the cgroup path:</p><pre>cat /proc/$PID/cgroup</pre><p>Expected output:</p><pre>0::/machine.slice/libpod-71d926d3c61c8c4a9beecc7d1a81a516d247dcf1c541bb71e9a39fb89721b7d5.scope</pre><p>Capture it:</p><pre>CGROUP_REL=$(awk -F: &#39;$1==&quot;0&quot;{print $3}&#39; /proc/$PID/cgroup)<br>CGROUP_DIR=&quot;/sys/fs/cgroup${CGROUP_REL}&quot;<br>echo &quot;$CGROUP_DIR&quot;</pre><p>Now inspect the actual cgroup files:</p><pre>cat &quot;$CGROUP_DIR/memory.max&quot;<br>cat &quot;$CGROUP_DIR/cpu.max&quot;<br>cat &quot;$CGROUP_DIR/pids.max&quot;</pre><p>You should see values that correspond to the limits we gave Podman.</p><p>For example:</p><pre># Not related but you will understand where is the fucking CGROUP_REL comes from!<br>root@kienlt-lab-utilities:~# cat /proc/890812/cgroup<br>0::/machine.slice/libpod-71d926d3c61c8c4a9beecc7d1a81a516d247dcf1c541bb71e9a39fb89721b7d5.scope<br># And then<br>root@kienlt-lab-utilities:~# cat &quot;$CGROUP_DIR/memory.max&quot;<br>134217728<br># Lets calculate xD<br>root@kienlt-lab-utilities:~# bytes=134217728<br>root@kienlt-lab-utilities:~# mib=$((bytes / 1024 / 1024))<br>root@kienlt-lab-utilities:~# echo &quot;${mib} MB&quot;<br>128 MB<br>root@kienlt-lab-utilities:~# cat &quot;$CGROUP_DIR/cpu.max&quot;<br>50000 100000<br>root@kienlt-lab-utilities:~# cat &quot;$CGROUP_DIR/pids.max&quot;<br>64</pre><ul><li>memory.max should be around 134217728 bytes for 128m</li><li>pids.max should be 64</li><li>cpu.max should show a quota/period pair that approximates 0.5 CPU</li></ul><p>Podman did not invent resource limits. It translated your CLI flags into cgroup settings the Linux kernel already understands.</p><p>You can also look at live usage (like command docker stats):</p><pre>podman stats --no-stream kienlt-lab<br>ID            NAME        CPU %       MEM USAGE / LIMIT  MEM %       NET IO          BLOCK IO    PIDS        CPU TIME    AVG CPU %<br>8c68368d643d  kienlt-lab  0.01%       53.25kB / 134.2MB  0.04%       1.626kB / 936B  0B / 0B     1           34.723ms    0.01%</pre><p>Again, the container is not that fucking special. It is just a process tree attached to a cgroup subtree.</p><h3>7. Step 4: Prove the Layered Filesystem Story</h3><p>This section is fucking great for me, I see overlayfs for a lot of fucking time, but I never really understand what the fuck are they!</p><p>Now let us inspect the container filesystem driver:</p><pre>podman inspect kienlt-lab | jq &#39;.[0].GraphDriver&#39;</pre><p>Expected output:</p><pre>{<br>  &quot;Name&quot;: &quot;overlay&quot;,<br>  &quot;Data&quot;: {<br>    &quot;LowerDir&quot;: &quot;/var/lib/containers/storage/overlay/cce92674e98722970ab3fdce76a2566f54db535beeb24f0b4397f070ab5f6987/diff&quot;,<br>    &quot;MergedDir&quot;: &quot;/var/lib/containers/storage/overlay/10fd7e8145ead961bd9f42c8402410974548da353e02474a93e5428d30962c5f/merged&quot;,<br>    &quot;UpperDir&quot;: &quot;/var/lib/containers/storage/overlay/10fd7e8145ead961bd9f42c8402410974548da353e02474a93e5428d30962c5f/diff&quot;,<br>    &quot;WorkDir&quot;: &quot;/var/lib/containers/storage/overlay/10fd7e8145ead961bd9f42c8402410974548da353e02474a93e5428d30962c5f/work&quot;<br>  }<br>}</pre><p>cce9267.... and 10fd7e8145.... are SHA256 of uncompressed image layers?</p><p>That maps directly to OverlayFS concepts:</p><ul><li><strong>LowerDir</strong>: read-only image layers</li><li><strong>UpperDir</strong>: writable layer for this container</li><li><strong>WorkDir</strong>: OverlayFS bookkeeping</li><li><strong>MergedDir</strong>: the final mounted root filesystem the process sees</li></ul><p>Extract those paths:</p><pre>LOWER_DIR=$(podman inspect kienlt-lab | jq -r &#39;.[0].GraphDriver.Data.LowerDir&#39;)<br>UPPER_DIR=$(podman inspect kienlt-lab | jq -r &#39;.[0].GraphDriver.Data.UpperDir&#39;)<br>MERGED_DIR=$(podman inspect kienlt-lab | jq -r &#39;.[0].GraphDriver.Data.MergedDir&#39;)<br><br>echo &quot;LOWER_DIR=$LOWER_DIR&quot;<br>echo &quot;UPPER_DIR=$UPPER_DIR&quot;<br>echo &quot;MERGED_DIR=$MERGED_DIR&quot;</pre><p>Now write a file from inside the container:</p><pre>podman exec kienlt-lab sh -c &#39;echo &quot;hello from upperdir&quot; &gt; /root/hello_overlayfs.txt&#39;</pre><p>Check the merged filesystem view:</p><pre>ls -l &quot;$MERGED_DIR/root/hello_overlayfs.txt&quot;<br>cat &quot;$MERGED_DIR/root/hello_overlayfs.txt&quot;</pre><p>Then inspect the upper layer:</p><pre>ls -l &quot;$UPPER_DIR/root/hello_overlayfs.txt&quot;<br>cat &quot;$UPPER_DIR/root/hello_overlayfs.txt&quot;</pre><p>Output in action:</p><pre>root@kienlt-lab-utilities:~# ls -l &quot;$MERGED_DIR/root/hello_overlayfs.txt&quot;<br>-rw-r--r-- 1 root root 20 Mar 25 01:08 /var/lib/containers/storage/overlay/10fd7e8145ead961bd9f42c8402410974548da353e02474a93e5428d30962c5f/merged/root/hello_overlayfs.txt<br>root@kienlt-lab-utilities:~# cat &quot;$MERGED_DIR/root/hello_overlayfs.txt&quot;<br>hello from upperdir<br>root@kienlt-lab-utilities:~# ls -l &quot;$UPPER_DIR/root/hello_overlayfs.txt&quot;<br>-rw-r--r-- 1 root root 20 Mar 25 01:08 /var/lib/containers/storage/overlay/10fd7e8145ead961bd9f42c8402410974548da353e02474a93e5428d30962c5f/diff/root/hello_overlayfs.txt<br>root@kienlt-lab-utilities:~# cat &quot;$UPPER_DIR/root/hello_overlayfs.txt&quot;<br>hello from upperdir<br># You can see clearly that the file is not in the lowerdir which is read-only. That is why echo to file in container doesn&#39;t show in lowerdir!<br>root@kienlt-lab-utilities:~# cat &quot;$LOWER_DIR/root/hello_overlayfs.txt&quot;<br>cat: /var/lib/containers/storage/overlay/cce92674e98722970ab3fdce76a2566f54db535beeb24f0b4397f070ab5f6987/diff/root/hello_overlayfs.txt: No such file or directory<br>root@kienlt-lab-utilities:~# ls -l &quot;$LOWER_DIR/root/hello_overlayfs.txt&quot;<br>ls: cannot access &#39;/var/lib/containers/storage/overlay/cce92674e98722970ab3fdce76a2566f54db535beeb24f0b4397f070ab5f6987/diff/root/hello_overlayfs.txt&#39;: No such file or directory</pre><p>That file was not baked into the image. It landed in the <strong>writable upper layer</strong> of this specific container.</p><p>A container image is not one giant mutable disk. It is usually a stack of read-only layers plus one writable layer on top.</p><p>A picture for easier imagination from <a href="https://blogs.cisco.com/developer/373-containerimages-03">blogs of cisco</a>:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*UnpDr4Cbd4PUJiK2.png" /></figure><h3>8. Put the 3 Pieces Together</h3><p>The fucking flow:</p><pre>Podman CLI<br>  -&gt; asks an OCI runtime to start a process<br>  -&gt; that process gets new namespaces<br>  -&gt; that process is attached to cgroups<br>  -&gt; that process sees a merged root filesystem<br>  -&gt; done</pre><p>Reason containers are usually lighter than VMs:</p><ul><li>no second kernel</li><li>no hardware virtualization boundary</li><li>no emulated machine</li><li>just a process with isolation, limits, and a filesystem view</li></ul><h3>9. Summary</h3><p>If you want one sentence to remember, make it this: A container is just a Linux process built from namespaces, cgroups, and a container-specific root filesystem.</p><p>So yes, “namespaces + cgroups + OverlayFS” is a useful teaching shortcut or telling your friend xD</p><p>But the most precise version is:</p><ul><li><strong>Namespaces</strong> provide isolation</li><li><strong>Cgroups</strong> provide resource control</li><li><strong>A layered root filesystem</strong> provides the file view</li></ul><h3>12. Cleanup</h3><pre>podman rm -f kienlt-lab</pre><p><strong>References:</strong></p><ul><li><a href="https://www.redhat.com/en/blog/containers-are-linux">Containers are Linux</a></li><li><a href="https://blog.quarkslab.com/digging-into-the-oci-image-specification.html">Digging into the OCI Image Specification</a></li><li><a href="https://docs.docker.com/engine/storage/drivers/overlayfs-driver/">OverlayFS driver</a></li><li><a href="https://blogs.cisco.com/developer/373-containerimages-03">Container images</a></li><li><a href="https://docs.podman.io/en/stable/">Podman: What is Podman?</a></li><li><a href="https://docs.kernel.org/admin-guide/cgroup-v2.html">Linux cgroup v2 documentation</a></li><li><a href="https://docs.kernel.org/filesystems/overlayfs.html">Linux OverlayFS documentation</a></li><li><a href="https://man7.org/linux/man-pages/man7/namespaces.7.html">Linux namespaces overview</a></li><li><a href="https://man7.org/linux/man-pages/man1/nsenter.1.html">Linux </a><a href="https://man7.org/linux/man-pages/man1/nsenter.1.html">nsenter</a></li></ul><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=4f61d6c78fbc" width="1" height="1" alt="">]]></content:encoded>
        </item>
    </channel>
</rss>