<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Vigilant Engineer on Medium]]></title>
        <description><![CDATA[Stories by Vigilant Engineer on Medium]]></description>
        <link>https://medium.com/@shivam2003?source=rss-8c3f1e81861d------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/1*Pa4_kzuXeCB7yomtmzvloA.jpeg</url>
            <title>Stories by Vigilant Engineer on Medium</title>
            <link>https://medium.com/@shivam2003?source=rss-8c3f1e81861d------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Fri, 20 Nov 2026 04:11:45 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@shivam2003/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[The Last-Minute CKA Crash Course: What to Know Before You Start the Exam]]></title>
            <link>https://blog.stackademic.com/the-last-minute-cka-crash-course-what-to-know-before-you-start-the-exam-1cd488630316?source=rss-8c3f1e81861d------2</link>
            <guid isPermaLink="false">https://medium.com/p/1cd488630316</guid>
            <category><![CDATA[cka-exam]]></category>
            <dc:creator><![CDATA[Vigilant Engineer]]></dc:creator>
            <pubDate>Fri, 07 Aug 2026 20:10:28 GMT</pubDate>
            <atom:updated>2026-08-14T06:41:08.932Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*CRiMr5DKQatOPV0wa02a6A.png" /></figure><p><em>A practical, command-first review of the concepts, workflows, exam rules, and small habits that can save you minutes — and marks — on the Certified Kubernetes Administrator exam.</em></p><p>The CKA is not a memory contest. It is a two-hour operations shift compressed into an exam: read a task, enter the correct host and cluster, change live state, verify the result, and move on.</p><p>That changes how you should prepare. Knowing Kubernetes vocabulary helps, but speed comes from repeatable troubleshooting loops, confident terminal work, and verification.</p><p>As of August 2026, the official CKA is an online, remotely proctored, performance-based exam with <strong>15–20 command-line tasks</strong>, a <strong>two-hour limit</strong>, and a <strong>66% passing score</strong>. The current environment uses <strong>Kubernetes v1.35</strong>. The certification is valid for two years. Always recheck the official pages before your appointment because the exam follows new Kubernetes minor releases on a rolling basis.</p><h3>First, know the weight of the exam</h3><p>The current curriculum is divided into five domains:</p><ul><li><strong>Troubleshooting — 30%</strong></li><li><strong>Cluster Architecture, Installation &amp; Configuration — 25%</strong></li><li><strong>Services &amp; Networking — 20%</strong></li><li><strong>Workloads &amp; Scheduling — 15%</strong></li><li><strong>Storage — 10%</strong></li></ul><p>The obvious lesson: troubleshooting, cluster administration, and networking deserve most of your final revision time. Do not spend your last evening memorizing obscure YAML fields while your service-debugging workflow is still slow.</p><h3>The six-line routine for every task</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*3LGEF5alz6zTTu_Hrtborg.png" /></figure><p>Use the same opening routine until it becomes automatic:</p><ol><li>Read the whole task, including the required host, context, namespace, names, and output path.</li><li>SSH to the designated host shown in the task’s infobox.</li><li>Confirm the current context and namespace before changing anything.</li><li>Solve the smallest possible unit of work.</li><li>Verify the requested end state from the API — and, when relevant, from the application path.</li><li>Exit the SSH session and move on.</li></ol><p>Useful orientation commands:</p><pre>hostname<br> kubectl config current-context<br> kubectl config get-contexts<br> kubectl get ns<br> kubectl api-resources</pre><p>The official environment provides kubectl, the k alias, Bash completion, yq, curl, wget, and man pages on each designated SSH host. The base host does not contain those tools. Nested SSH is not supported.</p><h3>1. Troubleshooting: follow the request path</h3><p>Troubleshooting carries the largest weight. Avoid random changes. Start with symptoms and walk through the layers.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*ZEkmhdKyWsWxKA0MdmM7fQ.png" /></figure><h3>A pod is failing</h3><pre>kubectl get pod -n &lt;ns&gt; -o wide<br> kubectl describe pod &lt;pod&gt; -n &lt;ns&gt;<br> kubectl logs &lt;pod&gt; -n &lt;ns&gt; --all-containers<br> kubectl logs &lt;pod&gt; -n &lt;ns&gt; --previous<br> kubectl get events -n &lt;ns&gt; --sort-by=.metadata.creationTimestamp</pre><p>Look for image errors, missing Secrets or ConfigMaps, bad commands or arguments, failed probes, volume-mount errors, scheduling constraints, resource pressure, and restarts. describe explains what Kubernetes tried to do; logs explains what the process did.</p><h3>A Service is unreachable</h3><p>Walk the chain in order:</p><p><strong>Pod labels → Service selector → EndpointSlice → DNS → NetworkPolicy → application port</strong></p><pre>kubectl get pod -n &lt;ns&gt; --show-labels<br> kubectl get svc -n &lt;ns&gt; -o wide<br> kubectl describe svc &lt;svc&gt; -n &lt;ns&gt;<br> kubectl get endpointslice -n &lt;ns&gt; -l kubernetes.io/service-name=&lt;svc&gt;<br> kubectl get networkpolicy -n &lt;ns&gt;</pre><p>Then test from inside the cluster with a temporary pod or an existing workload. Check both name resolution and the actual HTTP/TCP path. A Service with no endpoints is usually a selector/readiness problem, not a CoreDNS problem.</p><h3>A node or cluster component is unhealthy</h3><pre>kubectl get nodes -o wide<br> kubectl describe node &lt;node&gt;<br> sudo systemctl status kubelet<br> sudo journalctl -u kubelet --since &quot;20 min ago&quot;<br> sudo crictl ps -a</pre><p>Know where kubelet configuration, static Pod manifests, kubeconfig files, and control-plane certificates usually live. Be able to recognize failures caused by an expired certificate, broken static-pod manifest, stopped kubelet, container runtime issue, missing CNI, or resource pressure.</p><h3>2. Cluster administration: control the control plane</h3><p>This domain rewards calm, procedural work.</p><p>Be comfortable with:</p><ul><li>kubeadm init, join, token creation, and discovery-token hashes</li><li>control-plane and worker-node upgrades</li><li>draining and uncordoning nodes safely</li><li>etcd snapshots and restores</li><li>highly available control-plane concepts</li><li>RBAC with Roles, ClusterRoles, and bindings</li><li>Helm and Kustomize for installing or updating components</li><li>CNI, CSI, and CRI responsibilities</li><li>CRDs and operators</li></ul><h3>RBAC: test instead of guessing</h3><pre>kubectl auth can-i get pods --as=&lt;user&gt; -n &lt;ns&gt;<br> kubectl create role pod-reader --verb=get,list,watch --resource=pods -n &lt;ns&gt;<br> kubectl create rolebinding read-pods --role=pod-reader --user=&lt;user&gt; -n &lt;ns&gt;</pre><p>Remember the scope rule: a Role is namespaced; a ClusterRole is cluster-scoped but can also be bound inside a namespace.</p><h3>Node maintenance</h3><pre>kubectl cordon &lt;node&gt;<br> kubectl drain &lt;node&gt; --ignore-daemonsets --delete-emptydir-data<br> # perform maintenance<br> kubectl uncordon &lt;node&gt;</pre><p>Read the task carefully before using destructive flags. Verification matters: confirm the node state, workload rescheduling, and component health after the change.</p><h3>Backups and upgrades</h3><p>For etcd, know the snapshot workflow, the endpoint/certificate parameters, and that restore is not complete until the control plane is pointed at the restored data directory and becomes healthy. For kubeadm upgrades, think in stages: inspect the plan, upgrade the control plane, handle kubelet/kubectl packages, drain when required, restart kubelet, verify, and repeat for workers.</p><h3>3. Services and networking: trace traffic, do not stare at YAML</h3><p>You should be able to explain and operate:</p><ul><li>Pod-to-Pod connectivity</li><li>ClusterIP, NodePort, and LoadBalancer Services</li><li>selectors, ports, target ports, and EndpointSlices</li><li>CoreDNS and service discovery</li><li>ingress controllers and Ingress resources</li><li>Gateway API resources and routing</li><li>NetworkPolicy selectors and ingress/egress rules</li></ul><p>NetworkPolicy questions become easier when you ask four things:</p><ol><li>Which pods does the policy select?</li><li>Is the policy about ingress, egress, or both?</li><li>Which peers are allowed?</li><li>Which ports and protocols are allowed?</li></ol><p>Policies are additive. A selected pod can receive or send only the traffic allowed by the combined applicable policies. Also remember that policy enforcement depends on the CNI implementation.</p><p>For Gateway API, understand the relationship between a gateway implementation, GatewayClass, Gateway, and route resources such as HTTPRoute. When a route does not work, inspect acceptance conditions, parent references, listener rules, hostnames, backends, and Service ports.</p><h3>4. Workloads and scheduling: generate, edit, verify</h3><p>Imperative commands save time, especially when used to create a starting manifest:</p><pre>kubectl create deployment web --image=nginx --dry-run=client -o yaml &gt; web.yaml<br> kubectl run test --image=busybox:1.36 --restart=Never --dry-run=client -o yaml &gt; pod.yaml<br> kubectl expose deployment web --port=80 --target-port=8080 --dry-run=client -o yaml &gt; svc.yaml</pre><p>Then edit only what the task requires.</p><p>Practice:</p><ul><li>rolling updates, rollout status, history, and rollback</li><li>ConfigMaps and Secrets as environment variables and volumes</li><li>resource requests and limits</li><li>probes and self-healing behavior</li><li>horizontal pod autoscaling</li><li>node selectors, affinity/anti-affinity, taints, and tolerations</li><li>topology spread constraints</li><li>admission and scheduling behavior</li></ul><pre>kubectl rollout status deployment/&lt;name&gt; -n &lt;ns&gt;<br> kubectl rollout history deployment/&lt;name&gt; -n &lt;ns&gt;<br> kubectl rollout undo deployment/&lt;name&gt; -n &lt;ns&gt;<br> kubectl get pod -n &lt;ns&gt; -o wide</pre><p>Do not stop at “the YAML applied.” Confirm desired replicas, readiness, placement, rollout state, and application behavior.</p><h3>5. Storage: understand the binding story</h3><p>Know the roles of StorageClass, PersistentVolume, PersistentVolumeClaim, and the Pod that mounts the claim.</p><p>You should be able to reason about:</p><ul><li>static versus dynamic provisioning</li><li>access modes</li><li>reclaim policies</li><li>volume modes</li><li>capacity and selector mismatches</li><li>default StorageClasses</li><li>binding modes such as WaitForFirstConsumer</li><li>Pods stuck in Pending because storage cannot bind or mount</li></ul><pre>kubectl get sc,pv,pvc -A<br> kubectl describe pvc &lt;pvc&gt; -n &lt;ns&gt;<br> kubectl describe pv &lt;pv&gt;<br> kubectl describe pod &lt;pod&gt; -n &lt;ns&gt;</pre><p>When a claim is pending, compare its requested capacity, access modes, storage class, selector, and binding mode with the available provisioning path.</p><h3>Terminal speed that is actually worth practising</h3><p>The goal is not a giant alias file. It is reducing friction in the commands you use repeatedly.</p><pre>export do=&quot;--dry-run=client -o yaml&quot;<br> alias k=kubectl<br> source &lt;(kubectl completion bash)<br> complete -o default -F __start_kubectl k</pre><p>High-value commands include:</p><pre>kubectl explain deployment.spec.strategy<br> kubectl get &lt;resource&gt; -o yaml<br> kubectl edit &lt;resource&gt; &lt;name&gt;<br> kubectl diff -f file.yaml<br> kubectl apply -f file.yaml<br> kubectl delete -f file.yaml</pre><p>Use kubectl explain when you know the resource but not the field. Use the official documentation when you need a complete example. Avoid typing long manifests from memory.</p><h3>Documentation rules and navigation</h3><p>During the CKA, allowed documentation must be opened from the browser inside the exam VM. The official allowed list includes Kubernetes documentation and blog, Helm documentation, Gateway API documentation for CKA, and task-specific documentation supplied in the Quick Reference box. Searching within Kubernetes documentation is allowed, but opening external search results is not.</p><p>Practise finding pages by concept, not by memorized URL. Useful search phrases include:</p><ul><li>kubectl jsonpath</li><li>kubeadm upgrade</li><li>etcd snapshot restore</li><li>NetworkPolicy default deny</li><li>Gateway API HTTPRoute</li><li>persistent volume reclaim policy</li></ul><p>Do not rely on personal bookmarks, notes, AI tools, external search engines, or another browser window during the exam.</p><h3>Exam-day checklist</h3><p>Before the appointment:</p><ul><li>Run the PSI system check and remote testing tutorial.</li><li>Prefer a personal computer on which you can install the secure browser and stop prohibited processes.</li><li>Use one active monitor; dual monitors are not supported.</li><li>Plug in the laptop and use a stable internet connection — wired if possible.</li><li>Check webcam, microphone, WebRTC access, and required macOS permissions.</li><li>Clear the desk, walls, and area under the desk; no paper or writing tools.</li><li>Keep a valid, original, unexpired government-issued photo ID ready. The name must match your exam checklist.</li><li>Close other applications and browser windows.</li></ul><p>Inside the exam UI:</p><ul><li>Do not reboot the base host.</li><li>Use Ctrl+Shift+C and Ctrl+Shift+V in the Linux terminal.</li><li>Use Ctrl+Alt+W instead of Ctrl+W, which can close a browser tab.</li><li>Use i to enter insert mode in Vim; the Insert key is prohibited.</li><li>Resize the content panel and browser to create enough terminal space.</li><li>Use the built-in notepad only; making notes elsewhere is prohibited.</li></ul><h3>A final 24-hour plan</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*hTQ264RreZ5UMc_rwkvbiw.png" /></figure><p>Do not try to learn a new Kubernetes subsystem the night before. Use the final day to make known workflows automatic.</p><ol><li>Run one timed simulator session in the actual exam style.</li><li>Review every error, especially wrong contexts, namespaces, names, ports, and output paths.</li><li>Drill one pod failure, one broken Service, one RBAC task, one node issue, one rollout, and one PVC issue.</li><li>Practise documentation lookup inside the permitted sites.</li><li>Rehearse the PSI UI, system check, and room setup.</li><li>Stop early enough to sleep properly.</li></ol><p>The best last-minute mindset is simple: <strong>read precisely, change minimally, verify relentlessly</strong>.</p><p>You do not need to remember all of Kubernetes. You need to recognize the layer that is broken, reach the right tool quickly, and leave the cluster in the exact state the task requests.</p><h3>Official references</h3><ul><li><a href="https://training.linuxfoundation.org/certification/certified-kubernetes-administrator-cka/">Linux Foundation: CKA certification and current curriculum</a></li><li><a href="https://docs.linuxfoundation.org/tc-docs/certification/tips-cka-and-ckad">Linux Foundation: Important instructions for CKA and CKAD</a></li><li><a href="https://docs.linuxfoundation.org/tc-docs/certification/certification-resources-allowed">Linux Foundation: Resources allowed during the exam</a></li><li><a href="https://docs.linuxfoundation.org/tc-docs/certification/faq-cka-ckad-cks">Linux Foundation: CKA/CKAD/CKS FAQ</a></li><li><a href="https://github.com/cncf/curriculum">CNCF: Open-source certification curriculum</a></li></ul><p><em>Exam details in this article were checked on August 8, 2026. Recheck the official pages before your appointment because versions and policies can change.</em></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=1cd488630316" width="1" height="1" alt=""><hr><p><a href="https://blog.stackademic.com/the-last-minute-cka-crash-course-what-to-know-before-you-start-the-exam-1cd488630316">The Last-Minute CKA Crash Course: What to Know Before You Start the Exam</a> was originally published in <a href="https://blog.stackademic.com/">Stackademic</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[20 Generative AI Concepts Everyone Should Know in 2026]]></title>
            <link>https://blog.stackademic.com/20-generative-ai-concepts-everyone-should-know-in-2026-b6beaa209639?source=rss-8c3f1e81861d------2</link>
            <guid isPermaLink="false">https://medium.com/p/b6beaa209639</guid>
            <category><![CDATA[genai]]></category>
            <category><![CDATA[rags]]></category>
            <category><![CDATA[ai]]></category>
            <category><![CDATA[mcps]]></category>
            <dc:creator><![CDATA[Vigilant Engineer]]></dc:creator>
            <pubDate>Fri, 07 Aug 2026 19:31:30 GMT</pubDate>
            <atom:updated>2026-08-14T06:41:04.024Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*ZoFWfyoS4J7VUVJf4w1ZkQ.png" /></figure><p><em>A plain-English guide to the ideas behind reasoning models, multimodal AI, RAG, agents, MCP, LoRA, and the systems people are actually building now.</em></p><p>Generative AI is no longer just “a chatbot that writes.” It can search approved knowledge, inspect images, call tools, draft code, create media, and carry a task across several steps.</p><p>That makes the vocabulary more important. These are the 20 concepts that explain how modern GenAI products work — and where their limits still are.</p><h3>1. Generative AI</h3><p><strong>Generative AI</strong> creates new output: text, code, images, audio, video, or structured data. It learns patterns from existing data and generates a plausible continuation or new sample from those patterns.</p><p>The key word is <em>plausible</em>. A polished answer can still be wrong, incomplete, or unsafe for the task at hand.</p><h3>2. Foundation models</h3><p>A <strong>foundation model</strong> is a large, broadly trained model that can be adapted to many different tasks. Rather than training a new model for every job, teams start with a capable base model and add prompting, retrieval, tools, or targeted tuning.</p><p>This is why one model can summarize documents, answer questions, write code, and analyse images: it is a general base, not a single-purpose program.</p><h3>3. Large language models (LLMs)</h3><p>An <strong>LLM</strong> is a foundation model built to work with language. At a basic level, it predicts the most useful next token from the text and context it receives.</p><p>That prediction task produces surprisingly broad capabilities — drafting, extraction, translation, classification, and explanation — but it does not guarantee factual knowledge, intent, or access to your private systems.</p><h3>4. Transformers and attention</h3><p>Most modern LLMs use a <strong>Transformer</strong> architecture. Its key mechanism, <strong>attention</strong>, helps the model decide which parts of the input matter most when producing the next output.</p><p>You do not need the mathematics to use the idea: attention is why a model can connect a question at the end of a long prompt to a detail near the beginning — although it can still miss or misinterpret that detail.</p><h3>5. Tokens and tokenization</h3><p>Models process <strong>tokens</strong>, not necessarily whole words. A token may be a word, a piece of a word, punctuation, a code symbol, or an image patch.</p><p>Token count drives three practical things: cost, latency, and how much information fits into a request. Better GenAI design often starts with sending less irrelevant text, not simply buying a larger model.</p><h3>6. Context windows</h3><p>A <strong>context window</strong> is the amount of information a model can consider in one request: your prompt, prior messages, retrieved documents, tool results, and its own output.</p><p>Think of it as a working desk, not a permanent memory. A large desk helps, but dumping every document onto it can bury the relevant evidence. Good systems select, structure, and cite the context they provide.</p><h3>7. Prompts, system instructions, and examples</h3><p>A <strong>prompt</strong> is the task and context you give the model. <strong>System instructions</strong> set higher-level boundaries such as tone, safety rules, and what the assistant must not do. <strong>Few-shot examples</strong> demonstrate the exact input/output pattern you want.</p><p>The best prompt is usually not clever wording. It is a precise specification: audience, goal, source material, output format, constraints, and an explicit instruction to say when evidence is missing.</p><h3>8. Reasoning models and test-time compute</h3><p>Newer GenAI systems can spend additional effort at answer time — breaking down a problem, checking intermediate work, trying alternatives, or using tools before responding. This is often called <strong>reasoning</strong> or <strong>test-time compute</strong>.</p><p>It can improve hard tasks, but it comes with a trade-off: more latency and cost. More “thinking” is not proof of correctness; important answers still need evaluation and independent verification.</p><h3>9. Multimodal models</h3><p><strong>Multimodal</strong> models work across text, images, audio, video, and sometimes files or structured data. They can, for example, read a chart, discuss an image, transcribe a recording, or answer questions over a slide deck.</p><p>This is a major shift because real work is not only text. It also creates new privacy, copyright, and accuracy questions for every uploaded file or recording.</p><h3>10. Diffusion models</h3><p>Many image and video generators use <strong>diffusion</strong>. In simple terms, a model learns how to turn noise into a coherent image by repeatedly refining it toward the requested description.</p><p>That explains a familiar experience: image generation is iterative. You steer composition, style, subject, and constraints through successive prompts or edits rather than expecting the first output to be perfect.</p><h3>11. Embeddings</h3><p>An <strong>embedding</strong> represents the meaning of an item — text, image, audio, product, or record — as a list of numbers. Items with related meaning tend to sit near one another in this mathematical space.</p><p>Embeddings power semantic search. A search for “recover my account” can find a document titled “resetting access,” even when the words do not exactly match.</p><h3>12. Vector databases</h3><p>A <strong>vector database</strong> stores embeddings and quickly finds the closest matches to a new query. It is the retrieval layer behind many “chat with your documents” products.</p><p>It is not a magic truth store. Retrieval quality depends on clean documents, useful chunking, correct permissions, metadata, and a way to keep stale content out of results.</p><h3>13. RAG: retrieval-augmented generation</h3><p><strong>RAG</strong> combines retrieval with generation. First, the system searches approved documents. Then it supplies the most relevant passages to the model so that the answer is grounded in current, organization-specific information.</p><p>Use RAG when the knowledge changes often or must come from a governed source. It is usually a better answer than fine-tuning when the problem is “the model needs today’s policy,” not “the model needs a new behavior.”</p><h3>14. Grounding and citations</h3><p><strong>Grounding</strong> means tying an answer to evidence outside the model’s internal training. In a strong GenAI experience, the answer points to the documents, records, or tool outputs that support it.</p><p>Citations improve reviewability, but they are not decoration. Check that a cited source actually supports the claim, is current, and is visible to the person asking the question.</p><h3>15. Tool calling and function calling</h3><p><strong>Tool calling</strong> lets a model request a structured action: search a database, calculate a value, create a ticket, check inventory, or invoke an API. The model suggests the call; the application validates permissions and executes it.</p><p>This is the bridge between a convincing chat response and useful work. It also means authorization, input validation, logging, and approval boundaries become non-negotiable.</p><h3>16. AI agents and agentic workflows</h3><p>An <strong>agent</strong> can plan a sequence of steps, use tools, inspect results, and adapt its next action toward a goal. An agentic workflow might retrieve a policy, compare it with a request, prepare a draft, and ask a person for final approval.</p><p>An agent is not “a model with a personality.” It is a model embedded in a workflow with memory, tools, state, and constraints. The more authority it has, the more narrowly its access should be scoped.</p><h3>17. MCP: Model Context Protocol</h3><p><strong>MCP</strong> is an open protocol for connecting AI applications to external resources, prompts, and tools. A useful analogy is a standard port: it gives AI clients a consistent way to connect to files, databases, services, and workflows.</p><p>MCP makes integrations easier to reuse, but it does not remove security work. Treat every connected server as a real integration: review what it can read, what it can change, how it authenticates, and what data it exposes.</p><h3>18. Fine-tuning, instruction tuning, and LoRA</h3><p><strong>Fine-tuning</strong> adapts a base model with task-specific examples. <strong>Instruction tuning</strong> helps a model follow a desired task style. <strong>LoRA</strong> (low-rank adaptation) is a parameter-efficient approach that changes a small set of trainable weights rather than retraining everything.</p><p>Use tuning when you need repeatable behavior or a specialized output pattern at scale. Use RAG when you need current facts. Teams often waste time fine-tuning knowledge that should simply be retrieved.</p><h3>19. Inference: latency, throughput, and quantization</h3><p><strong>Inference</strong> is the live use of a model after training. Product teams care about three things: <strong>latency</strong> (how long one response takes), <strong>throughput</strong> (how much work the system handles), and <strong>cost</strong>.</p><p><strong>Quantization</strong> makes a model use lower-precision numbers so it can run with less memory and often lower cost. It can be a powerful optimization, but it must be measured against quality for the actual task — not assumed to be free.</p><h3>20. Evals, hallucinations, and prompt injection</h3><p>An <strong>eval</strong> is a deliberate test set and scoring process for AI behavior. It checks whether a system answers correctly, follows policy, uses tools safely, and fails in an acceptable way.</p><p>Two threats make evals essential. <strong>Hallucinations</strong> are plausible-looking but unsupported outputs. <strong>Prompt injection</strong> is malicious or untrusted content that tries to override a system’s instructions — for example, a webpage telling an agent to reveal data or ignore its rules. Treat untrusted content as data, never as authority.</p><h3>The 30-second GenAI cheat sheet</h3><ul><li><strong>Foundation model / LLM:</strong> the broadly trained base that works with language and other content.</li><li><strong>Tokens / context window:</strong> what the model processes and how much it can keep on its working desk.</li><li><strong>Prompt / examples:</strong> the task specification you give it.</li><li><strong>Embeddings / vector database:</strong> the semantic-search layer that finds related information.</li><li><strong>RAG / grounding:</strong> retrieve approved evidence, then answer with it.</li><li><strong>Tool calling / agents:</strong> let the model take multi-step action through controlled integrations.</li><li><strong>MCP:</strong> a reusable protocol for those AI-to-tool and AI-to-data connections.</li><li><strong>Fine-tuning / LoRA:</strong> adapt behavior; do not use them as a substitute for current information.</li><li><strong>Inference / quantization:</strong> the speed, cost, and hardware side of serving a model.</li><li><strong>Evals / security:</strong> prove the workflow is reliable and resistant to hallucinations and instruction attacks.</li></ul><h3>What matters most</h3><p>The current GenAI stack is not one model and one prompt. It is a system: model selection, retrieved evidence, tool permissions, user experience, evaluation, monitoring, and human accountability.</p><p>For everyday use, start small: use a model to draft, summarize, classify, or search. For anything that changes a record, spends money, exposes private data, or makes a high-impact recommendation, require trustworthy sources, narrow tool permissions, logs, evaluations, and a human approval checkpoint.</p><p>Generative AI is becoming more capable because it can act on more context and more tools. The winning skill is not memorizing buzzwords. It is knowing what each component is allowed to know, do, and decide.</p><h3>References</h3><ul><li><a href="https://developers.google.com/machine-learning/glossary/generative">Google Machine Learning Glossary: Generative AI</a></li><li><a href="https://developers.google.com/machine-learning/glossary">Google Machine Learning Glossary</a></li><li><a href="https://modelcontextprotocol.io/docs/getting-started/intro">Model Context Protocol introduction</a></li><li><a href="https://nvlpubs.nist.gov/nistpubs/ai/NIST.AI.600-1.pdf">NIST Generative AI Profile</a></li></ul><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=b6beaa209639" width="1" height="1" alt=""><hr><p><a href="https://blog.stackademic.com/20-generative-ai-concepts-everyone-should-know-in-2026-b6beaa209639">20 Generative AI Concepts Everyone Should Know in 2026</a> was originally published in <a href="https://blog.stackademic.com/">Stackademic</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[AWS FIS Part 2: Make One Instance Feel Slow — Before Your Customers Do]]></title>
            <link>https://medium.com/@shivam2003/aws-fis-part-2-make-one-instance-feel-slow-before-your-customers-do-dec679d4327e?source=rss-8c3f1e81861d------2</link>
            <guid isPermaLink="false">https://medium.com/p/dec679d4327e</guid>
            <category><![CDATA[aws]]></category>
            <category><![CDATA[cloud-computing]]></category>
            <category><![CDATA[aws-fis]]></category>
            <dc:creator><![CDATA[Vigilant Engineer]]></dc:creator>
            <pubDate>Sat, 01 Aug 2026 19:25:26 GMT</pubDate>
            <atom:updated>2026-08-01T19:25:26.314Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Cv4Sq2gWFNkweqH003TRwA.png" /></figure><p><em>A safety-first, CLI-driven AWS Fault Injection Service game day for testing dependency latency, retries, fallbacks, and recovery signals on Amazon EC2.</em></p><p>In Part 1, we tested whether a service could lose one EC2 instance. That is an important failure mode — but it is not the one that quietly damages user experience most often.</p><p>An instance can stay healthy, pass its load-balancer check, and still make every request feel slow because a dependency has become sluggish. The danger is rarely the extra 150 milliseconds by itself. It is the chain reaction: requests wait, worker threads pile up, retries multiply traffic, queues grow, and an ordinary dependency hiccup turns into a full-service incident.</p><p>This Part 2 experiment answers a narrower and more useful question: <strong>if one staging API instance experiences extra latency while calling a dependency, do our timeouts, retries, fallback paths, alarms, and dashboards behave the way we believe they do?</strong></p><p>We will inject a modest amount of <em>egress</em> latency into 25% of flows from exactly one tagged staging instance, observe the system for two minutes, and retain a report. Nothing here should be copied into production without a reviewed hypothesis, an approved window, and a real stop condition.</p><h3>The game-day contract: decide what success looks like first</h3><p>Do not begin with the fault. Begin with the behavior you expect.</p><p>For this example, write down a hypothesis such as: <strong>adding 150 ms of latency, with 50 ms jitter, to 25% of calls from one API instance to </strong><strong>payments.internal.example should not breach our staging availability SLO. p95 latency may increase, retry volume may rise slightly, and the error-rate alarm must remain </strong><strong>OK.</strong></p><h3>Define pass and abort signals</h3><ul><li><strong>Availability / 5xx rate</strong> — Expect no meaningful regression. Abort if the CloudWatch error alarm enters ALARM.</li><li><strong>p95 endpoint latency</strong> — Expect a bounded increase. Abort if it exceeds the team’s agreed staging threshold.</li><li><strong>Dependency timeout metric</strong> — A small increase can be acceptable. Abort if it grows continuously or triggers a retry storm.</li><li><strong>Queue depth / worker saturation</strong> — Expect it to remain stable or self-correct. Abort if backlog or saturation accelerates.</li><li><strong>Fallback path</strong> — If it is invoked, it must return a safe response. Abort if it fails, returns corrupt data, or hides an error silently.</li></ul><p>This is why controlled latency is useful: it tests the <em>system response</em>, not merely whether the fault command works.</p><h3>Guardrails before you touch AWS FIS</h3><p>Use a dedicated staging account or an explicitly approved non-production environment for your first run. Tag targets narrowly and use a single randomly selected instance.</p><p>This walkthrough assumes all of the following are true:</p><ul><li>The target is an Amazon EC2 instance running Amazon Linux 2/2023, Ubuntu, RHEL 8/9, or CentOS 9.</li><li>It is a Systems Manager managed node with a working SSM Agent and a suitable instance profile.</li><li>The AWS FIS experiment role has been reviewed for this SSM action. For the source-targeted latency document, AWS notes that the experiment role needs ec2:DescribeInstances and ec2:DescribeSubnets in addition to the Systems Manager permissions needed to run the document.</li><li>You have a CloudWatch alarm that is meaningful for the experiment — not an alarm that is already noisy or in INSUFFICIENT_DATA.</li><li>You can explain the dependency hostname, interface, target tags, duration, and stop condition to another engineer.</li></ul><p>Use a positive opt-in tag. Never target every instance that happens to carry a generic Environment=staging tag.</p><pre>aws ec2 describe-instances \<br>  --region ap-south-1 \<br>  --filters \<br>    &#39;Name=tag:ChaosReady,Values=true&#39; \<br>    &#39;Name=tag:Environment,Values=staging&#39; \<br>    &#39;Name=tag:Role,Values=api&#39; \<br>    &#39;Name=instance-state-name,Values=running&#39; \<br>  --query &#39;Reservations[].Instances[].{Id:InstanceId,AZ:Placement.AvailabilityZone,Name:Tags[?Key==`Name`]|[0].Value}&#39; \<br>  --output table</pre><p>If that table shows more — or different — instances than you expected, stop and fix the tags. The next command is not a substitute for checking target resolution.</p><h3>Check the node and the SSM document</h3><p>Pick one ID from the previous output only for validation. AWS FIS will still select the final target according to the template’s COUNT(1) rule.</p><pre>export AWS_REGION=ap-south-1<br>export CHECK_INSTANCE_ID=i-0123456789abcdef0</pre><pre>aws ssm describe-instance-information \<br>  --region &quot;$AWS_REGION&quot; \<br>  --filters &quot;Key=InstanceIds,Values=$CHECK_INSTANCE_ID&quot; \<br>  --query &#39;InstanceInformationList[0].{Ping:PingStatus,Platform:PlatformType,Agent:AgentVersion}&#39; \<br>  --output table</pre><pre>aws ssm describe-document \<br>  --region &quot;$AWS_REGION&quot; \<br>  --name AWSFIS-Run-Network-Latency-Sources \<br>  --query &#39;Document.Parameters&#39; \<br>  --output table</pre><p>The first command should report Online. The second gives you the current public-document parameters in your Region; check them rather than relying on an old screenshot or blog post.</p><p>AWSFIS-Run-Network-Latency-Sources uses Linux traffic control (tc) to add latency and jitter for traffic to or from named sources. It accepts a source such as a DNS name, IP address, CIDR, Availability Zone, S3, or DYNAMODB; it also supports a percentage of flows. Unlike the broader latency document, this lets you test one dependency path instead of slowing all network traffic.</p><h3>Create a deliberately small experiment template</h3><p>Save the following as fis-part2-network-latency.json. Replace every placeholder before running it.</p><pre>{<br>  &quot;description&quot;: &quot;Staging: inject bounded egress latency to one API instance for a dependency-resilience game day&quot;,<br>  &quot;tags&quot;: {<br>    &quot;Name&quot;: &quot;fis-part2-network-latency&quot;,<br>    &quot;Owner&quot;: &quot;platform-team&quot;,<br>    &quot;Environment&quot;: &quot;staging&quot;<br>  },<br>  &quot;targets&quot;: {<br>    &quot;oneStagingApi&quot;: {<br>      &quot;resourceType&quot;: &quot;aws:ec2:instance&quot;,<br>      &quot;resourceTags&quot;: {<br>        &quot;ChaosReady&quot;: &quot;true&quot;,<br>        &quot;Environment&quot;: &quot;staging&quot;,<br>        &quot;Role&quot;: &quot;api&quot;<br>      },<br>      &quot;filters&quot;: [<br>        {<br>          &quot;path&quot;: &quot;State.Name&quot;,<br>          &quot;values&quot;: [&quot;running&quot;]<br>        }<br>      ],<br>      &quot;selectionMode&quot;: &quot;COUNT(1)&quot;<br>    }<br>  },<br>  &quot;actions&quot;: {<br>    &quot;injectDependencyLatency&quot;: {<br>      &quot;actionId&quot;: &quot;aws:ssm:send-command&quot;,<br>      &quot;description&quot;: &quot;Delay 25 percent of egress flows from one API node to the chosen dependency&quot;,<br>      &quot;parameters&quot;: {<br>        &quot;duration&quot;: &quot;PT3M&quot;,<br>        &quot;documentArn&quot;: &quot;arn:aws:ssm:ap-south-1::document/AWSFIS-Run-Network-Latency-Sources&quot;,<br>        &quot;documentParameters&quot;: &quot;{\&quot;Interface\&quot;:\&quot;eth0\&quot;,\&quot;DelayMilliseconds\&quot;:\&quot;150\&quot;,\&quot;JitterMilliseconds\&quot;:\&quot;50\&quot;,\&quot;FlowsPercent\&quot;:\&quot;25\&quot;,\&quot;Sources\&quot;:\&quot;payments.internal.example\&quot;,\&quot;TrafficType\&quot;:\&quot;egress\&quot;,\&quot;DurationSeconds\&quot;:\&quot;120\&quot;,\&quot;InstallDependencies\&quot;:\&quot;True\&quot;}&quot;<br>      },<br>      &quot;targets&quot;: {<br>        &quot;Instances&quot;: &quot;oneStagingApi&quot;<br>      }<br>    }<br>  },<br>  &quot;stopConditions&quot;: [<br>    {<br>      &quot;source&quot;: &quot;aws:cloudwatch:alarm&quot;,<br>      &quot;value&quot;: &quot;arn:aws:cloudwatch:ap-south-1:123456789012:alarm:staging-api-5xx-high&quot;<br>    }<br>  ],<br>  &quot;roleArn&quot;: &quot;arn:aws:iam::123456789012:role/FISNetworkLatencyExperimentRole&quot;,<br>  &quot;experimentOptions&quot;: {<br>    &quot;accountTargeting&quot;: &quot;single-account&quot;,<br>    &quot;emptyTargetResolutionMode&quot;: &quot;fail&quot;<br>  },<br>  &quot;experimentReportConfiguration&quot;: {<br>    &quot;outputs&quot;: {<br>      &quot;s3Configuration&quot;: {<br>        &quot;bucketName&quot;: &quot;my-fis-experiment-reports&quot;,<br>        &quot;prefix&quot;: &quot;network-latency&quot;<br>      }<br>    },<br>    &quot;dataSources&quot;: {<br>      &quot;cloudWatchDashboards&quot;: [<br>        {<br>          &quot;dashboardIdentifier&quot;: &quot;arn:aws:cloudwatch::123456789012:dashboard/staging-api-resilience&quot;<br>        }<br>      ]<br>    },<br>    &quot;preExperimentDuration&quot;: &quot;PT15M&quot;,<br>    &quot;postExperimentDuration&quot;: &quot;PT15M&quot;<br>  }<br>}</pre><p>There are three intentional safety choices here:</p><ol><li><strong>COUNT(1)</strong> constrains the blast radius to one matching instance.</li><li><strong>FlowsPercent: 25</strong> is a gentle first probe. Increase the percentage only after results support it.</li><li><strong>Action duration versus fault duration.</strong> The FIS action is monitored for three minutes, while the SSM document injects latency for 120 seconds. AWS documents these as separate controls; allowing extra action time avoids treating dependency installation or command overhead as part of the fault window.</li></ol><p>Two practical notes:</p><ul><li>Replace eth0 only after confirming the primary interface on your AMI. The document supports DEFAULT too, but an explicit, verified interface makes the experiment easier to review.</li><li>Start with a real, non-public dependency hostname used by your staging application. Do not point the experiment at a made-up name and mistake DNS failure for resilience validation.</li></ul><h3>Review the alarm and create the template</h3><p>Before creation, verify that the exact alarm ARN in the JSON is healthy and that someone is watching the dashboard.</p><pre>aws cloudwatch describe-alarms \<br>  --region &quot;$AWS_REGION&quot; \<br>  --alarm-names staging-api-5xx-high \<br>  --query &#39;MetricAlarms[0].{State:StateValue,Arn:AlarmArn,Reason:StateReason}&#39; \<br>  --output table</pre><pre>aws fis create-experiment-template \<br>  --region &quot;$AWS_REGION&quot; \<br>  --cli-input-json file://fis-part2-network-latency.json</pre><p>Store the returned template ID.</p><pre>export TEMPLATE_ID=EXTxxxxxxxxxxxxxxxxxx</pre><h3>First run: preview targets with no fault injection</h3><p>AWS FIS lets you start an experiment with actionsMode=skip-all. This produces a target preview and verifies that the template resolves the resources you intend, while skipping actions on those resources.</p><pre>aws fis start-experiment \<br>  --region &quot;$AWS_REGION&quot; \<br>  --experiment-template-id &quot;$TEMPLATE_ID&quot; \<br>  --experiment-options actionsMode=skip-all \<br>  --query &#39;experiment.id&#39; \<br>  --output text</pre><p>Open the resulting experiment in the FIS console and inspect the resolved target. Confirm its account, Name tag, Availability Zone, and role. A preview is essential, but it does <strong>not</strong> prove that the experiment role can perform the real fault action — AWS explicitly calls out that limitation.</p><p>If the target is wrong, edit tags or filters and preview again. If it is right, schedule the run, name an observer, and make the stop command visible to everyone involved.</p><h3>Run the experiment and watch the behavior, not just the status</h3><pre>export EXPERIMENT_ID=$(aws fis start-experiment \<br>  --region &quot;$AWS_REGION&quot; \<br>  --experiment-template-id &quot;$TEMPLATE_ID&quot; \<br>  --experiment-options actionsMode=run-all \<br>  --query &#39;experiment.id&#39; \<br>  --output text)</pre><pre>echo &quot;$EXPERIMENT_ID&quot;</pre><p>During the next two minutes, have one person watch the application dashboard and another watch FIS/SSM execution state. The valuable observations are usually these:</p><ul><li>Do dependency calls time out at the expected boundary?</li><li>Are retries bounded, jittered, and visible in metrics?</li><li>Does the instance continue to serve healthy requests without causing connection-pool starvation?</li><li>Does a fallback produce a safe, clearly labelled result?</li><li>Did the expected alarm stay OK, and would the abort alarm actually fire if the service degraded?</li></ul><p>Use the CLI for a concise status check:</p><pre>aws fis get-experiment \<br>  --region &quot;$AWS_REGION&quot; \<br>  --id &quot;$EXPERIMENT_ID&quot; \<br>  --query &#39;experiment.{Status:state.status,Reason:state.reason,Actions:actions}&#39; \<br>  --output json</pre><p>If the abort condition is met — or the team simply loses confidence — stop it manually. A stopped experiment is still a useful experiment.</p><pre>aws fis stop-experiment \<br>  --region &quot;$AWS_REGION&quot; \<br>  --id &quot;$EXPERIMENT_ID&quot;</pre><p>AWS FIS’s pre-configured SSM documents create rollback scripts before fault injection begins. For network latency, the scripts live under /var/lib/amazon/ssm/; rollback activity is also logged to the system journal with the aws-fis-rollback tag. That safety net is valuable, but it is not a reason to skip monitoring.</p><h3>Turn the run into engineering evidence</h3><p>An experiment without evidence becomes a story somebody remembers differently next quarter.</p><p>The report configuration in the template writes a PDF report to S3 and captures the CloudWatch dashboard around the experiment. Make the dashboard include the signals in your hypothesis: endpoint latency, 5xx rate, dependency duration, timeout count, retry count, queue depth, and host saturation. Keep the pre- and post-experiment windows long enough to see both the baseline and recovery.</p><p>Finish the game day with four short answers:</p><ol><li><strong>Did the observed behavior match the hypothesis?</strong></li><li><strong>What customer-visible or operator-visible signal changed first?</strong></li><li><strong>Which safeguard worked, and which one was missing?</strong></li><li><strong>What single follow-up will we ship before raising the experiment intensity?</strong></li></ol><p>Examples of good follow-ups are reducing an overly generous client timeout, capping retries, adding a dependency-timeout metric, annotating the dashboard, or adding a circuit breaker. “Run more chaos” is not a follow-up; it is a wish.</p><h3>Cheat sheet</h3><ul><li><strong>List eligible targets:</strong> aws ec2 describe-instances with ChaosReady=true, Environment=staging, and Role=api filters.</li><li><strong>Confirm SSM health:</strong> aws ssm describe-instance-information --filters &quot;Key=InstanceIds,Values=…&quot;.</li><li><strong>Inspect document parameters:</strong> aws ssm describe-document --name AWSFIS-Run-Network-Latency-Sources.</li><li><strong>Constrain the blast radius:</strong> selectionMode: &quot;COUNT(1)&quot;.</li><li><strong>Apply only partial impact:</strong> FlowsPercent: &quot;25&quot;.</li><li><strong>Target a dependency path:</strong> Sources: &quot;payments.internal.example&quot;, TrafficType: &quot;egress&quot;.</li><li><strong>Preview without faults:</strong> aws fis start-experiment ... --experiment-options actionsMode=skip-all.</li><li><strong>Run the fault:</strong> aws fis start-experiment ... --experiment-options actionsMode=run-all.</li><li><strong>Abort immediately:</strong> aws fis stop-experiment --id &quot;$EXPERIMENT_ID&quot;.</li><li><strong>Check automatic rollback logs:</strong> sudo journalctl -t aws-fis-rollback.</li><li><strong>Preserve evidence:</strong> experimentReportConfiguration → S3 report + CloudWatch dashboard snapshot.</li></ul><h3>The next experiment</h3><p>Once one-instance, partial-flow latency is boring and predictable, change <strong>one variable at a time</strong>: increase affected flows, test packet loss, exercise a different dependency, or use a broader workload-level failure scenario. Every step should retain the same discipline: a measurable hypothesis, a bounded target set, a meaningful abort path, and evidence that the system recovered.</p><p>Reliability is not the claim that nothing will become slow. It is the confidence — earned in controlled conditions — that slow dependencies do not surprise you, your users, or your on-call team.</p><h3>References</h3><ul><li><a href="https://docs.aws.amazon.com/fis/latest/userguide/actions-ssm-agent.html">AWS FIS: use Systems Manager SSM documents</a></li><li><a href="https://docs.aws.amazon.com/fis/latest/userguide/experiment-options.html">AWS FIS experiment options and target previews</a></li><li><a href="https://docs.aws.amazon.com/fis/latest/userguide/experiment-report-configuration.html">AWS FIS experiment report configurations</a></li><li><a href="https://docs.aws.amazon.com/fis/latest/userguide/experiment-template-example.html">AWS FIS example experiment templates</a></li><li><a href="https://docs.aws.amazon.com/cli/latest/reference/fis/create-experiment-template.html">AWS CLI: create-experiment-template</a></li><li><a href="https://docs.aws.amazon.com/cli/latest/reference/fis/start-experiment.html">AWS CLI: start-experiment</a></li></ul><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=dec679d4327e" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Run Your First Safe AWS FIS Experiment: Stop One EC2 Instance, Learn Before an Outage Does]]></title>
            <link>https://medium.com/@shivam2003/run-your-first-safe-aws-fis-experiment-stop-one-ec2-instance-learn-before-an-outage-does-8cb66a765a3c?source=rss-8c3f1e81861d------2</link>
            <guid isPermaLink="false">https://medium.com/p/8cb66a765a3c</guid>
            <category><![CDATA[sre]]></category>
            <category><![CDATA[google]]></category>
            <category><![CDATA[aws]]></category>
            <category><![CDATA[chaos-engineering]]></category>
            <category><![CDATA[cloud-computing]]></category>
            <dc:creator><![CDATA[Vigilant Engineer]]></dc:creator>
            <pubDate>Sat, 01 Aug 2026 19:14:50 GMT</pubDate>
            <atom:updated>2026-08-01T19:14:50.650Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*b4onpvf4pfa2JpFhlqUliA.png" /></figure><p>A command-driven, guardrailed AWS Fault Injection Service walkthrough for testing whether a service can lose one instance without losing customers.</p><blockquote><strong><em>Safety first:</em></strong><em> run this in a non-production account or a deliberately approved, isolated workload. Do not copy the example into a production fleet until an owner has approved the hypothesis, target tag, alarm threshold, and rollback plan.</em></blockquote><p>The most valuable first chaos experiment is often a little boring: stop <strong>one</strong> pre-approved EC2 instance, watch the service, and let AWS FIS start it again after two minutes.</p><p>That small failure answers a useful question: <em>if one node disappears, do requests stay healthy enough for customers?</em> If the answer is no, the experiment has done its job before a real outage gets the chance.</p><p>AWS Fault Injection Service (AWS FIS) turns that question into an experiment template: targets, actions, stop conditions, and an IAM role. The rest of this article builds a safe first template around those parts.</p><h3>The experiment in one sentence</h3><p><strong>Hypothesis:</strong> In staging, the checkout service remains inside its latency and error-budget guardrails when one explicitly tagged EC2 instance is stopped for two minutes.</p><p>This is deliberately narrow:</p><ul><li><strong>Blast radius:</strong> exactly one instance, selected only from Environment=staging and ChaosReady=true.</li><li><strong>Fault:</strong> aws:ec2:stop-instances.</li><li><strong>Duration:</strong> PT2M; FIS starts the stopped instance afterward.</li><li><strong>Abort condition:</strong> a CloudWatch alarm that represents unacceptable customer impact.</li><li><strong>Fail closed:</strong> if FIS resolves no target, the experiment fails rather than silently succeeding.</li></ul><p>It does <em>not</em> prove multi-AZ resilience, database failover, or an Auto Scaling replacement path. Those deserve later, separate hypotheses.</p><h3>Before you touch FIS: write a safety contract</h3><p>Pause if you cannot fill in every item below. Chaos engineering is an exercise in learning, not an authorization shortcut.</p><ul><li><strong>Scope:</strong> a staging account and one non-critical service.</li><li><strong>Owner:</strong> a named application owner and an on-call observer.</li><li><strong>Success signal:</strong> 5xx rate stays below the agreed threshold; p95 latency stays within SLO.</li><li><strong>Stop signal:</strong> a CloudWatch alarm on the customer-impact metric.</li><li><strong>Target:</strong> instances tagged ChaosReady=true and Environment=staging.</li><li><strong>Duration:</strong> two minutes, during a quiet change window.</li><li><strong>Recovery:</strong> startInstancesAfterDuration=PT2M, plus a human who can stop the experiment.</li></ul><p>FIS resolves targets when an experiment starts. A preview is valuable, but tags and fleet membership can change before the real run, so repeat the pre-flight check immediately before injecting the fault.</p><h3>1. Confirm account, region, alarm, and eligible targets</h3><p>Use an explicit profile and region. Replace every placeholder before running a command.</p><pre>export AWS_PROFILE=staging-sre<br>export AWS_REGION=ap-south-1<br>export ALARM_NAME=checkout-p95-latency-high</pre><pre>aws sts get-caller-identity</pre><pre>aws cloudwatch describe-alarms \<br>  --alarm-names &quot;$ALARM_NAME&quot; \<br>  --query &#39;MetricAlarms[0].AlarmArn&#39; \<br>  --output text</pre><pre>aws ec2 describe-instances \<br>  --filters \<br>    &#39;Name=tag:ChaosReady,Values=true&#39; \<br>    &#39;Name=tag:Environment,Values=staging&#39; \<br>    &#39;Name=instance-state-name,Values=running&#39; \<br>  --query &#39;Reservations[].Instances[].{Id:InstanceId,AZ:Placement.AvailabilityZone,Name:Tags[?Key==`Name`]|[0].Value}&#39; \<br>  --output table</pre><p>The alarm must measure a user-relevant steady state: for example, a checkout 5xx rate, a synthetic transaction failure rate, or p95 latency. An instance CPU alarm is usually a weak emergency brake because it can be healthy while the user journey is broken.</p><p>Copy the alarm ARN into ALARM_ARN. If the target command returns anything surprising, stop here and fix the tagging; do not “see what happens.”</p><pre>export ALARM_ARN=&#39;arn:aws:cloudwatch:ap-south-1:123456789012:alarm:checkout-p95-latency-high&#39;<br>export ACCOUNT_ID=&#39;123456789012&#39;<br>export FIS_ROLE_NAME=&#39;FISStopOneStagingInstance&#39;</pre><h3>2. Give FIS a deliberately narrow role</h3><p>FIS assums an experiment role to take action on your behalf. Start with least privilege and keep the role separate from the humans who start experiments.</p><p>Create fis-trust-policy.json:</p><pre>{<br>  &quot;Version&quot;: &quot;2012-10-17&quot;,<br>  &quot;Statement&quot;: [{<br>    &quot;Effect&quot;: &quot;Allow&quot;,<br>    &quot;Principal&quot;: { &quot;Service&quot;: &quot;fis.amazonaws.com&quot; },<br>    &quot;Action&quot;: &quot;sts:AssumeRole&quot;,<br>    &quot;Condition&quot;: {<br>      &quot;StringEquals&quot;: { &quot;aws:SourceAccount&quot;: &quot;123456789012&quot; },<br>      &quot;ArnLike&quot;: { &quot;aws:SourceArn&quot;: &quot;arn:aws:fis:ap-south-1:123456789012:experiment/*&quot; }<br>    }<br>  }]<br>}</pre><p>Create fis-stop-one-policy.json. The tag condition is the important part: even if someone changes the template, this role should only stop or start instances explicitly marked as chaos-ready.</p><pre>{<br>  &quot;Version&quot;: &quot;2012-10-17&quot;,<br>  &quot;Statement&quot;: [<br>    {<br>      &quot;Sid&quot;: &quot;StopAndStartOnlyChaosReadyStagingInstances&quot;,<br>      &quot;Effect&quot;: &quot;Allow&quot;,<br>      &quot;Action&quot;: [&quot;ec2:StopInstances&quot;, &quot;ec2:StartInstances&quot;],<br>      &quot;Resource&quot;: &quot;*&quot;,<br>      &quot;Condition&quot;: {<br>        &quot;StringEquals&quot;: {<br>          &quot;ec2:ResourceTag/ChaosReady&quot;: &quot;true&quot;,<br>          &quot;ec2:ResourceTag/Environment&quot;: &quot;staging&quot;<br>        }<br>      }<br>    },<br>    {<br>      &quot;Sid&quot;: &quot;DescribeInstancesForFIS&quot;,<br>      &quot;Effect&quot;: &quot;Allow&quot;,<br>      &quot;Action&quot;: &quot;ec2:DescribeInstances&quot;,<br>      &quot;Resource&quot;: &quot;*&quot;<br>    }<br>  ]<br>}</pre><p>Then create the role and attach the inline policy:</p><pre>aws iam create-role \<br>  --role-name &quot;$FIS_ROLE_NAME&quot; \<br>  --assume-role-policy-document file://fis-trust-policy.json</pre><pre>aws iam put-role-policy \<br>  --role-name &quot;$FIS_ROLE_NAME&quot; \<br>  --policy-name StopOneTaggedStagingInstance \<br>  --policy-document file://fis-stop-one-policy.json</pre><pre>export FIS_ROLE_ARN=&quot;arn:aws:iam::$ACCOUNT_ID:role/$FIS_ROLE_NAME&quot;</pre><p>If the target has encrypted EBS volumes, FIS can also need permission to create a KMS grant and the KMS key policy must allow it. Treat that as an explicit design review, not a reason to broaden this example’s policy blindly.</p><h3>3. Define the experiment as code</h3><p>Create fis-stop-one.json. The COUNT(1) selection mode is your hard blast-radius boundary. emptyTargetResolutionMode: fail prevents a mistagged fleet from giving you a false-green experiment.</p><pre>{<br>  &quot;description&quot;: &quot;Staging: stop one ChaosReady EC2 instance for two minutes&quot;,<br>  &quot;targets&quot;: {<br>    &quot;oneTaggedStagingInstance&quot;: {<br>      &quot;resourceType&quot;: &quot;aws:ec2:instance&quot;,<br>      &quot;resourceTags&quot;: {<br>        &quot;ChaosReady&quot;: &quot;true&quot;,<br>        &quot;Environment&quot;: &quot;staging&quot;<br>      },<br>      &quot;selectionMode&quot;: &quot;COUNT(1)&quot;<br>    }<br>  },<br>  &quot;actions&quot;: {<br>    &quot;stopOneInstance&quot;: {<br>      &quot;actionId&quot;: &quot;aws:ec2:stop-instances&quot;,<br>      &quot;description&quot;: &quot;Stop one approved instance, then start it after two minutes&quot;,<br>      &quot;parameters&quot;: {<br>        &quot;startInstancesAfterDuration&quot;: &quot;PT2M&quot;<br>      },<br>      &quot;targets&quot;: {<br>        &quot;Instances&quot;: &quot;oneTaggedStagingInstance&quot;<br>      }<br>    }<br>  },<br>  &quot;stopConditions&quot;: [{<br>    &quot;source&quot;: &quot;aws:cloudwatch:alarm&quot;,<br>    &quot;value&quot;: &quot;REPLACE_WITH_ALARM_ARN&quot;<br>  }],<br>  &quot;roleArn&quot;: &quot;REPLACE_WITH_FIS_ROLE_ARN&quot;,<br>  &quot;experimentOptions&quot;: {<br>    &quot;accountTargeting&quot;: &quot;single-account&quot;,<br>    &quot;emptyTargetResolutionMode&quot;: &quot;fail&quot;<br>  },<br>  &quot;tags&quot;: {<br>    &quot;Environment&quot;: &quot;staging&quot;,<br>    &quot;Experiment&quot;: &quot;stop-one-instance&quot;,<br>    &quot;Owner&quot;: &quot;platform-team&quot;<br>  }<br>}</pre><p>Replace the two placeholder strings with $ALARM_ARN and $FIS_ROLE_ARN (or use your configuration-management tool to render the JSON), then create the template:</p><pre>aws fis create-experiment-template \<br>  --cli-input-json file://fis-stop-one.json \<br>  --query &#39;experimentTemplate.{Id:id,Arn:arn}&#39; \<br>  --output table</pre><p>Save the returned ID:</p><pre>export TEMPLATE_ID=&#39;REPLACE_WITH_TEMPLATE_ID&#39;</pre><h3>4. Run a no-fault target preview first</h3><p>skip-all tells FIS to resolve targets and skip all actions. It is a safe rehearsal for checking target selection, logging, and multi-account wiring. It does <strong>not</strong> prove that the experiment role can stop an instance, so it is a review gate—not a permission test.</p><pre>PREVIEW_ID=$(aws fis start-experiment \<br>  --experiment-template-id &quot;$TEMPLATE_ID&quot; \<br>  --experiment-options actionsMode=skip-all \<br>  --tags Phase=target-preview \<br>  --query &#39;experiment.id&#39; \<br>  --output text)</pre><pre>aws fis list-experiment-resolved-targets \<br>  --experiment-id &quot;$PREVIEW_ID&quot; \<br>  --query &#39;resolvedTargets[].{Target:targetName,Details:targetInformation}&#39; \<br>  --output json</pre><p>Read the output with the workload owner. It should contain only the intended staging instance. If it names an unexpected resource — or nothing at all — do not proceed to the real run.</p><h3>5. Run the experiment and observe the service, not just FIS</h3><p>When the owner, observer, alarm, target preview, and change window are all green, start the run. Use a client token so a repeated command does not accidentally create another experiment.</p><pre>EXPERIMENT_ID=$(aws fis start-experiment \<br>  --experiment-template-id &quot;$TEMPLATE_ID&quot; \<br>  --client-token &quot;stop-one-$(date +%s)&quot; \<br>  --tags Change=CHG-1234,Phase=approved-run \<br>  --query &#39;experiment.id&#39; \<br>  --output text)</pre><pre>echo &quot;Experiment: $EXPERIMENT_ID&quot;</pre><pre>aws fis get-experiment \<br>  --id &quot;$EXPERIMENT_ID&quot; \<br>  --query &#39;experiment.{Status:state.status,Reason:state.reason,Actions:actions}&#39; \<br>  --output json</pre><p>During the two minutes, keep three views open:</p><ol><li>The customer-impact dashboard that backs the stop alarm.</li><li>Load balancer / application metrics: error rate, p95 latency, request volume, and saturation.</li><li>The FIS experiment state and the resolved target list.</li></ol><p>If the service leaves the agreed safe range, FIS should stop after the alarm changes state. You can also stop it manually:</p><pre>aws fis stop-experiment --id &quot;$EXPERIMENT_ID&quot;</pre><p>Stopping is not a rewind button. FIS completes any pending post-actions before stopping, and a stopped experiment cannot be resumed. Wait for the final state, verify the instance is healthy again, then check application recovery — not merely that the EC2 instance says running.</p><h3>6. Close the loop: turn the result into engineering work</h3><p>Write down the result while the timeline is fresh:</p><ul><li><strong>Observed:</strong> Did load balance away? Did latency or errors move? Did the alarm fire?</li><li><strong>Expected vs. actual:</strong> What differed from the hypothesis?</li><li><strong>Evidence:</strong> experiment ID, target ID, dashboard links, deployment/change ID, and timestamps.</li><li><strong>Follow-up:</strong> a concrete owner and due date for every gap.</li></ul><p>A successful experiment is not “nothing broke.” It is evidence that your chosen steady-state metric stayed healthy under one precise failure. An unsuccessful one is often more valuable: it gives you a reproducible reliability bug with a clear trigger.</p><h3>AWS FIS cheat sheet</h3><ul><li><strong>Confirm identity:</strong> aws sts get-caller-identity — prevents the wrong-account incident.</li><li><strong>Restrict scope:</strong> resourceTags plus selectionMode: &quot;COUNT(1)&quot; — makes the blast radius explicit.</li><li><strong>Fail on no target:</strong> emptyTargetResolutionMode: &quot;fail&quot; — avoids a false pass.</li><li><strong>Add an emergency brake:</strong> an aws:cloudwatch:alarm stop condition — stops when a service metric crosses your limit.</li><li><strong>Preview targets:</strong> start-experiment --experiment-options actionsMode=skip-all — verifies selection without injecting the fault.</li><li><strong>Inspect selected resources:</strong> list-experiment-resolved-targets — gives the team a human review point.</li><li><strong>Start the real run:</strong> start-experiment --client-token ... — protects against accidental duplicate starts.</li><li><strong>Abort manually:</strong> aws fis stop-experiment --id ... — gives the operator immediate control.</li><li><strong>Collect evidence:</strong> get-experiment plus metrics/dashboard links — turns the exercise into learning.</li></ul><h3>Summary</h3><p>Start small enough that everyone can say “yes” with confidence: one tagged staging instance, two minutes, one customer-focused stop alarm, and one person watching the dashboard. Run a target preview, execute the approved experiment, then record what the system did — not what you hoped it would do.</p><p>That is the practical promise of AWS FIS: make failure intentional, bounded, observable, and useful.</p><h3>References</h3><ul><li>AWS FIS experiment template components: <a href="https://docs.aws.amazon.com/fis/latest/userguide/experiment-templates.html">https://docs.aws.amazon.com/fis/latest/userguide/experiment-templates.html</a></li><li>AWS CLI create-experiment-template: <a href="https://docs.aws.amazon.com/cli/latest/reference/fis/create-experiment-template.html">https://docs.aws.amazon.com/cli/latest/reference/fis/create-experiment-template.html</a></li><li>AWS FIS actions reference (aws:ec2:stop-instances): <a href="https://docs.aws.amazon.com/fis/latest/userguide/fis-actions-reference.html">https://docs.aws.amazon.com/fis/latest/userguide/fis-actions-reference.html</a></li><li>IAM roles for AWS FIS experiments: <a href="https://docs.aws.amazon.com/fis/latest/userguide/getting-started-iam-service-role.html">https://docs.aws.amazon.com/fis/latest/userguide/getting-started-iam-service-role.html</a></li><li>AWS FIS experiment options and target previews: <a href="https://docs.aws.amazon.com/fis/latest/userguide/experiment-options.html">https://docs.aws.amazon.com/fis/latest/userguide/experiment-options.html</a></li><li>AWS FIS stop conditions: <a href="https://docs.aws.amazon.com/fis/latest/userguide/stop-conditions.html">https://docs.aws.amazon.com/fis/latest/userguide/stop-conditions.html</a></li><li>AWS FIS resolved targets: <a href="https://docs.aws.amazon.com/fis/latest/userguide/list-experiment-resolved-targets.html">https://docs.aws.amazon.com/fis/latest/userguide/list-experiment-resolved-targets.html</a></li></ul><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=8cb66a765a3c" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Inside Amazon’s “Day 1” Machine: The 16 Principles That Built an Empire]]></title>
            <link>https://medium.com/@shivam2003/inside-amazons-day-1-machine-the-16-principles-that-built-an-empire-519b4bfd32c7?source=rss-8c3f1e81861d------2</link>
            <guid isPermaLink="false">https://medium.com/p/519b4bfd32c7</guid>
            <category><![CDATA[amazon]]></category>
            <category><![CDATA[jobs]]></category>
            <category><![CDATA[aws]]></category>
            <dc:creator><![CDATA[Vigilant Engineer]]></dc:creator>
            <pubDate>Sat, 18 Jul 2026 11:07:35 GMT</pubDate>
            <atom:updated>2026-07-18T11:07:35.154Z</atom:updated>
            <content:encoded><![CDATA[<h3>What happens when customer obsession, ownership, speed, and relentless standards become more than values — and start running the company?</h3><p><strong>Picture your first Monday at Amazon.</strong></p><p>You walk into a meeting expecting a presentation. Instead, someone hands you a six-page memo. The room goes silent. Everyone reads.</p><p>Then the questions begin.</p><p>What does the customer actually need? What does the data say? Who owns the problem? Is this a one-way door or a two-way door? Are we moving fast enough — or merely rushing?</p><p>Nobody points to a motivational poster. Nobody recites a mission statement. Yet the same invisible logic shapes every challenge, disagreement, and decision in the room.</p><p>That logic is Amazon’s real operating system: its <strong>16 Leadership Principles</strong>.</p><p>They helped turn an online bookstore into a company that reshaped retail, cloud computing, entertainment, logistics, and the very meaning of convenience. But they also created one of the most demanding corporate cultures in modern business.</p><p>To understand Amazon, don’t start with the smile on the box.</p><p>Start inside that meeting.</p><h3>Act I: The empty chair</h3><p>There is a famous idea associated with Amazon meetings: leave an empty chair for the most important person in the room — the customer.</p><p>Whether or not a chair is physically present, the mindset is. Every proposal must eventually answer a brutal question:</p><blockquote><em>How does this make life better for the customer?</em></blockquote><h3>1. Customer Obsession</h3><p>Most companies watch their competitors. Amazon watches customers.</p><p>Leaders are expected to begin with a customer problem and work backwards toward the product, process, or technology that solves it. That difference matters. A competitor-first company asks, “What feature are they launching?” A customer-obsessed company asks, “What frustration still feels normal only because nobody has fixed it?”</p><p>Prime is a perfect expression of that instinct. Customers did not arrive with a detailed specification for a membership program. They simply hated waiting and paying repeatedly for delivery. Amazon treated that irritation as an invitation.</p><p>Customer obsession is the North Star. But a North Star is useless if everyone waits for somebody else to navigate.</p><h3>2. Ownership</h3><p>So the second question arrives: <em>Who owns this?</em></p><p>At Amazon, the desired answer is not a department name. It is: <strong>I do.</strong></p><p>Ownership means thinking beyond your title, team, and quarterly target. If a broken process hurts the company, “that’s not my job” is not an acceptable escape route. An owner fixes the immediate problem, then asks why the system allowed it to happen.</p><p>This is how accountability scales. Jeff Bezos cannot sit in every room. But an ownership mindset can.</p><h3>Act II: The idea nobody understands — yet</h3><p>Now imagine the meeting moves to a strange proposal.</p><p>It is expensive. The market is uncertain. Some people in the room think customers may never want it. The safest option is to improve the existing business and move on.</p><p>Amazon’s next principles are designed for precisely this moment.</p><h3>3. Invent and Simplify</h3><p>Invention is glamorous in hindsight. In the beginning, it usually looks unnecessary, impractical, or misunderstood.</p><p>Amazon asks leaders to search everywhere for ideas, reject the comfort of “not invented here,” and simplify relentlessly. Complexity accumulates quietly: one approval becomes three, one manual report becomes a weekly ritual, one workaround becomes permanent infrastructure.</p><p>The inventor sees that friction and refuses to accept it as normal.</p><p>AWS emerged from Amazon’s own infrastructure challenges. Kindle tried to compress a bookstore into a device. Alexa placed a voice interface in the home. None was an obvious extension of selling books online. Each required Amazon to tolerate being misunderstood.</p><h3>4. Are Right, A Lot</h3><p>But invention cannot become a license for ego.</p><p>“Are Right, A Lot” sounds like a celebration of certainty. Its deeper meaning is almost the opposite. Strong judgment grows from seeking diverse perspectives and actively trying to disprove your own beliefs.</p><p>The best leader in the room is not the person who never changes their mind. It is the person who changes it when the evidence changes — and does so without protecting their pride.</p><h3>5. Learn and Be Curious</h3><p>That intellectual humility leads naturally to curiosity.</p><p>Amazon expects leaders to remain students. They explore unfamiliar technologies, ask uncomfortable questions, and step into subjects where they are not yet experts. Curiosity is not treated as a pleasant personality trait. In a market that keeps changing, it is a survival mechanism.</p><p>The moment a company says, “We already know how this works,” Day 1 begins to fade.</p><h3>Act III: Raising the bar</h3><p>The proposal survives. A team must now build it.</p><p>This is where Amazon’s culture becomes intensely human: who joins the team, what quality means, and how large the ambition is allowed to become.</p><h3>6. Hire and Develop the Best</h3><p>Amazon’s hiring philosophy asks whether every new person can raise the performance bar. The well-known “bar raiser” system is meant to protect that standard beyond the needs of one hiring manager or one urgent vacancy.</p><p>But hiring is only the opening chapter. Leaders must coach people, recognize exceptional talent, and help strong employees move to places where they can create even greater impact.</p><p>A great leader does not build followers. A great leader builds more leaders.</p><h3>7. Insist on the Highest Standards</h3><p>Then comes the uncomfortable part: the work may be good, but good is not always good enough.</p><p>High standards at Amazon are meant to be contagious. Defects should not travel downstream. Recurring problems should not receive recurring patches. The goal is to fix the system so the problem stays fixed.</p><p>This principle explains both Amazon’s reliability and some of its cultural tension. A standard that inspires one person can exhaust another. The line between excellence and relentless pressure is real — and leadership determines which side a team experiences.</p><h3>8. Think Big</h3><p>High standards protect the details. “Think Big” protects the horizon.</p><p>Leaders are asked to communicate a bold direction and search for possibilities beyond the immediate roadmap. Small thinking may feel safer, but it rarely produces a Kindle, Prime, or AWS.</p><p>Thinking big is not the same as making grand speeches. It means seeing how a solution for one team could help a thousand teams — or how an internal capability could become an entirely new business.</p><h3>Act IV: The clock starts ticking</h3><p>The vision is clear. The team is capable. Now reality arrives: limited time, incomplete information, and a deadline that will not negotiate.</p><h3>9. Bias for Action</h3><p>Amazon separates decisions into two kinds of doors.</p><p>A <strong>one-way door</strong> is difficult to reverse, so it deserves careful thought. A <strong>two-way door</strong> is reversible, so waiting for perfect certainty may be more dangerous than making a reasonable decision and learning.</p><p>Bias for Action does not mean recklessness. It means recognizing that speed carries value — and that many decisions can be corrected after you walk through the door.</p><h3>10. Frugality</h3><p>Then someone says, “We do not have enough people, time, or budget.”</p><p>Amazon’s answer is not always to add more. Constraints can force clarity. Frugality asks teams to accomplish more with less, because abundance often hides weak priorities and unnecessary complexity.</p><p>The principle is not about being cheap. It is about making resourcefulness a creative advantage.</p><h3>11. Earn Trust</h3><p>Speed and pressure only work when trust survives them.</p><p>Leaders earn trust by listening carefully, speaking honestly, treating people with respect, and being openly self-critical. They do not hide an uncomfortable metric or pretend a mistake belongs to somebody else.</p><p>Trust is built in small moments: admitting “I was wrong,” giving credit away, delivering on a promise, and telling the truth before the truth becomes convenient.</p><h3>12. Dive Deep</h3><p>Now the dashboard says everything is fine — but one customer story says otherwise.</p><p>Amazon leaders are expected to move comfortably between strategy and detail. They inspect the underlying data, question mismatched signals, and follow anomalies until the real mechanism becomes visible.</p><p>No task is beneath the leader when the truth is buried inside it.</p><h3>13. Have Backbone; Disagree and Commit</h3><p>The room divides.</p><p>One side wants to launch. The other believes the plan is flawed. Amazon expects leaders to challenge decisions respectfully, even when doing so is uncomfortable or socially expensive.</p><p>But debate cannot continue forever. Once the decision is made, everyone commits to making it work.</p><p>This is one of Amazon’s most powerful — and most difficult — ideas: <strong>intellectual honesty before the decision, unity after it.</strong></p><h3>14. Deliver Results</h3><p>Eventually, the memo, debate, and ambition must become an outcome.</p><p>Deliver Results is where every earlier principle faces reality. Leaders focus on the inputs they can control, adapt when setbacks arrive, and finish what matters with the required quality and timing.</p><p>Effort counts. Learning counts. But the customer ultimately experiences the result.</p><h3>Act V: When success becomes responsibility</h3><p>Early Amazon was fighting to survive. Modern Amazon operates at a scale where its decisions affect employees, communities, sellers, the environment, and entire industries.</p><p>That scale forced the operating system to evolve.</p><h3>15. Strive to Be Earth’s Best Employer</h3><p>This principle asks leaders to create a safer, more productive, more diverse, and more just workplace. It recognizes that a company cannot call itself successful if the people producing that success are consistently diminished by the process.</p><p>It is also where Amazon faces its sharpest test. The aspiration must coexist with a culture famous for high pressure, aggressive measurement, and relentless expectations. Writing the principle is easy. Making employees feel it in everyday decisions is the real work.</p><h3>16. Success and Scale Bring Broad Responsibility</h3><p>The final principle widens the lens.</p><p>A garage startup can focus mostly on survival. A global institution cannot. At Amazon’s scale, even unintended consequences can travel far. Leaders are expected to consider what their decisions leave behind and to act with humility about the impact they create.</p><p>The message is simple: when your footprint grows, so does your duty.</p><h3>The machine — and the mirror</h3><p>By the end of that imaginary Monday meeting, the 16 principles no longer look like a list.</p><p>They look like a chain reaction.</p><p><strong>Customer Obsession</strong> identifies the problem. <strong>Ownership</strong> makes someone responsible for it. <strong>Invent and Simplify</strong> creates a new path. <strong>Are Right, A Lot</strong> and <strong>Learn and Be Curious</strong> improve the judgment behind it.</p><p><strong>Hire and Develop the Best</strong> builds the team. <strong>Highest Standards</strong> protects quality. <strong>Think Big</strong> expands the opportunity. <strong>Bias for Action</strong> creates momentum, while <strong>Frugality</strong> keeps the solution disciplined.</p><p><strong>Earn Trust</strong> holds people together. <strong>Dive Deep</strong> finds the truth. <strong>Disagree and Commit</strong> turns conflict into coordinated action. <strong>Deliver Results</strong> closes the loop.</p><p>And the final two principles ask whether winning is enough — if the workplace and the wider world pay too much for the victory.</p><p>That is the power of Amazon’s Leadership Principles. They are not individually revolutionary. Customer focus, curiosity, accountability, and high standards existed long before Amazon.</p><p>The unusual part is how relentlessly the company connects them — and how deeply it embeds them into hiring, meetings, product decisions, performance reviews, and everyday language.</p><p>The result is both a machine and a mirror.</p><p>A machine, because the principles help thousands of teams make decisions without waiting for the founder.</p><p>A mirror, because they force every leader to confront difficult questions:</p><blockquote><em>Am I serving the customer — or protecting my plan?</em></blockquote><blockquote><em>Am I acting like an owner — or waiting for permission?</em></blockquote><blockquote><em>Am I defending a high standard — or simply creating pressure?</em></blockquote><blockquote><em>Am I willing to disagree?</em></blockquote><blockquote><em>And after the debate, can I commit?</em></blockquote><p>Amazon calls it <strong>Day 1</strong>: the belief that the company must keep the urgency, curiosity, and hunger of its first day.</p><p>The real lesson is not that every organization should copy Amazon.</p><p>It is that culture becomes powerful only when it stops being something a company says — and becomes the way people decide what to do next.</p><p><strong>That is when principles stop decorating the walls.</strong></p><p><strong>They start building the future.</strong></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=519b4bfd32c7" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Inside Kimi K3: How a 2.8T-Parameter Model Is Designed to Be Served]]></title>
            <link>https://blog.stackademic.com/inside-kimi-k3-how-a-2-8t-parameter-model-is-designed-to-be-served-58d72250a373?source=rss-8c3f1e81861d------2</link>
            <guid isPermaLink="false">https://medium.com/p/58d72250a373</guid>
            <category><![CDATA[kimi]]></category>
            <category><![CDATA[genai]]></category>
            <category><![CDATA[architecture]]></category>
            <category><![CDATA[ai]]></category>
            <category><![CDATA[kimi-k3]]></category>
            <dc:creator><![CDATA[Vigilant Engineer]]></dc:creator>
            <pubDate>Sat, 18 Jul 2026 06:25:49 GMT</pubDate>
            <atom:updated>2026-08-07T06:44:59.403Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*n8CD8OKzdMBinWaHsh3YJw.png" /></figure><p><em>From 16-of-896 sparse routing to Kimi Delta Attention, MXFP4 quantization and 64-accelerator supernodes — the real story is not scale alone, but the economics that make scale usable.</em></p><blockquote><strong><em>TL;DR:</em></strong><em> The architecture described for Kimi K3 combines a 2.8-trillion-parameter Mixture-of-Experts model, a 1-million-token context window, native vision, mostly-linear attention, low-precision training and large serving supernodes. Only 16 of 896 experts are activated for each token. That separation between total capacity and active compute is the central idea behind the system.</em></blockquote><h3>A note on the claims</h3><p>This article is based on the detailed architecture brief at <a href="https://shivam2003.com/posts/kimi-k3-architecture-and-infrastructure">shivam2003.com</a> and the launch material it cites. The source describes a July 27, 2026 weights release, which is later than the date of this draft. Specifications, benchmark results, cache-hit rates and pricing should therefore be read as announced figures until they can be independently verified.</p><h3>Why this architecture matters</h3><p>“2.8 trillion parameters” is the number that grabs attention. It is not the most important number.</p><p>The harder problem is serving a model of that size without making every request prohibitively expensive. Kimi K3’s reported design attacks cost at several layers at once:</p><ul><li><strong>Sparse computation:</strong> only 16 of 896 experts run for each token.</li><li><strong>Long-context efficiency:</strong> Kimi Delta Attention maintains a compact state instead of an ever-growing full KV cache in most layers.</li><li><strong>Low-precision weights:</strong> quantization-aware training targets MXFP4 weights and MXFP8 activations.</li><li><strong>Scale-up serving:</strong> the model is sharded across a tightly connected supernode with at least 64 accelerators.</li><li><strong>Prefix reuse:</strong> a prefill cache makes repeated agentic prompts dramatically cheaper.</li></ul><p>In other words, K3 is best understood as an <strong>economics design</strong>. The model is large, but every architectural choice tries to reduce how much of that largeness must be paid for on each token.</p><h3>Three concepts that make the rest easy</h3><h3>1. Mixture of Experts: a large model that wakes up only a little</h3><p>A dense transformer evaluates the same feed-forward parameters for every token. A Mixture-of-Experts, or MoE, layer replaces that single feed-forward network with many specialist networks and a router.</p><p>For each token, the router scores the available experts and selects only a small subset. Total parameters describe the model’s capacity; active parameters describe the compute bill. In the reported K3 design, 896 experts hold the capacity while only 16 are selected per token.</p><p>This is powerful, but it creates a new failure mode: <strong>expert collapse</strong>. If a few experts become popular early in training, they receive more data and improve faster, while less-used experts fall further behind. K3’s Stable LatentMoE framework and Quantile Balancing mechanism are intended to keep routing useful and utilization more even.</p><h3>2. The KV cache: why long context becomes expensive</h3><p>Standard attention lets every new token look back at previous tokens. During inference, the system stores key and value vectors for that history in a KV cache so it does not recompute the past.</p><p>The catch is simple: more context means a larger cache. At one million tokens, memory consumption and memory movement can dominate the workload. A huge advertised context window is easy to print on a product page; serving it with acceptable latency and cost is the real engineering challenge.</p><h3>3. Linear attention: a fixed-size memory instead of a full transcript</h3><p>Linear-attention systems compress history into a running state. Think of it as maintaining a continuously revised summary instead of keeping a searchable copy of every previous line.</p><p>The benefit is that state size does not grow with context length. The trade-off is fidelity: compressed memory can forget or blur exact details. Delta-rule attention improves this by writing the correction between what the current state predicts and what the new token actually provides.</p><h3>Kimi K3 and Kimi K2 at a glance</h3><pre>Dimension           Kimi K2                         Kimi K3<br>──────────────────  ──────────────────────────────  ───────────────────────────────<br>Total parameters    About 1T                        2.8T<br>Experts             384 routed + 1 shared           896; 16 active per token<br>Attention           MLA / Kimi Linear lineage       KDA + Gated MLA + AttnRes<br>Context window       256K class                      1M tokens<br>Modality             Text                            Native vision + text<br>Weight format        BF16 / integer variants         MXFP4 weights, MXFP8 activations<br>Scaling efficiency   Baseline                        About 2.5× better (vendor claim)</pre><p>The table reveals the core strategy: increase total capacity aggressively, then use sparsity, compact memory and low precision to keep active cost under control.</p><h3>The MoE design: why 16 of 896 is the headline architecture choice</h3><p>Adding experts increases what the model can store, but not necessarily what it computes for each token. That is the attraction of a very wide and very sparse MoE.</p><p>For the approach to work, the router must send each token to the right specialists while preventing traffic from concentrating on a handful of experts. The architecture brief names three pieces in this stability story:</p><ul><li><strong>Stable LatentMoE</strong> provides the framework for training extremely wide, sparse routing.</li><li><strong>Quantile Balancing</strong> uses the distribution of router scores to balance expert utilization without relying only on a blunt auxiliary penalty.</li><li><strong>Sigmoid Tanh Unit (SiTU)</strong> provides finer gated control over activation flow.</li></ul><p>The useful mental model is this: the knowledge remains distributed across all 896 experts, but the computation for a token visits only 16. The next token can visit a different set.</p><h3>Kimi Delta Attention: the load-bearing choice</h3><p>Kimi Delta Attention, or KDA, is the component that makes the one-million-token context claim economically interesting.</p><p>A KDA layer carries a state matrix S. When a token arrives with key k and value v, a naive linear-attention system would simply add the new association:</p><pre>S ← S + v·kᵀ</pre><p>Repeated additions cause old and new associations to overlap. The delta rule first asks what the state already predicts for the key, then writes only the correction:</p><pre>S ← S − β·(Ŝk − v)·kᵀ</pre><p>If the state already predicts the right value, very little changes. If it is wrong, the error becomes the update. KDA reportedly makes the gate β channel-wise, allowing different feature dimensions to retain or refresh information at different rates.</p><p>That produces three practical benefits:</p><ul><li><strong>Constant-size recurrent state:</strong> most attention layers do not need a per-token cache that grows without bound.</li><li><strong>Fine-grained forgetting:</strong> different channels can preserve information for different durations.</li><li><strong>Hardware-aware execution:</strong> blockwise updates map efficiently to accelerator matrix units.</li></ul><h3>Why K3 still keeps full attention</h3><p>Compressed state is efficient, but it is lossy. It can remember the topic of a long document while still struggling to reproduce an identifier from line 40,000 exactly.</p><p>K3 therefore uses a hybrid design. Most layers reportedly use KDA, while periodic <strong>Gated Multi-head Latent Attention</strong> layers preserve an exact-recall path. The Kimi Linear research lineage used roughly three linear layers for every full-attention layer; the source describes K3 as building on that pattern.</p><p><strong>Attention Residuals</strong> address a second accumulation problem: depth. In a very deep transformer, useful early representations can become buried in a residual stream containing hundreds of later updates. Attention Residuals allow deeper layers to selectively retrieve earlier representations instead of reading only the uniform accumulated sum.</p><h3>The architecture, component by component</h3><pre>Component             Where it acts              Why it matters<br>────────────────────  ─────────────────────────  ─────────────────────────────────<br>Kimi Delta Attention  Most attention layers      Compact state and long-context cost<br>Gated MLA             Periodic attention layers  Exact-recall escape hatch<br>Attention Residuals   Across model depth         Selective access to earlier features<br>Stable LatentMoE      Feed-forward blocks         Stable 896-expert sparse routing<br>Quantile Balancing    Router                      Reduces expert collapse<br>SiTU                  Activations                 Fine-grained gated signal flow<br>Per-Head Muon         Optimizer                   Head-specific matrix updates<br>MXFP4 QAT              Whole model                 Native low-precision serving</pre><p>The pattern is consistent: replace uniform work with selective work. Select experts instead of running them all. Retrieve useful earlier layers instead of accepting an undifferentiated residual sum. Keep a compact memory for most attention, then invoke full attention where exactness matters.</p><h3>Training infrastructure: quantization is part of the model</h3><p>The source says K3 uses quantization-aware training from supervised fine-tuning onward, targeting MXFP4 weights and MXFP8 activations.</p><p>Post-training quantization takes a model trained at higher precision and compresses it afterward. That saves memory but can reduce quality because the model never learned to operate on the lower-precision grid.</p><p>Quantization-aware training puts the constraint into the training loop. The model adapts to the precision in which it will be served. At 2.8 trillion parameters, this is not a cosmetic optimization. BF16 weights would occupy roughly 5.6 TB before overhead; four-bit weights are about 1.4 TB. The difference determines whether the model can fit inside one tightly coupled serving domain.</p><p>That leads to an important shift: the four-bit artifact is not merely a smaller derivative. It is intended to be the primary model.</p><h3>Serving infrastructure: why K3 wants a supernode</h3><p>The architecture brief recommends configurations with 64 or more accelerators connected by a high-bandwidth scale-up fabric. Two constraints explain the shape.</p><p>First, even at four bits, approximately 1.4 TB of weights must be sharded across many devices. Second, MoE routing creates all-to-all communication: every token’s hidden state must reach its selected 16 experts and return. Sparse compute saves arithmetic, but it increases dependence on a fast interconnect.</p><p>If expert traffic crosses a slow boundary, communication can erase the benefit of sparsity. A supernode keeps the exchange inside one low-latency domain.</p><h3>The prefill cache turns architecture into pricing</h3><p>Agentic applications repeatedly send nearly identical prefixes: the same system prompt, tool definitions, repository context and most of the conversation history. A prefill cache stores the already-computed state for that stable prefix, so the next request processes only the new suffix.</p><p>KDA makes those cached states more compact than a full million-token KV cache. The source reports a cache-hit rate above 90% for coding workloads and describes this pricing:</p><pre>Meter                Announced price / 1M tokens   Interpretation<br>───────────────────  ────────────────────────────  ───────────────────────────────<br>Input, cache hit     $0.30                         Reused stable prefix<br>Input, cache miss    $3.00                         Full prefix processing<br>Output               $15.00                        Generated tokens</pre><p>The tenfold hit-to-miss gap is an architectural lesson for developers: <strong>prompt stability is money</strong>.</p><ul><li>Keep system prompts and tool definitions byte-identical across turns.</li><li>Append new history instead of rewriting old history.</li><li>Place volatile per-request data near the end of the prompt.</li><li>Separate reusable repository context from frequently changing instructions.</li><li>Measure cache-hit rate as a product metric, not only an infrastructure metric.</li></ul><h3>What the design enables</h3><p>The combination of native vision and a one-million-token window is aimed at workloads that mix large bodies of text with screenshots, diagrams or frames: whole codebases, document collections, long transcripts, dashboard generation and multi-step autonomous engineering.</p><p>But long context does not automatically mean perfect recall, and benchmark strength does not automatically mean a polished agent. The source itself highlights likely limitations around thinking-history sensitivity, excessive proactiveness and a user-experience gap relative to leading closed systems.</p><p>Those caveats matter. An agentic model is an engine inside a harness. Context management, tool design, permissions, evaluation and recovery behavior can determine whether the same underlying model feels reliable or chaotic.</p><h3>The bigger lesson</h3><p>Kimi K3, as described, is not simply a parameter-count flex. It is an attempt to co-design model architecture, numerical precision, interconnect topology, cache behavior and API pricing.</p><p>The most important ideas are easy to summarize:</p><ul><li><strong>Capacity and compute are separated:</strong> 2.8T parameters exist, but only 16 of 896 experts activate for a token.</li><li><strong>Long context is treated as a systems problem:</strong> KDA limits state growth while periodic full attention preserves an exact-recall path.</li><li><strong>Quantization moves into training:</strong> MXFP4 is designed into the model rather than applied as an afterthought.</li><li><strong>The network is part of the model:</strong> expert parallelism depends on a tightly connected supernode.</li><li><strong>Pricing exposes the infrastructure:</strong> the cache-hit discount rewards applications that keep prefixes stable.</li></ul><blockquote><em>The real achievement is not building a 2.8-trillion-parameter model. It is making a model of that size behave like a product rather than a research exhibit.</em></blockquote><h3>Further reading</h3><ul><li><a href="https://shivam2003.com/posts/kimi-k3-architecture-and-infrastructure">Kimi K3 — Architecture &amp; Infrastructure, Explained</a> — the detailed source brief for this article.</li><li><a href="https://www.kimi.com/blog/kimi-k3">Moonshot AI — Kimi K3 launch post</a> — primary source for the announced specifications and pricing.</li></ul><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=58d72250a373" width="1" height="1" alt=""><hr><p><a href="https://blog.stackademic.com/inside-kimi-k3-how-a-2-8t-parameter-model-is-designed-to-be-served-58d72250a373">Inside Kimi K3: How a 2.8T-Parameter Model Is Designed to Be Served</a> was originally published in <a href="https://blog.stackademic.com/">Stackademic</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Grok Build : Inside xAI’s Architecture for an AI Coding Agent]]></title>
            <link>https://blog.stackademic.com/grok-build-inside-xais-architecture-for-an-ai-coding-agent-aa5e1efd8fea?source=rss-8c3f1e81861d------2</link>
            <guid isPermaLink="false">https://medium.com/p/aa5e1efd8fea</guid>
            <category><![CDATA[elon-musk]]></category>
            <category><![CDATA[xai]]></category>
            <category><![CDATA[ai]]></category>
            <dc:creator><![CDATA[Vigilant Engineer]]></dc:creator>
            <pubDate>Fri, 17 Jul 2026 06:22:30 GMT</pubDate>
            <atom:updated>2026-07-22T08:48:26.296Z</atom:updated>
            <content:encoded><![CDATA[<p><em>A software architect’s tour of </em><em>xai-org/grok-build: ports and adapters, actor-owned state, capability-based tools, rewindable workspaces, and kernel-enforced safety.</em></p><p>Most AI coding products can be sketched as a loop: collect a prompt, call a model, run a tool, append the result, and repeat.</p><p>That description is technically correct for Grok Build — and architecturally misleading.</p><p>The open-source <a href="https://github.com/xai-org/grok-build">xai-org/grok-build</a> repository shows a system designed for a much harder problem: how to let a probabilistic planner operate a developer’s machine without turning the application into a ball of shared state, UI assumptions, and irreversible side effects.</p><p>Its answer is not one clever algorithm. It is a set of reinforcing architectural patterns:</p><ul><li>multiple front ends over one application runtime;</li><li>actors that exclusively own mutable domains;</li><li>an effectively immutable agent assembled by a composition root;</li><li>a late-bound tool registry that doubles as a capability graph;</li><li>a workspace boundary around files, processes, version control, permissions, and checkpoints;</li><li>and defense in depth from tool exposure to policy evaluation to an OS-level sandbox.</li></ul><p>For software architects, that combination is more interesting than any individual feature. It is a concrete blueprint for building an agent that must be interactive, scriptable, extensible, cancellable, resumable, and safe at the same time.</p><h3>First, the architecture in one picture</h3><p>At the repository snapshot examined for this article, the released binary is assembled by xai-grok-pager-bin. The name is historical; the package is explicitly documented as the composition root.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*v0R2p-dmLZ-QYVPp8viOdA.png" /></figure><p>This is not a strict textbook layering diagram; real systems have cross-cutting telemetry, cancellation, configuration, and persistence. But it captures the most important dependency direction: interfaces adapt to sessions, sessions use agents and actors, agents expose tools, and tools reach the machine through the workspace boundary.</p><h3>Pattern 1: The user interface is an adapter, not the application</h3><p>The README describes four operating styles: a full-screen terminal UI, headless execution for scripts and CI, an ACP mode for editor integration, and a long-running leader/server path.</p><p>The composition root imports run_headless, run_leader, and run_stdio_agent from the shell runtime. The TUI lives in xai-grok-pager; rendering primitives are separated again into xai-grok-pager-render. The result is a ports-and-adapters shape: terminal rendering is important, but it is not the owner of the agent’s domain logic.</p><p>This is a subtle but high-value choice.</p><p>If the conversational loop lived inside the TUI, headless mode would become a second implementation. Editor support would become a third. Each mode would slowly diverge in permission behavior, cancellation semantics, model configuration, and tool results.</p><p>Grok Build instead treats each mode as a protocol projection over a shared runtime. Human-readable terminal text, a single JSON result, newline-delimited streaming events, and ACP notifications are different views of the same session.</p><p>For architects, the rule is simple: <strong>an agent interface should translate intent and events; it should not own the agent’s truth.</strong></p><h3>Pattern 2: Build an immutable agent; isolate the mutable machinery</h3><p>xai-grok-agent contains an AgentBuilder that combines an agent definition with session context: working directory, prompt audience, tools, disallowed tools, skills, permission mode, compaction policy, reminders, memory, MCP limits, backends, and subagent settings.</p><p>The built Agent is described in the source as effectively immutable. It contains:</p><ul><li>the parsed definition;</li><li>the structured prompt context;</li><li>a cached rendered system prompt;</li><li>session policies;</li><li>hosted tools;</li><li>and an Arc&lt;ToolBridge&gt;.</li></ul><p>That last field is the escape hatch for operational change. MCP tools may appear or disappear, completion state can evolve, and runtime resources need internal locks. Those mutations happen behind the bridge rather than by rewriting the agent object from many call sites.</p><p>This is a powerful hybrid of two familiar patterns:</p><ol><li><strong>Aggregate snapshot:</strong> the agent is a coherent, inspectable configuration produced once by a builder.</li><li><strong>Mutable gateway:</strong> the volatile tool environment is accessed through a narrow, synchronized object.</li></ol><p>The benefit is temporal consistency. A turn does not have to wonder whether half the prompt reflects one configuration while half the toolset reflects another. When a host intentionally changes tool overrides, it can explicitly re-finalize the prompt.</p><p>The design also hides forked-session implementation paths from the model. A subagent may execute inside an overlay or worktree, while the rendered prompt continues to show the original project path. That is an example of architectural information hiding applied to AI context: the model sees the stable domain identity; the runtime retains the physical execution detail.</p><h3>Pattern 3: Actors own mutable state, handles own communication</h3><p>Two small crates make the concurrency strategy unusually clear.</p><p>xai-chat-state runs a ChatStateActor in its own Tokio task. The actor exclusively owns the conversation, token accounting, sampling configuration, persistence implementation, turn capture, compaction markers, and repair operations. Other components send typed commands through a handle.</p><p>Commands include both mutations and request/reply queries. A query carries a one-shot response channel. Mutations that must be ordered — such as replacing the system head while tool results may be arriving — are performed inside the actor so the read-modify-write sequence cannot race another update.</p><p>xai-grok-sampler uses a related but deliberately different actor shape. Its actor serializes global sampling state and command handling, then spawns a Tokio task per inference request. Multiple model requests can stream concurrently, while registration, cancellation, config updates, and active-request bookkeeping remain serialized.</p><pre>Session ──command──▶ ChatStateActor ──owns──▶ conversation + persistence<br>        ◀─oneshot──</pre><pre>Session ──submit───▶ SamplerActor ──spawns──▶ request task A ─▶ stream<br>                              └─────────────▶ request task B ─▶ stream<br>Session ──cancel(id)────────────────────────▶ cancellation token</pre><p>This is more than “using channels.” The design identifies contention domains and gives each one a single writer.</p><p>The payoff is visible in operations such as history repair. The actor checks whether a turn is active at processing time, not only at the caller, because a caller-side check would race the next turn. In other words, correctness is placed at the serialization point.</p><p>That is the architectural lesson: <strong>put invariants where ordering is guaranteed, not where the request originates.</strong></p><p>There is a trade-off. The code uses unbounded channels in several internal paths. That keeps producers simple and prevents a UI or stream callback from deadlocking on a full queue, but it shifts backpressure responsibility to higher-level rate limits, lifecycle controls, and monitoring. In a long-running multi-session deployment, queue growth deserves explicit telemetry and load testing.</p><h3>Pattern 4: The tool registry is a capability graph, not a switch statement</h3><p>A naive agent implementation dispatches tools with a large match expression:</p><pre>if name == &quot;read_file&quot; ...<br>else if name == &quot;bash&quot; ...<br>else if name == &quot;search&quot; ...</pre><p>Grok Build has a more extensible abstraction. ToolBridge wraps a finalized registry containing tool implementations, schemas, metadata, session resources, name mappings, requirements, and reminders.</p><p>Tools have stable namespaces and semantic kinds. A ToolKind can represent reading, editing, execution, planning, web access, memory, task scheduling, media generation, deployment, or goal updates. The namespace distinguishes built-in dialects and MCP tools.</p><p>The distinction between <strong>tool name</strong> and <strong>tool kind</strong> is architecturally important. Prompts and policies can ask, “Does this agent have a way to edit?” without hard-coding whether the client-facing function is called search_replace, apply_patch, or something supplied through another tool dialect.</p><p>The prompt system is late-bound to the finalized registry. Templates can resolve placeholders such as “the registered tool for this kind.” Therefore, the prompt describes the capabilities the agent actually received, not an aspirational static list.</p><p>The registry also separates two result views:</p><ul><li>a clean structured output for serialization, ACP notifications, and hunk tracking;</li><li>prompt-ready text with cross-cutting reminders appended.</li></ul><p>That separation prevents a common leakage problem: UI/protocol consumers should not be forced to parse model-specific reminder text, while the model still needs those reminders in its context.</p><p>MCP fits naturally because the bridge can register and unregister tools dynamically. The architecture does not need to turn every external tool into a core dependency.</p><p>One especially pragmatic detail is the dedicated xai-grok-mcp crate. Its manifest states that it quarantines a newer rmcp and reqwest dependency family while the rest of the workspace remains on another reqwest version. This is an <strong>anti-corruption layer at the package-manager level</strong>: isolate dependency incompatibility behind a stable internal boundary instead of allowing it to infect the entire graph.</p><h3>Pattern 5: The workspace is an execution plane</h3><p>The xai-grok-workspace crate is not just a filesystem helper. It owns the host-local execution environment:</p><ul><li>filesystem implementations and indexes;</li><li>Git and Jujutsu status/operations;</li><li>terminal/process execution;</li><li>LSP integration;</li><li>hooks;</li><li>permissions;</li><li>session lifecycle;</li><li>events;</li><li>checkpoints and rewind;</li><li>worktrees and foreign sessions;</li><li>remote/client RPC types.</li></ul><p>This boundary resembles a control-plane/data-plane split.</p><p>The model-facing session decides <strong>what</strong> capability to invoke. The workspace decides <strong>how</strong> that capability touches the machine, under which identity, in which session, with which policy, and with which recovery metadata.</p><p>The repository reinforces the boundary with separate xai-grok-workspace-types and xai-grok-workspace-client crates. Requests, events, RPC envelopes, session identities, permission types, file types, tool types, and worktree operations can be shared without importing the entire host implementation.</p><p>That makes local and remote execution variations possible while keeping the domain protocol stable. It also makes testing easier: terminal and filesystem backends are traits, and the source tree includes mock filesystem and test session factories.</p><p>For enterprise agent platforms, this is one of the most reusable ideas in the project: <strong>treat the developer machine as a service boundary, even when everything runs in one process.</strong></p><h3>Pattern 6: Safety is three concentric rings</h3><p>Grok Build does not rely on a single “Are you sure?” dialog. It implements safety at three different layers.</p><h3>Ring 1: Capability exposure</h3><p>An agent can be built with an allowlist or denylist of tools. If shell execution is not in the finalized registry, the model cannot call it. Headless mode also supports removing all subagents or selected subagent types.</p><p>This is the smallest attack surface: do not expose capabilities the task does not need.</p><h3>Ring 2: Semantic permission policy</h3><p>When a tool remains available, permission rules evaluate the specific invocation. The policy supports shell commands, read/edit/search paths, MCP tools, web fetches, and web searches.</p><p>The implementation uses order-independent precedence: deny beats ask, and ask beats allow. It also parses chained shell segments, peels wrappers such as env or timeout, and recursively checks scripts passed to bash -c-style shells. If a restricted script cannot be decomposed, it fails closed to “ask.”</p><p>Another thoughtful rule closes a semantic bypass: a read restriction also governs grep, because grep reads file contents. Policies are organized around the effect, not merely the function name.</p><h3>Ring 3: Kernel-enforced sandbox</h3><p>The xai-grok-sandbox crate applies process-lifetime restrictions using Landlock on Linux and Seatbelt on macOS. Because the boundary is enforced by the operating system, it covers in-process file operations and spawned commands rather than trusting each tool implementation to behave.</p><p>Profiles range from normal workspace writes to read-only and strict modes. Custom profiles can deny credential paths or glob patterns. The sandbox is irreversible after application, and a resumed session retains its original profile rather than silently widening access.</p><p>This is real defense in depth:</p><pre>Model request<br>   │<br>   ▼<br>Is the capability present?        ── no ──▶ impossible to call<br>   │ yes<br>   ▼<br>Does policy allow this invocation?── no ──▶ reject or ask<br>   │ yes<br>   ▼<br>Does the kernel permit the effect? ── no ──▶ OS blocks access</pre><p>The limitations are documented and matter. Child-process network blocking is Linux-specific; on macOS it is a no-op. Built-in HTTP tools run in-process and still require network access. Linux glob-based deny rules are constrained by mount-namespace behavior for files created after startup, whereas macOS can enforce the pattern dynamically. An architect should read these as explicit trust-boundary definitions, not footnotes.</p><h3>Pattern 7: Rewind is a compensating transaction across domains</h3><p>AI agents produce a sequence of side effects, not one database transaction. A turn may edit several files, move Git state, and create tracked hunks before the user decides to rewind.</p><p>Grok Build models rewind at prompt boundaries. A checkpoint can combine:</p><ul><li>filesystem before/after state;</li><li>incremental hunk-tracker deltas;</li><li>optional Git HEAD/index state;</li><li>persistence through a checkpoint store.</li></ul><p>The workspace has one internal turn-boundary fan-out, so the relevant domains observe the same start and end. Restore composes prior deltas, truncates future checkpoints, and re-seeds tracking state.</p><p>This resembles a saga more than ACID. Each domain captures enough information to perform a compensating action, and the workspace coordinates them at a shared boundary.</p><p>Why not simply run git reset? Because not every workspace starts clean, not every file is tracked, not every edit maps neatly to one commit, and the UI needs hunk-level knowledge. A Git-only rollback would confuse version-control state with the complete state of an agent session.</p><p>The broader lesson is important: <strong>agent undo must be designed as a first-class distributed state problem — even inside a desktop process.</strong></p><h3>Pattern 8: The leader process separates interaction lifetime from session lifetime</h3><p>The binary exposes leader management and connection paths in addition to direct TUI and headless execution. The shell contains leader discovery, registration, client capabilities, control commands, WebSocket transport, and explicit shutdown flushing.</p><p>This allows the interactive client to be treated as a view onto a longer-lived runtime. The architecture can support reconnects, editor clients, and coordinated session ownership without forcing inference and session actors to die when one terminal surface disappears.</p><p>Shutdown handling reveals the priority: session actors are asked to flush before the cancellation token tears down their task set. That sequence turns lifecycle management into part of correctness. Persistence is not an afterthought attached to process exit.</p><p>For distributed-system architects, this is familiar territory in miniature: client lifecycle, worker lifecycle, and durable state lifecycle are different clocks.</p><h3>A request lifecycle, end to end</h3><p>Putting the pieces together, a typical turn looks approximately like this:</p><ol><li>A TUI, headless caller, or ACP client submits a prompt.</li><li>The session records turn metadata and sends the user item to the chat-state actor.</li><li>Chat state serializes the mutation and persists according to its journal strategy.</li><li>The session asks the agent for current tool definitions and hosted tools.</li><li>Chat state builds a model request from the conversation, pruning or repairing history when required.</li><li>The sampler actor registers the request and spawns a streaming task.</li><li>Streamed model events return text, reasoning, or tool calls to the session.</li><li>A tool call crosses ToolBridge, where schema, registry metadata, resources, reminders, permissions, and execution backends come into play.</li><li>Tool output is recorded in chat state and sent back into the next model round.</li><li>At turn completion, usage is finalized, persistence is flushed, checkpoint domains are finalized, and presentation adapters project the result.</li></ol><p>Cancellation cuts across this flow. Request tasks have cancellation tokens; terminal control is kept separately from the registry lock so foreground commands can be killed even while a tool call owns registry state. This is a good example of designing locks around cancellation paths rather than adding cancellation after the fact.</p><h3>What architects should challenge</h3><p>Strong architecture is not architecture without costs.</p><h3>The crate graph is expensive</h3><p>The repository contains dozens of focused crates and a generated root Cargo.toml. Boundaries improve isolation, parallel ownership, and build targeting, but navigation and change coordination become harder. The README explicitly recommends per-crate checks because full-workspace builds are slow.</p><h3>The shell remains a large integration surface</h3><p>Even after extracting agent, sampler, chat state, tools, and workspace concerns, xai-grok-shell still coordinates authentication, sessions, configuration, extensions, MCP, persistence, leader transport, and more. That is understandable for an application service layer, but it deserves continual pressure against becoming a new monolith.</p><h3>Unbounded messaging needs operational limits</h3><p>Actors simplify ownership, but unbounded queues can turn a slow consumer into memory pressure. Session quotas, per-stream limits, queue-depth metrics, and overload behavior are as important as race freedom.</p><h3>Platform guarantees are intentionally unequal</h3><p>The sandbox documents differences among Linux, macOS, and Windows. Product language should never flatten those differences into one universal “safe mode.” Architecture documentation, telemetry, and UI should report the guarantee actually active on the host.</p><h3>Open-source shape and product shape are not identical</h3><p>The repository is periodically synchronized from a larger monorepo, and the root manifest is generated. External contributions are not accepted. Architects studying the project should distinguish public boundaries from the internal organizational architecture that produced them.</p><h3>The most reusable design rules</h3><p>If I were extracting a playbook from Grok Build for another agent platform, it would be this:</p><ol><li><strong>Make every UI a protocol adapter.</strong> Keep the turn engine independent of terminal, IDE, and CI presentation.</li><li><strong>Give each mutable domain one owner.</strong> Use typed commands and handles; enforce invariants at the serialization point.</li><li><strong>Build a coherent agent snapshot.</strong> Put volatile integrations behind explicit gateways.</li><li><strong>Model tools semantically.</strong> Separate names, kinds, schemas, requirements, execution resources, and output projections.</li><li><strong>Late-bind prompts to real capabilities.</strong> Never tell the model it has a tool that the session did not receive.</li><li><strong>Treat the local machine as an execution plane.</strong> Files, processes, VCS, permissions, and recovery belong behind a boundary.</li><li><strong>Layer safety.</strong> Capability removal, invocation policy, and kernel enforcement solve different failure modes.</li><li><strong>Design undo before autonomy.</strong> Side-effecting agents need coordinated checkpoints, not only chat history.</li><li><strong>Separate client, worker, and persistence lifetimes.</strong> Reconnection and graceful shutdown are architectural features.</li><li><strong>Isolate dependency conflicts.</strong> A small quarantine crate can be an anti-corruption layer too.</li></ol><h3>Final perspective</h3><p>The most interesting thing about Grok Build is not that it can read files, execute commands, use MCP servers, or render a polished terminal interface. Many agents can do those things.</p><p>The interesting thing is where the repository places authority.</p><p>The UI does not own the session. The model does not own the machine. The agent snapshot does not expose arbitrary mutation. Mutable state belongs to actors. Tool calls pass through a capability registry. Machine effects pass through a workspace. Policy can stop an invocation. The kernel can still stop the effect. Turn boundaries create points for persistence and compensation.</p><p>That is what mature agent architecture looks like: not an attempt to make the model deterministic, but a system that remains understandable when the model is not.</p><p>Source: <a href="https://github.com/xai-org/grok-build">xai-org/grok-build</a>. Architectural observations in this article are based on the public source snapshot at commit 8adf901 (synced July 16, 2026).</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=aa5e1efd8fea" width="1" height="1" alt=""><hr><p><a href="https://blog.stackademic.com/grok-build-inside-xais-architecture-for-an-ai-coding-agent-aa5e1efd8fea">Grok Build : Inside xAI’s Architecture for an AI Coding Agent</a> was originally published in <a href="https://blog.stackademic.com/">Stackademic</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[When Five Fast Machines Became One Fragile Exchange]]></title>
            <link>https://blog.stackademic.com/when-five-fast-machines-became-one-fragile-exchange-422dcee2f072?source=rss-8c3f1e81861d------2</link>
            <guid isPermaLink="false">https://medium.com/p/422dcee2f072</guid>
            <category><![CDATA[cloud-computing]]></category>
            <category><![CDATA[devops]]></category>
            <category><![CDATA[computer-science]]></category>
            <category><![CDATA[aws]]></category>
            <category><![CDATA[system-design-interview]]></category>
            <dc:creator><![CDATA[Vigilant Engineer]]></dc:creator>
            <pubDate>Tue, 14 Jul 2026 07:38:17 GMT</pubDate>
            <atom:updated>2026-07-14T07:38:17.266Z</atom:updated>
            <content:encoded><![CDATA[<p><em>Reverse-engineering Coinbase’s May 7, 2026 outage from cooling loops and CPU caches to Raft quorums, Kafka leadership, order-book fairness, and disaster recovery.</em></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*1ZiBuyHTmfVoaALSQy13iQ.png" /></figure><p>At 7:48 p.m. Eastern time on May 7, 2026, nearly all trading on Coinbase stopped.</p><p>A trader tapping “Buy” did not see a chiller alarm. A market maker did not see three dead virtual machines. An engineer watching a Kafka producer did not see heat collecting above a server inlet. Each person saw a different symptom: an order that would not enter the book, a quote that stopped moving, a deposit that remained pending, a producer that could not append an event.</p><p>The initiating fault was physical. At 7:20 p.m., multiple chiller units failed in one data hall in AWS’s use1-az4 Availability Zone. Temperatures rose. Thermal protection removed power from affected racks. EC2 instances disappeared and EBS volumes on the affected hardware became unavailable. At 9:29 p.m., AWS terminated instances in the cluster placement group that held Coinbase’s matching-engine cluster. Three of five voting nodes went away. The remaining two could not form a majority, so they correctly stopped accepting writes.</p><p>But “a cooling failure took down Coinbase” is not an explanation. It skips the important questions.</p><p>Why did five machines need three acknowledgements? Why were the machines close enough to share a physical failure? Why could the service not simply start three replacements somewhere else? Why did restoring matching-engine quorum at 12:06 a.m. not restore trading? Why could a managed Kafka service block a fee service, which then block quoting? And why is refusing new orders the safe behavior when the market is moving and customers most want to trade?</p><p>This article follows those questions down the stack. The goal is not to summarize Coinbase’s postmortem. It is to build the mental model with which the outage becomes predictable.</p><blockquote><strong><em>Engineering note.</em></strong><em> Public facts in this article come from Coinbase’s June 1 postmortem. Packet traces, internal schemas, precise timeout values, and implementation details that Coinbase did not publish are explicitly presented as models, not claims about proprietary code.</em></blockquote><h3>Part 1 — The outage, minute by minute</h3><p>The most useful timeline separates the physical event, the matching engine, the event backbone, and the customer surface. “Recovered” meant different things for each layer.</p><pre>Time (ET)       Layer                 Event<br>May 7 19:20     Facility              Multiple chillers fail in one data hall,<br>                                      use1-az4. Temperature begins rising.<br>19:20–19:48     EC2/EBS + services    Thermal mitigation impairs instances and<br>                                      volumes; dependencies begin timing out.<br>19:48           Customer surface      Nearly all trading halts. Buying, selling,<br>                                      sends, receives, deposits and withdrawals<br>                                      are unavailable or degraded.<br>19:48–21:29     Application layer     Errors cascade through order routing,<br>                                      quoting, fees, ledger and payment paths.<br>21:29           Matching engine       AWS terminates instances in the placement<br>                                      group. Three of five Raft voters disappear.<br>21:29+          Consensus             Two survivors cannot commit writes. No<br>                                      automated cross-AZ failover exists.<br>Before 00:06    Recovery              Emergency startup change; a node group is<br>                                      created outside the impaired placement group.<br>May 8 00:06     Matching engine       A 3-of-5 quorum is restored.<br>00:06–02:25     Validation            Logs/state synchronize; dependent systems<br>                                      remain blocked, especially managed Kafka.<br>02:25           Market state          Markets reopen in cancel-only mode.<br>03:00           Kafka/MSK             Engineers manually reassign partitions away<br>                                      from impaired brokers.<br>03:49           Market state          Full trading resumes across all books.<br>05:30           Retail surface        Web and mobile trading return degraded.<br>09:30           Kafka/MSK             Priority-zero and priority-one topics recover.<br>09:53           Retail surface        Web and mobile fully recover.<br>14:00           Kafka/MSK             Remaining event-stream backlog clears.</pre><p>The interval from 7:20 to 7:48 matters. A facility does not fail like a Boolean variable. Cooling capacity falls, inlet temperature rises unevenly, fans accelerate, hardware throttles, network paths lose endpoints, retries increase load, and only then do safety systems remove power. This is a <em>gray failure</em>: some requests succeed while others stall, and different observers disagree about whether a component is alive.</p><p>The interval from 12:06 to 3:49 matters even more. Consensus had returned, but a market is not only its matching engine. Coinbase reported that two Amazon MSK clusters were stuck “healing.” A control-plane defect prevented Kafka partition-leader reelection, so producers could not write. The fee service was blocked; quoting was blocked behind it; ledger, payments, and data pipelines were affected. Restoring one critical state machine did not make the dependency graph healthy.</p><p>Users experienced global impact because the failed component was not a regional website cache. It was an authority. A matching engine decides which orders trade and in what order. You cannot safely run two independent authorities and merge their answers later. If New York says Alice bought the last bitcoin at $90,000 while Virginia says Bob bought it, reconciliation cannot make both executions true.</p><h3>Part 2 — What a crypto exchange actually is</h3><p>A centralized crypto exchange is a collection of state machines around one scarce resource: the right to change balances and orders in an unambiguous sequence.</p><pre>flowchart LR<br>  U[Web / Mobile UI] --&gt; G[API Gateway]<br>  MM[Market maker API] --&gt; G<br>  G --&gt; A[Authentication + Rate Limits]<br>  A --&gt; R[Pre-trade Risk Engine]<br>  R --&gt; OE[Order Entry]<br>  OE --&gt; ME[Matching Engine / Order Book]<br>  ME --&gt; ES[Event Stream: Kafka/MSK]<br>  ES --&gt; MD[Market Data]<br>  ES --&gt; F[Fee Service]<br>  ES --&gt; L[Double-entry Ledger]<br>  L --&gt; W[Wallet / Custody]<br>  W --&gt; C[Blockchain Consensus Networks]<br>  ES --&gt; S[Settlement + Reconciliation]<br>  MD --&gt; G<br>  L --&gt; G</pre><p><strong>The trading UI</strong> renders prices, balances, order forms, and status. It is not the exchange. It can display a cached price while order entry is dead, or appear broken while an API trader continues normally. A sound UI identifies freshness: “book sequence 8,412,771 received 240 ms ago” is more meaningful than a green dot.</p><p><strong>The API gateway</strong> terminates TLS, authenticates keys, checks signatures and timestamps, enforces rate limits, normalizes requests, and routes them. Inside the machine, encrypted TCP bytes arrive through the NIC; the kernel or user-space stack reconstructs a stream; TLS verifies message authentication and decrypts records; a parser turns bytes into fields; the gateway assigns or validates an idempotency key. A gateway should reject malformed traffic before it reaches scarce matching capacity.</p><p><strong>The risk engine</strong> answers “may this account place this order?” It checks available collateral, open exposure, position limits, price bands, self-trade rules, and liquidation constraints. The word <em>available</em> is subtle. A $10,000 balance with $7,000 reserved by open orders does not permit another $10,000 buy. Risk therefore needs a consistent view of reservations. If two concurrent requests both observe the old balance, they can overspend unless reservation is serialized or atomically conditional.</p><p><strong>The matching engine</strong> owns an order book for a product such as BTC-USD. Its job is deterministic: given the same ordered commands, every replica must produce the same fills, cancels, and book state. It does not ask a blockchain which trader came first. The venue’s sequencer decides arrival order.</p><p><strong>The ledger</strong> is the financial source of truth. A fill might debit USD from the buyer, credit BTC to the buyer, debit BTC from the seller, credit USD to the seller, and record fees. In a double-entry ledger, every transaction balances: the algebraic sum of postings is zero in each asset after accounting for venue accounts. The order book answers “what traded?” The ledger answers “who owns what now?”</p><p><strong>The wallet and custody layer</strong> manages cryptographic keys, deposit addresses, transaction construction, signing policy, hot/cold storage, and chain monitoring. An exchange balance is usually an internal ledger claim. No Bitcoin transaction occurs each time two Coinbase users trade BTC-USD.</p><p><strong>Settlement</strong> converts executions into final obligations. Internally, ledger postings may settle immediately. Externally, fiat rails and blockchains have different finality, reorg, cutoff, and reconciliation rules. Settlement workers must be replayable because crashes can occur after an external side effect but before a local acknowledgement.</p><p><strong>Market data</strong> transforms engine events into snapshots, incremental book updates, trades, candles, tickers, and feeds. Consumers apply deltas only when sequence numbers are contiguous. If they have sequence 100 and receive 102, they must stop, fetch a snapshot, and bridge the gap; silently applying 102 creates a fictional book.</p><p><strong>The price engine</strong> derives marks, indexes, reference prices, and bands from one or more venues. It is distinct from the matching engine’s last trade. A manipulation-resistant mark may use medians, time windows, outlier removal, and stale-feed rules.</p><p><strong>Liquidity</strong> is executable quantity near the current price, not merely a number on a screen. A book with one BTC at $90,000 and the next offer at $95,000 has a $90,000 best ask but poor depth. During an outage, stale visible liquidity is worse than no quote because it invites decisions based on inventory that cannot be executed.</p><p><strong>The blockchain consensus layer</strong> belongs to Bitcoin, Ethereum, Solana, and other networks. It determines which deposits and withdrawals become final on-chain. It does not order trades inside the centralized exchange. Coinbase’s matching cluster using Raft and Bitcoin miners using proof of work solve different consensus problems at different boundaries.</p><h3>Part 3 — Inside the matching engine</h3><p>An order book contains bids sorted by descending price and asks sorted by ascending price. At each price, orders form a FIFO queue. This is price-time priority: better price wins; among equal prices, earlier arrival wins.</p><p>A <strong>limit order</strong> specifies a worst acceptable price. “Buy 2 BTC at $90,000” can execute at $90,000 or less; unfilled quantity may rest. A <strong>market order</strong> expresses urgency and walks available prices; robust APIs often implement it as a marketable limit with a protection band because “any price” is dangerous during a gap. A <strong>stop order</strong> is dormant until a trigger price is observed, then releases a market or limit order. <strong>IOC</strong> means execute immediately and cancel the remainder. <strong>FOK</strong> means execute the entire amount immediately or execute nothing.</p><pre>match(incoming):<br>    opposite = asks if incoming.side == BUY else bids<br>    while incoming.remaining &gt; 0:<br>        level = opposite.best_price_level()<br>        if level == nil or !crosses(incoming.limit, level.price): break<br>        resting = level.front()<br>        quantity = min(incoming.remaining, resting.remaining)<br>        emit Fill(taker=incoming.id, maker=resting.id,<br>                  price=resting.price, quantity=quantity)<br>        decrement both orders<br>        if resting.remaining == 0: level.pop_front()<br>    if incoming.remaining &gt; 0 and incoming.time_in_force allows_resting:<br>        own_side[incoming.price].push_back(incoming)</pre><p>Why use the resting order’s price? Because it preserves the price already advertised by the maker. Why a single deterministic loop? Because parallel mutation creates races in priority. If two threads remove the same head order, the engine can double-fill it. High-performance venues often partition by product — one ordered execution lane per book or shard — rather than allowing many threads to mutate one book.</p><p>Memory layout is a latency feature. A pointer-heavy tree sends the CPU to unrelated addresses. An L1 cache hit is only a few cycles; a main-memory miss can cost roughly two orders of magnitude more time. Engines therefore use arrays, intrusive queues, object pools, compact structs, and integer price ticks. Converting 90000.01 dollars to integer cents or ticks avoids floating-point comparison surprises and makes keys cheap.</p><p>Cache locality means the next bytes are already in a cache line when the CPU needs them. Object pools avoid allocator locks and page faults. CPU affinity pins the engine thread so the scheduler does not migrate it to another core and discard warm caches. Isolating the core reduces involuntary context switches. NUMA awareness allocates memory from the socket attached to that core and NIC; otherwise a load crosses the inter-socket fabric.</p><p>The ordinary network receive path is long: NIC receives frame → DMA writes host memory → interrupt or polling wakes kernel → driver constructs metadata → IP validates and routes → TCP orders bytes and manages retransmission → socket buffer wakes process → scheduler runs process → copy or mapping exposes bytes. Kernel bypass frameworks such as DPDK map NIC queues into user space and poll them, removing interrupts, scheduler wakeups, and parts of the generic stack. RDMA lets one host access registered memory on another with little remote CPU involvement. AWS’s EFA exposes OS-bypass capabilities for supported HPC-style communication. None of these removes the need for application-level ordering and consensus.</p><p>TCP remains attractive because it provides an ordered, reliable byte stream and mature congestion control. But head-of-line blocking means a lost packet delays later bytes in that stream. Custom UDP or RDMA protocols can reduce overhead but must rebuild loss handling, flow control, authentication, and operational tooling. Faster is not automatically safer.</p><p>Why does every microsecond matter? Suppose a market maker receives a price change, computes a quote, sends it, and waits for an acknowledgement. If one venue’s round trip is 80 μs and another’s is 1.5 ms, the fast venue lets the maker update inventory more often and quote a tighter spread with less adverse-selection risk. Millions of small improvements compound into better prices.</p><p>The following is an engineering budget, not a Coinbase measurement. Values are representative ranges from modern tuned systems; hardware, packet size, load, C-states, hypervisor, and percentile can change them dramatically.</p><pre>Optimization                         Approx. saving per critical operation<br>Warm contiguous data vs DRAM miss    0.05–0.15 μs per avoided random miss<br>Pinned core vs unlucky migration     2–20 μs when a migration is avoided<br>Local NUMA vs remote NUMA access     0.05–0.15 μs per dependent remote access<br>Object pool vs contended allocation  0.1–5 μs, much more in tail cases<br>Polling/kernel bypass vs wakeup      3–20 μs per receive/send path<br>Same fabric vs cross-AZ hop          ~500–2,000+ μs one way, topology dependent<br>Batch 16 log records vs 16 fsyncs    often milliseconds of amortized savings</pre><pre>Warning: do not add the minima and call the result a benchmark. Savings overlap.<br>Measure p50, p99, and p99.99 under production-shaped load.</pre><h3>Part 4 — Consensus from first principles</h3><p>Replication exists because machines fail. If one process owns the only copy of the order book, a dead motherboard ends the market. Copying state to several machines helps only if the copies agree on the order of commands.</p><p>Imagine two orders arriving concurrently: Alice buys at $90,001; Bob cancels his resting sell at $90,000. Replica A sees buy then cancel and produces a fill. Replica B sees cancel then buy and produces no fill. Both processed valid commands, but their state diverged because order matters.</p><p>A replicated state machine solves this by separating <em>agreement</em> from <em>execution</em>. Replicas first agree that log index 42 contains Alice’s buy and index 43 contains Bob’s cancel. Each deterministic state machine then applies entries in index order.</p><p>Raft organizes replicas as follower, candidate, or leader. Followers expect heartbeats. If a follower receives none before a randomized election timeout, it increments its term, becomes a candidate, votes for itself, and asks peers for votes. A candidate that wins a majority becomes leader. Clients send commands to that leader. The leader appends each command locally and sends AppendEntries RPCs. Once an entry from the current term is stored by a majority, the leader can mark it committed and apply it.</p><pre>sequenceDiagram<br>  participant C as Client<br>  participant A as A (Leader)<br>  participant B as B (Follower)<br>  participant Cn as C (Follower)<br>  participant D as D (Follower)<br>  participant E as E (Follower)<br>  C-&gt;&gt;A: PlaceOrder(id=77)<br>  A-&gt;&gt;A: append log[1042]<br>  A-&gt;&gt;B: AppendEntries(term=18, index=1042)<br>  A-&gt;&gt;Cn: AppendEntries(term=18, index=1042)<br>  A-&gt;&gt;D: AppendEntries(...)<br>  A-&gt;&gt;E: AppendEntries(...)<br>  B--&gt;&gt;A: success<br>  Cn--&gt;&gt;A: success<br>  Note over A,Cn: 3 copies: majority reached<br>  A-&gt;&gt;A: commit and match order<br>  A--&gt;&gt;C: accepted + sequence</pre><p>For N voters, a majority is floor(N/2)+1. For five nodes, that is three. The mathematics is intersection. Any two sets of three chosen from five must share at least one node because 3 + 3 &gt; 5. If a committed value is known to one majority, a later elected leader must obtain votes from another majority that intersects it. Raft’s log-up-to-date voting rule uses that intersection to preserve committed history.</p><p>With only two survivors, the cluster cannot distinguish “the other three are dead” from “we are isolated and the other three are healthy.” If both sides accepted writes, the 2-node minority and 3-node majority could create incompatible histories: split brain. So two survivors stop. This sacrifices availability to preserve consistency.</p><pre>Five voters: {A,B,C,D,E}<br>Possible safe quorum: {A,B,C}<br>Another safe quorum:  {C,D,E}<br>Intersection:          {C}</pre><pre>After A,B,C fail:<br>Survivors:             {D,E}<br>Votes available:       2<br>Votes required:        3<br>Result:                no leader, no committed writes</pre><p>Heartbeats are not magical failure detectors. A missing response might mean a dead process, a paused runtime, a saturated NIC, packet loss, a long storage stall, or a partition. In an asynchronous network there is no fixed time after which silence proves death. Timeouts express an operational suspicion. Too short produces election storms; too long increases failover time.</p><p>Leases can let a leader serve reads without contacting a quorum when bounded clock and message-delay assumptions hold. If those assumptions fail, a deposed leader can serve stale data. A linearizable read instead confirms current leadership — often with a quorum round or a Raft ReadIndex — and waits until the local state machine has applied the required commit index.</p><p>Paxos and Raft both establish majority-backed decisions. Paxos is usually described in proposer, acceptor, and learner roles with numbered ballots; Multi-Paxos amortizes repeated agreement under a stable leader. Raft deliberately decomposes leader election, log replication, and safety for understandability. They are not “strong versus weak” consensus; correct implementations provide comparable safety goals.</p><p>CAP says that during a network partition a distributed system cannot provide both linearizable consistency and availability for every request. It does not say “pick two” during normal operation. The matching cluster chose consistency under partition: reject writes rather than risk conflicting executions. PACELC adds the normal case: <em>if Partition, Availability or Consistency; Else, Latency or Consistency</em>. Even without failure, synchronous replication pays network and durability latency for stronger agreement.</p><h3>Part 5 — Why put all voters together?</h3><p>AWS cluster placement groups pack instances close together within one Availability Zone and place them in a high-bisection-bandwidth network segment. AWS documents up to 10 Gbit/s for a single flow between enhanced-networking instances in such a group, versus 5 Gbit/s outside it; eligible ENA Express configurations can go higher. A cluster placement group cannot span Availability Zones.</p><p>That topology suits a consensus loop. A leader may need two follower acknowledgements before every externally visible command. Add one millisecond of one-way latency and the quorum round trip can add roughly two milliseconds before storage and application work. At 100,000 ordered commands per second, batching can preserve throughput, but individual command latency and tail jitter still rise. More importantly, fairness becomes noisier: orders arriving while a batch is waiting experience variable delay.</p><p>Where does latency come from?</p><p><strong>Serialization delay</strong> is packet bits divided by link rate. A 1,500-byte frame takes about 1.2 μs to serialize at 10 Gbit/s and 0.12 μs at 100 Gbit/s. <strong>Propagation delay</strong> is physical travel: light in fiber covers roughly 200 km per millisecond, or about 5 μs per kilometer of path. <strong>Switch delay</strong> includes ingress parsing, buffering, arbitration, and egress. <strong>Queue delay</strong> is near zero on an idle link and unbounded relative to a microsecond budget under congestion.</p><p>At the host, the NIC moves bytes over PCIe using DMA. Interrupt moderation may deliberately wait to batch completions. The kernel processes protocol state. A sleeping thread must be scheduled. Its cache may be cold. If the thread moves sockets, NUMA adds another fabric traversal. Each cost looks small in isolation; the p99 is the path where several coincide.</p><p>Coinbase’s design choice therefore had a rational local optimum: co-locate voters for predictable microsecond-scale communication. The architectural defect was not low latency. It was allowing the low-latency failure domain to also be the only active authority, without automated cross-zone continuity.</p><blockquote><strong><em>Warning.</em></strong><em> “Spread the five voters over three AZs” is not a free fix. It moves cross-AZ latency into every consensus commit and can make a trading venue structurally slower. A common alternative is a fast zonal quorum plus a warm, continuously replicated standby authority in another zone, with a carefully fenced failover protocol.</em></blockquote><h3>Part 6 — The AWS machinery below the API</h3><p>An AWS Region is a geographic area containing multiple Availability Zones. An AZ is one or more discrete data centers with independent and redundant power, networking, and connectivity. AWS says AZs in a Region can be up to roughly 100 km apart, are linked by dedicated metro fiber, and are close enough for single-digit-millisecond synchronous replication. A data hall is smaller than an AZ. This incident began in one hall inside one AZ.</p><pre>flowchart TB<br>  R[us-east-1 Region]<br>  R --&gt; AZ1[AZ use1-az4]<br>  R --&gt; AZ2[Another AZ]<br>  R --&gt; AZ3[Another AZ]<br>  AZ1 --&gt; DC1[Data center / hall A]<br>  AZ1 --&gt; DC2[Data center / hall B]<br>  DC1 --&gt; SP[Spine switches]<br>  SP --&gt; TOR1[Top-of-rack switch]<br>  SP --&gt; TOR2[Top-of-rack switch]<br>  TOR1 --&gt; Rack1[EC2 hosts + storage paths]<br>  TOR2 --&gt; Rack2[EC2 hosts + storage paths]<br>  DC1 --&gt; Cool[Chillers / cooling distribution]<br>  Cool -. thermal event .-&gt; Rack1<br>  Cool -. thermal event .-&gt; Rack2</pre><p>Power enters through utility feeds, switchgear, UPS systems, distribution units, and rack power units. Cooling removes almost the same energy that servers consume electrically. Chilled-water or refrigerant systems move heat from computer-room air handlers to an outside rejection system. Servers pull cool air through the front and exhaust hot air behind them. If heat removal falls below heat generation, inlet temperature rises.</p><p>Hardware protects itself in stages. Fans speed up. CPUs lower frequency and voltage. Controllers alarm. At unsafe thresholds, operators or automated safety systems remove load or power. Abrupt termination means volatile RAM disappears. A network peer sees missed heartbeats and connection resets or timeouts. EBS volumes are network-attached storage; if affected compute or storage paths are impaired, attachment and I/O can stall even though a volume abstraction still exists in an API.</p><p>EC2 Nitro moves many virtualization and I/O functions onto dedicated cards, leaving more host CPU for workloads and strengthening isolation. ENA is the enhanced network adapter exposed to instances. A top-of-rack switch connects servers in a rack; leaf/spine fabrics connect racks through multiple equal-cost paths. These layers reduce single-device failures, but they do not nullify a hall-level thermal safety action.</p><p>A control plane decides desired configuration: launch an instance, attach a volume, elect or move a managed-service partition. A data plane carries customer packets and storage operations. Good recovery plans minimize dependence on a control plane during its most stressed moment. Coinbase’s Kafka issue is a vivid example: brokers existed, but the managed control plane did not complete leader reelection and clusters remained stuck healing.</p><h3>Part 7 — Simulating the failure packet by packet</h3><p>Assume A is leader in term 18. A, B, and C share racks that are removed; D and E survive. Heartbeats run every 50 ms and election timeouts are randomized around 150–300 ms. Those values are illustrative.</p><pre>sequenceDiagram<br>  participant A as A leader (fails)<br>  participant B as B (fails)<br>  participant C as C (fails)<br>  participant D as D survivor<br>  participant E as E survivor<br>  A-&gt;&gt;D: AppendEntries term 18 (heartbeat)<br>  A-&gt;&gt;E: AppendEntries term 18 (heartbeat)<br>  Note over A,C: thermal shutdown / instances terminate<br>  D-xA: heartbeat response cannot be delivered<br>  E-xA: heartbeat response cannot be delivered<br>  Note over D: election timer expires<br>  D-&gt;&gt;D: term=19, vote D<br>  D-&gt;&gt;E: RequestVote(term=19, lastLog)<br>  E--&gt;&gt;D: vote granted<br>  Note over D,E: only 2 votes; 3 required<br>  Note over E: later timeout expires<br>  E-&gt;&gt;E: term=20, vote E<br>  E-&gt;&gt;D: RequestVote(term=20)<br>  D--&gt;&gt;E: vote granted<br>  Note over D,E: still only 2; election cannot succeed</pre><p>At the packet level, a heartbeat is typically a small RPC over an existing TCP connection. When A loses power, no FIN handshake is guaranteed; the connection simply stops making progress. D’s kernel may retransmit unacknowledged segments with backoff. Raft does not wait for TCP to prove death. Its application election timer expires first.</p><p>D sends RequestVote to E and attempts A, B, and C. DNS or service discovery may still return dead addresses. If startup code insists that all five names resolve before creating transport objects, replacement nodes can fail before Raft even has a chance to operate. Coinbase reported shipping an emergency change to remove an assumption that all five cluster nodes were resolvable.</p><p>Could D and E serve reads? They can serve explicitly stale, local diagnostic reads if the product permits it. They cannot safely claim linearizable current state because they cannot confirm that term 18’s leader has not committed an entry with another majority or that a newer leader does not exist. For a trading venue, stale order state can be financially dangerous. A safe public posture is to stop mutations and mark data stale.</p><p>Meanwhile Kafka has a different leadership layer. Each partition has a leader broker and follower replicas. Producers append to the partition leader. If the leader’s zone disappears, the controller should choose an in-sync replica, update metadata, and clients reconnect. Coinbase reported a managed-control-plane defect that prevented this automatic reelection on two clusters. Producers then failed or timed out even though some brokers survived.</p><pre>sequenceDiagram<br>  participant Q as Quoting Service<br>  participant F as Fee Service<br>  participant P as Kafka Producer<br>  participant M as MSK Control Plane<br>  participant B1 as Dead Leader Broker<br>  participant B2 as Surviving Replica<br>  Q-&gt;&gt;F: fee schedule / computed fee request<br>  F-&gt;&gt;P: publish or consume required event<br>  P-xB1: produce request times out<br>  P-&gt;&gt;M: refresh partition metadata<br>  M--&gt;&gt;P: cluster healing; no new leader<br>  Note over B2: data exists but is not authoritative leader<br>  P--&gt;&gt;F: cannot append<br>  F--&gt;&gt;Q: unavailable<br>  Q--&gt;&gt;Q: stop quoting</pre><p>This is how one building becomes global impact. The failed building held a unique low-latency authority; the surviving event infrastructure could not automatically establish new partition authorities; higher-level services correctly refused to fabricate fees, balances, quotes, or executions.</p><h3>Part 8 — Why recovery was not automatic</h3><p>Automatic recovery requires more than spare machines. It requires a rehearsed answer to identity, membership, data freshness, fencing, capacity, dependencies, and market reopening.</p><p>First, new nodes need durable state. A Raft follower can receive a snapshot containing state through index S, then replay write-ahead-log entries S+1…N. The WAL must be validated with checksums and term/index metadata. If a node has uncommitted suffix entries from an old leader, the new leader compares prevLogIndex and prevLogTerm, backs up to the last matching prefix, and overwrites the divergent suffix. Committed entries are never discarded by a correct protocol.</p><p>Second, membership changes are themselves consensus operations. Simply editing a config file from five addresses to three can create two clusters that both think they are legitimate. Safe systems use joint consensus or another proven reconfiguration method so old and new configurations overlap during transition.</p><p>Third, failover needs fencing. If a warm standby becomes active while the old primary can still reach some clients, both may accept orders. A fencing token — an epoch stored in a strongly consistent external authority, a lease with defensible timing assumptions, or network isolation verified by an independent mechanism — ensures only the highest epoch can publish executions.</p><p>Fourth, market reopening is a product protocol:</p><ol><li><strong>Recover quorum.</strong> Establish one authoritative log and healthy replicas.</li><li><strong>Synchronize state.</strong> Load snapshots, replay WAL, verify hashes and sequence numbers.</li><li><strong>Recover dependencies.</strong> Kafka, fees, risk, ledger, and market-data paths must accept and replay events.</li><li><strong>Cancel-only.</strong> Let customers reduce stale exposure without creating new matches.</li><li><strong>Auction.</strong> Collect orders, publish an indicative price, then uncross simultaneously to avoid rewarding the first connection after downtime.</li><li><strong>Continuous trading.</strong> Resume normal price-time matching only after feeds and safeguards are stable.</li></ol><p>Coinbase restored quorum at 12:06 a.m., opened cancel-only at 2:25, and resumed all books at 3:49. That is caution, not idle time. After a multi-hour market interruption, external prices have moved, resting orders may be stale, and clients reconnect in a burst. Reopening directly into continuous trading can transfer value arbitrarily to the fastest reconnecting participant.</p><p>Kafka recovery also requires ordering. A manual partition reassignment chooses surviving brokers, copies or identifies replicas, updates leadership and replica assignments, and lets clients refresh metadata. Priority topics should recover before analytics. Consumers then drain backlogs. If a consumer performs external effects, replay must be idempotent: store an event ID or source offset alongside the resulting ledger transaction so a duplicate does not create a second payment.</p><h3>Part 9 — Performance versus reliability</h3><p>Availability and latency are not enemies in every design, but the strongest failure isolation usually adds distance, coordination, or idle capacity.</p><pre>Design                         Normal commit      Survives hall/AZ?  Main cost<br>1 node in 1 hall               local only         No                 data/authority loss<br>5 voters, cluster placement   local quorum       No, if correlated  shared fate<br>5 voters across 3 AZs          cross-AZ quorum    Usually            millisecond latency<br>3 local + async standby        local quorum       Maybe              possible data loss<br>3 local + sync warm standby    local + remote ack Yes                remote tail latency<br>Local authority + fenced DR    local fast path    Yes after failover complexity + RTO</pre><p>Suppose local quorum round-trip time is 100 μs at p99 and cross-AZ is 2 ms. Moving the quorum path across AZs adds roughly 1.9 ms per unbatched commit, a 20× latency increase. Batching 100 commands into one replication round preserves throughput but not the time the first command waits for the batch. Tail latency matters because systems compose: if order entry, risk, fee, ledger, and event publication each have a p99 spike, the request tail is driven by the slowest correlated component, not by adding medians.</p><p>Replication cost appears in bandwidth, CPU, storage IOPS, and coordination. Three copies triple raw log bytes before compression and protocol overhead. Acknowledging after local memory is fast but loses data on power failure. Acknowledging after fsync is durable but pays device and queue latency. Group commit amortizes one flush across many records while adding a small waiting window.</p><p>Reliability engineering should express the contract explicitly:</p><pre>RPO (maximum accepted data loss):       0 committed orders<br>RTO (time to safe market authority):    target, e.g. &lt; 5 minutes<br>Failover trigger:                       independent multi-signal decision<br>Fencing guarantee:                     one active epoch<br>Standby freshness:                      commit index lag &lt; threshold<br>Reopen protocol:                        cancel-only → auction → continuous<br>Dependency readiness:                   fees, risk, ledger, Kafka, market data</pre><p>A zero RPO does not force every replica across AZs. It does force every acknowledged command to exist in a failure domain that survives the declared fault model — or to make the customer contract narrower. If the design claims AZ resilience, a committed order cannot live only in one hall.</p><h3>Part 10 — What changes after this class of outage</h3><p>Coinbase committed to improving warm cross-zone standby for its matching engines, running regular production failover exercises, building Kafka tooling and runbooks for the observed failure modes, working with AWS on the control-plane defect, and migrating a two-AZ Kafka cluster to three AZs.</p><p>A <strong>warm standby</strong> is already provisioned, has code and configuration loaded, receives state continuously, and can be promoted without waiting for capacity or a full snapshot copy. Warm is not the same as safe. The standby must prove its applied log index, dependencies, certificates, secrets, clock health, and network reachability. Promotion must fence the old authority.</p><p><strong>Chaos engineering</strong> should test hypotheses, not create random pain. “If one hall disappears, order authority moves within five minutes with zero committed-order loss” is testable. Fault injection should remove three voters, blackhole DNS, stall EBS, deny MSK control-plane progress, delay cross-AZ links, and restart consumers with a backlog.</p><p><strong>Game days</strong> test the sociotechnical system: paging, incident command, market-operations decisions, regulator notification, AWS escalation, runbook correctness, and authority to ship an emergency change. A technically valid standby that nobody is permitted to promote is not a recovery system.</p><p><strong>Regional failover</strong> is a larger problem than AZ failover. Regions change network distance, data sovereignty, capacity pools, service endpoints, and external dependencies. It is valuable only after zonal evacuation is boring. Otherwise it adds a second untested architecture.</p><h3>Part 11 — Lessons by role</h3><p><strong>For Kubernetes engineers:</strong> a PodDisruptionBudget does not protect against an AZ disappearing. Anti-affinity labels do not help if the application’s own consensus membership, storage, DNS, or managed dependencies share fate. Understand the failure domain behind each node label. Test loss of nodes, volumes, DNS answers, control planes, and admission paths. StatefulSets provide stable identities; they do not make membership changes safe.</p><p><strong>For SREs:</strong> monitor authority, not only process health. Useful signals include current leader and term, quorum reachability, commit-index lag, snapshot age, Kafka partitions without leaders, producer error rate, consumer lag by criticality, stale market-data sequence, and dependency-specific readiness. Page on customer invariants: “no authoritative quote path” is clearer than “CPU low.”</p><p><strong>For cloud architects:</strong> map components to physical fault domains. “Multi-AZ” on a diagram can hide a single-AZ cluster, two-AZ Kafka topic, zonal EBS attachment, or control-plane operation required during recovery. Separate data-plane survival from control-plane repair. Reserve capacity before the incident.</p><p><strong>For startup founders:</strong> availability is a product property, not a cloud-provider checkbox. Define what must keep working, what may become read-only, and how customers are protected when truth is uncertain. A safe degraded mode often beats a heroic attempt to remain writable.</p><p><strong>For trading-system designers:</strong> fairness is part of recovery. Preserve deterministic sequencing, publish freshness, stop on ambiguous authority, make cancels possible before new risk, and reopen with an auction. Latency optimizations need a paired failure-containment story.</p><h3>Part 12 — Reproduce the failure yourself</h3><p>You can learn most of this without running a real exchange. Build a disposable Kubernetes cluster, deploy etcd as a five-member Raft system, observe it, then remove three members.</p><pre># Prerequisites: Docker, kubectl, kind, Helm<br>kind create cluster --name quorum-lab --config - &lt;&lt;&#39;EOF&#39;<br>kind: Cluster<br>apiVersion: kind.x-k8s.io/v1alpha4<br>nodes:<br>- role: control-plane<br>- role: worker<br>- role: worker<br>- role: worker<br>- role: worker<br>- role: worker<br>EOF</pre><pre>kubectl create namespace quorum-lab<br>helm repo add bitnami <a href="https://charts.bitnami.com/bitnami">https://charts.bitnami.com/bitnami</a><br>helm install etcd bitnami/etcd -n quorum-lab \<br>  --set replicaCount=5 \<br>  --set auth.rbac.create=false \<br>  --set metrics.enabled=true</pre><pre>kubectl -n quorum-lab get pods -w<br>kubectl -n quorum-lab exec etcd-0 -- \<br>  etcdctl endpoint status --cluster -w table</pre><p>Write a key, remove two members, and show that the cluster still has three voters. Then remove a third.</p><pre>kubectl -n quorum-lab exec etcd-0 -- etcdctl put market BTC-USD</pre><pre># Failure 1 and 2: quorum remains 3/5<br>kubectl -n quorum-lab delete pod etcd-3 etcd-4 \<br>  --grace-period=0 --force</pre><pre># Prevent immediate recreation for a controlled lab<br>kubectl -n quorum-lab scale statefulset etcd --replicas=3<br>kubectl -n quorum-lab exec etcd-0 -- etcdctl put status degraded-but-writable</pre><pre># Now deliberately destroy quorum by isolating another member.<br># Use a NetworkPolicy or Chaos Mesh in preference to deleting persistent data.<br>kubectl create namespace chaos-mesh<br>helm repo add chaos-mesh <a href="https://charts.chaos-mesh.org/">https://charts.chaos-mesh.org</a><br>helm install chaos-mesh chaos-mesh/chaos-mesh -n chaos-mesh \<br>  --set chaosDaemon.runtime=containerd \<br>  --set chaosDaemon.socketPath=/run/containerd/containerd.sock</pre><pre>apiVersion: chaos-mesh.org/v1alpha1<br>kind: NetworkChaos<br>metadata:<br>  name: isolate-third-etcd-member<br>  namespace: quorum-lab<br>spec:<br>  action: partition<br>  mode: one<br>  selector:<br>    namespaces: [quorum-lab]<br>    labelSelectors:<br>      app.kubernetes.io/name: etcd<br>  direction: both<br>  target:<br>    mode: all<br>    selector:<br>      namespaces: [quorum-lab]<br>      labelSelectors:<br>        app.kubernetes.io/name: etcd<br>  duration: 5m</pre><p>Do not run broad selectors against a shared cluster. Use a disposable Kind environment and inspect the generated labels first.</p><p>Add Prometheus and Grafana with kube-prometheus-stack. Graph leader changes, proposals committed, proposals failed, peer round-trip time, WAL fsync duration, and backend commit duration. Correlate the moment quorum disappears with client timeouts. Then restore the network and confirm one leader, monotonic revisions, and successful writes.</p><p>LitmusChaos can express the same experiment as a workflow; the important part is the hypothesis and steady-state assertion. For example: before fault, five endpoints healthy; during loss of two, writes succeed; during loss of three, writes fail; after restoration, the last acknowledged value remains and writes resume.</p><p>For an AWS exercise, Terraform can create three subnets in three AZs, security groups, and small test instances. Do not pretend that stopping instances reproduces a cooling event. Combine instance termination, packet loss, storage delay, and blocked control-plane actions. Tag everything, set a budget, and destroy the lab.</p><pre>variable &quot;azs&quot; { default = [&quot;us-east-1a&quot;, &quot;us-east-1b&quot;, &quot;us-east-1c&quot;] }</pre><pre>resource &quot;aws_subnet&quot; &quot;lab&quot; {<br>  count             = 3<br>  vpc_id            = aws_vpc.lab.id<br>  availability_zone = var.azs[count.index]<br>  cidr_block        = cidrsubnet(aws_vpc.lab.cidr_block, 4, count.index)<br>  tags = { Name = &quot;quorum-lab-${count.index}&quot; }<br>}</pre><pre># Exercise questions:<br># 1. Which acknowledgements cross AZ boundaries?<br># 2. Can replacement capacity launch if one AZ is unavailable?<br># 3. What fences the old leader?<br># 4. Can recovery proceed if a managed control-plane call stalls?</pre><h3>Part 13 — A small Raft walkthrough in Go</h3><p>The following code is educational. It omits persistence, snapshots, membership change, flow control, authentication, retries, and many safety checks. Do not use it in production.</p><pre>type State int<br>const (<br>    Follower State = iota<br>    Candidate<br>    Leader<br>)</pre><pre>type Entry struct {<br>    Term    uint64<br>    Command []byte<br>}</pre><pre>type Node struct {<br>    mu          sync.Mutex<br>    id          string<br>    peers       []string<br>    state       State<br>    term        uint64<br>    votedFor    string<br>    log         []Entry<br>    commitIndex uint64<br>    lastApplied uint64<br>    heartbeatCh chan struct{}<br>}</pre><pre>func (n *Node) electionLoop(ctx context.Context) {<br>    for {<br>        timeout := randomized(150*time.Millisecond, 300*time.Millisecond)<br>        timer := time.NewTimer(timeout)<br>        select {<br>        case &lt;-ctx.Done():<br>            timer.Stop(); return<br>        case &lt;-n.heartbeatCh:<br>            timer.Stop() // valid leader contact resets suspicion<br>        case &lt;-timer.C:<br>            go n.startElection(ctx)<br>        }<br>    }<br>}</pre><pre>func (n *Node) startElection(ctx context.Context) {<br>    n.mu.Lock()<br>    n.state = Candidate<br>    n.term++<br>    term := n.term<br>    n.votedFor = n.id<br>    lastIndex, lastTerm := n.lastLogPosition()<br>    n.mu.Unlock()</pre><pre>    votes := int32(1)<br>    majority := len(n.peers)/2 + 1<br>    for _, peer := range n.peers {<br>        if peer == n.id { continue }<br>        go func(p string) {<br>            reply, err := requestVote(ctx, p, term, lastIndex, lastTerm)<br>            if err != nil { return }<br>            n.mu.Lock()<br>            defer n.mu.Unlock()<br>            if reply.Term &gt; n.term { n.becomeFollower(reply.Term); return }<br>            if n.term != term || n.state != Candidate { return }<br>            if reply.Granted &amp;&amp; atomic.AddInt32(&amp;votes, 1) &gt;= int32(majority) {<br>                n.state = Leader<br>                go n.heartbeatLoop(ctx, term)<br>            }<br>        }(peer)<br>    }<br>}</pre><p>Notice that silence does not authorize a minority leader. majority is computed from configured voters, not reachable peers. Changing it to “half of whoever answered” would create split brain.</p><pre>func (n *Node) replicate(ctx context.Context, cmd []byte) error {<br>    n.mu.Lock()<br>    if n.state != Leader { n.mu.Unlock(); return ErrNotLeader }<br>    entry := Entry{Term: n.term, Command: cmd}<br>    n.log = append(n.log, entry)<br>    index := uint64(len(n.log) - 1)<br>    term := n.term<br>    n.mu.Unlock()</pre><pre>    // Real Raft tracks nextIndex/matchIndex per follower and retries conflicts.<br>    acks := int32(1)<br>    majority := len(n.peers)/2 + 1<br>    committed := make(chan struct{}, 1)</pre><pre>    for _, peer := range n.peers {<br>        if peer == n.id { continue }<br>        go func(p string) {<br>            if appendEntry(ctx, p, index, term, entry) == nil &amp;&amp;<br>               atomic.AddInt32(&amp;acks, 1) == int32(majority) {<br>                committed &lt;- struct{}{}<br>            }<br>        }(peer)<br>    }</pre><pre>    select {<br>    case &lt;-ctx.Done():<br>        // Ambiguous to client: the entry may commit after timeout.<br>        // Retry must use the same idempotency key.<br>        return ctx.Err()<br>    case &lt;-committed:<br>        n.mu.Lock()<br>        if n.term != term || n.state != Leader {<br>            n.mu.Unlock(); return ErrLeadershipChanged<br>        }<br>        n.commitIndex = index<br>        n.applyCommittedInOrder()<br>        n.mu.Unlock()<br>        return nil<br>    }<br>}</pre><p>The ambiguous timeout comment is crucial. A client can lose the response after the command commits. Retrying with a new order ID can create a duplicate. The command must carry a stable idempotency key, and the state machine must remember the result for that key.</p><p>Recovery log synchronization compares terms, not just lengths:</p><pre>Leader sends: prevIndex=1041, prevTerm=18, entries=[1042@18]<br>Follower checks log[1041].term.</pre><pre>If equal: append, acknowledge.<br>If absent/different: reject with conflict hint.<br>Leader decreases nextIndex and retries from an earlier point.<br>Once prefixes match, follower deletes its uncommitted divergent suffix<br>and appends the leader&#39;s entries.</pre><p>A production implementation persists currentTerm, votedFor, and log entries before replying; installs snapshots safely; limits inflight replication; handles disk-full and corruption; verifies membership transitions; exposes metrics; and has been model-checked or battle-tested. Consensus code is compact only in diagrams.</p><h3>A closer look at durability, Kafka, and the ledger boundary</h3><p>Three logs appear in this story and they should not be confused: the Raft replication log, Kafka’s partition log, and the financial ledger. They may all contain an event called “fill,” but they solve different problems.</p><p>The <strong>Raft log</strong> is the input sequence for one replicated state machine. Its term and index establish consensus history. The leader may append an order command, replicate it, commit it, and then produce deterministic engine events. A follower’s Raft log is not a customer-facing audit statement; it contains protocol history and may include an uncommitted suffix that is later overwritten.</p><p>A <strong>Kafka partition log</strong> distributes events to many independent consumers. Records within one partition are ordered by offset. There is no total order across all partitions unless the application constructs one. A producer acknowledgement depends on acks, replica state, and broker configuration. Consumers advance offsets independently. Kafka makes fan-out and replay practical; it does not automatically make a multi-service business transaction atomic.</p><p>The <strong>financial ledger</strong> records ownership. Its invariants survive replay, service restarts, and reporting changes. If a consumer reads FillExecuted(order=77, trade=9001) twice, it must not post twice. One pattern is to insert the source event ID into the same database transaction as balanced postings under a uniqueness constraint:</p><pre>BEGIN;<br>INSERT INTO consumed_events(event_id) VALUES (&#39;trade-9001&#39;);<br>-- unique violation means this event was already applied<br>INSERT INTO postings(tx_id, account, asset, amount) VALUES<br>  (&#39;trade-9001&#39;, &#39;buyer-usd&#39;,  &#39;USD&#39;, -90000),<br>  (&#39;trade-9001&#39;, &#39;seller-usd&#39;, &#39;USD&#39;,  89910),<br>  (&#39;trade-9001&#39;, &#39;venue-fees&#39;, &#39;USD&#39;,     90),<br>  (&#39;trade-9001&#39;, &#39;seller-btc&#39;, &#39;BTC&#39;,     -1),<br>  (&#39;trade-9001&#39;, &#39;buyer-btc&#39;,  &#39;BTC&#39;,      1);<br>COMMIT;</pre><p>The example uses simplified units and omits clearing accounts, but its invariant is visible. USD sums to zero; BTC sums to zero. The uniqueness constraint converts at-least-once event delivery into exactly-once <em>effect</em> at this boundary. “Exactly once” is rarely a magical property of the whole pipeline. It is built from stable identities, atomic conditional writes, and deterministic retries at each side-effect boundary.</p><p>Now consider a crash sequence:</p><ol><li>The matching engine commits order 77 at Raft index 1042.</li><li>It emits trade 9001 to an outbox or event publisher.</li><li>Kafka accepts the record but the acknowledgement is lost.</li><li>The publisher retries trade 9001.</li><li>The ledger consumer sees two copies.</li></ol><p>Without the stable trade ID and ledger uniqueness check, recovery invents money. With them, the second delivery is a harmless no-op. This is why restoring Kafka is not merely making brokers green. Engineers must verify producer idempotence, partition ordering, consumer offsets, poison records, and every downstream deduplication contract.</p><h3>What a Kafka leader election actually changes</h3><p>A Kafka producer caches metadata mapping a topic-partition to its leader broker. It sends a Produce request containing records with sequence metadata. The leader appends to its local segment, followers fetch from it, and — depending on acknowledgement policy — the leader responds after enough in-sync replicas have persisted the data. If the leader disappears, a controller chooses an eligible replica and publishes new metadata. Producers receive errors such as “not leader,” refresh metadata, and retry.</p><p>A replica can be alive yet ineligible because it fell outside the in-sync replica set. Promoting a stale replica can lose acknowledged records. Enabling unclean leader election trades durability for availability and is generally unacceptable for financially critical streams. Therefore “a broker is up” is insufficient. The questions are: which replica contains the committed prefix, which broker is authorized leader, and do clients know the new epoch?</p><p>A managed-service control plane automates those transitions. If that automation stalls, application clients cannot safely appoint a leader themselves. Manual partition reassignment moves replicas or leadership according to an operator-selected plan. Under incident load, that plan must consider network capacity: copying many large partitions at once can saturate the same fabric needed for healthy traffic. Priority recovery is rational — restore order, fee, risk, and ledger topics before bulk analytics.</p><h3>Why snapshots do not eliminate recovery time</h3><p>A snapshot compresses a long history into a point-in-time state image. For an order book it might contain price levels, every resting order, client order IDs, sequence counters, and risk reservations at commit index 50 million. The snapshot needs a checksum, format version, configuration epoch, and the last included Raft term/index.</p><p>Installing it still costs time. Bytes must be read from durable storage or transferred over the network, decompressed, validated, allocated, and indexed. A 100 GB snapshot over an effective 5 Gbit/s path has a theoretical transfer floor near 160 seconds before protocol overhead, contention, validation, and replay. If 20 million WAL entries follow the snapshot, replay CPU may dominate. Faster snapshots reduce recovery point age but consume foreground I/O when created. Incremental snapshots reduce transfer size but complicate correctness.</p><p>The leader also needs flow control. Sending an entire log to a new follower can fill NIC queues and page cache, increasing the p99 latency of active order replication. Production systems throttle catch-up, prioritize quorum members already near the head, and prevent a recovering replica from becoming a voting dependency before it is ready.</p><h3>Readiness must represent a transaction path</h3><p>A Kubernetes readiness probe that returns 200 because the process event loop is running tells a load balancer almost nothing. For order entry, readiness may need to mean:</p><pre>process healthy<br>AND configuration epoch current<br>AND authenticated connection to risk service<br>AND matching leader known<br>AND commit lag below threshold<br>AND critical Kafka partitions have leaders<br>AND fee data fresh enough<br>AND ledger write path accepting idempotent transactions<br>AND market mode permits requested operation</pre><p>Even that should not become one brittle synchronous mega-probe. Expose component health separately, derive customer-capability states centrally, and route by operation. “Cancel available, new order unavailable, balances stale as of sequence X” is a more honest degraded mode than a single red or green status.</p><h3>The dangerous recovery shortcuts</h3><p>During a prolonged incident, several tempting actions can violate safety:</p><ul><li><strong>Lower quorum from three to two.</strong> Unless performed through a safe reconfiguration with proof the old majority is fenced, this can create a second authority.</li><li><strong>Promote the most responsive replica.</strong> Responsiveness says nothing about whether its log is most up to date.</li><li><strong>Delete a divergent WAL.</strong> An entry that looks uncommitted locally may have been committed by a previous majority. Protocol rules, not intuition, decide.</li><li><strong>Enable unclean Kafka election.</strong> This may recover writes by discarding acknowledged events.</li><li><strong>Skip replay to reduce backlog.</strong> Missing fills or ledger postings is not graceful degradation.</li><li><strong>Open continuous trading immediately.</strong> Stale resting orders and reconnect races create unfair transfers.</li></ul><p>Safe recovery is deliberately asymmetric: it is easy to stop accepting new risk and hard to create a new authority. The extra friction is the mechanism preserving history.</p><blockquote><strong><em>Engineering note.</em></strong><em> The strongest incident metric is not “time until servers restarted.” Track time to a provably unique authority, time to cancel-only, time to fair price discovery, time to continuous trading, time to critical-event catch-up, and time to full customer-surface recovery. Those clocks isolate different engineering obligations. A team can improve infrastructure restart time while leaving authority selection or backlog drainage untouched. Conversely, a venue may restore safe cancellation quickly while intentionally delaying matching. Reporting one recovery number hides the architecture; reporting the sequence turns the incident into an actionable design review.</em></blockquote><p>The same principle applies to service-level objectives. Availability for a trading system is not merely successful HTTP responses. Define separate indicators for authenticated reads, fresh market data, cancels accepted, new orders durably sequenced, fills ledgered, deposits recognized, and withdrawals submitted. During a partial failure, these capabilities diverge. Measuring them separately prevents a healthy landing page from masking a dead market and prevents a deliberately safe cancel-only mode from being mistaken for total recovery.</p><h3>Part 14 — The prediction you should now be able to make</h3><p>Before seeing a postmortem, inspect a low-latency exchange and ask:</p><ol><li>Where is the one authoritative order sequence produced?</li><li>How many failures remove its quorum, and are those failures physically correlated?</li><li>Does the design’s declared fault domain include the placement group, hall, AZ, and managed-service control plane?</li><li>Can a standby prove freshness and fence the former primary without an engineer editing code?</li><li>Which event topics sit between matching, fees, risk, ledger, and quoting?</li><li>What happens when brokers survive but automatic leadership does not?</li><li>Can the venue become cancel-only while truth is incomplete?</li><li>Has the full reopening sequence run in production-shaped conditions?</li></ol><p>If five voting nodes are packed into one low-latency failure domain, three can disappear together. If no automated cross-zone authority exists, consensus will stop. If critical services synchronously depend on a managed event backbone whose recovery path is itself impaired, restoring the first quorum will not restore the product. If reopening fairness is taken seriously, recovery will take longer than process startup.</p><p>That is the deeper lesson of May 7. The outage did not violate distributed-systems theory. It followed it exactly. The majority rule protected the order history. The placement group delivered the latency it was chosen for. Thermal safety protected hardware and people. Kafka clients refused to write without partition leaders. Every local mechanism behaved defensibly, yet the composition failed the customer.</p><p>Reliability lives in that composition.</p><h3>Research references</h3><ol><li><a href="https://www.coinbase.com/blog/a-postmortem-of-our-may-7-2026-outage">Coinbase — A postmortem of our May 7, 2026 outage</a>.</li><li><a href="https://raft.github.io/raft.pdf">Ongaro and Ousterhout — In Search of an Understandable Consensus Algorithm</a>.</li><li><a href="https://lamport.azurewebsites.net/pubs/paxos-simple.pdf">Leslie Lamport — Paxos Made Simple</a>.</li><li><a href="https://research.google/pubs/spanner-googles-globally-distributed-database-2/">Google — Spanner: Google’s Globally-Distributed Database</a>.</li><li><a href="https://www.allthingsdistributed.com/files/amazon-dynamo-sosp2007.pdf">Amazon — Dynamo: Amazon’s Highly Available Key-value Store</a>.</li><li><a href="https://users.ece.cmu.edu/~adrian/731-sp04/readings/GL-cap.pdf">Gilbert and Lynch — Brewer’s Conjecture and the Feasibility of Consistent, Available, Partition-Tolerant Web Services</a>.</li><li><a href="https://www.cs.umd.edu/~abadi/papers/abadi-pacelc.pdf">Daniel Abadi — Consistency Tradeoffs in Modern Distributed Database System Design (PACELC)</a>.</li><li><a href="https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/placement-groups.html">AWS — EC2 placement groups</a> and <a href="https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/placement-strategies.html">placement strategies</a>.</li><li><a href="https://docs.aws.amazon.com/whitepapers/latest/aws-fault-isolation-boundaries/availability-zones.html">AWS — Availability Zones as fault-isolation boundaries</a>.</li><li><a href="https://aws.amazon.com/ec2/nitro/">AWS — Nitro System</a>, <a href="https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/enhanced-networking-ena.html">ENA</a>, and <a href="https://aws.amazon.com/hpc/efa/">Elastic Fabric Adapter</a>.</li><li><a href="https://etcd.io/docs/v3.6/learning/">etcd learning documentation</a> and <a href="https://etcd.io/docs/v3.6/op-guide/recovery/">disaster recovery guidance</a>.</li><li><a href="https://www.cockroachlabs.com/docs/stable/architecture/replication-layer">CockroachDB replication layer</a>; <a href="https://tikv.org/deep-dive/scalability/multi-raft/">TiKV Multi-Raft</a>; <a href="https://apple.github.io/foundationdb/">FoundationDB documentation</a>.</li><li><a href="https://netflixtechblog.com/fit-failure-injection-testing-35d8e2a9bb2">Netflix — Failure Injection Testing</a>.</li><li><a href="https://kafka.apache.org/documentation/#replication">Apache Kafka — replication design</a>.</li><li><a href="https://doc.dpdk.org/guides/prog_guide/">DPDK Programmer’s Guide</a>.</li></ol><h3>Before you go</h3><ul><li>Please take a moment to like the post and follow the writer!</li><li>Did you know that over 400,000 developers share what they’re building, learning, and discovering across our platforms every month? Learn how you can contribute <a href="https://plainenglish.io/write-for-us">here</a></li></ul><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=422dcee2f072" width="1" height="1" alt=""><hr><p><a href="https://blog.stackademic.com/when-five-fast-machines-became-one-fragile-exchange-422dcee2f072">When Five Fast Machines Became One Fragile Exchange</a> was originally published in <a href="https://blog.stackademic.com/">Stackademic</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Linux System Monitoring: A Practical DevOps Guide to Finding Performance Bottlenecks]]></title>
            <link>https://medium.com/@shivam2003/linux-system-monitoring-a-practical-devops-guide-to-finding-performance-bottlenecks-7dd302e273e0?source=rss-8c3f1e81861d------2</link>
            <guid isPermaLink="false">https://medium.com/p/7dd302e273e0</guid>
            <dc:creator><![CDATA[Vigilant Engineer]]></dc:creator>
            <pubDate>Tue, 14 Jul 2026 04:02:41 GMT</pubDate>
            <atom:updated>2026-07-14T04:02:41.615Z</atom:updated>
            <content:encoded><![CDATA[<p>Linux system monitoring is most useful when it answers a specific question: what changed, where is the system waiting, and which workload is responsible? This guide gives you a repeatable way to investigate CPU, memory, storage, and network pressure without treating every high number as an incident.</p><h4>Start with a baseline</h4><p>Before diving into a single metric, capture the system’s shape and current load. Run the same commands during a healthy period and during an incident; comparison is far more useful than a one-off snapshot.</p><pre>hostnamectl<br>uptime<br>lscpu<br>free -h<br>lsblk<br>ip -br addr</pre><p>The load average in uptime is a queue-length signal, not a CPU percentage. Compare it with the number of logical CPUs from nproc, then confirm whether the queue is caused by runnable work, blocked I/O, or both.</p><h4>CPU: find saturation before blaming the application</h4><p>Use mpstat to see whether one core is overloaded or the whole machine is busy.</p><pre>mpstat -P ALL 1 5<br>top -o %CPU<br>ps aux --sort=-%cpu | head -n 10</pre><p>Focus on %usr, %sys, %iowait, and %idle. Sustained user or system CPU can mean real compute pressure; high iowait points to storage or network-backed I/O. A single hot core may indicate a single-threaded workload, uneven interrupt handling, or CPU affinity. On virtual machines, elevated %steal can indicate contention on the host.</p><h4>Memory: watch available memory and reclaim activity</h4><p>Linux deliberately uses spare RAM for cache, so low free memory alone is not a problem. Look instead at available memory, swap activity, and whether processes are being reclaimed or killed.</p><pre>free -h<br>vmstat 1 10<br>ps aux --sort=-%mem | head -n 10<br>cat /proc/pressure/memory</pre><p>In vmstat, non-zero si or so over several samples is a strong signal of memory pressure. Check the kernel log for out-of-memory events when the system becomes unstable.</p><pre>dmesg -T | grep -Ei &#39;out of memory|killed process&#39;</pre><h4>Storage: separate capacity problems from latency problems</h4><p>Disk-full alerts and slow disks are different failures, so check both. df shows filesystem capacity, while iostat shows request behaviour and device latency.</p><pre>df -hT<br>iostat -xz 1 10<br>pidstat -d 1 5</pre><p>On devices with extended statistics, await measures average request time and %util shows how busy the device is. Interpret them together: a busy device with rising await and queue depth is a likely bottleneck. Avoid a universal threshold — acceptable latency depends on the device, workload, and established baseline.</p><h4>Network: verify errors, drops, and connection pressure</h4><p>Start by confirming the interface name rather than assuming eth0 exists.</p><pre>ip -br link<br>sar -n DEV 1 10<br>ss -s<br>ip -s link show dev &lt;interface&gt;</pre><p>Look for increasing drops, errors, retransmissions, or a sudden increase in established connections. Use ss -tanp to identify the owning process when connection volume is the concern. Test a known dependency — not a public address — when measuring application connectivity.</p><h4>A practical incident workflow</h4><p>1. Establish the symptom: latency, error rate, throughput, or a host alert.<br>2. Capture a short sample with vmstat 1 10, mpstat -P ALL 1 10, and iostat -xz 1 10.<br>3. Identify the responsible process using top, ps, pidstat, or ss.<br>4. Correlate the time window with deploys, scheduled jobs, traffic changes, and kernel logs.<br>5. Change one thing at a time, then repeat the same sample to verify the outcome.</p><p>For deeper CPU investigation, perf can reveal hotspots when permissions and kernel configuration allow it.</p><pre>perf stat -p &lt;PID&gt; sleep 30<br>perf record -g -p &lt;PID&gt; -- sleep 30<br>perf report</pre><p>Use this after you have identified a CPU-bound process. Profiling every process during an outage adds noise and may create unnecessary overhead.</p><h4>Build a lightweight baseline</h4><p>A small scheduled capture makes trends visible and provides useful evidence during incident review.</p><pre>#!/usr/bin/env bash<br>set -euo pipefail<br>stamp=$(date +%Y%m%d-%H%M%S)<br>out=/var/log/system-baseline-$stamp.txt<br>{<br>  echo &quot;== $(date -Is) ==&quot;<br>  uptime<br>  free -h<br>  vmstat 1 2<br>  mpstat -P ALL 1 1<br>  iostat -xz 1 1<br>  df -hT<br>  ss -s<br>} &gt; &quot;$out&quot;</pre><p>Run it with the least privilege required, store logs where they can be retained and rotated, and avoid collecting sensitive process arguments in shared locations.</p><h4>Closing thought</h4><p>Good monitoring is not a dashboard full of thresholds. It is a disciplined habit of comparing current behaviour with a known baseline, correlating signals across the host, and verifying a change with fresh measurements. With that workflow, Linux tools such as vmstat, mpstat, iostat, ss, and perf become a practical path from symptom to cause.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=7dd302e273e0" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Building a Robust SMS Service with AWS SNS for Indian Markets]]></title>
            <link>https://blog.stackademic.com/building-a-robust-sms-service-with-aws-sns-for-indian-markets-0a9d50861499?source=rss-8c3f1e81861d------2</link>
            <guid isPermaLink="false">https://medium.com/p/0a9d50861499</guid>
            <category><![CDATA[sns]]></category>
            <category><![CDATA[sms]]></category>
            <category><![CDATA[aws]]></category>
            <category><![CDATA[india]]></category>
            <dc:creator><![CDATA[Vigilant Engineer]]></dc:creator>
            <pubDate>Sat, 02 Aug 2025 06:32:24 GMT</pubDate>
            <atom:updated>2025-08-04T05:59:03.729Z</atom:updated>
            <content:encoded><![CDATA[<p><em>Navigating TRAI compliance, AWS configuration, and common pitfalls when implementing SMS messaging for Indian users</em></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*dC-LwX45ZmrykACXoHhXDw.png" /></figure><h3>Introduction</h3><p>Sending SMS messages to Indian phone numbers isn’t as straightforward as you might think. India has one of the most regulated SMS ecosystems in the world, thanks to the Telecom Regulatory Authority of India (TRAI). If you’re building an application that needs to send OTPs, notifications, or marketing messages to Indian users, you’ll need to navigate through a complex web of registrations, templates, and compliance requirements.</p><p>In this comprehensive guide, I’ll walk you through the entire process of setting up AWS Simple Notification Service (SNS) for SMS delivery in India, sharing real-world insights, common pitfalls, and best practices I’ve learned while implementing this for production applications.</p><blockquote><strong><em>Why This Matters</em></strong><em>: India processes over 3 billion SMS messages daily. Getting this right means reaching your users reliably; getting it wrong means your messages never arrive.</em></blockquote><h3>Understanding the Indian SMS Landscape</h3><p>Before diving into the technical implementation, let’s understand why India requires special handling:</p><h3>The TRAI Framework</h3><p>The Telecom Regulatory Authority of India (TRAI) implemented strict regulations to combat spam and ensure message authenticity. Every commercial SMS must be:</p><ol><li><strong>Pre-approved</strong>: Your message template must be registered and approved</li><li><strong>Traceable</strong>: Each message must include specific identifiers</li><li><strong>Compliant</strong>: Content must match exactly with registered templates</li></ol><h3>Key Compliance Requirements</h3><p>When sending SMS in India, you need these four critical components:</p><ul><li><strong>PE-ID (Principal Entity ID)</strong>: Your unique business identifier with TRAI</li><li><strong>Template ID</strong>: Pre-approved message template identifier</li><li><strong>Entity ID</strong>: Your registered entity identifier</li><li><strong>Sender ID</strong>: Your brand name (up to 6 characters)</li></ul><h3>Prerequisites: What You Need Before Starting</h3><h3>1. TRAI Registration Process</h3><p><strong>Step 1: Register Your Business Entity</strong></p><pre>Visit: https://www.vilpower.in/ (or other DLT portals)<br>Documents Required:<br>- Business registration certificate<br>- PAN card<br>- Authorization letter<br>- Sample message templates</pre><p><strong>What You’ll Get:</strong></p><ul><li><strong>PE-ID</strong>: Principal Entity ID (your unique business identifier)</li><li><strong>Entity ID</strong>: Specific to your messaging use case</li><li><strong>Chain ID</strong>: Aggregator chain identifier</li></ul><p><strong>Timeline</strong>: 7–15 business days for approval</p><h3>2. Template Registration</h3><p>Every message you send must match a pre-approved template:</p><pre>Template Example:<br>&quot;Shivam: Your OTP for login is {#var#}\n for login&quot;</pre><pre>Approved Template ID: 1107162228xxxxxxxx</pre><blockquote><strong><em>Pro Tip</em></strong><em>: Register multiple template variations during the initial setup. Adding new templates later requires the same approval process.</em></blockquote><h3>3. AWS Account Setup</h3><p>Ensure your AWS account has:</p><ul><li><strong>End User Messaging</strong> service enabled</li><li><strong>SNS</strong> permissions configured</li><li><strong>Appropriate region</strong> selected (ap-south-1 for India)</li></ul><h3>Step-by-Step AWS SNS Configuration</h3><h3>Phase 1: Initial AWS Setup</h3><p><strong>1. Request SNS Spending Limits</strong></p><p>AWS starts with a low SMS spending limit. Here’s how to increase it:</p><pre># Contact AWS Support with this information:<br>- Use Case: SMS notifications for Indian users<br>- Expected Monthly Volume: [Your estimate]<br>- Business Justification: [Your use case]<br>- Initial Request: $50-100 limit</pre><p><strong>Timeline</strong>: 24–48 hours for initial approval</p><p><strong>Scaling Strategy</strong>:</p><ul><li>Start with $50 limit</li><li>Request increases when you reach 50% usage</li><li>Provide usage analytics for faster approvals</li></ul><h3>2. Configure Sender ID Registration</h3><p>Navigate to <strong>AWS End User Messaging Console</strong>:</p><pre>AWS Console → End User Messaging → SMS → Sender IDs</pre><p><strong>Register Your Sender ID</strong>:</p><ul><li><strong>Sender ID</strong>: Your brand name (max 6 chars, e.g., “YOURCO”)</li><li><strong>Country</strong>: India (IN)</li><li><strong>Use Case</strong>: Transactional/Promotional</li><li><strong>Monthly Volume</strong>: Your estimated volume</li></ul><h3>Phase 2: Resource Policy Configuration</h3><p>This is where most developers get stuck. You need to configure a resource policy that allows SNS to use your registered Sender ID for local (Indian) route delivery.</p><p><strong>Why This Matters</strong>: Without proper resource policy, your messages will either fail or route through expensive international gateways.</p><h4>Creating the Resource Policy</h4><p>Navigate to your Sender ID settings and configure this policy:</p><pre>{<br>  &quot;Version&quot;: &quot;2012-10-17&quot;,<br>  &quot;Statement&quot;: [<br>    {<br>      &quot;Effect&quot;: &quot;Allow&quot;,<br>      &quot;Principal&quot;: {<br>        &quot;Service&quot;: &quot;pinpoint.amazonaws.com&quot;<br>      },<br>      &quot;Action&quot;: &quot;sms-voice:SendTextMessage&quot;,<br>      &quot;Resource&quot;: &quot;arn:aws:sms-voice:ap-south-1:YOUR_ACCOUNT_ID:sender-id/YOUR_SENDER_ID/IN&quot;,<br>      &quot;Condition&quot;: {<br>        &quot;StringEquals&quot;: {<br>          &quot;aws:SourceAccount&quot;: &quot;YOUR_ACCOUNT_ID&quot;<br>        }<br>      }<br>    },<br>    {<br>      &quot;Effect&quot;: &quot;Allow&quot;,<br>      &quot;Principal&quot;: {<br>        &quot;Service&quot;: &quot;sns.amazonaws.com&quot;<br>      },<br>      &quot;Action&quot;: &quot;sms-voice:SendTextMessage&quot;,<br>      &quot;Resource&quot;: &quot;arn:aws:sms-voice:ap-south-1:YOUR_ACCOUNT_ID:sender-id/YOUR_SENDER_ID/IN&quot;,<br>      &quot;Condition&quot;: {<br>        &quot;StringEquals&quot;: {<br>          &quot;aws:SourceAccount&quot;: &quot;YOUR_ACCOUNT_ID&quot;<br>        }<br>      }<br>    }<br>  ]<br>}</pre><p><strong>Configuration Details</strong>:</p><ul><li>Replace YOUR_ACCOUNT_ID with your AWS account ID</li><li>Replace YOUR_SENDER_ID with your registered sender ID</li><li>Keep the region as ap-south-1 for India</li></ul><blockquote><strong><em>Security Note</em></strong><em>: This policy restricts SMS sending to your specific sender ID and account, preventing unauthorized usage.</em></blockquote><h3>Implementation: Sending Your First SMS</h3><h3>Method 1: AWS CLI Implementation</h3><p>Here’s a complete working example:</p><pre>aws sns publish \<br>  --phone-number &quot;+91XXXXXXXXXX&quot; \<br>  --message &quot;WFM: Your OTP for login is 123456\nYour Brand&quot; \<br>  --message-attributes &#39;{<br>    &quot;AWS.SNS.SMS.SMSType&quot;: {<br>      &quot;DataType&quot;: &quot;String&quot;,<br>      &quot;StringValue&quot;: &quot;Transactional&quot;<br>    },<br>    &quot;AWS.SNS.SMS.SenderID&quot;: {<br>      &quot;DataType&quot;: &quot;String&quot;,<br>      &quot;StringValue&quot;: &quot;YOURID&quot;<br>    },<br>    &quot;AWS.MM.SMS.TemplateId&quot;: {<br>      &quot;DataType&quot;: &quot;String&quot;,<br>      &quot;StringValue&quot;: &quot;110716222808xxxx&quot;<br>    },<br>    &quot;AWS.MM.SMS.EntityId&quot;: {<br>      &quot;DataType&quot;: &quot;String&quot;,<br>      &quot;StringValue&quot;: &quot;11014890400xxxx&quot;<br>    }<br>  }&#39; \<br>  --region ap-south-1</pre><h3>Method 2: Python SDK Implementation</h3><pre>import boto3<br>import json<br>def send_sms_india(phone_number, otp_code):<br>    &quot;&quot;&quot;<br>    Send OTP SMS to Indian phone number via AWS SNS<br>    &quot;&quot;&quot;<br>    sns_client = boto3.client(&#39;sns&#39;, region_name=&#39;ap-south-1&#39;)<br>    <br>    # Message must match your registered template exactly<br>    message = f&quot;WFM: Your OTP for login is {otp_code}\nYour Brand&quot;<br>    <br>    message_attributes = {<br>        &#39;AWS.SNS.SMS.SMSType&#39;: {<br>            &#39;DataType&#39;: &#39;String&#39;,<br>            &#39;StringValue&#39;: &#39;Transactional&#39;<br>        },<br>        &#39;AWS.SNS.SMS.SenderID&#39;: {<br>            &#39;DataType&#39;: &#39;String&#39;,<br>            &#39;StringValue&#39;: &#39;YOURID&#39;<br>        },<br>        &#39;AWS.MM.SMS.TemplateId&#39;: {<br>            &#39;DataType&#39;: &#39;String&#39;,<br>            &#39;StringValue&#39;: &#39;11071622280xxxxxxx&#39;<br>        },<br>        &#39;AWS.MM.SMS.EntityId&#39;: {<br>            &#39;DataType&#39;: &#39;String&#39;,<br>            &#39;StringValue&#39;: &#39;11014890xxxxxxx&#39;<br>        }<br>    }<br>    <br>    try:<br>        response = sns_client.publish(<br>            PhoneNumber=phone_number,<br>            Message=message,<br>            MessageAttributes=message_attributes<br>        )<br>        <br>        return {<br>            &#39;success&#39;: True,<br>            &#39;message_id&#39;: response[&#39;MessageId&#39;],<br>            &#39;response&#39;: response<br>        }<br>        <br>    except Exception as e:<br>        return {<br>            &#39;success&#39;: False,<br>            &#39;error&#39;: str(e)<br>        }<br># Usage<br>result = send_sms_india(&quot;+917503xxxxxxx&quot;, &quot;123456&quot;)<br>print(result)</pre><h3>Method 3: Node.js Implementation</h3><pre>const AWS = require(&#39;aws-sdk&#39;);<br>// Configure AWS SNS<br>const sns = new AWS.SNS({<br>    region: &#39;ap-south-1&#39;,<br>    accessKeyId: process.env.AWS_ACCESS_KEY_ID,<br>    secretAccessKey: process.env.AWS_SECRET_ACCESS_KEY<br>});<br>async function sendSMSIndia(phoneNumber, otpCode) {<br>    const message = `WFM: Your OTP for login is ${otpCode}\nYour Brand`;<br>    <br>    const params = {<br>        PhoneNumber: phoneNumber,<br>        Message: message,<br>        MessageAttributes: {<br>            &#39;AWS.SNS.SMS.SMSType&#39;: {<br>                DataType: &#39;String&#39;,<br>                StringValue: &#39;Transactional&#39;<br>            },<br>            &#39;AWS.SNS.SMS.SenderID&#39;: {<br>                DataType: &#39;String&#39;,<br>                StringValue: &#39;YOURID&#39;<br>            },<br>            &#39;AWS.MM.SMS.TemplateId&#39;: {<br>                DataType: &#39;String&#39;,<br>                StringValue: &#39;1107162228xxxxxxx&#39;<br>            },<br>            &#39;AWS.MM.SMS.EntityId&#39;: {<br>                DataType: &#39;String&#39;,<br>                StringValue: &#39;110148904xxxxxxxx&#39;<br>            }<br>        }<br>    };<br>    <br>    try {<br>        const result = await sns.publish(params).promise();<br>        return {<br>            success: true,<br>            messageId: result.MessageId<br>        };<br>    } catch (error) {<br>        return {<br>            success: false,<br>            error: error.message<br>        };<br>    }<br>}<br>// Usage<br>sendSMSIndia(&#39;+91750348xxxxx&#39;, &#39;123456&#39;)<br>    .then(result =&gt; console.log(result))<br>    .catch(error =&gt; console.error(error));</pre><h3>Understanding Message Attributes: The Critical Details</h3><p>Each message attribute serves a specific purpose in the TRAI compliance framework:</p><h3>Required Attributes Breakdown</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*bGOm6TLRDK9U_SMRAUc1-g.png" /></figure><h3>Message Type Guidelines</h3><p><strong>Transactional Messages</strong>:</p><ul><li>OTPs, alerts, confirmations</li><li>No promotional content</li><li>Higher delivery priority</li><li>24/7 delivery allowed</li></ul><p><strong>Promotional Messages</strong>:</p><ul><li>Marketing, offers, newsletters</li><li>Requires explicit user consent</li><li>Delivery restricted to 9 AM — 9 PM</li><li>Lower delivery priority</li></ul><h3>Testing Your Implementation</h3><h3>Manual Testing via AWS Console</h3><p>Before automating, test manually through AWS Console:</p><ol><li><strong>Navigate to SNS Console</strong></li></ol><pre>AWS Console → Simple Notification Service → Text Messaging (SMS)</pre><p><strong>Publish Test Message</strong></p><ul><li>Select “Publish text message”</li><li>Enter phone number with country code (+91)</li><li>Add your message (must match template)</li><li>Include all required message attributes</li></ul><p><strong>Monitor Delivery</strong></p><ul><li>Check message status in SNS console</li><li>Look for delivery receipts</li><li>Verify message received on test device</li></ul><h3>Automated Testing Script</h3><pre>import boto3<br>import time<br>from datetime import datetime<br>def test_sms_delivery():<br>    &quot;&quot;&quot;Comprehensive SMS testing function&quot;&quot;&quot;<br>    <br>    test_cases = [<br>        {<br>            &#39;phone&#39;: &#39;+9175034xxxx&#39;,<br>            &#39;otp&#39;: &#39;123456&#39;,<br>            &#39;expected&#39;: &#39;success&#39;<br>        },<br>        {<br>            &#39;phone&#39;: &#39;+9198765xxxxxx&#39;,<br>            &#39;otp&#39;: &#39;789012&#39;,<br>            &#39;expected&#39;: &#39;success&#39;<br>        }<br>    ]<br>    <br>    results = []<br>    <br>    for test in test_cases:<br>        print(f&quot;Testing SMS to {test[&#39;phone&#39;]}...&quot;)<br>        <br>        result = send_sms_india(test[&#39;phone&#39;], test[&#39;otp&#39;])<br>        <br>        results.append({<br>            &#39;phone&#39;: test[&#39;phone&#39;],<br>            &#39;timestamp&#39;: datetime.now(),<br>            &#39;result&#39;: result,<br>            &#39;expected&#39;: test[&#39;expected&#39;]<br>        })<br>        <br>        # Rate limiting - avoid hitting AWS limits<br>        time.sleep(1)<br>    <br>    return results<br># Run tests<br>test_results = test_sms_delivery()<br>for result in test_results:<br>    print(f&quot;Phone: {result[&#39;phone&#39;]}, Success: {result[&#39;result&#39;][&#39;success&#39;]}&quot;)</pre><h3>Monitoring and Delivery Status</h3><p>Setting up proper monitoring is crucial for production applications. Here’s how to implement comprehensive SMS tracking:</p><h3>CloudWatch Integration</h3><pre>import boto3<br>from datetime import datetime, timedelta<br>def setup_sms_monitoring():<br>    &quot;&quot;&quot;Set up CloudWatch alarms for SMS monitoring&quot;&quot;&quot;<br>    <br>    cloudwatch = boto3.client(&#39;cloudwatch&#39;, region_name=&#39;ap-south-1&#39;)<br>    <br>    # Create alarm for failed SMS deliveries<br>    cloudwatch.put_metric_alarm(<br>        AlarmName=&#39;SMS-Delivery-Failures&#39;,<br>        ComparisonOperator=&#39;GreaterThanThreshold&#39;,<br>        EvaluationPeriods=2,<br>        MetricName=&#39;NumberOfMessagesFailed&#39;,<br>        Namespace=&#39;AWS/SNS&#39;,<br>        Period=300,<br>        Statistic=&#39;Sum&#39;,<br>        Threshold=5.0,<br>        ActionsEnabled=True,<br>        AlarmActions=[<br>            &#39;arn:aws:sns:ap-south-1:YOUR_ACCOUNT:sms-alerts&#39;<br>        ],<br>        AlarmDescription=&#39;Alert when SMS delivery failures exceed threshold&#39;<br>    )<br>    <br>    # Create alarm for high SMS costs<br>    cloudwatch.put_metric_alarm(<br>        AlarmName=&#39;SMS-Cost-Alert&#39;,<br>        ComparisonOperator=&#39;GreaterThanThreshold&#39;,<br>        EvaluationPeriods=1,<br>        MetricName=&#39;SMSMonthlySpend&#39;,<br>        Namespace=&#39;AWS/SNS&#39;,<br>        Period=3600,<br>        Statistic=&#39;Maximum&#39;,<br>        Threshold=40.0,  # Alert at 80% of $50 limit<br>        ActionsEnabled=True,<br>        AlarmActions=[<br>            &#39;arn:aws:sns:ap-south-1:YOUR_ACCOUNT:cost-alerts&#39;<br>        ],<br>        AlarmDescription=&#39;Alert when SMS costs approach limit&#39;<br>    )</pre><h3>Delivery Receipt Handling</h3><pre>def track_delivery_status(message_id, phone_number):<br>    &quot;&quot;&quot;Track SMS delivery status using SNS message attributes&quot;&quot;&quot;<br>    <br>    sns = boto3.client(&#39;sns&#39;, region_name=&#39;ap-south-1&#39;)<br>    <br>    try:<br>        # Get message attributes (delivery status)<br>        response = sns.get_sms_attributes()<br>        <br>        delivery_status = {<br>            &#39;message_id&#39;: message_id,<br>            &#39;phone_number&#39;: phone_number,<br>            &#39;timestamp&#39;: datetime.now(),<br>            &#39;status&#39;: &#39;sent&#39;,<br>            &#39;delivery_status&#39;: response.get(&#39;attributes&#39;, {}).get(&#39;DeliveryStatusLogging&#39;, &#39;disabled&#39;)<br>        }<br>        <br>        # Store in your database for tracking<br>        store_delivery_status(delivery_status)<br>        <br>        return delivery_status<br>        <br>    except Exception as e:<br>        print(f&quot;Error tracking delivery: {e}&quot;)<br>        return None<br>def store_delivery_status(status):<br>    &quot;&quot;&quot;Store delivery status in your database&quot;&quot;&quot;<br>    # Implement your database storage logic here<br>    pass<br>    </pre><h3>Troubleshooting Common Issues</h3><p>From my experience implementing SMS services for multiple production applications, here are the most common issues and their solutions:</p><h3>1. “Invalid Message ID” Error</h3><p><strong>Symptoms</strong>:</p><pre>Error: invalid message id<br>HTTP Status: 400</pre><p><strong>Root Causes &amp; Solutions</strong>:</p><p><strong>A. Template ID Mismatch</strong></p><pre># ❌ Wrong - Template ID doesn&#39;t match message content<br>message = &quot;Your OTP is 123456&quot;  # Different from registered template<br>template_id = &quot;110716222xxxxxxxx&quot;<br><br><br># ✅ Correct - Message matches template exactly<br>message = &quot;XYZ: Your OTP for login is 123456\n.ABC&quot;<br>template_id = &quot;11071622xxxxxxxxx&quot;</pre><p><strong>B. Extra Spaces or Characters</strong></p><pre># ❌ Wrong - Extra space before message<br>message = &quot; XYZ: Your OTP for login is 123456\n ABC&quot;<br><br><br># ✅ Correct - Exact template match<br>message = &quot;XYZ: Your OTP for login is 123456\n ABC&quot;</pre><p><strong>C. Incorrect Entity ID</strong></p><pre># Verify your Entity ID matches TRAI registration<br>entity_id = &quot;1101489xxxxxxxxx&quot;  # Must match exactly</pre><h3>2. “Unknown Error Attempting to Reach Phone”</h3><p><strong>Symptoms</strong>:</p><pre>Error: Unknown error attempting to reach phone<br>HTTP Status: 500</pre><p><strong>Solution Checklist</strong>:</p><pre>def validate_sms_parameters(phone, message_attrs):<br>    &quot;&quot;&quot;Validate all SMS parameters before sending&quot;&quot;&quot;<br>    <br>    required_attrs = [<br>        &#39;AWS.SNS.SMS.SMSType&#39;,<br>        &#39;AWS.SNS.SMS.SenderID&#39;, <br>        &#39;AWS.MM.SMS.TemplateId&#39;,<br>        &#39;AWS.MM.SMS.EntityId&#39;<br>    ]<br>    <br>    # Check all required attributes present<br>    for attr in required_attrs:<br>        if attr not in message_attrs:<br>            raise ValueError(f&quot;Missing required attribute: {attr}&quot;)<br>    <br>    # Validate phone number format<br>    if not phone.startswith(&#39;+91&#39;):<br>        raise ValueError(&quot;Phone number must include +91 country code&quot;)<br>    <br>    # Validate sender ID<br>    sender_id = message_attrs[&#39;AWS.SNS.SMS.SenderID&#39;][&#39;StringValue&#39;]<br>    if len(sender_id) &gt; 6:<br>        raise ValueError(&quot;Sender ID cannot exceed 6 characters&quot;)<br>    <br>    return True</pre><h3>3. Messages Not Delivered (Silent Failures)</h3><p><strong>Debugging Steps</strong>:</p><pre>def debug_sms_delivery():<br>    &quot;&quot;&quot;Comprehensive SMS delivery debugging&quot;&quot;&quot;<br>    <br>    sns = boto3.client(&#39;sns&#39;, region_name=&#39;ap-south-1&#39;)<br>    <br>    # Check SNS service status<br>    try:<br>        response = sns.get_sms_attributes()<br>        print(&quot;SNS Service Status:&quot;, response)<br>    except Exception as e:<br>        print(f&quot;SNS Service Error: {e}&quot;)<br>    <br>    # Verify spending limits<br>    monthly_spend = response.get(&#39;attributes&#39;, {}).get(&#39;MonthlySpendLimit&#39;, &#39;0&#39;)<br>    print(f&quot;Monthly Spend Limit: ${monthly_spend}&quot;)<br>    <br>    # Check default SMS type<br>    default_sms_type = response.get(&#39;attributes&#39;, {}).get(&#39;DefaultSMSType&#39;, &#39;Promotional&#39;)<br>    print(f&quot;Default SMS Type: {default_sms_type}&quot;)<br>    <br>    # Verify delivery status logging<br>    delivery_logging = response.get(&#39;attributes&#39;, {}).get(&#39;DeliveryStatusLogging&#39;, &#39;disabled&#39;)<br>    print(f&quot;Delivery Status Logging: {delivery_logging}&quot;)</pre><h3>4. High Delivery Costs</h3><p><strong>Cost Optimization Strategies</strong>:</p><pre>def optimize_sms_costs():<br>    &quot;&quot;&quot;Implement cost optimization for SMS delivery&quot;&quot;&quot;<br>    <br>    # Use local route (cheaper) for Indian numbers<br>    message_attributes = {<br>        &#39;AWS.SNS.SMS.SMSType&#39;: {<br>            &#39;DataType&#39;: &#39;String&#39;,<br>            &#39;StringValue&#39;: &#39;Transactional&#39;  # Cheaper than Promotional<br>        },<br>        # Always include Sender ID for local routing<br>        &#39;AWS.SNS.SMS.SenderID&#39;: {<br>            &#39;DataType&#39;: &#39;String&#39;, <br>            &#39;StringValue&#39;: &#39;YOURID&#39;<br>        }<br>    }<br>    <br>    # Implement rate limiting to avoid burst charges<br>    import time<br>    time.sleep(0.1)  # 100ms delay between messages<br>    <br>    return message_attributes</pre><h3>Production Best Practices</h3><h3>1. Rate Limiting Implementation</h3><pre>import time<br>from collections import defaultdict<br>from datetime import datetime, timedelta<br>class SMSRateLimiter:<br>    def __init__(self, max_per_minute=60, max_per_hour=1000):<br>        self.max_per_minute = max_per_minute<br>        self.max_per_hour = max_per_hour<br>        self.minute_counter = defaultdict(int)<br>        self.hour_counter = defaultdict(int)<br>        self.last_cleanup = datetime.now()<br>    <br>    def can_send_sms(self, phone_number):<br>        &quot;&quot;&quot;Check if SMS can be sent based on rate limits&quot;&quot;&quot;<br>        <br>        now = datetime.now()<br>        current_minute = now.strftime(&quot;%Y-%m-%d-%H-%M&quot;)<br>        current_hour = now.strftime(&quot;%Y-%m-%d-%H&quot;)<br>        <br>        # Cleanup old counters<br>        if now - self.last_cleanup &gt; timedelta(hours=1):<br>            self._cleanup_counters()<br>        <br>        # Check limits<br>        if self.minute_counter[f&quot;{phone_number}-{current_minute}&quot;] &gt;= self.max_per_minute:<br>            return False, &quot;Rate limit exceeded: too many messages per minute&quot;<br>        <br>        if self.hour_counter[f&quot;{phone_number}-{current_hour}&quot;] &gt;= self.max_per_hour:<br>            return False, &quot;Rate limit exceeded: too many messages per hour&quot;<br>        <br>        return True, &quot;OK&quot;<br>    <br>    def record_sms_sent(self, phone_number):<br>        &quot;&quot;&quot;Record that an SMS was sent&quot;&quot;&quot;<br>        now = datetime.now()<br>        current_minute = now.strftime(&quot;%Y-%m-%d-%H-%M&quot;)<br>        current_hour = now.strftime(&quot;%Y-%m-%d-%H&quot;)<br>        <br>        self.minute_counter[f&quot;{phone_number}-{current_minute}&quot;] += 1<br>        self.hour_counter[f&quot;{phone_number}-{current_hour}&quot;] += 1<br>    <br>    def _cleanup_counters(self):<br>        &quot;&quot;&quot;Remove old counter entries&quot;&quot;&quot;<br>        now = datetime.now()<br>        cutoff_hour = (now - timedelta(hours=2)).strftime(&quot;%Y-%m-%d-%H&quot;)<br>        cutoff_minute = (now - timedelta(minutes=5)).strftime(&quot;%Y-%m-%d-%H-%M&quot;)<br>        <br>        # Clean hour counters<br>        self.hour_counter = {<br>            k: v for k, v in self.hour_counter.items() <br>            if k.split(&#39;-&#39;)[-1] &gt; cutoff_hour.split(&#39;-&#39;)[-1]<br>        }<br>        <br>        # Clean minute counters  <br>        self.minute_counter = {<br>            k: v for k, v in self.minute_counter.items()<br>            if &#39;-&#39;.join(k.split(&#39;-&#39;)[-2:]) &gt; &#39;-&#39;.join(cutoff_minute.split(&#39;-&#39;)[-2:])<br>        }<br>        <br>        self.last_cleanup = now<br># Usage<br>rate_limiter = SMSRateLimiter()<br>def send_rate_limited_sms(phone_number, otp):<br>    &quot;&quot;&quot;Send SMS with rate limiting&quot;&quot;&quot;<br>    <br>    can_send, reason = rate_limiter.can_send_sms(phone_number)<br>    <br>    if not can_send:<br>        return {<br>            &#39;success&#39;: False,<br>            &#39;error&#39;: reason<br>        }<br>    <br>    # Send SMS<br>    result = send_sms_india(phone_number, otp)<br>    <br>    if result[&#39;success&#39;]:<br>        rate_limiter.record_sms_sent(phone_number)<br>    <br>    return result</pre><h3>2. Template Management System</h3><pre>class SMSTemplateManager:<br>    &quot;&quot;&quot;Manage SMS templates and validation&quot;&quot;&quot;<br>    <br>    def __init__(self):<br>        self.templates = {<br>            &#39;otp_login&#39;: {<br>                &#39;template&#39;: &#39;XYZ: Your OTP for login is {otp}\n ABC&#39;,<br>                &#39;template_id&#39;: &#39;11071622280xxxxxx&#39;,<br>                &#39;entity_id&#39;: &#39;11014890400xxxxxxx&#39;,<br>                &#39;variables&#39;: [&#39;otp&#39;]<br>            },<br>            &#39;password_reset&#39;: {<br>                &#39;template&#39;: &#39;XYZ: Your password reset OTP is {otp}\n ABC&#39;,<br>                &#39;template_id&#39;: &#39;1107162228xxxxxxx&#39;, <br>                &#39;entity_id&#39;: &#39;110148904xxxxxx&#39;,<br>                &#39;variables&#39;: [&#39;otp&#39;]<br>            }<br>        }<br>    <br>    def get_template(self, template_name):<br>        &quot;&quot;&quot;Get template configuration&quot;&quot;&quot;<br>        return self.templates.get(template_name)<br>    <br>    def format_message(self, template_name, **variables):<br>        &quot;&quot;&quot;Format message with variables&quot;&quot;&quot;<br>        template_config = self.get_template(template_name)<br>        <br>        if not template_config:<br>            raise ValueError(f&quot;Template {template_name} not found&quot;)<br>        <br>        # Validate required variables<br>        missing_vars = set(template_config[&#39;variables&#39;]) - set(variables.keys())<br>        if missing_vars:<br>            raise ValueError(f&quot;Missing variables: {missing_vars}&quot;)<br>        <br>        # Format message<br>        message = template_config[&#39;template&#39;].format(**variables)<br>        <br>        return {<br>            &#39;message&#39;: message,<br>            &#39;template_id&#39;: template_config[&#39;template_id&#39;],<br>            &#39;entity_id&#39;: template_config[&#39;entity_id&#39;]<br>        }<br>    <br>    def send_templated_sms(self, phone_number, template_name, **variables):<br>        &quot;&quot;&quot;Send SMS using template&quot;&quot;&quot;<br>        <br>        try:<br>            formatted = self.format_message(template_name, **variables)<br>            <br>            message_attributes = {<br>                &#39;AWS.SNS.SMS.SMSType&#39;: {<br>                    &#39;DataType&#39;: &#39;String&#39;,<br>                    &#39;StringValue&#39;: &#39;Transactional&#39;<br>                },<br>                &#39;AWS.SNS.SMS.SenderID&#39;: {<br>                    &#39;DataType&#39;: &#39;String&#39;,<br>                    &#39;StringValue&#39;: &#39;SAVENG&#39;<br>                },<br>                &#39;AWS.MM.SMS.TemplateId&#39;: {<br>                    &#39;DataType&#39;: &#39;String&#39;,<br>                    &#39;StringValue&#39;: formatted[&#39;template_id&#39;]<br>                },<br>                &#39;AWS.MM.SMS.EntityId&#39;: {<br>                    &#39;DataType&#39;: &#39;String&#39;,<br>                    &#39;StringValue&#39;: formatted[&#39;entity_id&#39;]<br>                }<br>            }<br>            <br>            sns = boto3.client(&#39;sns&#39;, region_name=&#39;ap-south-1&#39;)<br>            <br>            response = sns.publish(<br>                PhoneNumber=phone_number,<br>                Message=formatted[&#39;message&#39;],<br>                MessageAttributes=message_attributes<br>            )<br>            <br>            return {<br>                &#39;success&#39;: True,<br>                &#39;message_id&#39;: response[&#39;MessageId&#39;],<br>                &#39;template_used&#39;: template_name<br>            }<br>            <br>        except Exception as e:<br>            return {<br>                &#39;success&#39;: False,<br>                &#39;error&#39;: str(e),<br>                &#39;template_used&#39;: template_name<br>            }<br>   # Usage<br>   template_manager = SMSTemplateManager()<br>   # Send OTP SMS<br>   result = template_manager.send_templated_sms(<br>     phone_number=&#39;+9175034xxxxxx&#39;,<br>     template_name=&#39;otp_login&#39;,<br>     otp=&#39;123456&#39;<br>   )</pre><h3>3. Error Handling and Retry Logic</h3><pre>import time<br>import random<br>from functools import wraps<br>def retry_with_backoff(max_retries=3, base_delay=1):<br>    &quot;&quot;&quot;Decorator for retrying SMS sending with exponential backoff&quot;&quot;&quot;<br>    <br>    def decorator(func):<br>        @wraps(func)<br>        def wrapper(*args, **kwargs):<br>            last_exception = None<br>            <br>            for attempt in range(max_retries + 1):<br>                try:<br>                    return func(*args, **kwargs)<br>                    <br>                except Exception as e:<br>                    last_exception = e<br>                    <br>                    if attempt == max_retries:<br>                        break<br>                    <br>                    # Exponential backoff with jitter<br>                    delay = base_delay * (2 ** attempt) + random.uniform(0, 1)<br>                    time.sleep(delay)<br>                    <br>                    print(f&quot;SMS attempt {attempt + 1} failed: {e}. Retrying in {delay:.2f}s...&quot;)<br>            <br>            # All retries failed<br>            return {<br>                &#39;success&#39;: False,<br>                &#39;error&#39;: f&quot;Failed after {max_retries + 1} attempts: {last_exception}&quot;,<br>                &#39;attempts&#39;: max_retries + 1<br>            }<br>        <br>        return wrapper<br>    return decorator<br>@retry_with_backoff(max_retries=3, base_delay=1)<br>def send_sms_with_retry(phone_number, message, message_attributes):<br>    &quot;&quot;&quot;Send SMS with automatic retry logic&quot;&quot;&quot;<br>    <br>    sns = boto3.client(&#39;sns&#39;, region_name=&#39;ap-south-1&#39;)<br>    <br>    response = sns.publish(<br>        PhoneNumber=phone_number,<br>        Message=message,<br>        MessageAttributes=message_attributes<br>    )<br>    <br>    return {<br>        &#39;success&#39;: True,<br>        &#39;message_id&#39;: response[&#39;MessageId&#39;]<br>    }</pre><h3>Cost Analysis and Optimization</h3><h3>Understanding SMS Pricing</h3><p><strong>AWS SNS Pricing for India (as of 2024)</strong>:</p><ul><li><strong>Local Route</strong>: $0.004 per SMS (with registered Sender ID)</li><li><strong>International Route</strong>: $ 0.069 per SMS (without Sender ID)</li><li><strong>Volume Discounts</strong>: Available for &gt;100K messages/month</li></ul><h3>Cost Calculation Example</h3><pre>def calculate_sms_costs(monthly_volume):<br>    &quot;&quot;&quot;Calculate estimated SMS costs&quot;&quot;&quot;<br>    <br>    # Pricing tiers (example rates)<br>    local_rate = 0.004  # $0.007 per SMS via local route<br>    international_rate = 0.069  # $0.045 per SMS via international route<br>    <br>    # Assume 95% local delivery with proper setup<br>    local_messages = monthly_volume * 0.95<br>    international_messages = monthly_volume * 0.05<br>    <br>    monthly_cost = (local_messages * local_rate) + (international_messages * international_rate)<br>    <br>    return {<br>        &#39;monthly_volume&#39;: monthly_volume,<br>        &#39;local_messages&#39;: int(local_messages),<br>        &#39;international_messages&#39;: int(international_messages),<br>        &#39;monthly_cost_usd&#39;: round(monthly_cost, 2),<br>        &#39;cost_per_message&#39;: round(monthly_cost / monthly_volume, 4),<br>        &#39;annual_cost_usd&#39;: round(monthly_cost * 12, 2)<br>    }<br># Example calculations<br>volumes = [1000, 10000, 50000, 100000]<br>for volume in volumes:<br>    cost_analysis = calculate_sms_costs(volume)<br>    print(f&quot;Volume: {cost_analysis[&#39;monthly_volume&#39;]:,}&quot;)<br>    print(f&quot;Monthly Cost: ${cost_analysis[&#39;monthly_cost_usd&#39;]}&quot;)<br>    print(f&quot;Cost per SMS: ${cost_analysis[&#39;cost_per_message&#39;]}&quot;)<br>    print(&quot;---&quot;)</pre><h3>Security Considerations</h3><h3>1. Credential Management</h3><pre>import os<br>from aws_secretsmanager_caching import SecretCache, SecretCacheConfig<br>class SecureSMSService:<br>    &quot;&quot;&quot;Secure SMS service with proper credential management&quot;&quot;&quot;<br>    <br>    def __init__(self):<br>        # Use AWS Secrets Manager for sensitive data<br>        cache_config = SecretCacheConfig()<br>        self.secret_cache = SecretCache(config=cache_config)<br>        <br>        # Load configuration from secrets<br>        self.config = self._load_secure_config()<br>    <br>    def _load_secure_config(self):<br>        &quot;&quot;&quot;Load configuration from AWS Secrets Manager&quot;&quot;&quot;<br>        try:<br>            secret = self.secret_cache.get_secret_string(&#39;sms-service/config&#39;)<br>            return json.loads(secret)<br>        except:<br>            # Fallback to environment variables<br>            return {<br>                &#39;sender_id&#39;: os.getenv(&#39;SMS_SENDER_ID&#39;),<br>                &#39;template_id&#39;: os.getenv(&#39;SMS_TEMPLATE_ID&#39;),<br>                &#39;entity_id&#39;: os.getenv(&#39;SMS_ENTITY_ID&#39;)<br>            }<br>    <br>    def send_secure_sms(self, phone_number, otp):<br>        &quot;&quot;&quot;Send SMS with secure configuration&quot;&quot;&quot;<br>        <br>        # Validate phone number format<br>        if not self._validate_phone_number(phone_number):<br>            return {&#39;success&#39;: False, &#39;error&#39;: &#39;Invalid phone number&#39;}<br>        <br>        # Rate limiting check<br>        if not self._check_rate_limit(phone_number):<br>            return {&#39;success&#39;: False, &#39;error&#39;: &#39;Rate limit exceeded&#39;}<br>        <br>        # Send SMS<br>        return self._send_sms(phone_number, otp)<br>    <br>    def _validate_phone_number(self, phone):<br>        &quot;&quot;&quot;Validate Indian phone number format&quot;&quot;&quot;<br>        import re<br>        pattern = r&#39;^\+91[6-9]\d{9}$&#39;<br>        return bool(re.match(pattern, phone))<br>    <br>    def _check_rate_limit(self, phone):<br>        &quot;&quot;&quot;Implement rate limiting logic&quot;&quot;&quot;<br>        # Implement your rate limiting logic here<br>        return True<br>    <br>    def _send_sms(self, phone, otp):<br>        &quot;&quot;&quot;Internal SMS sending method&quot;&quot;&quot;<br>        # Implementation details...<br>        pass</pre><h3>2. OTP Security Best Practices</h3><pre>import secrets<br>import string<br>from datetime import datetime, timedelta<br>class SecureOTPManager:<br>    &quot;&quot;&quot;Secure OTP generation and validation&quot;&quot;&quot;<br>    <br>    def __init__(self, redis_client=None):<br>        self.redis = redis_client  # For storing OTP data<br>        self.otp_length = 6<br>        self.otp_expiry = timedelta(minutes=5)<br>        self.max_attempts = 3<br>    <br>    def generate_otp(self, phone_number):<br>        &quot;&quot;&quot;Generate secure OTP&quot;&quot;&quot;<br>        <br>        # Generate cryptographically secure OTP<br>        otp = &#39;&#39;.join(secrets.choice(string.digits) for _ in range(self.otp_length))<br>        <br>        # Store OTP with expiry<br>        otp_data = {<br>            &#39;otp&#39;: otp,<br>            &#39;phone&#39;: phone_number,<br>            &#39;created_at&#39;: datetime.now().isoformat(),<br>            &#39;attempts&#39;: 0,<br>            &#39;max_attempts&#39;: self.max_attempts<br>        }<br>        <br>        # Store in Redis with expiry<br>        if self.redis:<br>            key = f&quot;otp:{phone_number}&quot;<br>            self.redis.setex(key, self.otp_expiry.total_seconds(), json.dumps(otp_data))<br>        <br>        return otp<br>    <br>    def verify_otp(self, phone_number, submitted_otp):<br>        &quot;&quot;&quot;Verify submitted OTP&quot;&quot;&quot;<br>        <br>        if not self.redis:<br>            return {&#39;valid&#39;: False, &#39;error&#39;: &#39;OTP storage not configured&#39;}<br>        <br>        key = f&quot;otp:{phone_number}&quot;<br>        stored_data = self.redis.get(key)<br>        <br>        if not stored_data:<br>            return {&#39;valid&#39;: False, &#39;error&#39;: &#39;OTP expired or not found&#39;}<br>        <br>        otp_data = json.loads(stored_data)<br>        <br>        # Check attempt limit<br>        if otp_data[&#39;attempts&#39;] &gt;= otp_data[&#39;max_attempts&#39;]:<br>            self.redis.delete(key)<br>            return {&#39;valid&#39;: False, &#39;error&#39;: &#39;Maximum attempts exceeded&#39;}<br>        <br>        # Verify OTP<br>        if otp_data[&#39;otp&#39;] == submitted_otp:<br>            self.redis.delete(key)  # OTP used, remove it<br>            return {&#39;valid&#39;: True, &#39;message&#39;: &#39;OTP verified successfully&#39;}<br>        else:<br>            # Increment attempts<br>            otp_data[&#39;attempts&#39;] += 1<br>            self.redis.setex(key, self.otp_expiry.total_seconds(), json.dumps(otp_data))<br>            <br>            remaining = otp_data[&#39;max_attempts&#39;] - otp_data[&#39;attempts&#39;]<br>            return {<br>                &#39;valid&#39;: False, <br>                &#39;error&#39;: f&#39;Invalid OTP. {remaining} attempts remaining&#39;<br>            }<br># Usage<br>otp_manager = SecureOTPManager(redis_client=your_redis_client)<br># Generate and send OTP<br>otp = otp_manager.generate_otp(&#39;+91750348xxx&#39;)<br>sms_result = send_sms_india(&#39;+91750xxxxx&#39;, otp)<br># Later, verify OTP<br>verification = otp_manager.verify_otp(&#39;+9175034xxxxx&#39;, user_submitted_otp)</pre><h3>Conclusion</h3><p>Implementing SMS services for the Indian market requires careful attention to TRAI compliance, proper AWS configuration, and robust error handling. The key takeaways from this comprehensive guide are:</p><h3>Critical Success Factors:</h3><ol><li><strong>Complete TRAI Registration</strong>: Don’t skip any steps in the registration process</li><li><strong>Exact Template Matching</strong>: Even a single extra space can cause failures</li><li><strong>Proper Resource Policies</strong>: Essential for local route delivery and cost optimization</li><li><strong>Comprehensive Error Handling</strong>: Implement retry logic and proper error responses</li><li><strong>Security First</strong>: Use secure OTP generation and proper credential management</li></ol><h3>Production Readiness Checklist:</h3><ul><li>✅ TRAI registration completed with all required IDs</li><li>✅ AWS Sender ID registered and resource policy configured</li><li>✅ Templates registered and message formatting validated</li><li>✅ Rate limiting implemented to prevent abuse</li><li>✅ Monitoring and alerting configured</li><li>✅ Error handling and retry logic in place</li><li>✅ Security measures implemented for OTP handling</li><li>✅ Cost monitoring and optimization strategies active</li></ul><h3>Next Steps:</h3><ol><li><strong>Scale Gradually</strong>: Start with small volumes and increase based on success rates</li><li><strong>Monitor Continuously</strong>: Set up comprehensive monitoring for delivery rates and costs</li><li><strong>Optimize Templates</strong>: A/B test different message formats for better engagement</li><li><strong>Stay Compliant</strong>: Keep up with TRAI regulation changes</li></ol><p>The Indian SMS market is complex but rewarding when done correctly. With proper implementation, you can achieve &gt;95% delivery rates at optimal costs while maintaining full regulatory compliance.</p><p><em>Have you implemented SMS services for Indian users? Share your experiences and challenges in the comments below. For more technical deep-dives and AWS tutorials, follow me on Medium.</em></p><p><strong>References:</strong></p><ul><li><a href="https://www.vilpower.in/">TRAI DLT Portal</a></li><li><a href="https://docs.aws.amazon.com/sms-voice/">AWS End User Messaging Documentation</a></li><li><a href="https://docs.aws.amazon.com/sns/">AWS SNS Developer Guide</a></li></ul><p><em>This article is based on practical experience implementing SMS services for multiple production applications serving Indian users. All code examples are production-tested and follow AWS best practices.</em></p><h3>Thank you for being a part of the community</h3><p><em>Before you go:</em></p><ul><li>Be sure to <strong>clap</strong> and <strong>follow</strong> the writer ️👏️<strong>️</strong></li><li>Follow us: <a href="https://x.com/inPlainEngHQ"><strong>X</strong></a> | <a href="https://www.linkedin.com/company/inplainenglish/"><strong>LinkedIn</strong></a> | <a href="https://www.youtube.com/@InPlainEnglish"><strong>YouTube</strong></a> | <a href="https://newsletter.plainenglish.io/"><strong>Newsletter</strong></a> | <a href="https://open.spotify.com/show/7qxylRWKhvZwMz2WuEoua0"><strong>Podcast</strong></a> | <a href="https://twitch.tv/inplainenglish"><strong>Twitch</strong></a></li><li><a href="https://differ.blog/"><strong>Start your own free AI-powered blog on Differ</strong></a> 🚀</li><li><a href="https://discord.gg/in-plain-english-709094664682340443"><strong>Join our content creators community on Discord</strong></a> 🧑🏻‍💻</li><li>For more content, visit <a href="https://plainenglish.io/"><strong>plainenglish.io</strong></a> + <a href="https://stackademic.com/"><strong>stackademic.com</strong></a></li></ul><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=0a9d50861499" width="1" height="1" alt=""><hr><p><a href="https://blog.stackademic.com/building-a-robust-sms-service-with-aws-sns-for-indian-markets-0a9d50861499">Building a Robust SMS Service with AWS SNS for Indian Markets</a> was originally published in <a href="https://blog.stackademic.com/">Stackademic</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
    </channel>
</rss>