<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by A. Yigit Ogun on Medium]]></title>
        <description><![CDATA[Stories by A. Yigit Ogun on Medium]]></description>
        <link>https://medium.com/@ayogun?source=rss-eda7d7339636------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/1*mXT3UesCzCAeNl4H7oWd4w.png</url>
            <title>Stories by A. Yigit Ogun on Medium</title>
            <link>https://medium.com/@ayogun?source=rss-eda7d7339636------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Tue, 17 Nov 2026 00:13:20 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@ayogun/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[Why You’ll Soon Be Talking to Your Infrastructure: The Rise of Conversational IaC]]></title>
            <link>https://medium.com/@ayogun/why-youll-soon-be-talking-to-your-infrastructure-the-rise-of-conversational-iac-c9dc6ba2f154?source=rss-eda7d7339636------2</link>
            <guid isPermaLink="false">https://medium.com/p/c9dc6ba2f154</guid>
            <category><![CDATA[ai]]></category>
            <category><![CDATA[cloud-computing]]></category>
            <category><![CDATA[terraform]]></category>
            <category><![CDATA[infrastructure-as-code]]></category>
            <category><![CDATA[mcp-server]]></category>
            <dc:creator><![CDATA[A. Yigit Ogun]]></dc:creator>
            <pubDate>Sat, 07 Feb 2026 10:14:03 GMT</pubDate>
            <atom:updated>2026-02-07T10:14:03.562Z</atom:updated>
            <content:encoded><![CDATA[<h3>Beyond the YAML Wall</h3><p>For a decade, the primary interface between an engineer and the cloud has been a wall of static templates. Wrestling with YAML in CloudFormation or JSON in Terraform is a precise but joyless exercise in syntax management. We have spent more time satisfying the parser than defining the architecture.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*EuzGok_xoLqiwCqLqiShDw.png" /></figure><p>The era of manual templating is ending. Infrastructure is moving from being “declared” to being “discussed.” This shift from static artifacts to active dialogue is powered by the Model Context Protocol (MCP) and the emergence of AWS MCP Servers.</p><h3>The Four-Stage Evolution of Infrastructure as Code</h3><p>We are moving from managing syntax to managing <strong>Intent</strong>. This journey encompasses four distinct architectural eras:</p><ol><li><strong>Declarative IaC:</strong> Static templates (CloudFormation, Terraform, OpenTofu). You define the resource; the engine handles the state.</li><li><strong>Programmatic IaC:</strong> Logic-driven templates (AWS CDK, Pulumi, SST). You use code to generate the underlying static templates.</li><li><strong>Generative IaC:</strong> Prompt-to-code (Amazon Q Developer, GitHub Copilot). LLMs transform a single prompt into a functional starting point.</li><li><strong>Conversational IaC:</strong> Dialogue-to-deployment (Claude Code, Kiro CLI). Continuous iteration where the environment evolves through real-time discussion.</li></ol><blockquote>“Infrastructure used to be declared, then coded, then generated. Now it can be simply discussed.”</blockquote><h3>MCP: The “USB-C” Standard for AI Context</h3><p>The Model Context Protocol (MCP) is the engine of this transition. Introduced by Anthropic in November 2024 and donated to the Linux Foundation in December 2025, MCP is the “USB-C” of AI context , standardized, hot-swappable, and long overdue.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*fBaDkELt8k6Apuna4TyDlQ.png" /></figure><p>It is a <strong>Universal Protocol designed to replace fragmented integrations</strong>, standardizing how AI models connect to disparate data sources and toolsets. In this architecture:</p><ul><li><strong>MCP Servers</strong> expose tools and data through a standard protocol.</li><li><strong>MCP Clients</strong> (AI agents like Kiro CLI or Claude Code) consume these tools to execute tasks.</li></ul><h3>The Power of AWS MCP Servers</h3><p>AWS MCP Servers act as a <strong>managed remote</strong> dynamic proxy between the LLM and the AWS ecosystem. They allow agents to interact directly with the control plane, bypassing the need for manual script generation.</p><p>Engineers can now orchestrate a wide range of services via dialogue:</p><ul><li><strong>Identity:</strong> IAM roles, policies, and cross-account permissions.</li><li><strong>Networking:</strong> VPC topology, security groups, and routing.</li><li><strong>Infrastructure Management:</strong> Managed CloudFormation, CDK, or Terraform execution.</li><li><strong>Compute &amp; State:</strong> Lambda functions and DynamoDB tables.</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Z15-glCmY66-acOb8wouOQ.png" /></figure><p>Consider a “Functionless Address Book” example. Rather than writing a 200-line CDK script, a conversational agent can coordinate a complex, event-driven architecture, orchestrating AWS CDK, DynamoDB, Amazon EventBridge, and Step Functions, through a single technical conversation.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*VHK-liKOHfUEO6SsftCaJQ.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*8386AfzdQqoETtWG7ox3Wg.png" /></figure><h3>The “Sudo” Problem: Security in a Conversational World</h3><p>“Vibe coding” relies on high-velocity trust, which is the natural enemy of the Zero Trust architectures we are paid to build. Granting an AI agent the ability to modify cloud environments requires a rigorous security framework:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*t2ucfA48TlJIpRO0-DogeQ.png" /></figure><ul><li><strong>Treat Installation as Sudo:</strong> Only install MCP servers from verified sources. Verification is not a one-time task; check for security updates frequently.</li><li><strong>Enforce Least Privilege:</strong> Use restrictive IAM roles. Isolate the agent’s access to the local file system and the network.</li><li><strong>Human-in-the-Loop:</strong> Use trust permissions for all destructive tools. Require explicit human approval (y/n) before any deployment.</li><li><strong>Credential Hygiene:</strong> Never hardcode API keys or tokens. Use environment variables, store secrets in secure vaults, and prioritize short-lived credentials.</li><li><strong>Sandboxing:</strong> Run local MCP servers inside Docker containers to isolate the agent from the host operating system.</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*M_dus0P-hctTT_oqFnesTQ.png" /></figure><h3>From “Vibe Coding” to Production Specs</h3><p>While “vibe coding” is excellent for rapid prototyping and PoCs, production environments require more than a chat; they require <strong>Spec-driven development (SDD)</strong>.</p><p>The industry is gravitating toward “Infrastructure from Intent,” utilizing specialized agent resources to maintain context:</p><ul><li><strong>PROJECT.md:</strong> High-level context (e.g., AWS CDK with TypeScript).</li><li><strong>TASKS.md:</strong> Sequential instructions for adding resources to stacks.</li><li><strong>AGENTS.md:</strong> Precise, agent-focused guidance and constraints.</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*df2HGk3wJoqRrhQNon6ijg.png" /></figure><p>Tools like <strong>GitHub Spec Kit</strong>, <strong>Claude Code Plans</strong>, and <strong>Kiro IDE Specs</strong> are the first indicators of this shift. In this model, the dialogue captures the intent, while the “Spec” ensures production-grade reliability and verifiability.</p><h3>Conclusion: The Future is a Dialogue</h3><p>The era of manual template writing is fading. We are entering a world where the barrier between an engineer’s intent and a live cloud environment is a simple conversation. AWS MCP Servers have turned infrastructure into a participant in the development process rather than a static artifact.</p><blockquote><strong>As infrastructure moves from templates to dialogue, will our primary job as engineers shift from writing code to mastering the art of the technical conversation?</strong></blockquote><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=c9dc6ba2f154" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[From YAML to Dialogue: 5 Takeaways from IaCConf 2026’s AI Spotlight]]></title>
            <link>https://medium.com/@ayogun/from-yaml-to-dialogue-5-surprising-takeaways-from-iacconf-2026s-ai-spotlight-96d020c1e3ad?source=rss-eda7d7339636------2</link>
            <guid isPermaLink="false">https://medium.com/p/96d020c1e3ad</guid>
            <category><![CDATA[ai]]></category>
            <category><![CDATA[cloud]]></category>
            <category><![CDATA[terraform]]></category>
            <category><![CDATA[iac]]></category>
            <category><![CDATA[cloud-computing]]></category>
            <dc:creator><![CDATA[A. Yigit Ogun]]></dc:creator>
            <pubDate>Sat, 07 Feb 2026 09:20:09 GMT</pubDate>
            <atom:updated>2026-02-07T09:20:21.704Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*G8v9mtvi0ugettMsekobGA.png" /></figure><p>For years, we’ve defined our professional worth by our ability to wrestle with the rigid syntax of HCL, YAML, and TypeScript. We’ve managed sprawling state files and hand-authored thousand-line CloudFormation templates, always acting as the bridge that translates human intent into machine-readable declarations.</p><p>The <strong>IaCConf Spotlight 2026: AI in IaC</strong> (held January 28, 2026) signaled the definitive end of that era. It was the architectural “aha” moment of the conference: infrastructure has moved past being “declared,” “coded,” or even “generated.” We have officially entered the age of the dialogue.</p><p>Here are the five most surprising takeaways from the spotlight that define our new conversational reality.</p><p>—</p><h3><strong>1. The Dawn of Conversational IaC</strong></h3><p>We’ve graduated from the primitive “Prompt to Template” workflow, where an AI spits out a static file that we then have to debug manually. The new standard is <strong>Dialogue to Deployment</strong>.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*7eJqW7CBN0oIf_6eoDf6sQ.png" /></figure><p>In this paradigm, infrastructure isn’t a static artifact; it’s a live, iterative conversation between the architect and the agent.</p><blockquote>“Infrastructure used to be declared, then coded, then generated. Now it can be simply discussed.”</blockquote><p>This shift is profound because dialogue allows for real-time refinement. Instead of a one-way generation, an engineer can now iterate on a networking stack or a WAFv2 configuration through a continuous exchange, adjusting rules, optimizing for cost, and troubleshooting security groups without ever leaving the conversation.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*IAq15q3SxuyhhWdjD-Ek0w.png" /></figure><h3>2. MCP is the “USB-C” for AI Applications</h3><p>If there was a single technical cornerstone at IaCConf, it was the <strong>Model Context Protocol (MCP)</strong>. Introduced by Anthropic in late 2024 and donated to the Linux Foundation in December 2025, MCP has become the universal connector for AI context.</p><p>Think of it as the “USB-C” for the AI era. It replaces the fragmented, custom integrations of the past with a standardized interface that allows AI models to safely plug into your cloud environment.</p><p>• <strong>MCP Servers:</strong> These act as standardized connectors to tools like AWS IAM, Network, and Cost Explorer. Crucially, the conference highlighted the distinction between <strong>AWS Core (dynamic proxy)</strong> and the <strong>AWS MCP Server (managed remote)</strong>, giving architects more control over how their context is managed.</p><p>• <strong>MCP Clients:</strong> These are the AI agents , like those living in your CLI, that connect to these servers to pull live data or execute infrastructure changes.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*55lMIBljww5SCF93y4uxiA.png" /></figure><p>By standardizing this protocol, we’ve effectively killed vendor lock-in for AI context, ensuring that our agents have a unified way to “see” and “touch” the cloud.</p><h3>3. “Vibe Coding” is Moving Toward Production</h3><p>The term “Vibe Coding”, building infrastructure through high-level intent and conversational guidance, has shed its “toy” reputation. While it started in the world of rapid prototyping, it is now maturing into a production-ready workflow.</p><p>The demo of the <strong>Functionless Address Book</strong> was a standout example. By utilizing agents to orchestrate complex, event-driven architectures involving <strong>AWS Step Functions</strong>, <strong>Amazon EventBridge</strong>, and <strong>DynamoDB</strong>, speakers proved that “vibing” can handle the heavy lifting of sophisticated enterprise patterns.</p><p>Current tools making this possible include:</p><p>• <strong>Kiro CLI:</strong> A specialized interface using agents like cdk-buddy to manage AWS CDK apps.</p><p>• <strong>Claude Code:</strong> A CLI-driven agent for direct codebase interaction.</p><p>• <strong>AWS MCP Servers:</strong> Providing the managed remote backbone for these agents to interact with AWS resources securely.</p><p>However, the “vibe” isn’t free. The conference underscored that while development is faster, <strong>prompt and context engineering</strong> still “takes time.” The skill has shifted from writing the code to defining the context.</p><h3>4. A New Hierarchy of Trust and Security</h3><p>As we grant AI agents the power to modify our environments, our security model must evolve. The conference established five “Golden Rules” for AI-driven infrastructure, emphasizing that conversational speed must never come at the cost of environment integrity.</p><p>1. <strong>Treat Server Installation like Sudo Access:</strong> Use only trusted sources. Adding an MCP server to your stack is a high-privilege event.</p><p>2. <strong>Apply Least-Privilege IAM:</strong> Agents should only have the permissions necessary for the task at hand.</p><p>3. <strong>Human-in-the-Loop:</strong> Always use trust permissions for tools that require explicit confirmation (y/t/n) before any destructive action.</p><p>4. <strong>Secure Credential Handling:</strong> Never hardcode API keys. Use environment variables and short-lived credentials.</p><p>5. <strong>Mandatory Sandboxing:</strong> Run local MCP servers in <strong>Docker containers</strong>. This limits the agent’s access to the host shell and local file system, providing a critical layer of isolation.</p><h3>5. The Shift to Spec-Driven Development (SDD)</h3><p>Perhaps the most visionary shift is the move from “Infrastructure as Code” to <strong>Infrastructure as Specs</strong>. In this Spec-Driven Development (SDD) model, we no longer memorize syntax; we orchestrate intent.</p><p>This is managed through a new hierarchy of specification files that provide the “brain” for our agents:</p><p>• <strong>PROJECT.md:</strong> Defines the environment context, such as the app type (AWS CDK/TypeScript) and the specific AWS CLI profiles or regions.</p><p>• <strong>TASKS.md:</strong> Defines the logic sequence, providing a detailed, sequential list of instructions for adding resources to stacks.</p><p>• <strong>AGENTS.md:</strong> A specialized README that provides agent-focused guidance and extra context.</p><p>With the rise of tools like the <strong>GitHub Spec Kit</strong>, <strong>Kiro IDE Specs</strong>, and <strong>Spacelift Intent</strong>, our job is becoming less about writing the “how” and more about defining the “what” and “why” within a structured spec.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*iTUjGfW8FhTUWLVSk51omQ.png" /></figure><p>The Future is a Dialogue! The road to 2026 has been an evolutionary sprint. We’ve moved through the <strong>Declarative</strong> era (CloudFormation), the <strong>Programmatic</strong> era (CDK), and the <strong>Generative</strong> era (Copilot), only to arrive at the <strong>Conversational</strong> era.</p><p>As we adopt tools like Kiro and protocols like MCP, the role of the engineer is fundamentally transforming. We are no longer the scribes of YAML; we are the orchestrators of intent. Our technical value is no longer tied to a memory bank of syntax, but to our ability to architect solutions through a shared dialogue with our agents.</p><p>—</p><blockquote><strong>In a world where we build infrastructure through dialogue, is our most valuable skill still knowing the syntax, or is it finally knowing what to ask for?</strong></blockquote><p>—</p><p><em>To dive deeper into the repositories and community profiles discussed at IaCConf, explore the following:</em></p><p>• <a href="https://serverless.ninja/">https://serverless.ninja</a></p><p>• <a href="https://github.com/ServerlessNinja">https://github.com/ServerlessNinja</a></p><p>• <a href="https://builder.aws.com/community/@jakgaj">https://builder.aws.com/community/@jakgaj</a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=96d020c1e3ad" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Product Authentication Platform for ETSY on AWS]]></title>
            <link>https://medium.com/@ayogun/product-authentication-platform-for-etsy-on-aws-8d408e157713?source=rss-eda7d7339636------2</link>
            <guid isPermaLink="false">https://medium.com/p/8d408e157713</guid>
            <category><![CDATA[etsy]]></category>
            <category><![CDATA[serverless]]></category>
            <category><![CDATA[aws]]></category>
            <category><![CDATA[nft]]></category>
            <category><![CDATA[cloud]]></category>
            <dc:creator><![CDATA[A. Yigit Ogun]]></dc:creator>
            <pubDate>Mon, 02 Sep 2024 14:24:08 GMT</pubDate>
            <atom:updated>2025-02-18T08:42:01.703Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*_5NmJLNoUSOKjHICxq_jeA.png" /><figcaption><a href="https://www.etsy.com/shop/PlatinAndGoldRecords?ref=l2-about-shopname&amp;from_page=listing">Copy Signed LP Vinyl — Metallica Memorabilia Gift</a></figcaption></figure><p>In today’s world of e-commerce and digital products, ensuring the authenticity of goods is more critical than ever. When my friend built a company named <a href="https://www.etsy.com/shop/PlatinAndGoldRecords?ref=l2-about-shopname&amp;from_page=listing"><strong><em>PGR(Platin and Gold Records)</em></strong></a> where he manufacture copy signed collectibles, he knew that eventually he will need a platform to prove authenticity of the products. The products he sells are copy signed and produced in certain amount. This makes those products special to owner and non fungible. He considered using blockchain and some other platforms but he dropped them all because of various reasons. So, I decided to help him with building an Authentication Platform with AWS services. Of course in a !!<strong>SERVERLESS</strong>!! manner.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*E66i59hyUTeS-0q2yP6I1Q.jpeg" /></figure><p>I knew that the solution had to be both secure and scalable to handle potentially high traffic volumes. Leveraging AWS’s powerful suite of services, I was able to design and deploy a robust platform that not only met these requirements but also exceeded expectations in terms of performance and reliability.</p><p>In this article, I’ll walk you throughhow I built this platform using AWS services, including API Gateway, Lambda, DynamoDB, Route 53, WAF and CloudFront.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/707/1*oc-xChJx5pVI9kWGdXqNHg.png" /></figure><p>The problem was straightforward. I needed to create a system where customers could enter a product ID/Unique ID and instantly receive a response confirming the product’s authenticity. The solution had to be scalable, secure, and fast, with the ability to handle potentially thousands of requests per second during peak times. Additionally, I wanted to ensure that the platform was accessible from anywhere in the world with minimal latency.</p><h4>1. Designing the Backend with API Gateway and Lambda</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/299/0*omav-BS0JgcI89GR" /></figure><p><strong>Why API Gateway and Lambda?</strong></p><p>The backend of the platform needed to be serverless, ensuring that it could scale automatically based on the number of requests without requiring manual intervention. AWS API Gateway and Lambda were perfect for this:</p><p>- <strong>API Gateway</strong> provided a scalable, secure interface for handling HTTP requests. It allowed me to define RESTful APIs that could trigger Lambda functions, which processed the authentication logic.</p><ul><li><strong>Lambda</strong> enabled me to execute code in response to API requests without provisioning or managing servers. I wrote a Python function that would take the product ID as input, query a DynamoDB table for authenticity, and return a response to the user.</li></ul><pre>import json<br>import boto3<br><br>def lambda_handler(event, context):<br>    # Handle both cases: API Gateway invocations and direct test invocations<br>    product_id = None<br>    <br>    if &#39;queryStringParameters&#39; in event and event[&#39;queryStringParameters&#39;]:<br>        product_id = event[&#39;queryStringParameters&#39;].get(&#39;product_id&#39;)<br>    elif &#39;product_id&#39; in event:  # Fallback for test invocation<br>        product_id = event[&#39;product_id&#39;]<br>    <br>    if not product_id:<br>        return {<br>            &#39;statusCode&#39;: 400,<br>            &#39;headers&#39;: {<br>                &#39;Access-Control-Allow-Origin&#39;: &#39;*&#39;,<br>                &#39;Access-Control-Allow-Methods&#39;: &#39;GET&#39;,<br>                &#39;Access-Control-Allow-Headers&#39;: &#39;Content-Type&#39;<br>            },<br>            &#39;body&#39;: json.dumps({&#39;message&#39;: &#39;Product ID is missing&#39;})<br>        }<br>    <br>    # Connect to DynamoDB<br>    dynamodb = boto3.resource(&#39;dynamodb&#39;)<br>    table = dynamodb.Table(&#39;Products&#39;)<br>    <br>    # Query the DynamoDB table<br>    try:<br>        response = table.get_item(Key={&#39;product_id&#39;: product_id})<br>        if &#39;Item&#39; in response:<br>            item = response[&#39;Item&#39;]<br>            product_name = item.get(&#39;product_name&#39;, &#39;Unknown Product&#39;)<br>            return {<br>                &#39;statusCode&#39;: 200,<br>                &#39;headers&#39;: {<br>                    &#39;Access-Control-Allow-Origin&#39;: &#39;*&#39;,<br>                    &#39;Access-Control-Allow-Methods&#39;: &#39;GET&#39;,<br>                    &#39;Access-Control-Allow-Headers&#39;: &#39;Content-Type&#39;<br>                },<br>                &#39;body&#39;: json.dumps({<br>                    &#39;message&#39;: f&#39;Product is authentic: {product_name}&#39;,<br>                    &#39;product_name&#39;: product_name<br>                })<br>            }<br>        else:<br>            return {<br>                &#39;statusCode&#39;: 404,<br>                &#39;headers&#39;: {<br>                    &#39;Access-Control-Allow-Origin&#39;: &#39;*&#39;,<br>                    &#39;Access-Control-Allow-Methods&#39;: &#39;GET&#39;,<br>                    &#39;Access-Control-Allow-Headers&#39;: &#39;Content-Type&#39;<br>                },<br>                &#39;body&#39;: json.dumps({&#39;message&#39;: &#39;Product not found&#39;})<br>            }<br>    except Exception as e:<br>        return {<br>            &#39;statusCode&#39;: 500,<br>            &#39;headers&#39;: {<br>                &#39;Access-Control-Allow-Origin&#39;: &#39;*&#39;,<br>                &#39;Access-Control-Allow-Methods&#39;: &#39;GET&#39;,<br>                &#39;Access-Control-Allow-Headers&#39;: &#39;Content-Type&#39;<br>            },<br>            &#39;body&#39;: json.dumps({&#39;message&#39;: f&#39;Error: {str(e)}&#39;})<br>        }</pre><p><strong>Implementation — </strong>I set up an API Gateway with a single `GET` endpoint that accepted a `product_id` as a query parameter. This API triggered a Lambda function that interacted with DynamoDB to check whether the product ID existed and returned a corresponding message, such as “Product is authentic” or “Product not found.”</p><p><strong>CORS Issues — </strong>During development, I encountered Cross-Origin Resource Sharing (CORS) issues when trying to fetch results from my frontend. Because when you access the API through a website, you need to configure CORS in accordingly. API Gateway’s built-in CORS support made it easy to configure the necessary headers, and by integrating these headers into my Lambda function’s responses, I was able to resolve the issue quickly.</p><h4>2. Secure and Fast Storage with DynamoDB</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/556/0*FwT5khGDT0Wa5tlH.png" /></figure><p><strong>Why DynamoDB?</strong></p><p>DynamoDB was the ideal choice for storing product data due to its fully managed nature and ability to scale on demand. With DynamoDB, I didn’t have to worry about provisioning or managing databases, which allowed me to focus on the core functionality of the platform.</p><p><strong>Implementation</strong></p><p>I created a DynamoDB table to store product IDs and associated details, such as product names and authenticity statuses. The table was designed with a partition key of `product_id`, ensuring that queries would be efficient and fast.</p><p><strong>Optimizing Query Performance</strong> — One of the challenges was ensuring that queries remained performant as the database grew. By carefully designing the table schema and using DynamoDB’s built-in indexing features, I was able to maintain fast query responses even as the dataset expanded.</p><h4>3. Enhancing Performance and Security with CloudFront and Route 53</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/571/0*robOqnBKl27FGit0" /></figure><p><strong>Why CloudFront and Route 53?</strong></p><p><a href="https://www.etsy.com/shop/PlatinAndGoldRecords?ref=l2-about-shopname&amp;from_page=listing">My friend’s store</a> is on ETSY and he sell his products to all around the globe. To ensure that users across the globe could access the platform with minimal latency, I integrated <em>Amazon CloudFront</em>, AWS’s Content Delivery Network (CDN), into the architecture. Additionally, I used Amazon Route 53 for domain management and DNS routing.</p><p><strong>Implementation</strong></p><p>- <strong>CloudFront</strong> was configured as a CDN for the platform, caching API responses and static assets (like the HTML, CSS, and JavaScript files) to edge locations around the world. This significantly reduced latency for users, especially those far from the primary AWS region where the Lambda function was executed.</p><p>- <strong>Route</strong> <strong>53</strong> managed the domain name for the platform, allowing me to easily route traffic to the CloudFront distribution. I configured health checks and routing policies to ensure high availability and failover support.</p><p><strong>-Managing Security with AWS WAF</strong> — One of the challenges I faced was ensuring that the platform was secure against web-based threats, such as DDoS attacks. AWS Web Application Firewall (WAF) proved invaluable in this regard. With WAF, I was able to create rules that filtered out malicious requests before they reached the application. I integrated WAF with my API Gateway and CloudFront distribution, adding a layer of protection that guarded the platform against potential threats and ensured that only legitimate traffic was processed. This setup provided peace of mind, knowing that my platform was protected by a robust security framework.</p><h4>4. Hosting the Frontend with S3</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/401/0*fHEsZYIi_IZI27I-.png" /></figure><p><strong>Why S3?</strong></p><p>For hosting the static frontend of the platform, I used Amazon S3. S3 is perfect for static websites, offering high availability, scalability, and durability without the need for managing any servers.</p><p><strong>Implementation</strong></p><p>I created an S3 bucket configured for static website hosting, where I uploaded my HTML, CSS, and JavaScript files. I then linked the S3 bucket to my CloudFront distribution, ensuring that content was delivered quickly to users around the globe.</p><p>The final product authentication platform was a triumph of AWS’s capabilities. By leveraging services like API Gateway, Lambda, DynamoDB, CloudFront, and Route 53, I was able to build a secure, scalable, and globally accessible system with minimal overhead. The platform now handles hundreds of requests per day, providing users with instant verification of product authenticity, regardless of their location.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*3n7P41LDyGKcosD6a7_-aw.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*mr66Q3cR_MzT_uCe3rZ5bQ.png" /></figure><p>Now there is a lot of work in front of me to make this perfect. Especially frontend is vanilla based and looks boring. But one thing at once. I will work on Front End at last.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/773/0*93oRDftJ38L0YRvf.png" /></figure><p>I will also integrate DynamoDb Streams and save logs in a long-lived manner to allow customers to resell their collectibles and which will allows end users to retro-respectively check the full sale history of the products. Like NFT…</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/180/0*s8ACujmJE2sqdgLW" /></figure><p>If you want to support my friend, here is the link: <a href="https://www.etsy.com/de-en/shop/Siczil">https://www.etsy.com/shop/PlatinAndGoldRecords?ref=l2-about-shopname&amp;from_page=listing</a></p><p>You can find many lovely collectibles. I think worth to take a look at it while Christmas is just around corner. 😊</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=8d408e157713" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[IDE Extension for AWS Application Composer]]></title>
            <link>https://medium.com/@ayogun/ide-extension-for-aws-application-composer-01431e596386?source=rss-eda7d7339636------2</link>
            <guid isPermaLink="false">https://medium.com/p/01431e596386</guid>
            <category><![CDATA[cloud]]></category>
            <category><![CDATA[awsreinvent]]></category>
            <category><![CDATA[aws]]></category>
            <category><![CDATA[ai]]></category>
            <category><![CDATA[iac]]></category>
            <dc:creator><![CDATA[A. Yigit Ogun]]></dc:creator>
            <pubDate>Tue, 19 Dec 2023 10:11:09 GMT</pubDate>
            <atom:updated>2023-12-19T10:36:39.607Z</atom:updated>
            <content:encoded><![CDATA[<h4>In re:Invent 2023, AWS announced IDE extension for AWS Application Composer. Let’s explore!</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*og7GElMbdHmSMON1SBvpwg.png" /></figure><p>About a year ago, <a href="https://openai.com/blog/chatgpt">OpenAI unveiled ChatGPT</a>, a Large Language Model(LLM) that drove a great attention, and the AI hype got rapidly integrated into our daily lives. In this wave of innovation, AWS — as a key player pioneer in tech advancements — has been leveraging AI in numerous ways, notably with their recent announcements in <a href="https://www.google.com/url?sa=t&amp;rct=j&amp;q=&amp;esrc=s&amp;source=web&amp;cd=&amp;cad=rja&amp;uact=8&amp;ved=2ahUKEwi_-9_FhZuDAxU7gf0HHSlGC2oQFnoECB8QAQ&amp;url=https%3A%2F%2Freinvent.awsevents.com%2F&amp;usg=AOvVaw2nNrcy7-_rOe6Ypfvyn3lg&amp;opi=89978449">AWS re:Invent 2023</a>, like <a href="https://aws.amazon.com/q/">AWS Q</a>. As a Cloud Engineer, Infrastructure as Code (IaC) remains a cornerstone of my daily operations, crucial for maintaining sustainable IT infrastructure. However, amidst the buzz, an announcement seems to me that it has slipped under the radar of the community.</p><p>The <strong>AWS Application Composer and its IDE Extension</strong>, fortified with AI enhancements. While Terraform dominates my workload, I can see the potential significance of this release, especially for those relying heavily on CloudFormation. So, shall we?</p><h4>What is AWS Application Composer?</h4><p><a href="https://aws.amazon.com/about-aws/whats-new/2022/12/aws-application-composer-preview/">In AWS re:Invent 2022</a> event, AWS announced that service and it’s available since March 2023. AWS Application Composer is a powerful tool that makes developing and deploying apps on the AWS cloud easier. It serves as a visual builder that lets developers quickly and effectively design, build, and coordinate different AWS services to create applications.</p><p>This tool makes it easier to integrate several AWS services into a coherent application architecture by offering a drag and drop interface. Using pre-built components, templates, and workflows, developers can shorten the time it takes to construct an application.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*IjGwgptlmXqwatNa" /></figure><p>Without requiring complex manual configuration, customers can use Application Composer to visually sketch out the architecture of their apps, integrating different AWS services including Lambda functions, API Gateways, databases, and more. However, it’s currently available on the AWS website. AWS has elevated its capabilities with an IDE extension. While it’s only available with Visual Studio Code at the moment, I expect broader IDE support in the future. Let’s take a look at it.</p><h3>What is IDE extension for AWS Application Composer?</h3><p>Now, AWS Application Composer seamlessly integrates into your IDE, allowing you to visually construct modern applications and refine your IaC(infrastructure as code) templates using<a href="https://aws.amazon.com/codewhisperer/"> AWS CodeWhisperer</a> (CodeWhisperer is a service like GitHub CoPilot, an AI-powered productivity tool for the IDE and command line that generates <strong>code</strong>suggestions based on comments and existing <strong>code.</strong> ).</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*eVdJESliWC64XO3w" /></figure><p><a href="https://aws.amazon.com/blogs/aws/ide-extension-for-aws-application-composer-enhances-visual-modern-applications-development-with-ai-generated-iac/">[How it looks like on IDE]</a></p><p>This IDE extension mirrors the <strong>familiar drag-and-drop experience</strong> found in the console, empowering you to swiftly prototype ideas and concentrate on your application code. Not only does it bring Application Composer to your IDE, but it also enables generative AI-powered code suggestions(CodeWhisperer) within the CloudFormation template, all while visualizing the application architecture in a split view. With synchronized visualization and CloudFormation template editing in the IDE, you can refine designs without switching between consoles, reducing manual coding and enhancing productivity. And it is available at no charge. Awesome, isn’t it?</p><blockquote><a href="https://aws.amazon.com/blogs/aws/ide-extension-for-aws-application-composer-enhances-visual-modern-applications-development-with-ai-generated-iac/"><em>With Application Composer running in your IDE, you can also use the various tools available in your IDE. For example, you can seamlessly integrate IaC templates generated real-time by Application Composer with AWS Serverless Application Model (AWS SAM) to manage and deploy your serverless applications.</em></a></blockquote><p>Beside from the use cases mentioned in the announcement of the <a href="https://aws.amazon.com/blogs/aws/ide-extension-for-aws-application-composer-enhances-visual-modern-applications-development-with-ai-generated-iac/">IDE extension for AWS Application Composer</a>, I believe <strong>of</strong> that this will make it far easier to quickly validate what resources will be created by a CloudFormation template. This will be especially useful in CDK stacks where it will work as a quick way to verify that all the right components of an architecture are present without digging through the actual generated template.</p><p>It also integrates directly with StepFunctions workflow studio on Visual Studio Code.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*rYlgJJGUlWbuJ8f8" /></figure><p>Before proceeding, let’s install it together.</p><h4>How to Install AWS Application Composer in Visual Studio Code?</h4><p>We’ll install the latest <a href="https://aws.amazon.com/visualstudiocode/?trk=ad553193-613c-4a84-acde-bf4fa6c173a5&amp;sc_channel=el">AWS Toolkit for Visual Studio Code</a> plugin. Don’t mind and go ahead if you already have the AWS Toolkit plugin installed.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*tQepucBMlqvOqn4a" /></figure><p>In Visual Studio Code Extensions Marketplace, you can search for AWS Toolkit and click the blue install button. Or, just <a href="https://marketplace.visualstudio.com/items?itemName=AmazonWebServices.aws-toolkit-vscode&amp;refid=ad553193-613c-4a84-acde-bf4fa6c173a5">click here</a>.</p><p>Now, on left hand side you can see on your Visual Studio Code that AWS and Q sections have been added. But as you can see, now we need to login oeither with AWS Builder ID or SSO(single sign-on). And that’s it. You have it.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*nJu8dAvI2guwMeen" /></figure><p>In order to start using Application Composer, no need to authenticate into your AWS account. With Application Composer available on your IDE, you can open your existing AWS CloudFormation or AWS SAM templates.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*RG0YQb5PKzWwfchh" /></figure><p>Or simply you can create a new empty file, and right-click on the file, then select “<em>Open with Application Composer</em>” to start developing with drag and drop interface of Application Composer. In my case, I created a “try.yaml” file. You can see in the picture below.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*fesIR9v1qMEOR5gS" /></figure><p>Voila! You have it.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*-UN5pSg6-bgY5oa2" /></figure><p>As you can see above, it provided me with a blank canvas which I can work on it. Here I have both code and visual editors at the same time. Any changes that I make on the canvas will also be reflected in real time on my IaC template.</p><p>As you can see below, my “try.yaml” file is empty for now.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*OvWmZpdlIdjYRLGY" /></figure><p>For instance, I just drag and drop on canvas an API Gateway and a Lambda function. Then I connected them simply with drag and drop functionality again.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*7fy6INoLATl2sK0Z" /></figure><p>Do you remember my empty “try.yaml” file? Now let’s go back and look at my YAML file. You can see below in the image that it’s not empty anymore. Generative AI already wrote the IaC for us.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*naF8xIsS3NwL9ru-" /></figure><p>With available IaC templates in your local, it’ll be easier for you to manage the applications with AWS SAM CLI. You can easily create CI/CD with sam pipeline or deploy your stack with sam deploy.</p><p>One of the features which will accelerate your dev workflow is the built-in Sync feature that easily integrates with AWS SAM command sam sync. This feature syncs your local app changes to your AWS account, that’s helpful for you to do test and validation before you deploy your app into the prod.</p><p>With the new development of IaC templates with generative AI capabilities, you can instantly launch over 1000 resources in CloudFormation with generative AI code suggestions. This simplifies the integration of standard IaC resources and allows you to easily scale your architecture.</p><p>For example, suppose you have configured Amazon MQ, a standard IaC resource, and you want to use Application Composer to adjust some settings in your AWS CloudFormation resource. Adjust the values ​​in the Resource Configuration section and select Generate. Application Composer provides code suggestions that seamlessly integrate with your IaC templates. This feature eliminates the need for context switching and streamlines the development process. Build modern applications with AWS Application Composer Canvas while leveraging tools like Amazon CodeWhisperer and AWS SAM to accelerate your development workflow.<a href="https://aws.amazon.com/blogs/aws/ide-extension-for-aws-application-composer-enhances-visual-modern-applications-development-with-ai-generated-iac/">[*]</a></p><h3>Some Possible Use Cases</h3><ul><li>Rapid Application Prototyping: In today’s Agile IT World, AWS Application Composer’s visual canvas within the IDE may allow for quick prototyping of application ideas. Developers can easily design, modify, and visualize architectures without relying on manual coding. This may accelerate the initial development phase, and enabling faster experimentation and validation of concepts.</li><li>Infrastructure as Code (IaC) Development: For CloudFormation users, the IDE extension powered by Gen-AI offers a game changing approach to IaC development. It generates AI code suggestions, which reducing the time spent old style coding CloudFormation templates. The split view feature combines visualization and template editing, streamlining the iteration process and booming productivity.</li><li>Enhanced Collaboration and Iteration: By eliminating the need for constant context switching between consoles, teams may collaboratively work on projects more efficiently. The synchronization of Application Composer’s visualization and CloudFormation template editing in the IDE helps easy collaboration, allowing for quicker iterations and better development cycles.</li><li>Code Efficiency and Resource Inclusion: The generative AI capabilities empower developers to easily incorporate CloudFormation’s extensive library of resources. This ensures effective coding by providing suggestions that align with specific resource configurations, enabling easy integration of standard IaC resources into architectural designs.</li><li>Streamlined Application Development Workflow: AWS Application Composer, in conjunction with Amazon CodeWhisperer and AWS SAM, facilitates a comprehensive and efficient application development workflow. Developers can harness visual design, AI-driven suggestions, and associated AWS tools to streamline the entire application development lifecycle.</li></ul><p>If you have any questions, please don’t hesitate to contact me.</p><h4>Resources</h4><ul><li><a href="https://aws.amazon.com/blogs/aws/ide-extension-for-aws-application-composer-enhances-visual-modern-applications-development-with-ai-generated-iac/">https://aws.amazon.com/blogs/aws/ide-extension-for-aws-application-composer-enhances-visual-modern-applications-development-with-ai-generated-iac/</a></li><li><a href="https://aws.amazon.com/about-aws/whats-new/2023/11/ide-extension-aws-application-composer/">https://aws.amazon.com/about-aws/whats-new/2023/11/ide-extension-aws-application-composer/</a></li></ul><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=01431e596386" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[S3 Express One Zone and Directory Bucket]]></title>
            <link>https://awstip.com/s3-express-one-zone-and-directory-bucket-dcf3fec06156?source=rss-eda7d7339636------2</link>
            <guid isPermaLink="false">https://medium.com/p/dcf3fec06156</guid>
            <category><![CDATA[s3]]></category>
            <category><![CDATA[cloud]]></category>
            <category><![CDATA[storage]]></category>
            <category><![CDATA[aws]]></category>
            <category><![CDATA[reinvent]]></category>
            <dc:creator><![CDATA[A. Yigit Ogun]]></dc:creator>
            <pubDate>Wed, 13 Dec 2023 08:41:29 GMT</pubDate>
            <atom:updated>2023-12-14T16:34:56.192Z</atom:updated>
            <content:encoded><![CDATA[<h4>In re:Invent 2023, AWS launched S3 Express One Zone, “reinvented” high-speed object storage. Let’s explore!</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*FY8INowlM-IFk30yyLslqA.png" /></figure><p>This advanced storage class stands out with three key features:</p><p>- Single-digit millisecond first byte latency for compute-intensive and latency-sensitive applications</p><p>- Consistent performance eliminates tail latencies, driving down query times</p><p>- Data access speeds up to 10x faster, and requested costs up to 50% lower than S3 standard.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/812/0*5-r0ZzbP10FipIex" /></figure><p>This storage class was designed not just to offer low-latency storage but also to ensure remarkable consistency, eliminating tail latencies. This is especially crucial for analytics workloads sensitive to such delays (a topic we’ll dive into shortly). <strong><em>S3 Express One Zone presents data access speeds up to 10 times faster than S3 standard, coupled with a potential 50% reduction in request costs.</em></strong><em> This</em> presents a notably distinct performance profile compared to what you’re used to get with S3.</p><p>One fundamental difference between Express One Zone and the traditional storage classes lies in the introduction of a new bucket type within S3. Until now, S3 buckets have always been, well, just S3 buckets. However, with Express One Zone, AWS introduced a new category called a “<strong>directory bucket</strong>” which differs in three key aspects from the buckets you’ve been using for years.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*mLif-nwR8PxT48v-" /></figure><p>Firstly, directory buckets enable you to store data in a single availability zone that you specifically select. Secondly, they follow a different request scaling model compared to traditional buckets. And thirdly, their authentication works based on sessions rather than on a per-request basis.</p><p>Let’s take a closer look into each of these differences to understand the significance of this new feature.</p><h3>One Zone Storage</h3><p>When you upload an object to AWS, it’s distributed across multiple availability zones. These AZs are essentially independent data centers, each separated by miles yet connected by high-speed and highly redundant networks. While this distribution offers resilience and availability benefits, the distance between these zones adds a degree of latency to your requests.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*e9xKXHAOtGCR4YQj" /></figure><p><em>-Traditional S3 Infrastructure-</em></p><p>However, with a <strong>directory bucket</strong>, you have the power to specify a single availability zone for storing all data within that bucket. This eliminates the latency associated with spreading data across multiple AZs, providing your applications with lower-latency storage.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*xG8ppXbuSdossOtN" /></figure><p><em>-One Zone Storage Infrastructure-</em></p><p>Moving to this new single-zone storage model involves two primary architectural considerations. Firstly, with S3 placement now a notable factor, unlike regional buckets where AWS handles object placement across availability zones for you. With directory buckets, you have the flexibility to place your data adjacent to your compute resources, reducing network distance and enhancing application speed.</p><p>Secondly, S3 Express One Zone offers a different durability model, an essential aspect for builders to comprehend. To explain, let’s step back and understand how AWS ensures durability in S3 across its services. AWS employs various techniques for 11 nines of data durability in S3 Standard and other regional storage classes spread across multiple zones. These techniques involve robust integrity checks through checksums at various stages of data handling, storing data redundantly across devices to tolerate failures, and periodic audits to maintain data correctness over time.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*7G8GVkP1hy-KjEAm" /></figure><p>However, it’s important to note that while S3 Express One Zone maintains the same standards for durability within a single zone, <strong>it doesn’t safeguard against the complete loss or damage of an entire availability zone</strong>. While such occurrences are highly unlikely, it’s a factor worth considering for your architecture.</p><h3>New Request Scaling Model</h3><p>The method of scaling requests for Directory Buckets differs significantly from that of regional buckets. With regional buckets, the process involves starting at 5,500 read requests per prefix, and under load, AWS dynamically adds additional transaction capacity behind the scenes to support your growing usage.</p><p>Let’s look at the image below. This gradual, linear scaling process (depicted by the orange and yellow lines) ensures elasticity, allowing for scaling up to hundreds of thousands of transactions per second as your application grows. However, in certain scenarios — like scaling up rapidly with a large GPU cluster — customers might outpace AWS’s provisioning pace.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*1hJxFSQFaqYVfhIl" /></figure><p>Recognizing this, AWS observed that in transactional analytics and machine learning workloads, customers often prefer instant scalability to make immediate use of their compute resources. With directory buckets, AWS takes a different approach. Instead of gradually adding capacity over time, directory buckets instantly provision hundreds of thousands of transactions per second upon creation. This instant scalability empowers users to immediately scale up their compute resources and efficiently handle substantial workloads. It’s a distinct scaling model designed to meet these specific workload demands efficiently.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*jnTkzp6gFCTzcveO" /></figure><h3>Session Based Authentication</h3><p>The third significant difference involves the security model. With regional buckets, every single request requires authentication, offering high granularity and expressive policies in IAM and bucket policies. However, this authentication process incurs some latency, which accumulates over numerous requests.</p><p>To address this, AWS introduced a new <strong>Create Session API</strong> for directory buckets. This API allows users to authenticate a session and distribute the associated authentication latency over subsequent requests by obtaining a token granting access to the entire bucket. Sessions can operate in three modes: read-only, write-only, or read-write, providing essential options for users.</p><p>While AWS abstracts most security complexities in the SDK, writing bucket policies for directory buckets varies due to different data path authentication. Simplifying policies for directory buckets involves actions like allowing session creation based on specific principles.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*auSm9DEIUveAsIKe" /></figure><p>Regarding performance improvements, AWS reports substantial enhancements across several services: up to 5.8x faster loading times for Amazon SageMaker, about 2.1x quicker query times for Amazon Athena, and up to a 4x performance boost for Amazon EMR. These improvements showcase the tangible benefits of leveraging S3 Express One Zone for various AWS services.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*dBXGDiW2aCVrcB4h" /></figure><h3>Use Cases</h3><p>Let’s take a look at the various scenarios where the new Express One Zone and Directory Bucket services can play a pivotal role. From expediting machine learning and AI training by bolstering data access speeds to speeding up model dataset processing and development, these services usher in dynamic capabilities. They empower interactive data analytics by facilitating rapid insights and processing of extensive data volumes with exceptional speed and minimal latency for swift query execution. Seamlessly handling compute-intensive HPC workloads becomes effortless with scalable, high-performance storage that seamlessly integrates with computing resources. Additionally, these services enhance financial modeling, enabling scalable operations with heightened granularity and faster modeling speeds through low-latency operations. Furthermore, they optimize real-time advertising efforts by swiftly delivering personalized content with minimal latency, refining personalization strategies, and accelerating ad deployment. Moreover, they efficiently manage media content workloads by meeting tighter timelines for VFX, rendering, and transcoding needs through scalable storage that syncs seamlessly with computational demands. <a href="https://aws.amazon.com/s3/storage-classes/express-one-zone/#:~:text=S3%20Express%20One%20Zone%20can,Region%20to%20store%20your%20data.">[*]</a></p><p>Understanding how S3 Express Zone is handy and how it work is really important for making the most of them. Figuring out the best situations to use them is key to using them well. Learning about how the infrastructure actually works can help us prevent future problems before they happen, making things run more smoothly. Remember, while this service excels in speed, it may involve trade-offs in terms of resilience. It’s our responsibility to enhance the system’s resilience by designing it accordingly.</p><h3>Anti-patterns</h3><p>When deciding against utilizing AWS Directory buckets, it’s essential to consider specific limitations that might hinder their support for your particular use case. These constraints include several aspects:</p><ul><li>Objects within Directory buckets cannot have tags applied to them. Consequently, attempts to copy an object with a tag to a Directory bucket will result in a 501 Not Implemented response.</li><li>Directory buckets become inactive after remaining idle without request activity for 3 months. During this inactive state, the buckets are inaccessible for both read and write operations. Reactivation occurs upon access request, which might take a few minutes, leading to 503 slowdown responses for read and write requests.</li><li>Only Server Side Encryption with S3 Managed keys (SSE-S3) is supported for Directory buckets. Other encryption methods like SSE-KMS and SSE-C are not compatible.</li><li>Several essential S3 features such as Multi-Factor Authentication, S3 Versioning, Replication, Inventory reports, and S3 event notifications are not supported in conjunction with Directory buckets.</li><li>The authorization model differs for Directory buckets, lacking object-level authorization; instead, authorization must occur at the bucket level.</li><li>Currently, Directory buckets are supported only in specific regions: us-east-1, us-west-2, ap-northeast-1, and eu-north-1.</li></ul><p>If you have any questions, please don’t hesitate to contact me.</p><h4>Additional resources that you can take a look:</h4><ul><li><a href="https://aws.amazon.com/blogs/storage/tag/amazon-s3-express-one-zone/">https://aws.amazon.com/blogs/storage/tag/amazon-s3-express-one-zone/</a></li><li><a href="https://youtu.be/IGQtG-7kbbM">https://youtu.be/IGQtG-7kbbM</a></li><li><a href="https://aws.amazon.com/s3/storage-classes/express-one-zone/integrations/">https://aws.amazon.com/s3/storage-classes/express-one-zone/integrations/</a></li></ul><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=dcf3fec06156" width="1" height="1" alt=""><hr><p><a href="https://awstip.com/s3-express-one-zone-and-directory-bucket-dcf3fec06156">S3 Express One Zone and Directory Bucket</a> was originally published in <a href="https://awstip.com/">AWS Tip</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[The DIY Cloud At Home]]></title>
            <link>https://medium.com/@ayogun/the-diy-cloud-at-home-b8a0d1e6d6e9?source=rss-eda7d7339636------2</link>
            <guid isPermaLink="false">https://medium.com/p/b8a0d1e6d6e9</guid>
            <category><![CDATA[devops]]></category>
            <category><![CDATA[cloud-computing]]></category>
            <category><![CDATA[google-cloud-platform]]></category>
            <category><![CDATA[aws]]></category>
            <category><![CDATA[cloud]]></category>
            <dc:creator><![CDATA[A. Yigit Ogun]]></dc:creator>
            <pubDate>Sun, 10 Sep 2023 21:59:30 GMT</pubDate>
            <atom:updated>2023-09-10T21:59:30.472Z</atom:updated>
            <content:encoded><![CDATA[<p>An Epic Journey with the Cloudfryer!</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*pKOnCawQpOb2F6mXIp0ImQ.png" /></figure><p>I’ve been a long-time follower of <a href="https://twitter.com/MetinSeylan">Metin Seylan</a>, a Senior Software Developer known for sharing both entertaining and educational content. Today, I stumbled upon Metin Seylan’s tweet explaining his homegrown ‘Cloudfryer’ project. I was captivated by the insights he shared and felt compelled to spread the knowledge within the community.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/316/1*Pju9SdtL39p1BqcSOo2cwA.jpeg" /></figure><p>In this article, we will find answers to these questions:</p><ul><li>Why did he start this project?</li><li>What hardware did he use?</li><li>What kind of cloud experience did he create at home?</li><li>How much investment did he make in the hardware?</li><li>Was all this effort worth it for him to avoid using the cloud?</li></ul><p>So, let’s start.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*USGZoy_fncuLr2UaGAjSEA.jpeg" /></figure><p>He started learning TensorFlow and realized that his primary need wasn’t a massive GPU but rather an efficient data pipeline.</p><p>So, what did he need to do?</p><p>According to him;</p><h4>1) It involved:</h4><ul><li>Continuously scraping data from websites using bots.</li><li>Using Kafka Streams to preprocess this data, making it ideal for model training.</li></ul><h4>2) His hardware requirements included:</h4><p>- Approximately 36 CPU cores<br>- Around 200GB of RAM<br>- 10Gbit network bandwidth<br>- Space-saving</p><p>At first, he considered using a Raspberry Pi, but he found it to be:</p><p>- Insufficient<br>- Costly</p><p>His preference was the Lenovo Thinkcentre Tiny M720Q, which he described as a real beast.</p><h4>3) So, why did he choose it? Let’s take a look</h4><p>According to him, the Lenovo Thinkcentre Tiny M720Q had several appealing features:</p><ul><li>All components are upgradeable.</li><li>It supports 8th and 9th generation CPUs, including the i9–9900T (8 Core 16 Thread).</li><li>It can handle up to 64GB of RAM.</li><li>It has a PCI-E slot, which is a significant advantage.</li><li>It offers a variety of storage options, including NVME and SATA slots.</li></ul><h4>4) To meet his needs, he made the following changes:</h4><ul><li>Upgraded the CPU to an i5–9500T, offering 6 cores and 6 threads.</li><li>Increased RAM to 64GB with Corsair Vengeance SODIMM.</li><li>Opted for a Samsung 980 NVME SSD for storage.</li><li>Enhanced networking with a 10GB Cluster Networking Intel X540-T2 NIC.</li></ul><h4>5)Now let’s talk about the software and the automation</h4><ul><li>All software components were automated using Ansible, eliminating manual installations. It’s entirely open source!</li><li>You can find the details and configurations on <a href="https://github.com/MetinSeylan/homelab"><em>his GitHub repository</em></a>.</li></ul><p>He also mentioned that he used various software components, and you can learn more about which ones and why by checking out the repository.</p><h4>6) Which software he used and why?</h4><p>According to him, he used the following software components and the reasons behind their selection:</p><p>1. <a href="http://kubernetes.io/"><strong>Kubernetes</strong></a> — A well-known choice for container orchestration.<br>2. <a href="http://cilium.io/"><strong>Cilium</strong></a> — Selected for Kubernetes networking and bare-metal load balancing support.<br>3. <a href="http://helm.sh/"><strong>Helm</strong></a> — Chosen as the Kubernetes package manager.<br>4. <strong>Prometheus and Grafana</strong> — Utilized for cluster observability.<br>5. <a href="http://longhorn.io/"><strong>Longhorn</strong></a>] — Referred to as “SSD for K8s,” it allows distributed use of all disks within the cluster and automates backup and snapshot processes.<br>6. <a href="https://argoproj.github.io/argo-cd/"><strong>ArgoCD</strong></a> — For GitOps.</p><p>These software choices were made based on their respective functionalities and suitability for the project’s requirements.</p><h4>7) So, what is he programming and what are the dependencies?</h4><p>For his web scraper bot, he is writing code in Kotlin and outlined his approach:</p><ul><li>He’s using <strong>Kotlin</strong> for web scraping.</li><li>To build a lightweight and efficient native executable application, he relies on a combination of Spring Boot, Spring Native, and GraalVM.</li></ul><p>For storage and pipeline management, he mentioned using:</p><ul><li>Kafka for handling messaging.</li><li>Kafka Streams for stream processing.</li></ul><p>These choices reflect his strategy for building an efficient and resource-friendly web scraper bot and pipeline.</p><h4>8) The cost of the setup</h4><p>Regarding the costs of the hardware components, here’s the breakdown:</p><p>- Think Centre Tiny: Approximately 65 Euro<br>- i5–9500T CPU: Ranging from 30 to 65 Euro<br>- 64GB RAM: Approximately 100 Euro<br>- X540 T2 NIC: Ranging from 30 to 70 Euro<br>- NVME: About 30 Euro</p><p>He highlights that it’s a budget-friendly setup, mainly consisting of second-hand purchases, making it a cost-effective solution. Roughly around 255 to 330 euro.</p><h4>9) How about electric bill?</h4><p>When it comes to electricity consumption, he shared:</p><ul><li>At idle, it consumes only about ~66 watts.</li><li>Under heavy usage, it’s around ~250 watts or so.</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/472/1*m7bIZtFGPFQToGK7Xwh6Xw.jpeg" /></figure><h4>10) Why just not cloud?</h4><p>He explained why he opted for an on-premises setup instead of the cloud:</p><ul><li>He doesn’t have critical, customer-focused work, so concerns about downtime and security are minimal.</li><li>His home has a parallel internet connection with 1GBit up/down speeds, which makes it feasible for him to run his own infrastructure effectively.</li></ul><p>These factors contributed to his decision to choose an on-premises solution over a cloud-based one.</p><h4>11) How much would it cost on the cloud?</h4><p>He shared his estimation on Google Cloud Platform(GCP). By the way, it’s time to mention that he is a Google Cloud Developer Expert. Don’t forget to visit <a href="https://metin.sh/">his personal website.</a> And here is the GCP bill:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1000/1*4qKI4NeY225Uc13-JCz-yw.jpeg" /></figure><h4><strong>12) How about AWS?</strong></h4><p>As an AWS Community Builder, I want to share the cost of a similar infra on AWS(Amazon Web Services). If you go with an on-demand machine like the Metin did(I assume), the cost is:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/484/1*bVZTXjOCtbacf0fh-bPDOQ.png" /></figure><p>And this would make around 2,694 USD. It’s slightly more expensive than GCP. But since we compare it with a home setup, there will be an initial investment. Hence, we can choose Reserve Instances with a 3-year commitment. And the price goes down to:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*VKMVHI__SK-EKngzamB89w.png" /></figure><p>It’s monthly 2,403 USD. With proper adjustments, this price can go even more down. It’s just a matter of adjustment according to your use cases.</p><p>By making informed plan choices, cloud bills can be significantly reduced. This underscores the invaluable role of consulting with cloud professionals who can guide you toward optimizing your infrastructure and keeping costs in check.</p><p>As you can see with specific requirements, a meticulously crafted ‘Cloudfryer’ setup at home might just be the perfect recipe. However, the cloud offers unparalleled scalability, convenience, and cost-efficiency, making it an enticing choice for many. So, as you embark on your own journey of technological exploration, remember that the answer lies in aligning your choice with your goals, resources, and ambitions. Whichever path you choose, whether it’s the cozy corner of your home or the boundless skies of the cloud, let it serve as the canvas upon which you craft your digital masterpiece.</p><p>To see the original Twitter thread(in Turkish) <a href="https://twitter.com/MetinSeylan/status/1700770030508560474">click here</a>.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=b8a0d1e6d6e9" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[What Byzantine Generals Teach Us About AWS SQS]]></title>
            <link>https://awstip.com/what-byzantine-generals-teach-us-about-aws-sqs-8122230072a7?source=rss-eda7d7339636------2</link>
            <guid isPermaLink="false">https://medium.com/p/8122230072a7</guid>
            <category><![CDATA[aws]]></category>
            <category><![CDATA[devops]]></category>
            <category><![CDATA[cloud]]></category>
            <category><![CDATA[sqs]]></category>
            <category><![CDATA[cloud-computing]]></category>
            <dc:creator><![CDATA[A. Yigit Ogun]]></dc:creator>
            <pubDate>Sat, 09 Sep 2023 12:27:51 GMT</pubDate>
            <atom:updated>2023-09-12T18:12:12.691Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*R4L_IpsnKF2Um89kWcJjWQ.png" /></figure><p>Ever heard of the Byzantine Generals? <a href="https://medium.com/@ayogun/byzantine-generals-problem-a47b33ef87fc">Click here</a> and check it out. But in a nutshell: Generals trying to launch a coordinated attack on a city, but some of them are secretly plotting to confuse the troops. It’s like trying to plan a surprise party when a few friends are determined to spill the beans.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/275/0*z0XR7rVjj5tHAQFG.jpeg" /></figure><p>Now, let’s step into the tech world. We’re talking about messages traveling through a web of computers and bouncing around. They need to arrive in the right order. This is where the <a href="https://aws.amazon.com/message-queue/#:~:text=A%20message%20queue%20is%20a,once%2C%20by%20a%20single%20consumer.">message queues</a> comes in. In AWS the service is called Amazon Simple Queue Service (SQS).</p><p>So, how come they are related you are wondering. Right? Stay still. Today, we’ll uncover the connection between Byzantine Generals and SQS. It’s a story of how ancient problems can teach us a lot about modern tech. So grab your coffee, and let’s dive in!</p><p><strong>AWS SQS and Message Ordering:</strong></p><p>In AWS, SQS is the very first service that was published for public use in 2004. In cloud computing, where information flows like a river through Amazon Web Services (AWS). Among its many offerings, Amazon Simple Queue Service (SQS) stands as a message queue service for Asynchronous messaging. It’s essential to create decoupled resilient applications.</p><p>Within AWS SQS, two distinct queue types emerge: Standard Queues and FIFO (First-In-First-Out) Queues.</p><p><strong>Standard Queues</strong> : It offers a <strong>best-effort</strong> approach to message ordering. The reason is here standard queues <strong>prioritize scalability and high throughput</strong>, which may lead to messages being processed out of order. But why? We’ll come to that in a moment.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/800/0*UDxwAF7twDmJdImm.png" /></figure><p><strong>FIFO Queues</strong> : These queues guarantee that messages are delivered in the exact same order. When message ordering is a crucial assurance for applications, such as processing financial transactions or completing e-commerce sales, this service comes in handy.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/800/0*ki72x8KWWJq8OIZ6.png" /></figure><p>Okay, here that question comes to mind. What is <strong>best-effort</strong> and why my messages come out of order in the standard queue?</p><p>To make things easy to understand, we use often illustrations to grab that kind of abstract concept like SQS. You can see that it looks like a solid pipe. But is it? What is going on under the hood really? Let’s take a look at the image below.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/800/0*OaAG2606mmFjeV_H.png" /></figure><p>This image is from AWS reinvent event. You can see that it’s represented as a solid pipe. But let’s take a look at the next image.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/800/0*v4aoAmG9-Hla0Kwh.png" /></figure><p>In this image, messages seem randomly distributed but of course, sorted with best-effort. <a href="https://en.wikipedia.org/wiki/Best-effort_delivery">What is best-effort</a>? Why order is not guaranteed? Let’s zoom into the pipe itself.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/614/0*iRFim7xnEyg_7knm.png" /></figure><p>If we look closer, we understand the reason why it’s not a solid pipe. <a href="https://docs.aws.amazon.com/AWSSimpleQueueService/latest/SQSDeveloperGuide/sqs-basic-architecture.html">*</a>Because it is distributed on Amazon SQS servers!</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/596/0*BcuHBvQRz2MpSD63.png" /></figure><p>What do I mean? When you push a message into the queue, Amazon SQS redundantly stores a message in more than one availability zone (AZ). Message copies are stored in multiple AZs, no single computer, network, or AZ failure can make messages inaccessible. It’s perfect right! A standard queue makes a best effort to preserve the order of messages, but more than one copy of a message might be delivered out of order.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/736/0*03YXYw24tgZNHNGS.jpg" /></figure><p>Amazon SQS stores copies of your messages on multiple servers for redundancy and high availability.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/750/0*a1nDq_6ikrN4fV_w.png" /></figure><p>On rare occasions, one of the servers that stores a copy of a message might be unavailable when you receive or delete a message. If this occurs, the copy of the message isn’t deleted on that unavailable server, and you might get that message copy again when you receive messages. <a href="https://docs.aws.amazon.com/AWSSimpleQueueService/latest/SQSDeveloperGuide/standard-queues.html#standard-queues-at-least-once-delivery">*</a></p><p>If you don’t mind the order of the messages in your use case, it brings you some superiority in terms of scalability. As long as you keep pushing new messages, AWS can spin up more and more SQS servers and keep handling your messages. Well, here the price that you pay is an ordering issue.</p><h3>Okay, but how does SQS FIFO Queue work its magic?</h3><p>Here the AWS team put in hard work to guarantee and to provide <a href="https://docs.aws.amazon.com/AWSSimpleQueueService/latest/SQSDeveloperGuide/FIFO-queues.html#FIFO-queues-exactly-once-processing">exactly-once processing</a>. The exact details of how AWS SQS maintains the order of messages in a FIFO queue are not publicly disclosed, as this is considered part of AWS’s internal architecture and proprietary information.</p><p>It’s a blend of intelligent mechanisms. Each message entering an SQS FIFO queue carries a unique <strong>Message Deduplication ID</strong> to prevent duplicates. Additionally, messages are organized into <strong>Message Groups</strong> defined by a <strong>Message Group ID</strong>. Messages within the same group are processed sequentially, respecting the order of entry. The AWS team here making an extra effort to make sure the order is maintained which comes with a cost. It’s more expensive than Standard Queue and has a limited number of transactions per second (TPS).</p><p>The second thing is the maximum timeout for storing a given message deduplication ID is 5 minutes. A duplicate message would be accepted and continued processing in the SQS FIFO queue if it arrived more than five minutes after the original. If exactly-once were a crucial requirement, other distributed system components would have to be responsible for ensuring message uniqueness.</p><p>So, let’s come back to our topic. How all those are related to the Byzantine Generals?</p><p><strong>Understanding the Byzantine Generals Problem:</strong></p><p>To grasp the connection between Byzantine Generals and the modern-day challenges of message ordering, let’s take a closer look at the historical problem.</p><p>Imagine an ancient army preparing to lay siege to an enemy’s city. The success of their mission depends on synchronized attacks from multiple fronts. But here’s the catch: not all the generals can be trusted. Some of them might be traitors and they can sabotage the plan.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/800/0*-7rH1M5DIRZZOIb1.png" /></figure><p>In the face of this problem, the loyal generals must somehow agree on a common strategy, even though they can’t be sure if the orders they receive from their fellow commanders are genuine or deceitful. The challenge, known as the “<a href="https://en.wikipedia.org/wiki/Byzantine_fault">Byzantine Generals Problem</a>”, is to achieve consensus in a distributed and potentially unreliable communication network.</p><p>The complexity of contemporary distributed systems corresponds to this historical issue in a world where integrity is of utmost importance. Our digital systems struggle with faulty parts, network issues, and the necessity of maintaining the right sequence of events, much like the devoted generals trying to plan their attack.</p><p>The Byzantine Generals Problem serves as a timeless parable of the difficulties inherent in distributed decision-making. It underscores the vital importance of achieving consensus and maintaining order, whether on the battlefield of ancient Byzantium or in today’s cloud computing infrastructure.</p><p><strong>Message Ordering Challenges in Distributed Systems such as SQS:</strong></p><p>Maintaining the proper sequence of messages is comparable to orchestrating a difficult symphony in the interconnected world of distributed computing, where data passes across networks. Consider a situation where you must make sure a set of activities takes place exactly as planned.</p><p>Distributed systems(such as SQS) often involve multiple nodes(most probably different AZs) or components that communicate and collaborate to achieve a common goal. However, these systems operate in a volatile environment with uncertainty. Network delays, hardware failures, and asynchronous communication can introduce chaos into what should be an orderly sequence of events. This quest for precise message ordering is a fundamental challenge in distributed systems.</p><p>In the AWS world, AWS Simple Queue Service (SQS) comes to the rescue and AWS team manages this chaos for us. SQS offers a solution to the solve the message ordering problem. However, it’s not the end of the story. SQS FIFO queues guarantee in-order and exactly-once processing only within the queue itself, not in the entire distributed system that would use the queue.</p><p>The <a href="https://en.wikipedia.org/wiki/Atomic_broadcast">Total Order Broadcast</a> problem appears in the field of distributed computing. According to the Total sequence Broadcast, messages must be sent to every participant in the same sequence, regardless of who they are to. In a distributed system, FIFO delivery of messages is basically a special case of the issue with extra order restrictions. Understanding the constraints of potential FIFO delivery problem solutions is aided by research on the Total Order Broadcast problem. Simply put, the Total Order Broadcast problem is equivalent to the problem of achieving distributed consensus, i.e. having all participants in a distributed system agree on message delivery order.</p><p>Although a SQS FIFO queue is undoubtedly a very helpful tool in developing a solution to this issue, it is not a solution in and of itself. The task of guaranteeing exactly-once publication outside the queue typically falls to us developers.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=8122230072a7" width="1" height="1" alt=""><hr><p><a href="https://awstip.com/what-byzantine-generals-teach-us-about-aws-sqs-8122230072a7">What Byzantine Generals Teach Us About AWS SQS</a> was originally published in <a href="https://awstip.com/">AWS Tip</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Byzantine Generals Problem]]></title>
            <link>https://medium.com/@ayogun/byzantine-generals-problem-a47b33ef87fc?source=rss-eda7d7339636------2</link>
            <guid isPermaLink="false">https://medium.com/p/a47b33ef87fc</guid>
            <category><![CDATA[cloud]]></category>
            <category><![CDATA[byzantine-fault-tolerance]]></category>
            <category><![CDATA[distributed-systems]]></category>
            <category><![CDATA[blockchain]]></category>
            <category><![CDATA[smart-contracts]]></category>
            <dc:creator><![CDATA[A. Yigit Ogun]]></dc:creator>
            <pubDate>Sat, 09 Sep 2023 08:13:14 GMT</pubDate>
            <atom:updated>2023-09-09T08:13:14.040Z</atom:updated>
            <content:encoded><![CDATA[<p>A Byzantine fault is a state of a computer system, particularly distributed computing systems, where components may fail and there is imperfect information on whether a component has failed. It is also known as a Byzantine generals problem, interactive consistency, source congruency, error avalanche, Byzantine agreement problem, and Byzantine failure. The phrase gets its name from an allegory known as the “Byzantine generals problem”, which was created to explain a scenario in which the system’s participants must agree on a coordinated approach to prevent catastrophic system failure, yet some of these individuals are unreliable.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/275/1*vuoViwfNXDWj4Q2Czh6t-g.jpeg" /></figure><p>A server, for example, may inconsistently appear to failure-detection systems as both malfunctioning and working while exhibiting various symptoms to various observers in a Byzantine fault. It is challenging for the other components to declare it failed and block it from the network since they first need to agree on which component actually failed. A fault-tolerant computer system’s resistance to such circumstances is known as byzantine fault tolerance (BFT).</p><p>Consider a group of generals attacking a fort as an example of the flaw in its most basic form. The generals must determine whether to advance or retire as a group; some may favour advance while others favour retreat. The most crucial factor is that all generals come to an agreement on a strategy since a few generals’ haphazard attack would result in a rout and would be worse than either a coordinated attack or a coordinated retreat.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*xJGKNqJVrkFVtZ6NggwHbw.png" /></figure><p>The issue is made more difficult by the existence of treacherous generals who might not only vote for a poor strategy, but also do so selectively. For instance, the ninth general may send a vote of attack to the remaining generals and a vote of retreat to the four generals who support attacking while the remaining four generals support retreating. While the remainder will attack (which might not go well for the attackers), those who obtained a vote to retreat from the ninth general will retreat. The generals’ physical separation from one another and requirement to convey their votes via messengers who might not deliver them or might fabricate votes adds to the difficulty of the situation.</p><h4>Some Solutions</h4><p>In 1982, Lamport, Shostak, and Pease described a number of pioneering solutions. They started off by noticing that the Generals’ Problem may be reduced to solving a “Commander and Lieutenants” problem, in which all faithful Lieutenants must act together and that their actions must match what the Commander ordered, assuming the Commander is loyal:</p><ul><li>One method takes into account circumstances in which signals may be faked, but they would still be Byzantine-fault tolerant so long as there were less than one-third of the generals who were disloyal. It basically comes down to showing that the one Commander and two Lieutenants problem cannot be handled if the Commander is a traitor. Dealing with one-third or more traitors is impossible. To illustrate this, imagine that we have a traitorous Commander A and two Lieutenants, B and C. When A orders B to attack and C to retreat, and B and C communicate with one another by forwardeding A’s message, neither B nor C can determine who the traitor is because it may not necessarily be A — the other Lieutenant may have forged the message claiming to be from A. It can be shown that if <em>n</em> is the number of generals in total, and <em>t</em> is the number of traitors in that <em>n</em>, then there are solutions to the problem only when <em>n</em> &gt; 3<em>t</em> and the communication is synchronous (bounded delay)</li><li>Unforgeable message signatures are required for the second solution. Digital signatures (in contemporary computer systems, this may be accomplished in practise via public-key cryptography) can offer Byzantine fault tolerance in the presence of an arbitrary number of betraying generals for security-critical systems.</li></ul><p>You may hear the term “Byzantine Generals Problem” also in Blockchains. Blockchain is a public, distributed ledger that contains the records of all transactions. If all users of the Bitcoin network, known as nodes, could agree on which transactions occurred and in what order, they could verify the ownership and create a functioning, trustless money system without the need for a centralized authority. Due to its decentralized nature, blockchain relies heavily on a consensus technique to validate transactions. It is a peer-to-peer network that offers its users transparency as well as trust. Its distributed ledger is what sets it apart from other systems. Blockchain technology can be applied to any system that requires proper verification.<a href="https://www.geeksforgeeks.org/byzantine-generals-problem-in-blockchain/">[*]</a></p><p>In terms of distrbiuted systems; a source processor is assumed to broadcast its initial value to another processor in the system in accordance with the Byzantine agreement problem notion. To select a source processor, sequential ordering is used and a source processor is chosen at random.</p><p>All non-faulty processors must agree on the same value in order to obtain the byzantine agreement procedure. The source processor might not always be defective. Then, other processors do not use its starting value.</p><p>The goal of the Byzantine problem is to develop defences against failures in which system components fail in unpredictable ways, such as by processing requests erroneously, corrupting their local information, or generating inconsistent or wrong outputs. The Byzantine failure simulates real-world conditions in which computer and network hardware failures, network congestion, disconnection, and malicious attacks can cause computers and networks to behave in unexpected ways.</p><p><a href="https://groups.csail.mit.edu/tds/papers/Lynch/jacm85.pdf">In an article</a> published by MICHAEL J. FISCHER, NANCY A. LYNCH, and MICHAEL S. PATERSON; they have shown that a natural and important problem of fault-tolerant cooperative computing cannot be solved in a totally asynchronous model of computation. These results do not show that such problems cannot be “solved” in practice; rather, they point up the need for more refined models of distributed computing that better reflect realistic assumptions about processor and communication tim- ings, and for less stringent requirements on the solution to such problems.</p><p>To sum up, we find ourselves at the nexus of creativity and history as we reach the end of our adventure through the complex world of the Byzantine Generals Problem. This age-old puzzle, which was inspired by the generals of the Byzantine Empire, has survived the test of time and now paves the way for contemporary technologies.</p><p>The underlying ideas of blockchain technology and distributed systems are intimately resonant with the nature of the challenge, which is attaining consensus and order in the face of deceit and ambiguity. Just as it previously directed commanders seeking victory behind the walls of Byzantium, The Byzantine Generals Problem acts as a beacon, directing us through the maze-like intricacies of networked communication.</p><p>The Byzantine Generals Problem is an enduring challenge in the blockchain realm, necessitating creative solutions to guarantee that ledgers are immutable, trust is upheld, and digital transactions are secure. This emphasises the importance of the Byzantine Fault Tolerance (BFT) algorithms, consensus mechanisms, and cryptographic protocols that now underpin the decentralised currency and smart contract foundations.</p><p>The Byzantine Generals Problem is a signal for distributed systems engineers and architects to take up arms. It serves as a reminder that reaching consensus in a networked environment where components may malfunction or act maliciously is a riddle requiring creative solutions. It encourages us to create systems that, like those ancient generals, can cooperate and have faith in the face of uncertainty.</p><p>As we leave the Byzantine Generals to their ancient strategies, we step into the future, armed with the wisdom they imparted. Their challenge has evolved, but the quest for consensus and order endures. It’s a reminder that in the ever-expanding landscape of technology, the lessons of the past continue to shape the possibilities of tomorrow. So, with Byzantium as our guide, let us continue forging ahead, navigating the Byzantine labyrinth of the digital age, and seeking innovation amidst the echoes of history.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=a47b33ef87fc" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Comparison of ELK vs Jaeger]]></title>
            <link>https://medium.com/@ayogun/comparison-of-elk-vs-jaeger-b920c2a65327?source=rss-eda7d7339636------2</link>
            <guid isPermaLink="false">https://medium.com/p/b920c2a65327</guid>
            <category><![CDATA[jaeger]]></category>
            <category><![CDATA[elk-stack]]></category>
            <category><![CDATA[elasticsearch]]></category>
            <category><![CDATA[kibana]]></category>
            <category><![CDATA[devops]]></category>
            <dc:creator><![CDATA[A. Yigit Ogun]]></dc:creator>
            <pubDate>Fri, 30 Jun 2023 16:41:24 GMT</pubDate>
            <atom:updated>2023-06-30T16:41:24.655Z</atom:updated>
            <content:encoded><![CDATA[<h4>Comparison of Tracing Capabilities</h4><p>To track and examine the flow of requests among many components in distributed systems, tracing is a vital and essential component. It enables the detection of performance-related bottlenecks and other problems that might be impeding the system’s overall effectiveness. Additionally, the information gleaned by tracing can be quite helpful for deciphering user behavior and pinpointing areas in need of development.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*_xMmNRf9wAVzlOh8Nlu0VQ.jpeg" /><figcaption><a href="https://logz.io/blog/jaeger-and-the-elk-stack/">[*]</a></figcaption></figure><p>While there are many tools available for tracing in distributed systems, ELK and Jaeger are two of the most well-liked open-source choices. The tracing procedure can be substantially aided by the variety of features and capabilities that both programs offer.</p><p>In this report, we will explore and compare the tracing capabilities of ELK and Jaeger, examining their strengths and limitations in order to provide a comprehensive overview of their respective tracing capabilities. By doing so, I hope to provide readers with a better understanding of the different options available for tracing in distributed systems and help them make an informed decision about which tool is best suited to their needs.</p><p>But first thing first; what is ELK Stack and what is Jaeger? Let me walk you through it in a nutshell.</p><h3>What is Jaeger?</h3><p>Uber created Jaeger, which was released as an open-source in 2016. Jaeger, which enables users to conduct root cause analysis, performance optimization, and distributed transaction monitoring, was inspired by two other existing tracing tools, Zipkin and Dapper. It is swiftly rising in popularity among open-source distributed tracing tools.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*wRuwJaJz0hORgluYq4X6vw.jpeg" /><figcaption><a href="https://www.travelandleisure.com/uber-opentable-viator-travel-apps-6828479">[*]</a></figcaption></figure><p>For Go, Java, Node, Python, and C++ projects, Jaeger offers OpenTracing-based instrumentation. It also employs consistent upfront sampling with unique probability for each service or endpoint and supports a variety of storage backends, including memory, Elasticsearch, Kafka, and Cassandra.</p><h3>What is ELK (aka ELK Stack)?</h3><p>ELK stack refers to a combination of three open-source software tools: Elasticsearch, Logstash, and Kibana. Together, these tools create a powerful data processing and visualization platform commonly used for log and data analysis.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/600/1*SXT3XUMp1iDvvUoIzWa0Qg.png" /><figcaption><a href="https://www.open4dev.com/journal/2017/11/16/elk-stack-with-docker">[*]</a></figcaption></figure><p>The central component of the ELK stack is the search and analytics engine <a href="https://www.elastic.co/?ultron=B-Stack-Trials-EMEA-C-Exact&amp;gambit=Stack-Core&amp;blade=adwords-s&amp;hulk=paid&amp;Device=c&amp;thor=elasticsearch&amp;gclid=CjwKCAjw-vmkBhBMEiwAlrMeF1qOABkKp-o8FilPvxrH7ZNj2jtR3-uKkjl2Gv4MpjAqCWeQ-v82FhoCLo4QAvD_BwE">Elasticsearch</a>. Large volumes of data can be efficiently stored, indexed, and retrieved in real time. Elasticsearch is perfect for log management, monitoring, and full-text search since it handles organized, unstructured, and time-series data with ease.</p><p>Within the ELK stack, <a href="https://www.elastic.co/logstash/">Logstash</a> serves as a pipeline for data processing. From a variety of sources, including logs, metrics, and event streams, it gathers, parses, and transforms data. In order to enable smooth integration with various data sources and formats, Logstash provides a large variety of plugins that simplify data intake and filtering.</p><p><a href="https://www.elastic.co/kibana?ultron=B-Stack-Trials-EMEA-S-Exact&amp;gambit=Stack-Kibana&amp;blade=adwords-s&amp;hulk=paid&amp;Device=c&amp;thor=kibana&amp;gclid=CjwKCAjw-vmkBhBMEiwAlrMeF49_OwxV885Erf4vldYxdOHu_aiHGM1b7ghiMlBYr0fszA1kGVnWwRoCauoQAvD_BwE">Kibana</a> is the data visualization component of the ELK stack. It provides a web-based interface that allows users to explore, analyze, and visualize data stored in Elasticsearch. With its intuitive dashboards, charts, and maps, Kibana helps users gain insights from their data by creating interactive visual representations and performing advanced analytics.</p><p>Cut the story short, the ELK stack is a versatile and scalable solution for managing and analyzing data. It is widely used in various industries for log analysis, security monitoring, operational intelligence, and business analytics, empowering organizations to leverage their data effectively.</p><p>Okay, it’s enough trivia information. Let’s dive into the comparison. I’ll start with the capabilities of the ELK Stack.</p><h3>ELK Tracing Capabilities</h3><p>As I previously mentioned, ELK is a powerful tool that provides a wide range of capabilities for log analytics and tracing in distributed systems. With its three key components — Elasticsearch, Logstash, and Kibana — ELK offers a comprehensive solution for monitoring and analyzing system performance.</p><p>ELK uses the OpenTracing API for distributed tracing, which is one of its distinguishing characteristics. This gives developers a simple way to track requests across many services and a thorough understanding of system performance. ELK is a versatile solution for system monitoring and analysis since it supports a range of data sources, such as logs, metrics, and APM.</p><p>Additionally, ELK is quite adaptable, enabling developers to customize its features to meet their unique requirements. ELK is the best tool for locating system problems and improving system performance because of its strong search and visualization features.</p><p>ELK is a crucial tool for any business that uses distributed systems and wants to track and examine system performance.Its powerful tracing capabilities, extensive data source support, and customizable features make it a must-have for developers and IT professionals alike.</p><h4>Trace Collection</h4><p>With the comprehensive platform ELK, you may gather traces from a range of data sources, such as logs, metrics, and application performance information. You can then see a full picture of your system’s performance and pinpoint its weak points as a result. ELK employs Logstash, a strong tool with the ability to gather data from a variety of sources, to gather this data. You can enable Logstash to monitor every aspect of your system’s performance by configuring it to gather data from any source that produces data in a supported format.</p><p>Elasticsearch, a search and analytics engine that is incredibly scalable and adaptable, is used to store and index the data after it has been gathered. Elasticsearch enables you to search and analyze your data in real-time, giving you the ability to quickly identify issues and take corrective action. In addition to supporting a wide range of tracing formats such as Zipkin and Jaeger, ELK also provides a powerful search and analytics engine based on Elasticsearch.</p><h4>Trace Analysis</h4><p>For the analysis of your trace data, ELK offers a wide range of tools and functions. The traces are shown using Kibana, which also offers a variety of tools for data analysis and querying. You may filter and aggregate your data with Kibana to get a better understanding of how well your system is doing. To measure important performance indicators and monitor particular metrics, you can also make your own dashboards.</p><p>ELK offers machine learning capabilities for anomaly detection and predictive analytics in addition to its strong search and analytics capabilities. You can automatically find anomalies in your data using ELK’s machine-learning techniques, and you’ll get alerts when strange patterns are found. This makes it possible for you to anticipate prospective problems and treat them before they become serious.</p><h4>Scalability</h4><p>One of the key benefits of ELK is its ability to scale to handle large volumes of data. ELK provides horizontal scaling by allowing you to add more nodes to the Elasticsearch cluster. This enables you to handle increasing volumes of trace data as your system grows. ELK also provides sharding and replication capabilities for high availability and fault tolerance, ensuring that your trace data is always available when you need it.</p><h3>Jaeger Tracing Capabilities</h3><p>Jaeger provides visibility into the entire distributed system, allowing developers to identify and troubleshoot issues that may arise at the service level. Jaeger uses the OpenTracing API for tracing, which enables it to trace requests across multiple services and identify the root cause of issues in real-time.</p><p>As mentioned previously, Jaeger supports various languages such as Java, Python, and Go, making it a versatile tool for developers working in different programming languages. The tool provides a user-friendly interface that allows developers to visualize the flow of requests and identify bottlenecks in the system. Additionally, Jaeger provides integration with other popular tools such as Prometheus and Grafana, allowing developers to correlate tracing data with metrics and logs to gain a comprehensive view of the system.</p><h4>Trace Collection</h4><p>Jaeger is a powerful tool for collecting traces. With support for various agents and libraries, it can collect traces in a range of formats such as Zipkin and OpenCensus. Jaeger has a flexible and extensible architecture for trace collection, which includes features like support for sampling and rate limiting. This means you can collect detailed traces even in high-traffic environments.</p><h4>Trace Analysis</h4><p>Jaeger provides a web-based user interface for trace analysis. The UI includes a range of features, such as trace search, trace visualization, and service dependency analysis. You can use these features to gain insights into how your system is behaving and identify areas for improvement. Additionally, Jaeger integrates with Prometheus for metrics and Grafana for dashboarding, so you can get a complete picture of your system’s performance.</p><h4>Scalability</h4><p>Jaeger is designed to be highly scalable as well and can handle large volumes of data. It uses a distributed architecture with a collector, query, and storage layer. This architecture allows you to scale Jaeger horizontally by adding more nodes to the cluster. With Jaeger, you can collect and analyze traces from even the most complex systems, making it an indispensable tool for any organization.</p><h3>Comparison</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*R0R1WAIgMxfxdtu5tJpFPA.png" /></figure><p>Now let’s look from another angle. I will take a look at the differences and similarities quickly.</p><h4>Differences:</h4><ol><li>Purpose: ELK Stack is primarily designed for log management and analysis, providing a comprehensive platform for storing, indexing, and visualizing log data. Jaeger, on the other hand, is specifically focused on distributed tracing, enabling the collection, visualization, and analysis of traces in complex microservices architectures.</li><li>Data Focus: ELK Stack primarily handles structured and unstructured log data, which may include system logs, application logs, or any other log sources. Jaeger, on the other hand, is designed specifically for distributed traces, capturing and analyzing the flow of requests across microservices and providing insights into latency and performance.</li><li>Components: ELK Stack comprises Elasticsearch, Logstash, and Kibana as its core components, with each serving specific functions within the data processing pipeline. In contrast, Jaeger consists of components like the Jaeger Collector, Jaeger Agent, Jaeger Query, and Jaeger UI, which work together to collect, store, and visualize traces.</li></ol><h4>Similarities:</h4><ol><li>Open-source: Both ELK Stack and Jaeger are open-source projects, allowing users to access, modify, and contribute to the source code. This fosters a strong community of users and developers who provide support and continuously enhance the tools.</li><li>Scalability: Both ELK Stack and Jaeger are designed to be scalable. They can handle large volumes of data and support horizontal scaling by distributing the workload across multiple nodes, ensuring high availability and performance.</li><li>Visualization: Both ELK Stack (via Kibana) and Jaeger offer powerful visualization capabilities. Kibana allows users to create custom dashboards, charts, and graphs to visualize log data. Similarly, Jaeger provides a user-friendly interface to visualize traces, understand the flow of requests, and analyze latency and performance across distributed systems.</li><li>Integration: Both tools offer integrations with various technologies and frameworks. ELK Stack supports integration with numerous data sources, log formats, and APIs. Similarly, Jaeger provides libraries and SDKs for popular programming languages, making it easier to instrument applications and collect distributed traces.</li></ol><h3>Conclusion</h3><p>Both ELK and Jaeger are powerful tools for tracing in distributed systems. ELK is a widely used data analytics platform that can be used for log analytics. It provides a powerful search and analytics engine and can be used to monitor, analyze, and graph vast amounts of log data from various sources. ELK is highly customizable and can be used with a wide variety of plugins and third-party tools.</p><p>On the other hand, Jaeger is more focused on distributed tracing and provides a web-based UI for trace visualization. The Jaeger tracing system is optimized for use with microservices architectures and is designed to handle high volumes of trace data. Jaeger also provides a powerful query language that allows users to extract insights from trace data.</p><p>Both ELK and Jaeger are highly scalable and support horizontal scaling, making them suitable for use in large distributed systems. The choice between ELK and Jaeger depends on the specific requirements of the system and the use case. For systems that generate large amounts of log data, ELK may be the better choice. For systems that require detailed trace analysis, Jaeger may be more suitable. It’s up to you!</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=b920c2a65327" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[I chose EC2 Storage over EBS despite it’s ephemeral —  Here’s Why…]]></title>
            <link>https://medium.com/@ayogun/i-chose-ec2-storage-over-ebs-despite-its-ephemeral-here-s-why-377f3280f5c2?source=rss-eda7d7339636------2</link>
            <guid isPermaLink="false">https://medium.com/p/377f3280f5c2</guid>
            <category><![CDATA[ec2]]></category>
            <category><![CDATA[cloud]]></category>
            <category><![CDATA[amazon-web-services]]></category>
            <category><![CDATA[cloud-storage]]></category>
            <category><![CDATA[aws]]></category>
            <dc:creator><![CDATA[A. Yigit Ogun]]></dc:creator>
            <pubDate>Sat, 20 May 2023 13:27:34 GMT</pubDate>
            <atom:updated>2023-05-20T13:27:34.081Z</atom:updated>
            <content:encoded><![CDATA[<h3>I chose EC2 Storage over EBS despite it’s ephemeral — Here’s Why…</h3><p>In the world of cloud infrastructure, selecting the right storage solution is essential to getting the best performance and reliability. I don’t say it as a AWS Storage Community Builder, I tell it also as a regular AWS customer. I started to think about writing this article when I had to choose between EC2 Storage and EBS (Elastic Block Store).</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/800/1*SM-BpiSgKspHrYra3eFvKw.jpeg" /></figure><p>Contrary to popular belief, I decided on EC2 Storage despite its ephemeral nature. I’ll explain the motivations behind my unusual choice in this article, as well as the benefits that EC2 Storage offers.</p><p>First thing first;</p><h4><strong>1- EC2 Storage vs EBS:</strong></h4><p>Before diving into the logic behind my choice, let’s establish a foundational understanding of EC2 Storage and EBS. If you are already familiar with the services, you can simply skip this section. The local instance storage that comes with Amazon EC2 instances is referred to as “EC2 Storage”; it provides high-speed performance temporary block-level storage but lacks durability. EBS, on the other hand, offers robust and long-lasting block storage that is compatible with EC2 instances. While EBS is frequently the preferred option, I needed to go against it because of my client’s requirement, but I decided to investigate the unrealized potential of EC2 Storage. If you want to read more about other possible storage solutions on AWS, I recommend you to <a href="https://medium.com/@ayogun/ebs-efs-s3-comparison-from-aws-storage-solutions-aaae9bde731d">read my article.</a></p><p>Main reason is;</p><h4><strong>2- Superiority of Performance:</strong></h4><p>One of the main reasons and also the ultimate one, for choosing EC2 Storage was its exceptional performance. With EC2 Storage, you can simply experience lightning-fast data access, accelerated disk I/O, and remarkably low latency. Me personally in this use case, I had a database requires a block storage with the throughput to support several million transactions per second. <a href="https://aws.amazon.com/ebs/features/">If you look closer to the EBS</a>; you can get more information about EBS types. Although let me give a quick answer; 350,000 IOPS. Well if you don’t know what IOPS is, <strong>Input/output operations per second</strong> (<strong>IOPS</strong>, pronounced <em>eye-ops</em>) is an <a href="https://en.wikipedia.org/wiki/Input/output">input/output</a> performance measurement used to characterize <a href="https://en.wikipedia.org/wiki/Data_storage_device">computer storage</a> devices like <a href="https://en.wikipedia.org/wiki/Hard_disk_drive">hard disk drives</a> (HDD), <a href="https://en.wikipedia.org/wiki/Solid_state_drives">solid state drives</a> (SSD), and <a href="https://en.wikipedia.org/wiki/Storage_area_network">storage area networks</a> (SAN). Like <a href="https://en.wikipedia.org/wiki/Benchmark_(computing)">benchmarks</a>, IOPS numbers published by storage device manufacturers do not directly relate to real-world application performance.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*9s1bjNOdrFqbZQjW9iYYXg.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*4BKJaMQTFYBN2pvqFQlhaA.png" /></figure><p>And unrevealed potential;</p><h4>3- Flexible and Cost-Efficient:</h4><p>EC2 Storage’s inherent flexibility and cost effectiveness were further strong arguments in its favor. As an ephemeral storage option, EC2 Storage can be dynamically provisioned and terminated as needed. This flexibility empowered me to design highly scalable and fault-tolerant architectures, leveraging auto-scaling capabilities and reducing unnecessary costs associated with persistent storage.</p><p>I chose a different route by embracing the ephemeral nature of EC2 Storage in a world where EBS is frequently seen as the de facto option for persistent storage. The client requirement for superior performance, flexibility, and cost effectiveness motivated me to architect this solution. I wanted to use this blog to show off EC2 Storage’s latent potential and encourage readers to choose a creative storage solution. Even while EC2 Storage might not be appropriate for all use cases, it has changed the game for me. We can open new doors and significantly improve our cloud infrastructure by questioning the established quo and looking at other solutions.</p><p>Remember, sometimes it’s worth venturing into uncharted territory to uncover hidden gems that can revolutionize our technological landscapes. I also encourage you to read this article to get more detailed <a href="https://www.percona.com/blog/using-aws-ec2-instance-store-vs-ebs-for-mysql-how-to-increase-performance-and-decrease-cost/">about using AWS EC2 instance store vs EBS for MySQL.</a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=377f3280f5c2" width="1" height="1" alt="">]]></content:encoded>
        </item>
    </channel>
</rss>