<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Anil Augustine Chalissery on Medium]]></title>
        <description><![CDATA[Stories by Anil Augustine Chalissery on Medium]]></description>
        <link>https://medium.com/@anilaugustinechalissery?source=rss-b6da988055dc------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/1*XwSbscxzFm-EzUn4MqEVMw.jpeg</url>
            <title>Stories by Anil Augustine Chalissery on Medium</title>
            <link>https://medium.com/@anilaugustinechalissery?source=rss-b6da988055dc------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Thu, 19 Nov 2026 09:43:25 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@anilaugustinechalissery/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[We Had bull-board in Every Service. Here’s Why We Replaced It All with One Dashboard.]]></title>
            <link>https://anilaugustinechalissery.medium.com/we-had-bull-board-in-every-service-heres-why-we-replaced-it-all-with-one-dashboard-1dca550bf75e?source=rss-b6da988055dc------2</link>
            <guid isPermaLink="false">https://medium.com/p/1dca550bf75e</guid>
            <category><![CDATA[message-queue]]></category>
            <category><![CDATA[sqs]]></category>
            <dc:creator><![CDATA[Anil Augustine Chalissery]]></dc:creator>
            <pubDate>Tue, 07 Jul 2026 16:48:12 GMT</pubDate>
            <atom:updated>2026-07-07T16:48:12.460Z</atom:updated>
            <content:encoded><![CDATA[<p><em>How bullmq-control-dashboard gave us a single control plane for BullMQ queues across multiple Redis instances and environments, and why every other option fell short.</em></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*qm93PxkRVZJiGINS" /><figcaption>Photo by <a href="https://unsplash.com/@d_ks11?utm_source=medium&amp;utm_medium=referral">Dominic Kurniawan Suryaputra</a> on <a href="https://unsplash.com/?utm_source=medium&amp;utm_medium=referral">Unsplash</a></figcaption></figure><p>If you’re running BullMQ in production across multiple services, you’ve probably been here:</p><p>You’re deploying a new version of a service. Something feels off — jobs seem slow, or a queue that should be draining isn’t. You open the bull-board UI embedded in Service A. Looks fine. Switch to the tab for Service B. Fine there too. But you’re running four services, across two environments, on three different Redis instances. By the time you’ve clicked through everything, five minutes have passed and you still don’t have a clear picture.</p><p>That was us. And it was death by a thousand clicks.</p><h3>The Problem with bull-board Embedded Per Service</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*bVrOf8o8P_nE0q5V" /><figcaption>Photo by <a href="https://unsplash.com/@silverkblack?utm_source=medium&amp;utm_medium=referral">Vitaly Gariev</a> on <a href="https://unsplash.com/?utm_source=medium&amp;utm_medium=referral">Unsplash</a></figcaption></figure><p>Don’t get me wrong — <strong>bull-board is excellent</strong>. It’s the right choice for visualizing BullMQ queues and we still use it under the hood. But embedding it per service creates a fundamental visibility problem as your infrastructure grows.</p><p>Each embedded instance only knows about the Redis connection and queues you’ve manually wired into <em>that</em> service. There’s no cross-service view. No way to see all your failed jobs in one place. No overview that answers “is anything on fire right now?” with a single glance.</p><p>For a team running many Redis instances across multiple environments — dev, staging, production — this means you’re constantly context-switching between browser tabs, each showing you a slice of the truth.</p><p>We needed something different.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*ZG6BNI_gloS0MNy8ZacHYQ.png" /></figure><p><a href="https://github.com/imatefx/bullmq-control-dashboard">bullmq-control-dashboard</a> is a <strong>standalone central management dashboard</strong> built on top of bull-board. Same great visualization — but wrapped with everything bull-board intentionally leaves out.</p><p>The single most important thing it does differently from every other option we evaluated: <strong>it supports multiple Redis connections simultaneously, in one UI.</strong></p><p>Sounds obvious. Almost no other tool does it.</p><p>Taskforce.sh is cloud-hosted with pricing. Arena hasn’t been actively maintained. Every embedded bull-board instance is siloed by design. bullmq-control-dashboard is the only open-source, self-hosted option that lets you add as many Redis connections as you want and manage all your queues — across all of them — from a single place.</p><h3>What It Actually Gives You</h3><h3>One Overview Page for Everything</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*_xGCwynPI_0-SiW8rqKjZA.png" /></figure><p>The first thing you see after adding your Redis connections is an <strong>Overview page</strong> that aggregates job counts — active, waiting, completed, failed, delayed — across every queue on every Redis instance.</p><p>During a deployment, this page is the first thing we check. Instead of clicking through four separate bull-board instances, we look at one screen and immediately know if something is backing up or failing at an unusual rate.</p><h3>Auto-Discovery — No Manual Queue Wiring</h3><p>This was a surprise feature we didn’t expect to love as much as we do. The dashboard scans Redis using SCAN &lt;prefix&gt;:*:meta and <strong>automatically discovers all BullMQ queues</strong>. As your application adds or removes queues, the dashboard picks them up live.</p><p>With our old embedded bull-board setup, adding a new queue meant updating the configuration in the host service and redeploying. Now we add a queue in our app code and it just appears in the dashboard.</p><h3>Multi-Environment Support</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*07eEgTEpyIQxTl_gV9eokQ.png" /></figure><p>We have the same logical queues running in dev, staging, and production — all on separate Redis instances. With bullmq-control-dashboard, we connect all three and use queue <strong>grouping</strong> to organize them by environment. One tool, every environment, clear separation.</p><h3>Read-Only Mode for Production Queues</h3><p>This one is underrated for team safety. You can toggle any queue into <strong>read-only mode</strong>, which means anyone with dashboard access can observe it but nobody can accidentally retry, clean, or pause jobs on a production queue. We use this for our most critical queues and it removes a whole category of “oops” incidents during debugging sessions.</p><h3>Role-Based Access</h3><p>Two roles out of the box — <strong>admin</strong> (full access) and <strong>reader</strong> (view only). Readers see everything but all mutations return 403. We give reader access to anyone who needs observability into the queue system without needing to touch it. The server refuses to start without admin credentials configured, which means no accidentally public dashboards.</p><h3>Config Import/Export</h3><p>The entire configuration — connections, groups, queue settings — exports as a JSON file. We version-control ours and import it when spinning up the dashboard in a new environment. Secrets are redacted by default, with support for ${ENV_VAR} placeholders.</p><h3>Getting Started</h3><h3>Local (for evaluation)</h3><pre>git clone https://github.com/imatefx/bullmq-control-dashboard<br>cd bullmq-control-dashboard</pre><pre>cp .env.example .env<br># Set ADMIN_USERNAME and ADMIN_PASSWORD in .env</pre><pre>npm install<br>npm run dev<br># UI → <a href="http://localhost:5173/">http://localhost:5173</a>   API → <a href="http://localhost:3010/">http://localhost:3010</a></pre><p>Want to try it with realistic data before pointing it at your own Redis?</p><pre>npm run seed:demo   # seeds demo jobs into Redis db 5</pre><h3>Docker Compose (recommended)</h3><pre>cp .env.example .env<br>docker compose up --build<br># http://localhost:3010 — includes a bundled Redis container</pre><p>Config is mounted as a volume at ./config, so your connections and settings persist across restarts.</p><h3>Who Should Use This</h3><p>If any of these describe you, this is worth an afternoon to set up:</p><ul><li>You’re running BullMQ across <strong>multiple services</strong>, each with their own embedded bull-board</li><li>You have <strong>multiple Redis instances</strong> — per environment, per service, or both</li><li>You want <strong>one place to check queue health</strong> during deployments or incidents</li><li>You need to give teammates <strong>read-only visibility</strong> into queues without giving them full access</li></ul><p>If you’re running a single Redis instance with a handful of queues, embedded bull-board is probably still the right call. But the moment your infrastructure grows beyond that, the fragmentation starts costing you time every single day.</p><h3>Final Thoughts</h3><p>We didn’t set out to replace bull-board, we set out to fix the fragmentation that came with growing beyond a single Redis instance. bullmq-control-dashboard solved exactly that, without requiring us to change anything in our application code.</p><p>The auto-discovery alone was worth the switch. Not having to redeploy a service every time we add a new queue type is a small thing that compounds into a lot of saved time.</p><p>If you’re in a similar situation, give it a try: <a href="https://github.com/imatefx/bullmq-control-dashboard">github.com/imatefx/bullmq-control-dashboard</a></p><p>And if you’ve found other ways to handle multi-Redis queue observability, I’d genuinely like to hear about it in the comments.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=1dca550bf75e" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[How Kiro Supercharges My Productivity as a Systems Engineer]]></title>
            <link>https://faun.pub/how-kiro-supercharges-my-productivity-as-a-systems-engineer-9166b12903ab?source=rss-b6da988055dc------2</link>
            <guid isPermaLink="false">https://medium.com/p/9166b12903ab</guid>
            <category><![CDATA[amazon-web-services]]></category>
            <category><![CDATA[ide]]></category>
            <category><![CDATA[mcp-server]]></category>
            <category><![CDATA[agentic-ai]]></category>
            <category><![CDATA[ai]]></category>
            <dc:creator><![CDATA[Anil Augustine Chalissery]]></dc:creator>
            <pubDate>Wed, 03 Jun 2026 12:59:43 GMT</pubDate>
            <atom:updated>2026-06-03T12:59:43.932Z</atom:updated>
            <content:encoded><![CDATA[<p><strong>From Terraform modules to incident response scripts, how an agentic IDE changed my daily workflow</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Tk38ywuC7TipXf2kXMI5xA.jpeg" /></figure><h3>Introduction</h3><p>As a systems engineer, my day isn’t about building web apps. It’s shell scripts, Terraform modules, CloudFormation stacks, RDS configurations, ECS deployments, and the occasional “why is this subnet not routing” rabbit hole.</p><p>When AWS launched <a href="https://kiro.dev/">Kiro</a>, a spec-driven, agentic IDE, I was skeptical. Another AI code tool? But after using it for real infrastructure work, I realised it’s built differently. Kiro doesn’t just autocomplete code. It understands intent, follows specifications, and automates the repetitive parts of my workflow while keeping me in control.</p><p>Here’s how Kiro fits into my daily life as a systems engineer.</p><h3>1. Steering Files: My Team’s Standards, Always Applied</h3><p>Every team has conventions like naming patterns for resources, tagging policies, and preferred Terraform module structures. Before Kiro, these lived in a Confluence page nobody read.</p><p>With Kiro’s <strong>steering files</strong> (.kiro/steering/*.md), I encode those standards directly into the IDE. When Kiro generates or modifies infrastructure code, it follows these rules automatically.</p><p><strong>Example:</strong> I have a steering file that specifies:</p><ul><li>All Terraform resources must include Environment, Team, and CostCenter tags</li><li>S3 bucket names follow the pattern {company}-{env}-{service}-{purpose}</li><li>Security groups must never allow 0.0.0.0/0 on port 22</li></ul><p>Now, when I ask Kiro to scaffold a new module, it applies these constraints without me repeating them every time. This is especially powerful in a team setting — new engineers get guardrails from day one.</p><p><strong>How to set it up:</strong></p><p>Create a file at .kiro/steering/infra-standards.md in your workspace:</p><pre># Infrastructure Standards<br>## Tagging Policy<br>Every Terraform resource must include the following tags:<br>- Environment: dev | staging | prod<br>- Team: the owning team name<br>- CostCenter: the billing cost center code<br>## Naming Conventions<br>- S3 buckets: {company}-{env}-{service}-{purpose}<br>- IAM roles: {service}-{env}-{action}-role<br>- Security groups: {service}-{env}-sg<br>## Security Rules<br>- Never allow 0.0.0.0/0 ingress on port 22<br>- All RDS instances must have encryption at rest enabled<br>- ECS tasks must use awsvpc networking mode</pre><p>Kiro reads this on every interaction and applies it consistently.</p><h3>2. Specs: Turning Infrastructure Requirements into Executable Plans</h3><p>The spec workflow is where Kiro really differentiates itself from “vibe coding” tools. Instead of prompting and hoping, you define:</p><ol><li><strong>Requirements: </strong>what the infrastructure needs to do (in plain English or EARS notation)</li><li><strong>Design: </strong>how it will be structured (modules, dependencies, networking)</li><li><strong>Tasks: </strong>discrete implementation steps the agent works through</li></ol><h3>Real Scenario: ECR Cross-Region Replication</h3><p>I needed to set up ECR cross-region replication with a toggle mechanism for DR failover. Instead of writing the script from scratch, I created a spec:</p><p><strong>Requirement:</strong></p><blockquote><em>“ECR images in us-east-1 must replicate to us-west-2. Replication must be toggleable without redeploying infrastructure.”</em></blockquote><p><strong>Design:</strong> Kiro proposed using ECR replication configuration with a Terraform variable flag and a wrapper shell script.</p><p><strong>Tasks:</strong> It broke this into 4 steps:</p><ol><li>Terraform module for ECR replication configuration</li><li>Variable definitions with toggle flag</li><li>Shell wrapper script for operational use</li><li>Documentation and usage examples</li></ol><p>Kiro then executed each task, and I reviewed the output at each checkpoint. The result was production-ready in 20 minutes instead of the 2 hours it would have taken me manually.</p><h3>Why This Matters for Infrastructure</h3><p>Infrastructure code has real consequences. A misconfigured security group can expose a database. A wrong CIDR block can take down connectivity. The spec-driven approach forces you to think through requirements and design <em>before</em> any code is written, which is exactly what infrastructure changes should have anyway.</p><h3>3. Hooks: Automating the Boring Stuff</h3><p>Hooks let you trigger actions automatically based on IDE events. For infrastructure work, this is gold.</p><p><strong>My setup:</strong></p><p>Trigger Action Save a .tf file Run terraform fmt and terraform validate Save a .sh file Run shellcheck against it After completing a spec task Run tflint on the module Create a new .tf file Remind me to add required tags</p><p><strong>Example hook configuration</strong> (.kiro/hooks/terraform-validate.json):</p><pre>{<br>  &quot;name&quot;: &quot;Terraform Validate on Save&quot;,<br>  &quot;version&quot;: &quot;1.0.0&quot;,<br>  &quot;when&quot;: {<br>    &quot;type&quot;: &quot;fileEdited&quot;,<br>    &quot;patterns&quot;: [&quot;*.tf&quot;]<br>  },<br>  &quot;then&quot;: {<br>    &quot;type&quot;: &quot;runCommand&quot;,<br>    &quot;command&quot;: &quot;terraform fmt -check &amp;&amp; terraform validate&quot;<br>  }<br>}</pre><p>This means I never push malformed Terraform or a script with a missing quote. The feedback loop is instant, no context switching to a terminal, no forgetting to run the linter before committing.</p><h3>4. MCP Servers: Connecting AWS Context Directly</h3><p>Kiro supports <strong>Model Context Protocol (MCP)</strong> servers, which let the agent access external tools and data sources. For AWS work, this means Kiro can:</p><ul><li>Query AWS documentation for the latest best practices</li><li>Reference your actual infrastructure state</li><li>Pull in CloudFormation or Terraform registry docs on demand</li></ul><p><strong>My configuration</strong> (.kiro/settings/mcp.json):</p><pre>{<br>  &quot;mcpServers&quot;: {<br>    &quot;aws-docs&quot;: {<br>      &quot;command&quot;: &quot;uvx&quot;,<br>      &quot;args&quot;: [&quot;awslabs.aws-documentation-mcp-server@latest&quot;],<br>      &quot;env&quot;: {<br>        &quot;FASTMCP_LOG_LEVEL&quot;: &quot;ERROR&quot;<br>      },<br>      &quot;disabled&quot;: false,<br>      &quot;autoApprove&quot;: []<br>    }<br>  }<br>}</pre><p>Now, when I’m writing a Lake Formation permissions policy or configuring an RDS parameter group, Kiro has the latest API reference without me alt-tabbing to documentation. It’s like having the AWS docs embedded in your thought process.</p><h3>5. Day-to-Day Wins: Real Examples</h3><p>Here are concrete tasks where Kiro saved me significant time:</p><h3>Checking RDS AZ Placement Across Clusters</h3><p>I needed to verify which RDS instances were in us-east-1d (a problematic AZ for our account). I described the requirement to Kiro, and it generated the AWS CLI loop, parsed the JSON output, and formatted results — including edge cases I&#39;d have missed like multi-AZ replicas reporting their primary AZ only.</p><h3>Cognito User Pool Audit</h3><p>Needed to list all Cognito pools and their identity providers across multiple accounts. Kiro generated the script with proper pagination handling (something I always forget with AWS CLI) and added CSV output formatting for easy sharing with the security team.</p><h3>ECS Force-New-Deployment Automation</h3><p>When we need to rotate tasks without code changes (secret rotation, config updates), I used Kiro to build a wrapper that:</p><ul><li>Targets specific services by pattern matching</li><li>Waits for deployment stability</li><li>Reports rollback if the new deployment fails health checks</li><li>Sends a Slack notification on completion</li></ul><h3>Terraform Module Publishing</h3><p>Kiro walked me through the module registry publishing workflow, generated the proper directory structure, wrote the README with input/output documentation, and created the CI pipeline config and all following the conventions in my steering file.</p><h3>6. Kiro vs. Traditional AI Coding Tools</h3><p>Aspect Traditional AI Tools Kiro Context Single file or conversation Entire workspace + steering rules Workflow Prompt → hope for the best Spec → Design → Tasks → Execute Automation Manual copy-paste Hooks trigger actions automatically Control Accept/reject suggestions Autopilot or supervised mode with checkpoints AWS Integration Generic knowledge MCP servers for AWS-native context Team Standards Repeated in every prompt Encoded once in steering files</p><p>For systems engineers, the structured approach matters. We don’t have the luxury of “move fast and break things” when a wrong configuration can take down production.</p><h3>7. Tips for Getting Started as a Systems Engineer</h3><ol><li><strong>Start with steering files.</strong> Encode your team’s Terraform conventions, naming patterns, and security rules. This gives you immediate value on every interaction.</li><li><strong>Use specs for anything non-trivial.</strong> If a task involves more than one file or has dependencies, create a spec. The upfront thinking pays off in fewer iterations.</li><li><strong>Set up hooks for validation.</strong> terraform validate, shellcheck, tflint — automate the checks you always forget to run manually.</li><li><strong>Add the AWS docs MCP server.</strong> It’s a one-time setup that makes every AWS-related conversation more accurate.</li><li><strong>Use supervised mode for production changes.</strong> Kiro’s supervised mode lets you review each change as individual hunks before applying. Use this for anything touching prod infrastructure.</li></ol><h3>Conclusion</h3><p>Kiro isn’t replacing my expertise; it’s amplifying it. The spec-driven approach forces me to think clearly about requirements before writing code. The steering files encode team knowledge that would otherwise live in someone’s head. The hooks eliminate manual validation steps. And the agentic execution handles the tedious parts while I focus on architecture decisions.</p><p>If you’re a systems engineer spending hours on repetitive Terraform modules, shell scripts, or AWS configurations, give Kiro a try. It’s not about writing code faster; it’s about building infrastructure with more rigour and less friction.</p><p>The combination of structured specifications, persistent context, and automated guardrails is exactly what infrastructure-as-code needs from an AI tool. Not more autocomplete, more engineering discipline.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/700/0*z7i6gKASFZFILQay.png" /></figure><h4>👋 If you find this helpful, please click the clap 👏 button below a few times to show your support for the author 👇</h4><h4>🚀<a href="http://from.faun.to/r/8zxxd">Join FAUN.dev() &amp; get similar stories in your inbox each week for free!</a></h4><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=9166b12903ab" width="1" height="1" alt=""><hr><p><a href="https://faun.pub/how-kiro-supercharges-my-productivity-as-a-systems-engineer-9166b12903ab">How Kiro Supercharges My Productivity as a Systems Engineer</a> was originally published in <a href="https://faun.pub/">FAUN.dev() 🐾</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[10 Misconfigured AWS Resources That Could Get You Breached]]></title>
            <link>https://aws.plainenglish.io/10-misconfigured-aws-resources-that-could-get-you-breached-70f40a18a88a?source=rss-b6da988055dc------2</link>
            <guid isPermaLink="false">https://medium.com/p/70f40a18a88a</guid>
            <category><![CDATA[the-secops-group]]></category>
            <category><![CDATA[aws]]></category>
            <category><![CDATA[aws-iam]]></category>
            <category><![CDATA[cloud-security]]></category>
            <dc:creator><![CDATA[Anil Augustine Chalissery]]></dc:creator>
            <pubDate>Fri, 08 Aug 2025 01:10:07 GMT</pubDate>
            <atom:updated>2025-08-08T01:10:07.090Z</atom:updated>
            <content:encoded><![CDATA[<h4>The cloud gives you power. But with great power comes great misconfiguration</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*rnYqgQ-QKkRfRNQl" /><figcaption>Photo by <a href="https://unsplash.com/@flyd2069?utm_source=medium&amp;utm_medium=referral">FlyD</a> on <a href="https://unsplash.com/?utm_source=medium&amp;utm_medium=referral">Unsplash</a></figcaption></figure><p>Hey there, fellow cloud wranglers! Whether you’re a DevOps newbie spinning up your first EC2 instance or an intermediate AWS user confidently deploying production stacks with Terraform, there’s one thing we all have in common: <strong>we’re all just one misconfiguration away from a bad day.</strong></p><p>After spending years in the AWS ecosystem, helping companies build, scale, and yes, sometimes accidentally break things, I’ve seen the same scary patterns pop up again and again.</p><p>So here’s a rundown of the <strong>10 most common AWS misconfigurations that can get you breached, leaked, or just generally in trouble with your CISO.</strong> Buckle up.</p><h3>1. S3 Buckets with Public Access: The Classic</h3><p>S3: Simple Storage Service. Too simple sometimes.</p><p><strong>What goes wrong:</strong> You spin up a bucket for some public assets. Then someone decides, “Let’s just turn off that pesky Block Public Access setting.” Suddenly, <em>everything</em> in that bucket — logs, backups, even customer data — is out there for the world to browse like it’s a public Google Drive folder.</p><p><strong>Fix it:</strong></p><ul><li>Enable BlockPublicAccess (globally and per-bucket)</li><li>Use bucket policies <em>only when needed</em></li><li>Turn on access logging (so you know who peeked)</li></ul><h3>2. IAM Policies That Say ‘Yes’ to Everything</h3><p>You know what’s easier than writing a scoped IAM policy? Just using &quot;Action&quot;: &quot;*&quot;, &quot;Resource&quot;: &quot;*&quot; and calling it a day.<br> You know what else it does? Opens the door to <strong>absolute mayhem</strong>.</p><p><strong>Fix it:</strong></p><ul><li>Follow <strong>least privilege</strong></li><li>Use IAM Access Analyzer</li><li>Don’t give your Lambda function AdministratorAccess just because it’s misbehaving</li></ul><blockquote><em>IAM isn’t just a permission system. It’s your security perimeter. Don’t let it be a jelly fence.</em></blockquote><h3>3. Open Security Groups: All Are Welcome (Even the Hackers)</h3><p>“Oh, I’ll just open port 22 to 0.0.0.0 for now…”<br>- Famous Last Words</p><p><strong>What goes wrong:</strong> Your EC2 instance becomes a honeypot for every bot and brute-force script on the internet.</p><p><strong>Fix it:</strong></p><ul><li>Restrict security groups to specific IPs</li><li>Use Systems Manager Session Manager instead of SSH</li><li>Disable unused ports</li></ul><h3>4. Secrets in Plain Sight</h3><p>AWS keys in your Git repo? RDS passwords in your user data script?<br> You’re not alone. But you’re also not secure.</p><p><strong>Fix it:</strong></p><ul><li>Store secrets in <strong>AWS Secrets Manager</strong> or <strong>SSM Parameter Store</strong></li><li>Set up automatic Git secrets scanning</li><li>Use environment variables with encrypted secrets, not plaintext</li></ul><h3>5. Publicly Accessible RDS or ElastiCache Instances</h3><p>“Oh, I can connect to my DB directly from home now!”<br> Cool. So can everyone else.</p><p><strong>Fix it:</strong></p><ul><li>Keep databases in <strong>private subnets</strong></li><li>Use Bastion hosts or VPNs for access</li><li>Secure with security groups and VPC peering rules</li></ul><h3>6. IAM Roles with Loose Trust Policies</h3><p>Cross-account access is powerful. It’s also dangerous when misconfigured.</p><p><strong>What goes wrong:</strong> Trusting the wrong AWS account or principal with overly broad conditions.<br> Congratulations, you just gave access to someone you didn’t mean to.</p><p><strong>Fix it:</strong></p><ul><li>Use conditions like aws:PrincipalOrgID</li><li>Avoid &quot;Principal&quot;: &quot;*&quot;</li><li>Regularly audit assumed roles</li></ul><h3>7. Lambda Functions with Too Much Power</h3><p>“Eh, just give it * permissions. What’s the worst that can happen?”</p><p>Spoiler: a <strong>lot</strong>.</p><p><strong>What goes wrong:</strong> If that function is vulnerable (or worse, public-facing), attackers now have access to every AWS service your role allows.</p><p><strong>Fix it:</strong></p><ul><li>Use least-privilege roles for Lambda</li><li>Split functions by responsibility</li><li>Monitor with CloudTrail</li></ul><h3>8. Long-Lived IAM Users with Static Access Keys</h3><p>IAM users are like leftovers — you forget about them until they start to smell.<br> Especially when they have 3-year-old access keys still active.</p><p><strong>Fix it:</strong></p><ul><li>Use <strong>IAM roles</strong> or <strong>SSO</strong> whenever possible</li><li>Rotate keys regularly</li><li>Enable <strong>MFA</strong></li><li>Audit unused users</li></ul><h3>9. CloudTrail is Off or Logging to the Wrong Place</h3><p>No logs? No clue.<br> CloudTrail is your <strong>black box recorder</strong>. Don’t fly without it.</p><p><strong>What goes wrong:</strong> You forget to enable it in all regions, or log files go to an S3 bucket that anyone can delete.</p><p><strong>Fix it:</strong></p><ul><li>Enable CloudTrail across <strong>all regions</strong></li><li>Send logs to a secure, <strong>immutable S3 bucket</strong></li><li>Consider duplicating logs to a <strong>security account</strong></li></ul><blockquote><em>🧯 CloudTrail is like CCTV. You don’t need it… until you REALLY need it.</em></blockquote><h3>10. No MFA on the Root Account</h3><p>AWS gives you the keys to the kingdom — and then you leave the front door unlocked? Bold move.</p><p><strong>What goes wrong:</strong> The root account — the <em>most powerful identity in your AWS account</em> — doesn’t have Multi-Factor Authentication (MFA) enabled.<br> If someone gets that password, they own your cloud. Not just one service. All of it. Forever.</p><p><strong>Why it’s risky:</strong></p><ul><li>The root account can bypass SCPs, delete users, nuke infra</li><li>Attackers who compromise it can’t be stopped easily</li><li>Even AWS says: <em>“Don’t use root unless absolutely necessary.”</em></li></ul><p><strong>Fix it:</strong></p><ul><li>Log in right now and enable <strong>MFA on the root user</strong> (hardware key or authenticator app)</li><li>Store the backup codes securely</li><li>Use the root account only for critical one-time tasks (like setting up billing or support plans)</li></ul><h3>Wrapping Up: Don’t Be the Next Headline</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*bEGZDJkIWe6-f1yf" /><figcaption>Photo by <a href="https://unsplash.com/@ev?utm_source=medium&amp;utm_medium=referral">ev</a> on <a href="https://unsplash.com/?utm_source=medium&amp;utm_medium=referral">Unsplash</a></figcaption></figure><p>The truth is, <strong>AWS gives you all the tools to be secure</strong>. But like a box of IKEA furniture, they don’t always come pre-assembled, and using them wrong can cost you.</p><p>If you take anything away from this post:</p><ul><li>Don’t ignore IAM best practices.</li><li>S3 isn’t private by default.</li><li>And logs are like receipts, keep them.</li></ul><h4>Additional AWS Security Best Practices to Evaluate</h4><ul><li>Rotate IAM access keys regularly</li><li>Enable AWS Config rules for continuous compliance monitoring</li><li>Use Amazon Inspector for automated vulnerability scanning</li><li>Implement centralized logging with CloudTrail, VPC Flow Logs, and CloudWatch</li><li>Enforce S3 bucket security with encryption, policies, and SCPs</li><li>Run periodic AWS Well-Architected Reviews focusing on security</li></ul><p>Happy building (securely). And may your security groups always be tight, your keys always rotated, and your CloudTrail always enabled.</p><p>📣 <em>If you found this helpful, drop a comment or share it with someone who needs a gentle IAM slap. And if you’ve got your own AWS horror story, I’m all ears</em></p><h3>A message from our Founder</h3><p><strong>Hey, </strong><a href="https://linkedin.com/in/sunilsandhu"><strong>Sunil</strong></a><strong> here.</strong> I wanted to take a moment to thank you for reading until the end and for being a part of this community.</p><p>Did you know that our team run these publications as a volunteer effort to over 200k supporters? <strong>We do not get paid by Medium</strong>!</p><p>If you want to show some love, please take a moment to <strong>follow me on </strong><a href="https://linkedin.com/in/sunilsandhu"><strong>LinkedIn</strong></a><strong>, </strong><a href="https://tiktok.com/@messyfounder"><strong>TikTok</strong></a><strong> and </strong><a href="https://instagram.com/sunilsandhu"><strong>Instagram</strong></a>. And before you go, don’t forget to <strong>clap</strong> and <strong>follow</strong> the writer️!</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=70f40a18a88a" width="1" height="1" alt=""><hr><p><a href="https://aws.plainenglish.io/10-misconfigured-aws-resources-that-could-get-you-breached-70f40a18a88a">10 Misconfigured AWS Resources That Could Get You Breached</a> was originally published in <a href="https://aws.plainenglish.io/">AWS in Plain English</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Multi-Region DR for ECS Fargate: A Practical Guide]]></title>
            <link>https://aws.plainenglish.io/multi-region-dr-for-ecs-fargate-a-practical-guide-0b59a9893bdb?source=rss-b6da988055dc------2</link>
            <guid isPermaLink="false">https://medium.com/p/0b59a9893bdb</guid>
            <category><![CDATA[disaster-recovery]]></category>
            <category><![CDATA[terraform]]></category>
            <category><![CDATA[cloud]]></category>
            <category><![CDATA[aws]]></category>
            <category><![CDATA[ecs-fargate]]></category>
            <dc:creator><![CDATA[Anil Augustine Chalissery]]></dc:creator>
            <pubDate>Mon, 12 May 2025 04:16:37 GMT</pubDate>
            <atom:updated>2025-05-12T04:16:37.209Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*DCJUcSy28uIZ35U6" /><figcaption>Photo by <a href="https://unsplash.com/@punttim?utm_source=medium&amp;utm_medium=referral">Tim Gouw</a> on <a href="https://unsplash.com/?utm_source=medium&amp;utm_medium=referral">Unsplash</a></figcaption></figure><p>In today’s cloud-first world, building a system that simply “works” isn’t enough. Teams are increasingly expected to design infrastructure that remains <strong>resilient and available even in the face of regional failures</strong>. If you’re running services on AWS, you’ve probably thought about high availability — but have you thought seriously about <strong>disaster recovery (DR)</strong>?</p><p>Over the past few months, I’ve had the opportunity to architect a <strong>multi-region disaster recovery strategy</strong> for a customer-facing application running on <strong>Amazon ECS Fargate</strong>. The system also involved components like <strong>Application Load Balancers</strong>, <strong>Amazon SQS/SNS</strong>, <strong>DynamoDB</strong>, <strong>Redis</strong>, and <strong>Aurora PostgreSQL Serverless v2</strong>. Through this journey, I encountered both the power of AWS services and the subtle complexities involved in making them truly resilient across regions.</p><p>In this blog, I’ll walk you through the <strong>real-world decisions</strong> we made, the <strong>design principles</strong> we followed, and the <strong>AWS services</strong> that enabled us to build a DR-ready architecture.</p><h3>🧩 The Problem We Set Out to Solve</h3><p>The business requirement was straightforward on paper:</p><blockquote><em>“Ensure that the application can </em><strong><em>withstand a regional outage</em></strong><em> with </em><strong><em>minimal downtime</em></strong><em> and </em><strong><em>no data loss</em></strong><em> for critical services.”</em></blockquote><p>As simple as that sounds, it has serious implications. We had to look beyond just <strong>multi-AZ redundancy</strong> and design for <strong>multi-region failover</strong>.</p><p>We evaluated the four disaster recovery strategies outlined in AWS’s <a href="https://aws.amazon.com/blogs/architecture/disaster-recovery-dr-architecture-on-aws-part-i-strategies-for-recovery-in-the-cloud/">Disaster Recovery Architecture blog</a>:</p><ol><li><strong>Backup and Restore</strong></li><li><strong>Pilot Light</strong></li><li><strong>Warm Standby</strong></li><li><strong>Multi-Site Active-Active</strong></li></ol><p>Given our constraints around cost, complexity, and time-to-recovery, we landed on a <strong>Warm Standby</strong> model — where we keep a <strong>scaled-down version of the production environment running in a secondary region</strong>, ready to scale up if disaster strikes.</p><h3>🏗️ Our Architecture at a Glance</h3><p>We designed our architecture with two AWS regions:</p><h3>🔹 Primary Region (Region A)</h3><p>This is where the application runs at full scale:</p><ul><li><strong>Amazon ECS Fargate</strong> hosts REST APIs and background workers.</li><li><strong>Application Load Balancer</strong> routes incoming traffic to ECS tasks.</li><li><strong>Amazon SQS (FIFO)</strong> manages background job queues.</li><li><strong>Amazon SNS</strong> is used for broadcasting events to subscribers.</li><li><strong>DynamoDB</strong> holds structured, business-critical data.</li><li><strong>ElastiCache Redis</strong> is used for caching and state.</li><li><strong>Amazon Aurora PostgreSQL Serverless v2</strong> for transactional, relational data.</li></ul><h3>🔹 Secondary Region (Region B)</h3><p>Here, we maintain a live but minimal setup:</p><ul><li>All infrastructure is <strong>pre-provisioned using Terraform</strong>, but services are scaled down.</li><li>DynamoDB Global Tables replicate data automatically.</li><li>Redis and Aurora are provisioned but scaled to minimal capacity.</li><li>Route 53 is configured to detect outages and redirect DNS traffic.</li></ul><h3>⚙️ Service-by-Service Strategy</h3><h3>✅ ECS Fargate + ALB</h3><p>ECS Fargate simplifies container management and runs tasks across multiple AZs by default, We enhanced it by using 3-AZ minimum. However, it’s important to note that <strong>Fargate is regional</strong>. If the entire region goes down, so does the compute layer.</p><p>To tackle this:</p><ul><li>We replicated ECS services in the secondary region.</li><li>Used <strong>environment-specific configuration files</strong> and <strong>modular Terraform patterns</strong> to ensure consistency.</li><li>ECS tasks are kept at <strong>minimal count (1 or 0)</strong> in Region B, ready to scale up based on demand.</li></ul><h3>✅ Amazon Aurora PostgreSQL Serverless v2 with Global Database</h3><p>Aurora PostgreSQL Serverless v2 gave us the elasticity we needed for transactional workloads without the overhead of capacity planning, but what really unlocked cross-region resilience was <strong>Aurora Global Database</strong>.</p><p><strong>Aurora Global Database</strong> is designed specifically for <strong>low-latency global reads</strong> and <strong>fast cross-region disaster recovery</strong>. It uses <strong>storage-based replication,</strong> not snapshotting, to asynchronously replicate data from the primary writer in Region A to a read-only cluster in Region B, typically with latency under 1 second.</p><p>Our setup:</p><ul><li>Region A hosts the <strong>writer cluster</strong> (multi-AZ enabled), which handles all application writes and reads.</li><li>Region B maintains a <strong>read-only replica</strong>, which can be <strong>promoted to a standalone writer</strong> in the event of a failover.</li></ul><p>Failover Process:</p><ul><li>When Region A becomes unavailable, the read replica in Region B is <strong>manually or automatically promoted</strong> to become the new writer.</li><li>Route 53 is updated to point application traffic to the newly promoted endpoint.</li><li>ECS and Redis in Region B are scaled up, and services resume with minimal disruption.</li></ul><p>Why we chose this:</p><ul><li><strong>No need to restore snapshots</strong> or handle backup replication manually.</li><li>Promoting a global database takes <strong>less than a minute</strong>, making it ideal for our RTO goals.</li><li>Seamless integration with our Fargate workloads and IAM-based access control.</li></ul><p>Caveats to keep in mind:</p><ul><li>Aurora Global Database currently supports <strong>one primary and up to five read-only secondary regions</strong>.</li><li><strong>Writes are only allowed in one region</strong>, so active-active scenarios require special consideration or partitioned workloads.</li></ul><p>By using Aurora Global Database with Serverless v2, we achieved <strong>enterprise-grade availability</strong> with <strong>cost-efficient scaling</strong>, perfectly aligned with our warm standby DR model.</p><h3>✅ Amazon SQS and SNS</h3><p>These are <strong>region-scoped</strong> services. Even though AWS manages multi-AZ resilience, there is no native cross-region SQS/SNS.</p><p>Our approach:</p><ul><li>Identical queues and topics are deployed in Region B.</li><li>Applications are designed to be <strong>idempotent</strong>, so they can reprocess messages if needed.</li><li>For failover, a new set of producers/consumers are activated in the secondary region.</li></ul><h3>✅ DynamoDB Global Tables</h3><p>This was one of the most valuable pieces of the architecture. Global Tables provide <strong>multi-master, cross-region replication</strong> out of the box.</p><p>What worked well:</p><ul><li>Seamless replication with minimal latency.</li><li>Conflict resolution handled by AWS.</li><li>We configured IAM roles carefully to restrict writes during normal operations from Region B (to avoid unintended usage).</li></ul><h3>✅ Redis (ElastiCache)</h3><p>Redis doesn’t offer native cross-region replication. We treated this as a <strong>stateful, volatile component</strong>, which is rehydrated during failover.</p><p>Strategy:</p><ul><li>In Region B, Redis is <strong>provisioned but in small scale</strong>.</li><li>On failover, Redis is brought online, and the app repopulates necessary cache from DynamoDB and Aurora.</li></ul><h3>🔁 Failover Workflow</h3><p>Here’s what happens during a regional outage:</p><ol><li><strong>Route 53 health checks</strong> detect ALB unavailability in Region A.</li><li>DNS traffic is redirected to Region B’s ALB.</li><li>ECS services and Redis are scaled up in Region B via automation.</li><li>Aurora PostgreSQL is promoted from Region B’s read-replica.</li><li>Applications begin processing using Region B’s infrastructure.</li><li>DynamoDB Global Tables ensure business-critical data remains consistent.</li></ol><h3>🧠 Lessons from the Trenches</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*t2gV9cNsJpX5UZYE" /><figcaption>Photo by <a href="https://unsplash.com/@joszczepanska?utm_source=medium&amp;utm_medium=referral">Jo Szczepanska</a> on <a href="https://unsplash.com/?utm_source=medium&amp;utm_medium=referral">Unsplash</a></figcaption></figure><h3>🟡 Automation is Everything</h3><p>Manual DR steps are risky under pressure. We relied heavily on Terraform + Terragrunt for reproducibility and automation scripts for scaling and failover.</p><h3>🟡 Aurora Needs a Plan</h3><p>Aurora Serverless is a powerful piece, but you must plan:</p><ul><li><strong>Manual Promotion:</strong> For Aurora Serverless v2, promoting a read replica to become the primary instance is a manual operation initiated through the AWS Management Console or the AWS CLI.</li><li><strong>DNS Update (if applicable):</strong> If your application connects to Aurora via a custom DNS record, update the DNS record to point to the new cluster endpoint of the promoted instance in Region B. This might involve adjusting TTL (Time-To-Live) settings beforehand to expedite the switch.</li></ul><h3>🟡 Failover Isn’t Just for Show</h3><p>It’s easy to deploy infra in a second region and call it DR-ready. But until you’ve simulated a region outage, you won’t know what you’re missing.<br> We scheduled <strong>quarterly DR drills</strong> — and every time, we found something to improve.</p><h3>📌 Final Thoughts</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*f1cebQWsiM5asJbS" /><figcaption>Photo by <a href="https://unsplash.com/@uns__nstudio?utm_source=medium&amp;utm_medium=referral">Unseen Studio</a> on <a href="https://unsplash.com/?utm_source=medium&amp;utm_medium=referral">Unsplash</a></figcaption></figure><p>A disaster recovery plan is not a checkbox, it’s a living part of your cloud architecture. Our use of a <strong>warm standby model</strong>, <strong>modular IaC</strong>, and <strong>strategic service replication</strong> helped us build an infrastructure that’s resilient and cost-effective.</p><p>If you’re working with ECS Fargate and services like Aurora, Redis, and SQS/SNS, you don’t need to reinvent the wheel. AWS gives you the building blocks — you just need to <strong>orchestrate them smartly and test them thoroughly</strong>.</p><h3>🙋‍♂️ Let’s Connect</h3><p>If you’re working on a similar DR strategy, or want to dive deeper into Terraform patterns, feel free to drop a comment. I’d love to hear your thoughts or trade notes.</p><h3>Thank you for being a part of the community</h3><p><em>Before you go:</em></p><ul><li>Be sure to <strong>clap</strong> and <strong>follow</strong> the writer ️👏<strong>️️</strong></li><li>Follow us: <a href="https://x.com/inPlainEngHQ"><strong>X</strong></a> | <a href="https://www.linkedin.com/company/inplainenglish/"><strong>LinkedIn</strong></a> | <a href="https://www.youtube.com/@InPlainEnglish"><strong>YouTube</strong></a> | <a href="https://newsletter.plainenglish.io/"><strong>Newsletter</strong></a> | <a href="https://open.spotify.com/show/7qxylRWKhvZwMz2WuEoua0"><strong>Podcast</strong></a> | <a href="https://differ.blog/inplainenglish"><strong>Differ</strong></a> | <a href="https://twitch.tv/inplainenglish"><strong>Twitch</strong></a></li><li><a href="https://differ.blog/"><strong>Start your own free AI-powered blog on Differ</strong></a> 🚀</li><li><a href="https://discord.gg/in-plain-english-709094664682340443"><strong>Join our content creators community on Discord</strong></a> 🧑🏻‍💻</li><li>For more content, visit <a href="https://plainenglish.io/"><strong>plainenglish.io</strong></a> + <a href="https://stackademic.com/"><strong>stackademic.com</strong></a></li></ul><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=0b59a9893bdb" width="1" height="1" alt=""><hr><p><a href="https://aws.plainenglish.io/multi-region-dr-for-ecs-fargate-a-practical-guide-0b59a9893bdb">Multi-Region DR for ECS Fargate: A Practical Guide</a> was originally published in <a href="https://aws.plainenglish.io/">AWS in Plain English</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Terraform Modules: Should You Use the Terraform Registry or Build Internally?]]></title>
            <link>https://faun.pub/terraform-modules-should-you-use-the-terraform-registry-or-build-internally-8c68bd9c6b9a?source=rss-b6da988055dc------2</link>
            <guid isPermaLink="false">https://medium.com/p/8c68bd9c6b9a</guid>
            <category><![CDATA[iac]]></category>
            <category><![CDATA[terraform]]></category>
            <dc:creator><![CDATA[Anil Augustine Chalissery]]></dc:creator>
            <pubDate>Wed, 14 Aug 2024 18:23:36 GMT</pubDate>
            <atom:updated>2024-08-14T18:23:36.677Z</atom:updated>
            <content:encoded><![CDATA[<p>In the world of infrastructure as code (IaC), Terraform has become a go-to tool for managing cloud infrastructure. One of Terraform’s strengths is its modularity, allowing you to reuse configurations across projects. But when it comes to using modules, you face a key decision: should you rely on the Terraform Registry, or should you create and maintain your own internal modules?</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*uxVnAAsWMuya-vPL" /><figcaption>Photo by <a href="https://unsplash.com/@ffstop?utm_source=medium&amp;utm_medium=referral">Fotis Fotopoulos</a> on <a href="https://unsplash.com/?utm_source=medium&amp;utm_medium=referral">Unsplash</a></figcaption></figure><p>Let’s explore both options and identify best practices for making the right choice for your organization.</p><h3>The Terraform Registry: Convenience at Your Fingertips</h3><p>The Terraform Registry is a vast repository of pre-built modules. These modules are designed to be plug-and-play, allowing you to quickly set up everything from simple AWS S3 buckets to complex multi-region Kubernetes clusters.</p><h4>Advantages of Using the Terraform Registry:</h4><ol><li><strong>Time-Saving:</strong> The most obvious benefit of using registry modules is the time saved. Rather than writing code from scratch, you can deploy an industry-standard setup in a fraction of the time.</li><li><strong>Community-Driven Quality:</strong> Modules on the registry are often maintained by cloud providers, open-source communities, or DevOps professionals. These modules are usually well-tested, regularly updated, and adhere to best practices.</li><li><strong>Ease of Use:</strong> Terraform Registry modules come with detailed documentation, making them accessible even to those who may not be deeply familiar with the specific service or resource.</li></ol><p>However, while registry modules offer many conveniences, they’re not without potential drawbacks.</p><h4>Challenges of Using the Terraform Registry:</h4><ol><li><strong>Limited Customization:</strong> Pre-built modules are designed to be generic, which can mean they don’t perfectly align with your specific needs. You might end up with resources you don’t need or lacking features that are essential to your use case.</li><li><strong>Security Risks:</strong> Using external modules means you’re bringing third-party code into your environment. If not thoroughly vetted, these modules could introduce security vulnerabilities.</li><li><strong>Dependency on Maintainers:</strong> Updates and bug fixes are at the mercy of the module maintainers. If a critical change or a breaking update is released, it can disrupt your infrastructure unexpectedly.</li></ol><h3>Building Internal Modules: Tailored to Your Needs</h3><p>For organizations with unique requirements or a focus on security, creating and maintaining internal Terraform modules can be a more suitable approach. This allows you to fully control the code and ensure it aligns perfectly with your needs.</p><h4>Advantages of Creating Internal Modules:</h4><ol><li><strong>Full Customization:</strong> Internal modules can be tailored to fit your specific requirements, ensuring you get exactly what you need without any unnecessary components.</li><li><strong>Enhanced Security:</strong> By writing your own modules, you control the entire codebase, reducing the risk of introducing vulnerabilities from external sources.</li><li><strong>Consistency Across Projects:</strong> Internal modules can enforce consistent standards, naming conventions, and architecture across your organization, ensuring uniformity and reducing errors.</li></ol><p>But with this control comes responsibility.</p><h4>Challenges of Internal Modules:</h4><ol><li><strong>Maintenance Overhead:</strong> Unlike registry modules, you’re responsible for keeping your modules up to date, fixing bugs, and ensuring compatibility with new Terraform and provider releases.</li><li><strong>Initial Development Effort:</strong> Creating custom modules requires significant time and expertise, which could slow down your initial deployment.</li><li><strong>Knowledge Requirement:</strong> Your team needs to have a deep understanding of Terraform and the services you’re working with to create effective modules.</li></ol><h3>Best Practices: Finding the Right Balance</h3><p>So, should you use the Terraform Registry or build your modules internally? In many cases, the answer lies in a balanced approach.</p><ol><li><strong>Hybrid Strategy:</strong> Use registry modules for common, generic infrastructure components — like setting up a VPC or an S3 bucket — where your requirements are likely similar to others. For more unique or sensitive parts of your infrastructure, develop internal modules that can be fully customized.</li><li><strong>Customize Registry Modules:</strong> Even when using registry modules, consider forking them into your internal repositories. This allows you to make modifications as needed while still benefiting from the initial time savings of a pre-built module.</li><li><strong>Security First:</strong> Whether using external or internal modules, always review the code for security vulnerabilities and compliance with your organization’s standards. Establish a process for regular reviews and updates to avoid potential risks.</li></ol><h3>Conclusion</h3><p>Terraform’s modular approach is a powerful way to manage your cloud infrastructure, but the choice between using the Terraform Registry or building internal modules isn’t always straightforward. By understanding the benefits and challenges of each approach, and by adopting a balanced strategy, you can harness the full potential of Terraform while keeping your infrastructure secure, scalable, and tailored to your needs.</p><h3>References</h3><ul><li><a href="https://developer.hashicorp.com/terraform/registry/modules/use">https://developer.hashicorp.com/terraform/registry/modules/use</a></li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/700/0*O78X7f-EU5W4XTss.png" /></figure><h4>👋 If you find this helpful, please click the clap 👏 button below a few times to show your support for the author 👇</h4><h4>🚀<a href="http://from.faun.to/r/8zxxd">Join FAUN Developer Community &amp; Get Similar Stories in your Inbox Each Week</a></h4><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=8c68bd9c6b9a" width="1" height="1" alt=""><hr><p><a href="https://faun.pub/terraform-modules-should-you-use-the-terraform-registry-or-build-internally-8c68bd9c6b9a">Terraform Modules: Should You Use the Terraform Registry or Build Internally?</a> was originally published in <a href="https://faun.pub/">FAUN.dev() 🐾</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[How to Deploy Apache Superset on AWS ECS?]]></title>
            <link>https://aws.plainenglish.io/how-to-deploy-apache-superset-on-aws-ecs-08da76bedd32?source=rss-b6da988055dc------2</link>
            <guid isPermaLink="false">https://medium.com/p/08da76bedd32</guid>
            <category><![CDATA[open-source]]></category>
            <category><![CDATA[aws]]></category>
            <category><![CDATA[apache-superset]]></category>
            <category><![CDATA[aws-ecs]]></category>
            <category><![CDATA[data-visualization]]></category>
            <dc:creator><![CDATA[Anil Augustine Chalissery]]></dc:creator>
            <pubDate>Mon, 12 Aug 2024 12:59:22 GMT</pubDate>
            <atom:updated>2024-09-03T04:50:23.290Z</atom:updated>
            <content:encoded><![CDATA[<h4>A Step-by-Step Guide to Setting Up and Scaling Apache Superset with Amazon ECS and Athena</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*sZpWJq45jX82mlhfhqQFsA.png" /></figure><h3>Introduction</h3><p>Apache Superset has rapidly gained traction as a versatile and powerful tool for data exploration and visualization. As organizations increasingly rely on data-driven decision-making, the ability to create interactive, real-time dashboards becomes crucial. Apache Superset fills this gap with its rich feature set, offering users an intuitive platform for building complex visualizations. Deploying Superset on Amazon Web Services (AWS) using Elastic Container Service (ECS) further enhances its capabilities, providing scalability, security, and seamless integration with other AWS services. This guide will walk you through the process of deploying Superset on ECS, highlighting its key features and the advantages of leveraging AWS for your deployment.</p><p><strong>Key Features of Apache Superset:</strong></p><ol><li><strong>Interactive Dashboards:</strong> Superset enables users to create dashboards with a wide variety of visualizations, such as pie charts, line graphs, heatmaps, and geographical maps. The drag-and-drop interface makes it easy to build and customize dashboards, which can then be shared with others for collaborative data exploration.</li><li><strong>Advanced SQL Editor:</strong> Superset’s SQL editor allows users to write and execute SQL queries directly against their databases. It supports a range of SQL dialects, making it compatible with various database systems. The editor includes features like syntax highlighting, auto-completion, and query history, streamlining the data exploration process.</li><li><strong>Role-Based Access Control (RBAC):</strong> Security is a critical concern in any data-centric application. Superset addresses this with its robust RBAC system, which allows administrators to define roles and permissions at a granular level. This ensures that sensitive data is only accessible to authorized users, protecting your organization’s information.</li><li><strong>Extensibility with Plugins:</strong> Superset’s architecture is designed to be extensible. Users can develop and integrate custom plugins to add new functionalities, such as custom visualizations or data connectors. This flexibility allows organizations to tailor Superset to their specific needs.</li><li><strong>Optimized Performance and Scalability:</strong> Superset is built to handle large datasets and complex queries efficiently. It supports caching mechanisms to speed up data retrieval and can be deployed in a distributed environment for horizontal scaling, ensuring consistent performance even under heavy loads.</li></ol><h3>Architecture</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/751/1*rH2xDaYPE_VOAT8vBKVTug.png" /></figure><p>The provided diagram visually represents the architecture of deploying Superset on AWS ECS. Here’s a breakdown:</p><ul><li><strong>Public Subnet:</strong> Hosts the Internet Gateway and Application Load Balancer (ALB). The Internet Gateway provides internet access, while the ALB distributes incoming user requests to the Superset instances running in the private subnet.</li><li><strong>Private Subnet:</strong> Contains the ECS service running Superset. This subnet is isolated from the internet to enhance security, with outbound traffic routed through a NAT Gateway. The ECS tasks can connect to internal resources like the database and cache services in the data subnet.</li><li><strong>Data Subnet:</strong> Contains Amazon RDS (Aurora) for the Superset metadata database and ElastiCache for caching. These resources are critical for storing and retrieving data efficiently.</li><li><strong>AWS Services:</strong> Various AWS services are integrated into the architecture. Secrets Manager manages sensitive information like database credentials, CloudWatch monitors logs and metrics, and Athena allows querying data stored in S3.</li></ul><p>This architecture ensures that your Superset deployment is secure, scalable, and integrated with other AWS services, providing a robust solution for data visualization needs.</p><p><strong>Step-by-Step Guide to Deploying Superset on AWS ECS:</strong></p><p><strong>1. Containerizing Superset:</strong></p><ul><li><strong>Create a Docker Image:</strong> Begin by creating a Dockerfile that installs Superset and its dependencies. You can base this on a superset image and include steps to install other dependencies via pip, along with any necessary configuration files. Here is an example where we install the Athena driver. T<a href="https://superset.apache.org/docs/configuration/databases/">o check out database drivers supported by Apache Superset</a></li></ul><pre># Use the official Apache Superset image as the base<br># FROM apache/superset:latest<br>FROM apache/superset:3.1.3<br><br><br># Install dependencies for Athena<br>USER root<br>RUN apt-get update &amp;&amp; apt-get install -y sudo vim \<br>    &amp;&amp; pip install &quot;PyAthena[SQLAlchemy]&quot; \<br>    &amp;&amp; pip install &quot;pyathena[pandas]&quot; \<br>    &amp;&amp; pip install &quot;PyAthenaJDBC&quot;    \<br>    &amp;&amp; pip install prophet  <br><br># Add the superset_config.py file from the current directory<br># this helps us to overide default parameters and configurations<br>COPY superset_config.py /app/pythonpath/superset_config.py<br><br># Switch back to the superset user<br>USER superset<br><br># Expose the port that superset runs on<br>EXPOSE 8088<br><br></pre><p>Default superset_config.py -&gt; <a href="https://github.com/apache/superset/blob/master/docker/pythonpath_dev/superset_config.py">https://github.com/apache/superset/blob/master/docker/pythonpath_dev/superset_config.py</a></p><ul><li><strong>Push to Amazon ECR:</strong> After building the Docker image, push it to Amazon Elastic Container Registry (ECR). ECR is a fully managed Docker container registry that makes it easy to store, manage, and deploy container images. Once you create a ECR repo you can get the push commands for the console itself or follow doc -&gt; <a href="https://docs.aws.amazon.com/AmazonECR/latest/userguide/docker-push-ecr-image.html">https://docs.aws.amazon.com/AmazonECR/latest/userguide/docker-push-ecr-image.html</a></li></ul><p><strong>2. Setting Up an ECS Cluster:</strong></p><ul><li><strong>Create the ECS Cluster:</strong> Navigate to the ECS console and create a new cluster. Depending on your needs, you can choose between a cluster backed by EC2 instances or Fargate, which provides serverless compute for containers. <a href="https://docs.aws.amazon.com/AmazonECS/latest/developerguide/getting-started-fargate.html#get-started-fargate-cluster">To know more about creating ECS cluster and service</a>. Here we use Fargate cluster.</li><li><strong>Networking Setup:</strong> Choose a VPC for your cluster, and set up subnets. Typically, you’ll want to use private subnets for your ECS tasks to enhance security. Ensure that these subnets have access to a NAT Gateway if your containers need to pull updates from the internet.</li></ul><p><strong>3. Defining Task Definitions:</strong></p><p><strong>Create a Task Definition:</strong> In ECS, create a new task definition that specifies the container image, CPU, memory, and environment variables. This task definition acts as a blueprint for your containers.</p><p>We need to run a initialization create admin command to create admin user and to initialize db, Thus we create the task definition with the following json and once db user is created we create a updated version of task definition without command section in json</p><pre>{<br>  &quot;executionRoleArn&quot;: &quot;arn:aws:iam::123456789101:role/superset-ecstaskexecutionrole&quot;,<br>  &quot;containerDefinitions&quot;: [<br>    {<br>      &quot;logConfiguration&quot;: {<br>        &quot;logDriver&quot;: &quot;awslogs&quot;,<br>        &quot;options&quot;: {<br>          &quot;awslogs-group&quot;: &quot;/ecs/superset&quot;,<br>          &quot;awslogs-region&quot;: &quot;us-east-1&quot;,<br>          &quot;awslogs-stream-prefix&quot;: &quot;ecs&quot;<br>        }<br>      },<br>      &quot;portMappings&quot;: [<br>        {<br>          &quot;hostPort&quot;: 8088,<br>          &quot;protocol&quot;: &quot;tcp&quot;,<br>          &quot;containerPort&quot;: 8088<br>        }<br>      ],<br>      &quot;environment&quot;: [<br>        {<br>          &quot;name&quot;: &quot;REDIS_HOST&quot;,<br>          &quot;value&quot;: &quot;superset-redis.abcdef.0001.use1.cache.amazonaws.com&quot;<br>      },<br>      {<br>          &quot;name&quot;: &quot;DATABASE_DIALECT&quot;,<br>          &quot;value&quot;: &quot;postgresql&quot;<br>      },<br>      {<br>          &quot;name&quot;: &quot;REDIS_PORT&quot;,<br>          &quot;value&quot;: &quot;6379&quot;<br>      },<br>      {<br>          &quot;name&quot;: &quot;DATABASE_HOST&quot;,<br>          &quot;value&quot;: &quot;supersetdb.cluster-abcd01efg0ey.us-east-1.rds.amazonaws.com&quot;<br>      },<br>      {<br>          &quot;name&quot;: &quot;REDIS_RESULTS_DB&quot;,<br>          &quot;value&quot;: &quot;1&quot;<br>      },<br>      {<br>          &quot;name&quot;: &quot;SUPERSET_ENV&quot;,<br>          &quot;value&quot;: &quot;production&quot;<br>      },<br>      {<br>          &quot;name&quot;: &quot;SUPERSET_LOAD_EXAMPLES&quot;,<br>          &quot;value&quot;: &quot;no&quot;<br>      },<br>      {<br>          &quot;name&quot;: &quot;DATABASE_DB&quot;,<br>          &quot;value&quot;: &quot;superset_db&quot;<br>      },<br>      {<br>          &quot;name&quot;: &quot;REDIS_CELERY_DB&quot;,<br>          &quot;value&quot;: &quot;0&quot;<br>      },<br>      {<br>          &quot;name&quot;: &quot;DATABASE_USER&quot;,<br>          &quot;value&quot;: &quot;superset_user&quot;<br>      },<br>      {<br>          &quot;name&quot;: &quot;DATABASE_PORT&quot;,<br>          &quot;value&quot;: &quot;5432&quot;<br>      }<br>      <br>      ],<br>      &quot;secrets&quot;: [<br>        {<br>          &quot;valueFrom&quot;: &quot;arn:aws:secretsmanager:us-east-1:123456789101:secret:superset-SECRETS-ABcdEf:DATABASE_PASSWORD:AWSCURRENT:&quot;,<br>          &quot;name&quot;: &quot;DATABASE_PASSWORD&quot;<br>        },<br>        {<br>          &quot;valueFrom&quot;: &quot;arn:aws:secretsmanager:us-east-1:123456789101:secret:superset-SECRETS-ABcdEf:SECRET_KEY:AWSCURRENT:&quot;,<br>          &quot;name&quot;: &quot;SUPERSET_SECRET_KEY&quot;<br>        }<br>      ],<br>      &quot;entryPoint&quot;: [<br>        &quot;sh&quot;,<br>        &quot;-c&quot;<br>    ],<br>    &quot;command&quot;: [<br>        &quot;/usr/bin/run-server.sh &amp; sleep 30 &amp;&amp; superset fab create-admin --username admin --firstname Superset --lastname Admin --email admin@superset.com --password admin &amp;&amp; superset db upgrade &amp;&amp; superset init &amp;&amp; tail -f /dev/null&quot;<br>    ],<br>      &quot;image&quot;: &quot;123456789101.dkr.ecr.us-east-1.amazonaws.com/superset:latest&quot;,<br>      &quot;name&quot;: &quot;superset&quot;<br>    }<br>    ],<br>    &quot;memory&quot;: &quot;2048&quot;,<br>    &quot;taskRoleArn&quot;: &quot;arn:aws:iam::123456789101:role/superset-ecstaskrole&quot;,<br>    &quot;family&quot;: &quot;superset&quot;,<br>    &quot;requiresCompatibilities&quot;: [<br>      &quot;FARGATE&quot;<br>    ],<br>    &quot;networkMode&quot;: &quot;awsvpc&quot;,<br>    &quot;cpu&quot;: &quot;1024&quot;,<br>    &quot;volumes&quot;: []<br>  }</pre><p>The command section in above json will be creating the admin user with username and password as <em>admin</em>. Do change the password on first login itself for security.</p><ul><li><strong>Secrets and Configuration:</strong> Use AWS Secrets Manager to store and manage sensitive information like database credentials and SUPERSET_SECRET_KEY. In your task definition, reference these secrets to ensure secure access to your databases. Also remember to provide access to task execution role for secrets manager.</li></ul><p>After the first run and creation of the admin user, you can modify the command part to avoid running Create admin again, the below part can be inserted to command section of task definition json</p><pre> &quot;command&quot;: [<br>        &quot;/usr/bin/run-server.sh &amp;&amp; superset db upgrade &amp;&amp; superset init&quot;<br>    ],</pre><p><strong>4. Service Creation and Load Balancing:</strong></p><ul><li><strong>Set Up the Service:</strong> Create a service in ECS that runs the Superset task definition. ECS services manage the lifecycle of your tasks, ensuring that the desired number of tasks are running at all times. Follow <a href="https://docs.aws.amazon.com/AmazonECS/latest/developerguide/ECS_AWSCLI_Fargate.html#ECS_AWSCLI_Fargate_create_service">this doc</a> to create a service</li><li><strong>Application Load Balancer (ALB):</strong> Configure an ALB to route incoming traffic to your ECS service. The ALB can distribute traffic across multiple containers and handle SSL termination, ensuring secure communication between users and Superset.</li></ul><p><strong>5. Integration with AWS Services:</strong></p><ul><li><strong>Database and Cache:</strong> Connect Superset to Amazon RDS for storing metadata and ElastiCache for caching. These integrations enhance performance and reliability. You can follow <a href="https://docs.aws.amazon.com/AmazonElastiCache/latest/red-ug/GettingStarted.CreateCluster.html#Clusters.Create.CON.Redis-gs">this doc</a> to create a a elastic cache which is in non-cluster mode. you’ll need to setup subnet group in private/data subnets and a Security group</li><li><strong>Data Storage and Querying:</strong> Use Amazon S3 for data storage and AWS Athena for querying large datasets. Superset can connect directly to Athena, allowing users to visualize data stored in S3. Make sure you have your data in the data catalog db if you are planning query with athena. you’ll also need to provide access to ecstask role to query with athena, this way you can avoid using access keys from superset console.</li></ul><p>Since installed athena database driver with docker when you try to add the database in other databases you’ll be able to select athena from drop down</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/545/1*3bYLhNnEtu42y6tFHEESSA.png" /></figure><p>You can connect athena with access keys or with IAM role, the following is the syntax to connect with access keys</p><pre>awsathena+jdbc://{aws_access_key_id}:{aws_secret_access_key}@athena.{region_name}.amazonaws.com/{schema_name}?s3_staging_dir={s3_staging_dir}&amp;...</pre><p>Since ECS service have ecs task role attached we can make use of it to access Athena, which you can define by adding following parameters in Superset’s Athena database connection UI under ADVANCED → Other → ENGINE PARAMETERS.</p><pre>{<br>    &quot;connect_args&quot;: {<br>        &quot;role_arn&quot;: &quot;&lt;role arn&gt;&quot;<br>    }<br>}</pre><p>We need to fine grain access to IAM identity. The follow IAMpolucy need to be attached for accessing athena and to query catalog.</p><pre>{<br>    &quot;Version&quot;: &quot;2012-10-17&quot;,<br>    &quot;Statement&quot;: [<br>        {<br>            &quot;Effect&quot;: &quot;Allow&quot;,<br>            &quot;Action&quot;: [<br>                &quot;logs:CreateLogStream&quot;,<br>                &quot;logs:PutLogEvents&quot;,<br>                &quot;athena:ListDataCatalogs&quot;,<br>                &quot;athena:ListDatabases&quot;,<br>                &quot;s3:GetBucketLocation&quot;,<br>                &quot;glue:GetDatabases&quot;,<br>                &quot;athena:ListTableMetadata&quot;,<br>                &quot;athena:GetTableMetadata&quot;,<br>                &quot;glue:GetTables&quot;,<br>                &quot;glue:GetTable&quot;,<br>                &quot;glue:GetPartitions&quot;,<br>                &quot;lakeformation:GetDataAccess&quot;,<br>                &quot;s3:List*&quot;<br><br>            ],<br>            &quot;Resource&quot;: &quot;*&quot;<br>        },<br>        {<br>            &quot;Action&quot;: [<br>                &quot;athena:StartQueryExecution&quot;,<br>                &quot;s3:ListMultipartUploadParts&quot;,<br>                &quot;athena:GetQueryResults&quot;,<br>                &quot;athena:GetWorkGroup&quot;,<br>                &quot;s3:PutObject&quot;,<br>                &quot;s3:GetObject&quot;,<br>                &quot;s3:AbortMultipartUpload&quot;,<br>                &quot;athena:StopQueryExecution&quot;,<br>                &quot;athena:GetQueryExecution&quot;,<br>                &quot;athena:StopQueryExecution&quot;,<br>                &quot;athena:GetDatabase&quot;,<br>                &quot;athena:GetDataCatalog&quot;,<br>                &quot;athena:ListQueryExecutions&quot;<br>                <br>            ],<br>            &quot;Effect&quot;: &quot;Allow&quot;,<br>            &quot;Resource&quot;: [<br>                  &quot;arn:aws:athena:us-east-1:123456789101:workgroup/*&quot;,<br>                  &quot;arn:aws:s3:::&lt;aws-athena-query-results-bucket&gt;/*&quot;,<br>                  &quot;arn:aws:s3:::&lt;catalog-metadata-bucket&gt;/*&quot;<br>                    ]<br>        }<br>    ]<br>}</pre><p><strong>6. Monitoring, Logging, and Scaling:</strong></p><ul><li><strong>CloudWatch:</strong> Set up monitoring and logging with CloudWatch. You can create alarms to trigger scaling actions or notify you of performance issues.</li><li><strong>Auto Scaling:</strong> Define auto-scaling policies based on metrics like CPU utilization. ECS will automatically add or remove containers to maintain optimal performance as traffic fluctuates.</li></ul><h3><strong>Conclusion</strong></h3><p>Deploying Apache Superset on AWS ECS offers a scalable, secure, and cost-effective solution for organizations looking to harness the power of data visualization. By leveraging AWS’s robust infrastructure and services, you can ensure that your Superset deployment is not only powerful but also highly available and easy to manage. Whether you’re a small startup or a large enterprise, this deployment strategy allows you to focus on what matters most — turning data into actionable insights. With this guide, you’re well-equipped to deploy Superset on ECS and unlock the full potential of your data.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*vkbefa9nMVGB7T7S" /><figcaption>Photo by <a href="https://unsplash.com/@hannynaibaho?utm_source=medium&amp;utm_medium=referral">Hanny Naibaho</a> on <a href="https://unsplash.com/?utm_source=medium&amp;utm_medium=referral">Unsplash</a></figcaption></figure><h3>References</h3><ul><li><a href="https://superset.apache.org/docs/configuration/databases/">Database drivers supported</a></li><li><a href="https://medium.com/@mrflk/setting-up-apache-superset-on-ecs-to-connect-with-athena-8ad2f56eb41b">https://medium.com/@mrflk/setting-up-apache-superset-on-ecs-to-connect-with-athena-8ad2f56eb41b</a></li><li><a href="https://aws.amazon.com/solutions/implementations/apache-superset/">https://aws.amazon.com/solutions/implementations/apache-superset/</a></li></ul><h3>In Plain English 🚀</h3><p><em>Thank you for being a part of the </em><a href="https://plainenglish.io/"><strong><em>In Plain English</em></strong></a><em> community! Before you go:</em></p><ul><li>Be sure to <strong>clap</strong> and <strong>follow</strong> the writer ️👏<strong>️️</strong></li><li>Follow us: <a href="https://twitter.com/inPlainEngHQ"><strong>X</strong></a> | <a href="https://www.linkedin.com/company/inplainenglish/"><strong>LinkedIn</strong></a> | <a href="https://www.youtube.com/channel/UCtipWUghju290NWcn8jhyAw"><strong>YouTube</strong></a> | <a href="https://discord.gg/in-plain-english-709094664682340443"><strong>Discord</strong></a> | <a href="https://newsletter.plainenglish.io/"><strong>Newsletter</strong></a></li><li>Visit our other platforms: <a href="https://cofeed.app/"><strong>CoFeed</strong></a> | <a href="https://differ.blog/"><strong>Differ</strong></a></li><li>More content at <a href="https://plainenglish.io/"><strong>PlainEnglish.io</strong></a></li></ul><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=08da76bedd32" width="1" height="1" alt=""><hr><p><a href="https://aws.plainenglish.io/how-to-deploy-apache-superset-on-aws-ecs-08da76bedd32">How to Deploy Apache Superset on AWS ECS?</a> was originally published in <a href="https://aws.plainenglish.io/">AWS in Plain English</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Workaround for Amazon Lex V2 with Terraform]]></title>
            <link>https://aws.plainenglish.io/workaround-for-amazon-lex-v2-with-terraform-87d87b808129?source=rss-b6da988055dc------2</link>
            <guid isPermaLink="false">https://medium.com/p/87d87b808129</guid>
            <category><![CDATA[chatbots]]></category>
            <category><![CDATA[iac]]></category>
            <category><![CDATA[aws]]></category>
            <category><![CDATA[aws-lex]]></category>
            <category><![CDATA[terraform]]></category>
            <dc:creator><![CDATA[Anil Augustine Chalissery]]></dc:creator>
            <pubDate>Sat, 05 Aug 2023 12:20:08 GMT</pubDate>
            <atom:updated>2023-08-05T12:20:08.724Z</atom:updated>
            <content:encoded><![CDATA[<p>Unleashing the Power of Amazon Lex V2 with Terraform: Overcoming Limitations and Ingenious Workarounds</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*WjfaSc8UBkdFkixHyRFcJw.jpeg" /></figure><h4><strong>Introduction</strong></h4><p>Conversational bots have become increasingly popular for automating customer interactions, streamlining support processes, and enhancing user experiences. Amazon Lex V2 is a powerful natural language understanding service that allows you to build sophisticated bots that can understand and respond to user input in a human-like manner. In this blog post, we’ll guide you through the process of creating a chatbot with Amazon Lex V2 using Terraform, a popular infrastructure-as-code tool.</p><h4>Understanding the Requirements</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*cWv9v5PKVen07VJC" /><figcaption>Photo by <a href="https://unsplash.com/@ric3b0wl?utm_source=medium&amp;utm_medium=referral">Sam Xu</a> on <a href="https://unsplash.com/?utm_source=medium&amp;utm_medium=referral">Unsplash</a></figcaption></figure><p>One of the key requirements we had, was to use Terraform to create Lex also we needed the Lex bot in V2. In case you are wondering what&#39;s with Lex V2 you can find it <a href="https://docs.aws.amazon.com/lexv2/latest/dg/what-is.html">here</a>. Basically, Lex v2 is a better and simpler version of Lex V1. So during my R&amp;D, I figured out there is some strong separation between v1 and v2 lex. You can’t even list the bots you created in the v2 console, in the V1 console. Another thing that I figured out was Terraform aws provider doesn&#39;t support the creation of a lex bot in V2(as of today). You can see the open GitHub issue <a href="https://github.com/hashicorp/terraform-provider-aws/issues/21375">here</a> -&gt; <a href="https://github.com/hashicorp/terraform-provider-aws/issues/21375">https://github.com/hashicorp/terraform-provider-aws/issues/21375</a></p><h4>Overcoming Limitations — Workaround for Amazon Lex V2 with Terraform</h4><p>Now let&#39;s see the available options we have, along with its pros and cons,</p><p><strong>1.Use the AWS console to create lex bot v2</strong></p><p>Using the AWS console to create the Lex bot in V2 provides a straightforward approach but lacks the benefits of infrastructure-as-code (IaC) and version control. It is essential to maintain personalized documentation to ensure consistency across multiple environments.</p><p><strong>2. Use Terraform aws provider and create lex bot in v1</strong></p><p>Before implementing this we need to understand</p><ul><li>What all features in V2 are we compromising?</li><li>Will we be able to implement all the required features in V1?</li><li>How do we adapt the migration once AWS roll-out V1 depreciation? How much of developer support do we require on this migration?</li></ul><p>Creating the Lex bot in V1 using the Terraform AWS provider is a viable workaround, but it comes with the trade-off of potentially missing out on advanced features offered by Lex V2. Migration considerations must be taken into account when AWS eventually deprecates V1. Based on the answers to these questions you can select this choice.</p><p><strong>3. Use Terraform awscc provider and create Lex bot in V2</strong></p><p>The Terraform <a href="https://registry.terraform.io/providers/hashicorp/awscc/latest/docs">awscc provider</a> presents another option, but being in technical preview, it is not recommended for production usage, according to AWS. While it is an official HashiCorp provider, it may not receive the same level of support and updates as the AWS-native providers.</p><p><strong>4. Use Cloudformation to create a Lex.</strong></p><p>Leveraging CloudFormation to create the Lex bot in V2 offers a robust solution. By combining Terraform and CloudFormation, you can enjoy the benefits of IaC while utilizing CloudFormation’s support for Lex V2. This approach allows easier management and deployment across multiple environments, reducing the chances of human errors.</p><pre>resource &quot;aws_cloudformation_stack&quot; &quot;this&quot; {<br>  name = var.name<br><br>  parameters = var.parameters<br>  capabilities = var.capabilities <br>  template_body = file(&quot;${path.module}/Lex.yml&quot;)<br>}</pre><p><strong>5. Use AWS CLI in Terraform to create a Lex</strong></p><p>Another option is to use the AWS CLI in Terraform through local execution provisioners. While this method can work effectively, it requires the AWS CLI to be installed on the host executing Terraform, which might introduce additional dependencies.</p><pre>resource &quot;null_resource&quot; &quot;create-endpoint&quot; {<br>  provisioner &quot;local-exec&quot; {<br>    command = &quot; aws lexv2-models create-bot --bot-name &lt;botname&gt; --role-arn &lt;value&gt; --data-privacy &lt;value&gt; --cli-input-json file://&lt;file.json&gt;&quot;<br>  }<br>}</pre><h3>Conclusion</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*qRPugGH3mQ64Gnxl" /><figcaption>Photo by <a href="https://unsplash.com/@srz?utm_source=medium&amp;utm_medium=referral">sydney Rae</a> on <a href="https://unsplash.com/?utm_source=medium&amp;utm_medium=referral">Unsplash</a></figcaption></figure><p>In this blog post, we explored various workarounds to address the current limitation of creating an Amazon Lex V2 bot directly with Terraform. Each option comes with its own set of advantages and considerations, allowing you to make an informed decision based on your specific requirements and priorities.</p><p>In conclusion, selecting the most appropriate workaround depends on your project’s specific needs, future scalability requirements, and familiarity with the tools. We recommend carefully evaluating each option and understanding its implications before proceeding. As the cloud computing landscape evolves, it is essential to stay up-to-date with AWS provider updates and any new features introduced to facilitate smoother integration between Amazon Lex V2 and Terraform. Always strive for a well-architected and maintainable solution to achieve success in your conversational bot deployment endeavors.</p><h3>Reference</h3><ul><li><a href="https://github.com/hashicorp/terraform-provider-aws/issues/21375">Github issue</a></li><li><a href="https://docs.aws.amazon.com/lexv2/latest/dg/what-is.html">What is Lex v2?</a></li><li><a href="https://docs.aws.amazon.com/cli/latest/reference/lexv2-models/index.html#lexv2-models">Lex V2 AWS CLI doc</a></li></ul><h4>In Plain English</h4><p><em>Thank you for being a part of our community! Before you go:</em></p><ul><li><em>Be sure to </em><strong><em>clap</em></strong><em> and </em><strong><em>follow</em></strong><em> the writer! 👏</em></li><li><em>You can find even more content at </em><a href="https://plainenglish.io/"><strong><em>PlainEnglish.io</em></strong></a><strong><em> 🚀</em></strong></li><li><em>Sign up for our </em><a href="http://newsletter.plainenglish.io/"><strong><em>free weekly newsletter</em></strong></a><em>. 🗞️</em></li><li><em>Follow us on </em><a href="https://twitter.com/inPlainEngHQ"><strong><em>Twitter</em></strong></a>, <a href="https://www.linkedin.com/company/inplainenglish/"><strong><em>LinkedIn</em></strong></a>, <a href="https://www.youtube.com/channel/UCtipWUghju290NWcn8jhyAw"><strong><em>YouTube</em></strong></a>, and <a href="https://discord.gg/GtDtUAvyhW"><strong><em>Discord</em></strong></a><strong><em>.</em></strong></li></ul><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=87d87b808129" width="1" height="1" alt=""><hr><p><a href="https://aws.plainenglish.io/workaround-for-amazon-lex-v2-with-terraform-87d87b808129">Workaround for Amazon Lex V2 with Terraform</a> was originally published in <a href="https://aws.plainenglish.io/">AWS in Plain English</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Building Multi-Environment Infrastructure with Terragrunt]]></title>
            <link>https://faun.pub/building-multi-environment-infrastructure-with-terragrunt-7ee1074004aa?source=rss-b6da988055dc------2</link>
            <guid isPermaLink="false">https://medium.com/p/7ee1074004aa</guid>
            <category><![CDATA[aws]]></category>
            <category><![CDATA[terraform]]></category>
            <category><![CDATA[terragrunt]]></category>
            <category><![CDATA[iac]]></category>
            <category><![CDATA[devops]]></category>
            <dc:creator><![CDATA[Anil Augustine Chalissery]]></dc:creator>
            <pubDate>Tue, 21 Mar 2023 16:33:44 GMT</pubDate>
            <atom:updated>2023-03-21T16:33:44.738Z</atom:updated>
            <content:encoded><![CDATA[<h4>A walkthrough in Terragrunt with a multi-environment example</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*sBhRxEvZ8FU746W59SJifA.png" /></figure><h3>Why Terragrunt?</h3><p>Terraform is a powerful tool for infrastructure as code, but it can be challenging to manage larger projects with multiple modules and environments. That’s where Terragrunt comes in. Terragrunt is a tool that provides extra functionality on top of Terraform, making it easier to manage complex infrastructure projects.</p><p>One of the most significant benefits of Terragrunt is that it allows you to create reusable modules that can be shared across multiple projects. This can save you a lot of time and effort, especially if you are working on multiple projects that have similar infrastructure requirements.</p><p>Another benefit of Terragrunt is that it allows you to manage multiple environments (such as dev, staging, and production) with ease. With Terragrunt, you can define your infrastructure in a single file and then use it to create environments with different variables, such as region or size.</p><p>Terragrunt also provides a way to manage dependencies between modules, which is essential for projects with a large number of interdependent modules. You can use Terragrunt to ensure that each module is applied in the correct order and that any changes to one module are properly propagated to its dependencies.</p><p>Additionally, Terragrunt provides a convenient way to manage remote state files, which are necessary for collaborating on infrastructure projects with other team members. With Terragrunt, you can store your state files in a shared location (such as an S3 bucket) and automatically lock them to prevent conflicts when multiple people are working on the same project.</p><p>Finally, Terragrunt includes a number of other useful features, such as support for dynamic variables, built-in error checking, and the ability to run Terraform commands across multiple modules at once.</p><h3>TL;DR</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*BD0rTZXoOY0pWA6o" /><figcaption>Photo by <a href="https://unsplash.com/@quinoal?utm_source=medium&amp;utm_medium=referral">Quino Al</a> on <a href="https://unsplash.com/?utm_source=medium&amp;utm_medium=referral">Unsplash</a></figcaption></figure><p>Let&#39;s get our hands dirty by cloning this sample repo and launching some resources with Terragrunt.</p><p><a href="https://github.com/anilchalissery/terragrunt-sample-project">GitHub - anilchalissery/terragrunt-sample-project: A sample project to work on terragrunt</a></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/930/1*D-1fMDhNqANroySihHkFYg.png" /><figcaption>The tree structure of GitHub repo</figcaption></figure><p>Here in this Sample repository, we are creating a VPC, S3, and EC2. We’ll make use of modules from the module registry in this example.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/feb5ba1de685f0ad006a230857963047/href">https://medium.com/media/feb5ba1de685f0ad006a230857963047/href</a></iframe><p>We use an S3 bucket as the remote backend here and also use a DynamoDB for state locking.</p><p>Now Let&#39;s move to the dev folder and run plan and apply</p><pre>cd dev<br>terragrunt run-all plan<br>terragrunt run-all apply</pre><p>If you are using the <a href="https://www.terraform.io/docs/backends/types/s3.html">S3 backend</a> for remote state storage and the bucket you specify in remote_state.config doesn’t already exist, Terragrunt will create it automatically, with <a href="https://docs.aws.amazon.com/AmazonS3/latest/dev/Versioning.html">versioning</a>, <a href="https://docs.aws.amazon.com/AmazonS3/latest/dev/UsingServerSideEncryption.html">server-side encryption</a>, and <a href="https://docs.aws.amazon.com/AmazonS3/latest/dev/ServerLogs.html">access logging</a> enabled.</p><p>In addition, you can let terragrunt tag the bucket with custom tags that you specify in remote_state.config.s3_bucket_tags.</p><p>If you are using the <a href="https://www.terraform.io/docs/backends/types/s3.html">S3 backend</a> for remote state storage and you specify a dynamodb_table (a <a href="https://www.terraform.io/docs/backends/types/s3.html#dynamodb_table">DynamoDB table used for locking</a>) in remote_state.config, if that table doesn’t already exist, Terragrunt will create it automatically, with <a href="https://docs.aws.amazon.com/amazondynamodb/latest/developerguide/EncryptionAtRest.html">server-side encryption</a> enabled, including a primary key called LockID.</p><p>The function path_relative_to_include()returns the relative path between the current terragrunt.hcl file and the path specified in its include block. We typically use this in a root terragrunt.hcl file so that each Terraform child module stores its Terraform state at a different key. For example, the vpc module will have its key parameter resolve to vpc/terraform.tfstate and the ec2 module will have its key parameter resolve to ec2/terraform.tfstate</p><p>This <em>run-all </em>will run the plan and apply the command in each of the subfolders in the directory.</p><p>Now if we check our S3 bucket we can see a similar file structure in S3 bucket with each module having its statefile</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*dbB8GZXCEuM-oMbVGKeiKw.png" /><figcaption>S3 bucket folder structure</figcaption></figure><h3>Conclusion</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*EoZ91T0SuRa4_ncy" /><figcaption>Photo by <a href="https://unsplash.com/@etiennegirardet?utm_source=medium&amp;utm_medium=referral">Etienne Girardet</a> on <a href="https://unsplash.com/?utm_source=medium&amp;utm_medium=referral">Unsplash</a></figcaption></figure><p>Terragrunt is a powerful tool that can make it much easier to manage complex infrastructure projects with Terraform. With its support for reusable modules, multiple environments, dependencies, remote state files, and more, Terragrunt can help you save time and effort while ensuring that your infrastructure is properly managed and maintained.</p><h3>Reference</h3><ul><li><a href="http://i.imgur.com/iKtNmyp.png">Terragrunt Getting Started</a></li><li><a href="https://github.com/anilchalissery/terragrunt-sample-project">Sample Repo</a></li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/700/0*LZv0HimRZJs28-BS.png" /></figure><h4>👋 If you find this helpful, please click the clap 👏 button below a few times to show your support for the author 👇</h4><h4>🚀<a href="http://from.faun.to/r/8zxxd">Join FAUN Developer Community &amp; Get Similar Stories in your Inbox Each Week</a></h4><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=7ee1074004aa" width="1" height="1" alt=""><hr><p><a href="https://faun.pub/building-multi-environment-infrastructure-with-terragrunt-7ee1074004aa">Building Multi-Environment Infrastructure with Terragrunt</a> was originally published in <a href="https://faun.pub/">FAUN.dev() 🐾</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Apache Spark on AWS EKS]]></title>
            <link>https://faun.pub/apache-spark-on-aws-eks-3f6737d61d35?source=rss-b6da988055dc------2</link>
            <guid isPermaLink="false">https://medium.com/p/3f6737d61d35</guid>
            <category><![CDATA[apache-spark]]></category>
            <category><![CDATA[devops-tool]]></category>
            <category><![CDATA[aws-eks]]></category>
            <category><![CDATA[aws]]></category>
            <category><![CDATA[kubernetes]]></category>
            <dc:creator><![CDATA[Anil Augustine Chalissery]]></dc:creator>
            <pubDate>Fri, 04 Nov 2022 13:16:45 GMT</pubDate>
            <atom:updated>2022-11-04T13:16:45.371Z</atom:updated>
            <content:encoded><![CDATA[<h4>Creating EKS cluster with eksctl and Deploying Apache spark on EKS using Helm chart</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*2HEx75bI8GNVHWcDY8f1pg.jpeg" /></figure><h3><strong>Introduction</strong></h3><p>Apache Spark is an open-source analytical engine for executing data engineering, data science, and machine learning on single-node machines or clusters. Apache Spark natively supports Python, Java, R, and Scala, giving you a variety of languages for building your applications. Here we deploy Apache Spark with Helm which is a package manager of Kubernetes.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/761/1*tkWi16Clt_6RjLJNmyLVkg.png" /><figcaption>Image from <a href="https://spark.apache.org/docs/latest/running-on-kubernetes.html">Apache Spark on Kubernetes</a></figcaption></figure><p>With spark-submitwe can directly submit to the Kubernetes cluster. Spark creates a Spark driver running within a Kubernetes pod. The driver creates executors which are also running within Kubernetes pods and connects to them and executes the application code. When the application completes, the executor pods terminate and are cleaned up, but the driver pod persists logs and remains in a “completed” state in the Kubernetes API until it’s eventually garbage collected or manually cleaned up.</p><h4><strong>Step 1: Creating Kubernetes cluster with EKS</strong></h4><p>First, we create a cluster.yaml file and this file is used to create EKS cluster, Node groups, and relevant roles.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/ec2b0d363c4701ce4c3c9e2a28cbf347/href">https://medium.com/media/ec2b0d363c4701ce4c3c9e2a28cbf347/href</a></iframe><p>Once cluster.yaml is in place, use the following command to create EKS cluster.</p><pre>eksctl create cluster -f cluster.yaml</pre><p>This command will create CloudFormation stacks and clusters and roles are created. It takes around 15mins to complete this execution. In case of any errors checking CloudFormation stack events could help. The expected output is added below.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*ILlC295aTUwv7XkPD_Zptg.png" /><figcaption>outputs while creating the cluster</figcaption></figure><p>If we navigate to CloudFormation we could see the stacks compleated</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*2lcR_PhbrOtG2H5yttRrsg.png" /><figcaption>CloudFormations stacks completed execution</figcaption></figure><p>If we navigate to EC2 console we could see our instance deployed and in running state</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*V4gZuQvOIkccIirtojFA7A.png" /><figcaption>Node groups created with eksctl</figcaption></figure><p>If we check the terminal logs we could see that kubeconfig is added to home directory .kube/config</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*ohX6ECuegy_Du7Vuv4HLzw.png" /><figcaption>eksctl command execution completes</figcaption></figure><p>we can check cluster access by checking cluster-info or getting nodes</p><pre>kubectl cluster-info</pre><pre>kubectl get nodes</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*kC6Rk9xDZn1SUvkm89mthA.png" /><figcaption>checking cluster access</figcaption></figure><p>Now we have our cluster in place so let’s move on to the next step which is deploying apache spark</p><h4>Step 2: Deploy Apache Spark on EKS Cluster</h4><p>We add Apache Spark with <a href="https://helm.sh/">Helm charts</a>. Helm chart is a package manager for Kubernetes. With the help of Helm charts, we will be deploying <a href="https://bitnami.com/stack/spark/helm">Apache Spark packaged by Bitnami</a>.</p><p>First, we need to add the repo to the helm</p><pre>helm repo add bitnami https://charts.bitnami.com/bitnami</pre><p>(Optional)Then we can verify the repo by</p><pre>helm repo list</pre><p>Then to install this chart.</p><pre>helm install myspark bitnami/spark</pre><blockquote>myspark is a changable name. <a href="https://github.com/bitnami/charts/tree/main/bitnami/spark/#installing-the-chart">more info</a></blockquote><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*GrK-9a6th56r0X5_W8SgEw.png" /><figcaption>helm chart installed</figcaption></figure><p>To confirm Spark is deployed and pods are running we can list pods</p><pre>kubectl get pods</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Xkj7xtwCjCZc8DdRH0wfCg.png" /><figcaption>listing pods</figcaption></figure><p>Now lets port-forward and see the spark console.</p><pre>kubectl port-forward — namespace default svc/myspark-master-svc 8080:80</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*lSvB1chwyLIamNoJyQa2mA.png" /><figcaption>Spark UI</figcaption></figure><p>Now that our spark cluster is running let&#39;s try spark-submit.</p><h4>Step 3: Submitting a spark job</h4><p>To submit an application to the Apache Spark cluster, use the spark-submit <a href="https://github.com/apache/spark/tree/master/bin">script</a>. we need to <a href="https://spark.apache.org/downloads.html">download apache spark</a> to run spark-submit</p><pre>spark-submit \<br>    --class org.apache.spark.examples.SparkPi \<br>    --conf spark.kubernetes.container.image=bitnami/spark:3 \<br>    --master k8s://https://k8s-apiserver-host:k8s-apiserver-port \<br>    --conf spark.kubernetes.driverEnv.SPARK_MASTER_URL=spark://spark-master-svc:spark-master-port \<br>    --deploy-mode cluster \<br>    ./examples/jars/spark-examples_2.12-3.2.0.jar 1000</pre><p>As we have created a Kubernetes cluster setup, one way to discover the k8s-apiserver-host URL is by executing kubectl cluster-info.</p><pre>$ kubectl cluster-info<br>Kubernetes control plane is running at <a href="https://ec01035d6c8e74a267443b86721830fe.sk1.us-west-1.eks.amazonaws.com/">https://EC01035D6C8E74A267443B86721830FE.sk1.us-west-1.eks.amazonaws.com</a><br>CoreDNS is running at <a href="https://ec01035d6c8e74a267443b86721830fe.sk1.us-west-1.eks.amazonaws.com/api/v1/namespaces/kube-system/services/kube-dns:dns/proxy">https://EC01035D6C8E74A267443B86721830FE.sk1.us-west-1.eks.amazonaws.com/api/v1/namespaces/kube-system/services/kube-dns:dns/proxy</a></pre><p>In the above example, the specific Kubernetes cluster can be used with spark-submit by specifying --master k8s://https://EC01035D6C8E74A267443B86721830FE.sk1.us-west-1.eks.amazonaws.com:443 as an argument to spark-submit.</p><p>SPARK_MASTER_URL can be captured from the Spark console</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*YPKeJM5tA9_HNKnYSL0lwA.png" /><figcaption>SPARK_MASTER_URL</figcaption></figure><p>Here we use bitnami/spark:3 docker image which already has spark-examples_2.12–3.3.0.jar in /opt/bitnami/spark/examples/jars directory. so the command we use is as follows</p><pre>spark-submit \<br>    --class org.apache.spark.examples.SparkPi \<br>    --conf spark.kubernetes.container.image=bitnami/spark:3 \<br>    --master k8s://https://EC01035D6C8E74A267443B86721830FE.sk1.us-west-1.eks.amazonaws.com:443 \<br>    --conf spark.kubernetes.driverEnv.SPARK_MASTER_URL=spark://myspark-master-0.myspark-headless.default.svc.cluster.local:7077 \<br>    --deploy-mode cluster \<br>    local:///opt/bitnami/spark/examples/jars/spark-examples_2.12-3.3.0.jar 1000</pre><p>This will be showing some logs in real-time.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*IM3h5CpKdiO1DH0yXVvdSw.png" /><figcaption>Logs ending</figcaption></figure><p>Now in Spark UI, we can see this in completed applications</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*GeAnGpD0CJXJgWTNfJfoSA.png" /><figcaption>It is added to completed applications</figcaption></figure><p>As we use EKS we can make use of ECR too to store docker images.</p><h3>Reference</h3><ul><li><a href="https://spark.apache.org/docs/latest/running-on-kubernetes.html">Running Spark on Kubernetes</a></li><li><a href="https://aws.amazon.com/blogs/opensource/deploying-spark-jobs-on-amazon-eks/">Deploying Spark jobs on Amazon EKS</a></li><li><a href="https://aws.amazon.com/blogs/containers/best-practices-for-running-spark-on-amazon-eks/">Best practices for running Spark on Amazon EKS</a></li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/700/0*w7ESu0_-YGYCJ9Jg.png" /></figure><p>If you find this helpful, please click the clap 👏 button below a few times to show your support for the author 👇</p><h3>🚀<a href="http://from.faun.to/r/8zxxd">Join FAUN &amp; get similar stories in your inbox each week</a></h3><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=3f6737d61d35" width="1" height="1" alt=""><hr><p><a href="https://faun.pub/apache-spark-on-aws-eks-3f6737d61d35">Apache Spark on AWS EKS</a> was originally published in <a href="https://faun.pub/">FAUN.dev() 🐾</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[SonarQube deployed in AWS ECS Fargate]]></title>
            <link>https://aws.plainenglish.io/sonarqube-deployed-in-aws-ecs-fargate-e86fee056aae?source=rss-b6da988055dc------2</link>
            <guid isPermaLink="false">https://medium.com/p/e86fee056aae</guid>
            <category><![CDATA[docker]]></category>
            <category><![CDATA[sonarqube]]></category>
            <category><![CDATA[serverless]]></category>
            <category><![CDATA[aws]]></category>
            <category><![CDATA[devops-practice]]></category>
            <dc:creator><![CDATA[Anil Augustine Chalissery]]></dc:creator>
            <pubDate>Mon, 31 Oct 2022 06:16:54 GMT</pubDate>
            <atom:updated>2022-10-31T06:16:54.608Z</atom:updated>
            <content:encoded><![CDATA[<h4>SonarQube is an open-source code quality-checking tool. Here we deploy Sonarqube in ECS Fargate</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*fu0wR9U2jBa-EZ2Tdx1CBw.png" /></figure><p><a href="https://www.sonarqube.org/">SonarQube</a> is an open-source code quality-checking tool. Usually, we install this in some medium-type instances or run it in Docker. Here we are using a serverless model which is also cost-effective. We use a public docker hub image to create this.</p><h4>Prerequisite</h4><ul><li>Aws access to create ECS service</li><li>Postgres DB for storing data</li></ul><h4>Step 1: Create an ECR repo</h4><p>Ecr Repo is used to store the docker image of SonarQube. Here we used SonarQube <a href="https://hub.docker.com/layers/library/sonarqube/lts-community/images/sha256-538c77f1fd80b6d1f67e142d6aa50fc144714ceb1fcb678fb20f82d050844dea?context=explore">lts-community</a> version. we pull this docker image from the Docker hub and is kept in the ECR repo. We can also use the Docker hub image in ECS directly, but to avoid any issues while updates in the docker image we use ECR.</p><p>We create a private repository named <em>sonarqube</em></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*qeTmBappxFzvcq2qN1eC3Q.png" /><figcaption>ecr repo created</figcaption></figure><p>Once we have the ECR repo in place let’s pull the docker image from the <a href="https://hub.docker.com/_/sonarqube">Docker hub</a>.</p><pre>docker pull sonarqube:lts-community</pre><p>Once the docker image is pulled let&#39;s push it to the ECR repo. For this, from the ECR repository, we can obtain the push commands</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*5uIme2pCe3p_UNOF44r9DA.png" /><figcaption>push commands from ECR</figcaption></figure><pre>aws ecr get-login-password --region &lt;region&gt; | docker login --username AWS --password-stdin &lt;account-id&gt;.dkr.ecr.&lt;region&gt;.amazonaws.com</pre><pre>#since we already pulled docker image we can avoid docker build #command instead we proceed to tag image</pre><pre>docker tag sonarqube:lts-community &lt;account-id&gt;.dkr.ecr.&lt;region&gt;.amazonaws.com/sonarqube:latest</pre><pre>docker push &lt;account-id&gt;.dkr.ecr.&lt;region&gt;.amazonaws.com/sonarqube:latest</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*sCOZjVKmUJ7tlshJuhxX7w.png" /><figcaption>Pushing the docker image</figcaption></figure><p>After a few mins, the docker image will be there in the ECR repo</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*a3d1cDlJCNSaSy7Z_riKaQ.png" /><figcaption>docker image pushed</figcaption></figure><p>Now that we have our docker image in the ECR repo let&#39;s move to create a Task definition.</p><h4>Step 2: Creating Task Definition</h4><p>In the ECS console move to the Task definition section where we can click on <strong>Create</strong> to start creating the task definition. We start by launch type which is F<strong>argate</strong> here.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*mTe06E46-FwpcJYgjjmWIg.png" /><figcaption>Choosing Fargate launch type</figcaption></figure><p>Now moving to next we can name the task def</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*_E1c0Q7NEoFPOm1vwavL4Q.png" /><figcaption>Naming Task definitions</figcaption></figure><p>Next, we allocate the vCPU and Memory for our task.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*pZJnXAI5qC4B_oO1L6is0Q.png" /><figcaption>Setting Task size</figcaption></figure><p>Next, we move to set up container definitions where we specify the container name and image. We also add our port 9000 here which is the default port of SonarQube.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/899/1*06UjWncSnfu-28FM-1QVfg.png" /><figcaption>Adding container details and port</figcaption></figure><p>We add this in command to <a href="https://stackoverflow.com/questions/62860516/how-to-increase-the-vm-max-map-count-in-aws-ecs-fargate">eliminate some issues</a> we add this in the command section.</p><pre>-Dsonar.search.javaAdditionalOpts=-Dnode.store.allow_mmapfs=false</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/904/1*OwCw_ymMeakgBtsXuQiGoQ.png" /></figure><p>We use PostgreSQL DB to store the data of SonarQube. To secure the DB password we can use the parameter store optionally. if using parameter store make sure you provide ssm:getParameter access to ecstaskexecutionrole</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/907/1*WTQau0styUsuybceCFgk2w.png" /><figcaption>setting db creds as env variables</figcaption></figure><p>Now the rest part can be set as default and create task definition</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*aGq8oevtRKTgd01QmjgqRQ.png" /><figcaption>task definition created</figcaption></figure><h4>Step 3: Create ECS Cluster</h4><p>Now let&#39;s create a Network only ECS cluster. If you have a cluster feel free to use that</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*csixfdFUJT6eXLaByMxObA.png" /><figcaption>creating cluster</figcaption></figure><h4>Step 4: Creating ECS service</h4><p>Once we create a cluster let&#39;s move to create an ECS service in that cluster. As we create a Fargate service choose the launch type as Fargate while the service creation. Also, select the task definition we created and the cluster.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*lV5WukBTrhvvyLWpE-G-Pw.png" /><figcaption>Setting launch type and name of service</figcaption></figure><p>In the next part specify the task count and healthy percentage. we can set the task count as one and keep all others as default and move to the next step.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*_S4U7_6RS5y3W1NyfgSRaA.png" /></figure><p>In the next stage, we set up the VPC of service. Here we use private subnets and will not assign public IPs. Keep in mind to allow traffic from the ALB security group to this service security group for health checks.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*mUv4uYmfPe82Yp3WCdgzjg.png" /></figure><p>The next step is to set up a load balancer. Here we use an Application Load balancer.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*iAEB15si4yjqN8QO9R_Zxg.png" /><figcaption>Attaching load balancer</figcaption></figure><p>As we have an HTTPS listener already set we adding it to the HTTPS listener. It is also ok to assign an HTTP listener which can be edited later via the load balancer</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*rlzp2Nzp7ivv2IIwBVtLuw.png" /><figcaption>Attaching container to load balancer</figcaption></figure><p>The rest can be kept as default and proceed to the next. The next step will be the autoscaling of this service. As it is POC let&#39;s leave this and move forward to the next step. The last step is to review this and click on create.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*pBSaSEtIZQt3s0LcP8E6jw.png" /><figcaption>service created</figcaption></figure><p>Now in the load balancer, we can route to point it in our subdomain</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*IMsJzEqnYz6Ci6XKu4PWVQ.png" /><figcaption>Routing with load balancer</figcaption></figure><blockquote>Make sure you add this as record in DNS.</blockquote><p>With a few mins, the service will be up and running.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*mItozK5ZDuoR9QKP8GFUZQ.png" /><figcaption>SonarQube is up and running</figcaption></figure><h4>Troubleshooting</h4><ul><li>In case your service is not in a running state there might be some error in the task. You can see this in stopped tasks.</li><li>If service is not getting healthy in the target group make sure alb sg is allowed in ECS sg</li><li>Check logs for any error in connecting DB or other issues</li><li>As SonarQube takes some time to turn up it would be better to allow 300 seconds as the <strong>health check grace period </strong>would wait 300 secs before the health check.</li></ul><h4>Reference</h4><ul><li><a href="https://hub.docker.com/_/sonarqube">https://hub.docker.com/_/sonarqube</a></li><li><a href="https://docs.aws.amazon.com/AmazonECS/latest/developerguide/create-service.html">https://docs.aws.amazon.com/AmazonECS/latest/developerguide/create-service.html</a></li><li><a href="https://stackoverflow.com/questions/62860516/how-to-increase-the-vm-max-map-count-in-aws-ecs-fargate">https://stackoverflow.com/questions/62860516/how-to-increase-the-vm-max-map-count-in-aws-ecs-fargate</a></li></ul><p><em>More content at </em><a href="https://plainenglish.io/"><strong><em>PlainEnglish.io</em></strong></a><em>. Sign up for our </em><a href="http://newsletter.plainenglish.io/"><strong><em>free weekly newsletter</em></strong></a><em>. Follow us on </em><a href="https://twitter.com/inPlainEngHQ"><strong><em>Twitter</em></strong></a>, <a href="https://www.linkedin.com/company/inplainenglish/"><strong><em>LinkedIn</em></strong></a><em>, </em><a href="https://www.youtube.com/channel/UCtipWUghju290NWcn8jhyAw"><strong><em>YouTube</em></strong></a><em>, and </em><a href="https://discord.gg/GtDtUAvyhW"><strong><em>Discord</em></strong></a><em>. Interested in Growth Hacking? Check out </em><a href="https://circuit.ooo/"><strong><em>Circuit</em></strong></a><em>.</em></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=e86fee056aae" width="1" height="1" alt=""><hr><p><a href="https://aws.plainenglish.io/sonarqube-deployed-in-aws-ecs-fargate-e86fee056aae">SonarQube deployed in AWS ECS Fargate</a> was originally published in <a href="https://aws.plainenglish.io/">AWS in Plain English</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
    </channel>
</rss>