<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Abdul Rahim Shaikh on Medium]]></title>
        <description><![CDATA[Stories by Abdul Rahim Shaikh on Medium]]></description>
        <link>https://medium.com/@abdulshaikhr99?source=rss-c47a96f96b0a------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/1*Lr9_W53YZzaU8GBzLqF3kw.png</url>
            <title>Stories by Abdul Rahim Shaikh on Medium</title>
            <link>https://medium.com/@abdulshaikhr99?source=rss-c47a96f96b0a------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Tue, 17 Nov 2026 20:18:31 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@abdulshaikhr99/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[Building an Automated Cloud Cost Optimization and CI/CD Pipeline on AWS.]]></title>
            <link>https://medium.com/@abdulshaikhr99/building-an-automated-cloud-cost-optimization-and-ci-cd-pipeline-on-aws-169a871a0311?source=rss-c47a96f96b0a------2</link>
            <guid isPermaLink="false">https://medium.com/p/169a871a0311</guid>
            <category><![CDATA[software-development]]></category>
            <category><![CDATA[cloud]]></category>
            <category><![CDATA[devops]]></category>
            <category><![CDATA[aws]]></category>
            <dc:creator><![CDATA[Abdul Rahim Shaikh]]></dc:creator>
            <pubDate>Sat, 08 Nov 2025 18:23:57 GMT</pubDate>
            <atom:updated>2025-11-08T18:23:57.682Z</atom:updated>
            <content:encoded><![CDATA[<h3>Building an Automated Cloud Cost Optimization and CI/CD Pipeline on AWS. A Serverless, AI-Driven Approach</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*kC_HJQQEmMOW32rBr1_P3A.png" /></figure><p>In large-scale cloud environments, one of the most persistent challenges is controlling costs without sacrificing visibility or automation. Idle resources, underutilized instances, and neglected storage buckets can silently drain budgets over time. This project demonstrates how a cost-aware engineering team could tackle that challenge by building a <strong>fully automated, serverless cloud cost optimization and CI/CD system</strong> on <strong>AWS</strong>. Designed to mirror a production-grade workflow, the solution combines <strong>Terraform</strong> for Infrastructure as Code (IaC), <strong>AWS Lambda</strong> for serverless automation, <strong>Amazon CloudWatch</strong> for observability, and <strong>Amazon Bedrock</strong> for AI-powered insights — all orchestrated through <strong>GitHub Actions CI/CD pipelines</strong>. The result is a hands-free, environment-segmented (staging and production) system that continuously scans for idle resources, stops or tags them automatically, and delivers real-time AI summaries of potential savings and optimization opportunities.</p><p>Let’s get started.</p><h3><strong>Phase 1: Infra with Terraform</strong></h3><p>The first step in building any reliable automation pipeline is to establish a consistent and isolated infrastructure foundation. In this phase, the goal is to provision all core AWS components that the cost optimization system will depend on — fully through <strong>Terraform</strong> to ensure reproducibility and version control. Two separate environments, <strong>staging</strong> and <strong>production</strong>, are defined using Terraform workspaces, allowing new changes to be tested safely before promotion. Key resources are deployed at this stage: an <strong>Amazon S3 bucket</strong> structured with environment-specific prefixes for reports, logs, and AI summaries; a <strong>DynamoDB table</strong> to store resource scan history and action metadata; <strong>IAM roles and policies</strong> with least-privilege access for the Lambda functions that will be added later; an <strong>SNS topic</strong> for sending AI insights and cost alerts; and a placeholder <strong>CloudWatch dashboard</strong> to be populated in later phases. By completing this foundation first, we ensure all subsequent components — from the serverless Lambdas to the AI-driven insights — operate in a secure, well-defined, and production-ready environment.</p><p>We’ll start off by creating the project structure and specifically the Terraform setup for now. The workspace will look like:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/213/1*_YmcPHGwJSa0J0Swhy8wlw.png" /></figure><p>We also have the GitHub actions related files but we will work on this later as well as the Lambda related ones.</p><p>This Terraform setup provisions the foundational infrastructure for a <strong>Cloud Cost Optimizer</strong> system on AWS. It configures the AWS provider and remote backend to store Terraform’s state file securely in S3 with DynamoDB-based locking, ensuring reliable deployments across environments like <em>staging</em> or <em>production</em>. The configuration uses variables such as region, environment, and project name to make the deployment flexible and reusable across different stages of development.</p><p>The core resources provisioned include an <strong>S3 bucket</strong> for storing cost reports and AI-generated summaries, a <strong>DynamoDB table</strong> to track historical resource and cost data with scalable on-demand capacity, and an <strong>SNS topic</strong> to broadcast insights or alerts generated by AI components. Each environment gets its own logically separated S3 prefixes, allowing clean organization of reports and summaries.</p><p>Finally, <strong>placeholder IAM roles</strong> are created for upcoming Lambda functions that will handle resource scanning, cost optimization, and AI insights in the next phase. These roles include trust policies for Lambda but will receive detailed permissions later. Altogether, this phase sets up the <strong>data storage, history tracking, and communication backbone</strong> of the system, preparing it for the automation logic that follows.</p><p>After all that we can run our basic Terraform commands to get the infra up and running (in the process we will create separate workspaces for different environments):</p><pre>cd terraform<br>terraform init<br>terraform workspace new staging<br>terraform workspace new production</pre><p>After this we can switch to each respective workspace and then set the value for “environment” at runtime and do terraform apply.</p><pre>terraform apply -var=&quot;environment=staging&quot;</pre><p>Once we do all these steps we can confirm in our AWS console if the following are created:</p><ul><li>S3: cloud-cost-optimizer-staging-reports</li><li>DynamoDB: history table created</li><li>SNS: topic for summaries</li><li>IAM roles (placeholders)</li></ul><p>Once we check all this, phase 1 is complete.</p><p>In this phase, we laid the groundwork for the entire project by creating a <strong>repeatable, traceable, and environment-aware Terraform infrastructure</strong>. All core AWS services, S3, DynamoDB, IAM, SNS, and CloudWatch — are now provisioned and version-controlled through code, ensuring full <strong>reproducibility</strong> and <strong>granularity</strong> in future updates. With both <strong>staging</strong> and <strong>production</strong> environments defined via Terraform workspaces, we’ve established a clean separation for safe testing and deployment. This foundational setup acts as the backbone for everything that follows — the Lambda automation, AI integration, CI/CD pipelines, and monitoring — all of which will now build on top of this stable infrastructure. With our environments ready, we can confidently move on to <strong>Phase 2: Serverless Logic with AWS Lambda.</strong></p><h3>Phase 2: Serverless Logic with Lambda</h3><p>With the infrastructure foundation in place, Phase 2 focuses on bringing the system to life through <strong>serverless compute logic</strong>. In this stage, we design and implement three specialized <strong>AWS Lambda functions</strong> — <strong>Scanner</strong>, <strong>Action</strong>, and <strong>AI-Insights</strong> — each serving a critical role in the cost optimization workflow. The Scanner Lambda identifies idle or underutilized resources across EC2 and S3; the Action Lambda applies intelligent remediation by tagging or stopping them; and the AI-Insights Lambda leverages <strong>Amazon Bedrock</strong> to generate actionable summaries and recommendations. All functions are written in <strong>Python (using Boto3)</strong> and designed to run independently within their IAM boundaries. This phase transforms our static Terraform infrastructure into a dynamic, automated system capable of continuous cost monitoring and intelligent decision-making.</p><p>We will add three Python scripts inside the <strong>lambdas/</strong> directory, each representing one of the core automation functions. The <strong>Scanner Lambda</strong> script will continuously inspect AWS resources, evaluate EC2 utilization metrics and S3 activity, and log idle findings in a structured JSON format. The <strong>Action Lambda</strong> script will then consume those findings, safely stop or tag idle instances and buckets, and record every action in DynamoDB and S3 for traceability. Finally, the <strong>AI-Insights Lambda</strong> script will process the latest reports, generate concise executive summaries using <strong>Amazon Bedrock</strong> (mocked locally for now), and prepare those outputs for future publishing through SNS. Together, these scripts form the heart of the system’s cost-optimization engine, turning passive monitoring into proactive, data-driven automation.</p><p>action/main.py:</p><pre>import boto3, json, os, datetime<br><br>ec2 = boto3.client(&#39;ec2&#39;)<br>dynamodb = boto3.resource(&#39;dynamodb&#39;)<br>s3 = boto3.client(&#39;s3&#39;)<br><br>TABLE_NAME = os.environ.get(&#39;RESOURCE_TABLE&#39;)<br>BUCKET_NAME = os.environ.get(&#39;REPORT_BUCKET&#39;)<br>ENV = os.environ.get(&#39;ENVIRONMENT&#39;, &#39;staging&#39;)<br><br>def lambda_handler(event, context):<br>    now = datetime.datetime.utcnow().isoformat()<br>    actions = []<br><br>    table = dynamodb.Table(TABLE_NAME)<br>    # In real setup, you&#39;d query last scan results<br>    # Here we simulate by stopping one dummy instance<br>    sample_instance = &quot;i-0123456789abcdef0&quot;<br><br>    try:<br>        ec2.stop_instances(InstanceIds=[sample_instance])<br>        action = {&quot;resource_id&quot;: sample_instance, &quot;action&quot;: &quot;stop_instance&quot;, &quot;timestamp&quot;: now}<br>        actions.append(action)<br>        table.put_item(Item=action)<br>    except Exception as e:<br>        actions.append({&quot;error&quot;: str(e)})<br><br>    key = f&quot;{ENV}/reports/action_{now}.json&quot;<br>    s3.put_object(Bucket=BUCKET_NAME, Key=key, Body=json.dumps(actions))<br>    return {&quot;statusCode&quot;: 200, &quot;body&quot;: json.dumps(actions)}</pre><p>scanner/main.py:</p><pre>import boto3, json, os, datetime<br><br>dynamodb = boto3.resource(&#39;dynamodb&#39;)<br>s3 = boto3.client(&#39;s3&#39;)<br>ec2 = boto3.client(&#39;ec2&#39;)<br><br>TABLE_NAME = os.environ.get(&#39;RESOURCE_TABLE&#39;)<br>BUCKET_NAME = os.environ.get(&#39;REPORT_BUCKET&#39;)<br>ENV = os.environ.get(&#39;ENVIRONMENT&#39;, &#39;staging&#39;)<br><br>def lambda_handler(event, context):<br>    now = datetime.datetime.utcnow().isoformat()<br>    findings = []<br><br>    # --- Scan EC2 ---<br>    instances = ec2.describe_instances(Filters=[{&#39;Name&#39;: &#39;instance-state-name&#39;,&#39;Values&#39;: [&#39;running&#39;]}])<br>    print(instances)<br>    for reservation in instances[&#39;Reservations&#39;]:<br>        for inst in reservation[&#39;Instances&#39;]:<br>            instance_id = inst[&#39;InstanceId&#39;]<br>            metrics = {&quot;cpu&quot;: 3.5}  # placeholder metric<br>            if metrics[&quot;cpu&quot;] &lt; 5:  # idle threshold<br>                findings.append({<br>                    &quot;resource_id&quot;: instance_id,<br>                    &quot;resource_type&quot;: &quot;EC2&quot;,<br>                    &quot;idle_reason&quot;: &quot;Low CPU &lt;5%&quot;,<br>                    &quot;timestamp&quot;: now<br>                })<br><br>    # --- Scan S3 ---<br>    buckets = s3.list_buckets()[&#39;Buckets&#39;]<br>    for b in buckets:<br>        bucket_name = b[&#39;Name&#39;]<br>        objs = s3.list_objects_v2(Bucket=bucket_name).get(&#39;KeyCount&#39;, 0)<br>        if objs == 0:<br>            findings.append({<br>                &quot;resource_id&quot;: bucket_name,<br>                &quot;resource_type&quot;: &quot;S3&quot;,<br>                &quot;idle_reason&quot;: &quot;Empty bucket&quot;,<br>                &quot;timestamp&quot;: now<br>            })<br><br>    # --- Write to DynamoDB ---<br>    table = dynamodb.Table(TABLE_NAME)<br>    for f in findings:<br>        table.put_item(Item=f)<br><br>    # --- Write report to S3 ---<br>    key = f&quot;{ENV}/reports/report_{now}.json&quot;<br>    s3.put_object(Bucket=BUCKET_NAME, Key=key, Body=json.dumps(findings))<br><br>    return {&quot;statusCode&quot;: 200, &quot;body&quot;: json.dumps({&quot;findings&quot;: len(findings)})}</pre><p>ai_insights/main.py:</p><pre>import boto3, json, os, datetime<br><br>s3 = boto3.client(&#39;s3&#39;)<br>sns = boto3.client(&#39;sns&#39;)<br><br>BUCKET_NAME = os.environ.get(&#39;REPORT_BUCKET&#39;)<br>SNS_TOPIC = os.environ.get(&#39;SNS_TOPIC_ARN&#39;)<br>ENV = os.environ.get(&#39;ENVIRONMENT&#39;, &#39;staging&#39;)<br><br>def lambda_handler(event, context):<br>    now = datetime.datetime.utcnow().isoformat()<br><br>    # Load the latest report (simplified)<br>    key = f&quot;{ENV}/reports/&quot;<br>    objs = s3.list_objects_v2(Bucket=BUCKET_NAME, Prefix=key).get(&#39;Contents&#39;, [])<br>    latest = sorted(objs, key=lambda x: x[&#39;LastModified&#39;], reverse=True)[0][&#39;Key&#39;]<br>    data = json.loads(s3.get_object(Bucket=BUCKET_NAME, Key=latest)[&#39;Body&#39;].read())<br><br>    summary = {<br>        &quot;timestamp&quot;: now,<br>        &quot;summary&quot;: f&quot;{len(data)} idle resources found. Recommend verifying EC2 tags before shutdown.&quot;<br>    }<br><br>    ai_key = f&quot;{ENV}/ai_summaries/summary_{now}.json&quot;<br>    s3.put_object(Bucket=BUCKET_NAME, Key=ai_key, Body=json.dumps(summary))<br>    sns.publish(TopicArn=SNS_TOPIC, Message=json.dumps(summary), Subject=&quot;AI Insights&quot;)<br><br>    return {&quot;statusCode&quot;: 200, &quot;body&quot;: json.dumps(summary)}</pre><p>adding these 3 scripts into our lambda folder with the structure given above will now allow us to move onto the next step whihc is to test everything:</p><p>in our lambdas/scanner/tests/ we will create a fille called test_scanner.py and input the following script:</p><pre>from moto import mock_aws<br>import boto3, json, os<br>from importlib import reload<br>import lambdas.scanner.main as scanner_main<br><br>@mock_aws<br>def test_lambda():<br>    # --- Set environment variables before re-loading the module ---<br>    os.environ[&quot;RESOURCE_TABLE&quot;] = &quot;test-table&quot;<br>    os.environ[&quot;REPORT_BUCKET&quot;] = &quot;test-reports&quot;<br>    os.environ[&quot;ENVIRONMENT&quot;] = &quot;staging&quot;<br><br>    # Re-import to ensure the env vars are picked up inside main.py<br>    reload(scanner_main)<br>    from lambdas.scanner.main import lambda_handler<br><br>    # --- Create mock AWS resources ---<br>    dynamodb = boto3.client(&quot;dynamodb&quot;, region_name=&quot;us-east-1&quot;)<br>    s3 = boto3.client(&quot;s3&quot;, region_name=&quot;us-east-1&quot;)<br><br>    s3.create_bucket(Bucket=&quot;test-reports&quot;)<br>    dynamodb.create_table(<br>        TableName=&quot;test-table&quot;,<br>        KeySchema=[{&quot;AttributeName&quot;: &quot;resource_id&quot;, &quot;KeyType&quot;: &quot;HASH&quot;}],<br>        AttributeDefinitions=[{&quot;AttributeName&quot;: &quot;resource_id&quot;, &quot;AttributeType&quot;: &quot;S&quot;}],<br>        BillingMode=&quot;PAY_PER_REQUEST&quot;<br>    )<br><br>    # --- Run Lambda handler ---<br>    result = lambda_handler({}, {})<br>    assert result[&quot;statusCode&quot;] == 200<br>    body = json.loads(result[&quot;body&quot;])<br>    assert &quot;findings&quot; in body</pre><p>now we can run:</p><pre>python -m pytest -v</pre><p>and we should get passed tests results.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/955/1*T3UMDIaNdJyaELaVY8y0nA.png" /></figure><p>In this phase, we transformed our Terraform-defined environment into an intelligent, serverless automation layer. By developing three modular Lambda functions — <strong>Scanner</strong>, <strong>Action</strong>, and <strong>AI-Insights</strong> — we gave the system the ability to detect, respond, and reason about cloud resource usage. Each function was written and tested locally with <strong>Python 3.12</strong>, <strong>Boto3</strong>, and optionally <strong>Moto</strong> for safe simulations, ensuring correctness before deployment. Together, these Lambdas form the engine of our cost-optimization pipeline, turning raw AWS metrics into actionable operations and AI-generated insights. With the serverless logic now complete and verified, we’re ready to move on to <strong>Phase 3: Containerization and Deployment with Docker &amp; ECR.</strong></p><h3><strong>Phase 3: Containerization &amp; Deployment via Docker + ECR</strong></h3><p>In this phase, we package each Lambda function into lightweight, production-ready Docker images and store them in <strong>Amazon Elastic Container Registry (ECR)</strong>. Instead of uploading ZIP files directly to Lambda, this approach ensures greater consistency, dependency control, and version traceability — mirroring how modern serverless applications are deployed in real environments. Each function (Scanner, Action, and AI-Insights) is built from its own Dockerfile using <strong>Python 3.12 as the base image</strong>, tested locally, and then pushed to its respective ECR repository. By referencing these container images in Terraform, AWS Lambda will now pull the latest image versions automatically during deployment. This containerized workflow not only simplifies dependency management but also makes the Lambdas portable, reproducible, and aligned with real-world CI/CD practices.</p><p>We will add 3 Dockerfiles to our lambda directories:</p><pre>FROM public.ecr.aws/lambda/python:3.12<br><br>COPY main.py ${LAMBDA_TASK_ROOT}<br>COPY requirements.txt ./<br><br>RUN pip install -r requirements.txt --target &quot;${LAMBDA_TASK_ROOT}&quot;<br><br>CMD [&quot;main.lambda_handler&quot;]</pre><p>Once we do that we can build our Docker images locally so we have our custom images built from our base image of public.ecr.aws/lambda/python:3.12:</p><pre>docker build -t scanner-lambda ./lambdas/scanner</pre><p>Now we can authenticate as well as push to ECR (Elastic Container Registry — AWS’ managed container registry similar to docker hub but native to AWS).</p><pre>aws ecr get-login-password --region &lt;region&gt; | docker login --username AWS --password-stdin &lt;account-id&gt;.dkr.ecr.us-east-1.amazonaws.com</pre><p>We can replace &lt;account-id&gt; with our actual AWS account, in my case:</p><pre>aws ecr get-login-password --region us-east-1 | docker login --username AWS --password-stdin 982081047984.dkr.ecr.us-east-1.amazonaws.com</pre><p>After making sure we have created that connection, we can then create the ECR repositories remotely:</p><pre>aws ecr create-repository --repository-name scanner-lambda</pre><p>Now we can create an alias for the local image because <strong>tagging maps the local Docker image to its destination in Amazon ECR</strong>, allowing AWS to recognize where the image should be stored. This ensures that when we push the image, the correct version is uploaded to the intended repository, maintaining consistency between local builds and deployed Lambda functions. After that we can push to the remote.</p><pre>docker tag scanner-lambda:latest 982081047984.dkr.ecr.us-east-1.amazonaws.com/scanner-lambda:latest<br>docker push 982081047984.dkr.ecr.us-east-1.amazonaws.com/scanner-lambda:latest</pre><p>After this we can update our Terraform module to deploy using IaC which defines the <strong>Scanner Lambda function</strong> to run from a <strong>container image</strong> stored in <strong>Amazon ECR</strong>, rather than a ZIP package. It sets the function name dynamically per environment, assigns an <strong>IAM execution role</strong>, and configures a <strong>300-second timeout</strong>. The Lambda receives environment variables pointing to the <strong>DynamoDB table</strong>, <strong>S3 bucket</strong>, and current <strong>environment name</strong>, enabling it to interact with the correct resources for each deployment.</p><p>Once we do that we can do our usual terraform apply and we should have all our resources (ECR repositories and Lambda functions) created in our AWS console.</p><p>*IMPORTANT*</p><p>If we don’t see our Lambda functions or ECR images appear in the AWS console after running terraform apply, two common issues may be the cause. The first is <strong>PowerShell pipe behavior</strong> — unlike Linux terminals, PowerShell can sometimes mishandle the pipe (|) used in the ECR login command, causing authentication to fail silently. Running the same command in an elevated PowerShell session or using <strong>cmd.exe</strong> often resolves it. The second issue involves <strong>Docker BuildKit</strong>, which is enabled by default in newer Docker versions. BuildKit sometimes produces image manifests in a format that <strong>AWS Lambda doesn’t support</strong>, leading to errors like <em>“The image manifest or layer media type is not supported.”</em> Disabling BuildKit temporarily with:</p><pre>$env:DOCKER_BUILDKIT=0</pre><p>before rebuilding ensures the image uses a compatible schema for Lambda.</p><p>After this doing terraform apply should work.</p><p>In this phase, we transitioned our Lambda functions from traditional ZIP-based deployments to <strong>containerized images hosted on Amazon ECR</strong>, aligning our workflow with modern, production-grade DevOps standards. Each Lambda — Scanner, Action, and AI-Insights — was packaged into its own Docker image, built locally, and pushed to a dedicated ECR repository. We then updated our Terraform configuration to deploy these container-based Lambdas directly, ensuring version consistency and easy rollbacks. Along the way, we addressed common deployment challenges such as PowerShell piping issues during ECR authentication and Docker BuildKit’s manifest incompatibility with Lambda. With containerization complete, our serverless functions are now <strong>fully portable, reproducible, and ready for automated CI/CD deployment.</strong></p><h3><strong>Phase 4 — Continuous Integration &amp; Deployment with GitHub Actions</strong></h3><p>With our infrastructure and containerized Lambda functions in place, the next step is to <strong>automate the build, test, and deployment process</strong> using <strong>GitHub Actions</strong>. This phase establishes a seamless CI/CD pipeline that ensures every code change is validated, built, and deployed consistently across environments. The CI workflow runs on pull requests, executing unit tests, linting, and building Docker images before pushing them to <strong>Amazon ECR</strong>. The CD workflow, triggered on merges to the main branch, automatically runs Terraform to apply infrastructure changes and update Lambda functions with the latest image tags. By integrating GitHub Actions, we remove manual deployment steps, maintain continuous delivery discipline, and guarantee that our AWS environment always reflects the most up-to-date, tested version of our cost-optimization system.</p><p>We’ll make two files in our .github/workflows/ directory, which will be:</p><pre>cloud-cost-optimizer/<br>│<br>└── .github/<br>    └── workflows/<br>        ├── ci.yml<br>        └── cd.yml</pre><p>ci.yml will be for the build process, it handles <strong>Continuous Integration (CI)</strong> — the build and validation stage of our pipeline. It ensures that every pull request triggers automated checks like linting, unit testing, and Docker image builds, guaranteeing that only verified, production-ready code is pushed to <strong>Amazon ECR</strong> before deployment.</p><p>cd.yml handles <strong>Continuous Deployment (CD)</strong> — the release and delivery stage of our pipeline. It runs automatically on merges to the main branch, executing Terraform commands to apply infrastructure updates and redeploy the latest Lambda container images, ensuring production always reflects the newest stable build.</p><p>Our workflow files will be basic but powerful:</p><p>ci.yml:</p><pre>name: CI - Build and Push Lambda Images<br><br>on:<br>  pull_request:<br>    branches:<br>      - main<br><br>jobs:<br>  build-and-push:<br>    runs-on: ubuntu-latest<br><br>    steps:<br>      - name: Checkout repo<br>        uses: actions/checkout@v4<br><br>      - name: Configure AWS credentials<br>        uses: aws-actions/configure-aws-credentials@v4<br>        with:<br>          aws-access-key-id: ${{ secrets.AWS_ACCESS_KEY_ID }}<br>          aws-secret-access-key: ${{ secrets.AWS_SECRET_ACCESS_KEY }}<br>          aws-region: us-east-1<br><br>      - name: Log in to Amazon ECR<br>        uses: aws-actions/amazon-ecr-login@v2<br><br>      - name: Build and push Lambda images<br>        run: |<br>          IMAGE_TAG=${{ github.sha }}<br>          for FUNC in scanner action ai_insights; do<br>            if [ &quot;$FUNC&quot; = &quot;ai_insights&quot; ]; then<br>              docker build -t ai_insights-lambda ./lambdas/ai_insights<br>              docker tag ai_insights-lambda 982081047984.dkr.ecr.us-east-1.amazonaws.com/ai-insights-lambda:$IMAGE_TAG<br>              docker push 982081047984.dkr.ecr.us-east-1.amazonaws.com/ai-insights-lambda:$IMAGE_TAG<br>            else<br>              echo &quot;Skipping $FUNC&quot;<br>              docker build -t $FUNC-lambda ./lambdas/$FUNC<br>              docker tag $FUNC-lambda:latest 982081047984.dkr.ecr.us-east-1.amazonaws.com/$FUNC-lambda:$IMAGE_TAG<br>              docker push 982081047984.dkr.ecr.us-east-1.amazonaws.com/$FUNC-lambda:$IMAGE_TAG<br>            fi<br>          done</pre><p>This <strong>CI workflow</strong> automates the build and publishing process for our Lambda container images whenever a pull request targets the main branch. It first checks out the repository, configures AWS credentials, and logs into <strong>Amazon ECR</strong>. Then, for each Lambda function directory, it builds a Docker image, tags it with the unique Git commit SHA for version traceability, and pushes it to its respective <strong>ECR repository</strong>, ensuring that every code change produces a traceable, deployable image ready for release.</p><ul><li>The conditional in the loop is there due to the the naming of the local files, and the ECR repository.</li></ul><p>cd.yml:</p><pre>name: CD - Terraform Deploy<br><br>on:<br>  push:<br>    branches:<br>      - main<br><br>jobs:<br>  deploy:<br>    runs-on: ubuntu-latest<br><br>    steps:<br>      - name: Checkout repo<br>        uses: actions/checkout@v4<br><br>      - name: Setup Terraform<br>        uses: hashicorp/setup-terraform@v3<br>        with:<br>          terraform_version: 1.8.5<br><br>      - name: Configure AWS credentials<br>        uses: aws-actions/configure-aws-credentials@v4<br>        with:<br>          aws-access-key-id: ${{ secrets.AWS_ACCESS_KEY_ID }}<br>          aws-secret-access-key: ${{ secrets.AWS_SECRET_ACCESS_KEY }}<br>          aws-region: us-east-1<br><br>      - name: Terraform Init<br>        run: cd terraform &amp;&amp; terraform init -input=false<br><br>      - name: Terraform Plan<br>        run: cd terraform &amp;&amp; terraform plan -input=false -out=tfplan<br><br>      - name: Terraform Apply<br>        run: |<br>          cd terraform<br>          terraform import aws_dynamodb_table.resource_tracker resource-tracker-staging<br>          terraform import aws_iam_role.lambda_execution_role lambda-exec-staging<br>          terraform import aws_s3_bucket.reports levelup-reports-staging<br>          terraform import aws_s3_bucket.logs levelup-logs-staging<br>          terraform import aws_s3_bucket.ai_summaries levelup-ai-summaries-staging<br>          terraform apply -input=false -auto-approve tfplan</pre><p>This <strong>CD workflow</strong> automates infrastructure deployment whenever changes are merged into the main branch. It checks out the latest code, sets up Terraform, and configures AWS credentials for authenticated operations. The workflow then initializes Terraform, runs a plan to preview changes, imports any pre-existing resources to maintain state consistency, and finally applies the plan — ensuring that the latest infrastructure and Lambda image updates are automatically deployed to AWS in a controlled, repeatable way.</p><ul><li>The terraform import statements are present so that we don’t get a “resource already exists” error.</li></ul><p>After this we can add our GitHub repo secrets:</p><ul><li>AWS_ACCESS_KEY_ID</li><li>AWS_SECRET_ACCESS_KEY</li><li>AWS_REGION</li></ul><p>Once we do that we can test the workflow by creating a branch and pushing our code to the remote on that branch (this will trigger our CI workflow). And after we create a PR and merge into the main branch and if everything checks out and works fine we have successfully set up a full CI/CD pipeline for our application.</p><p>In this phase, we established a <strong>CI/CD pipeline with GitHub Actions</strong> to automate building, testing, and deploying our cloud cost optimization system. The <strong>CI workflow</strong> builds and pushes Lambda container images to Amazon ECR on every pull request, ensuring versioned, validated code artifacts. The <strong>CD workflow</strong> runs Terraform automatically on merges to main, applying infrastructure updates and redeploying the latest images — achieving a seamless, production-grade continuous delivery process.</p><h3><strong>Phase 5: AI Insights Integration with Amazon Bedrock</strong></h3><p>In this phase, the project evolved from pure automation into an <strong>intelligent, insight-driven system</strong>. By integrating <strong>Amazon Bedrock</strong> into the AI-Insights Lambda, the pipeline now transforms raw resource data into clear, executive-level summaries and recommendations. Instead of static JSON reports, each run now produces concise, AI-generated insights on cost trends, idle resources, and optimization opportunities. This addition bridges the gap between automation and analysis — enabling decision-makers to understand cloud efficiency at a glance.</p><p>We’ll start off by creating an IAM policy that will allow out Labda function to securely interact with Bedrock:</p><pre>resource &quot;aws_iam_role_policy&quot; &quot;lambda_bedrock_policy&quot; {<br>  name = &quot;lambda-bedrock-policy-${var.environment}&quot;<br>  role = aws_iam_role.lambda_execution_role.id<br><br>  policy = jsonencode({<br>    Version = &quot;2012-10-17&quot;<br>    Statement = [<br>      {<br>        Effect = &quot;Allow&quot;<br>        Action = [<br>          &quot;bedrock:InvokeModel&quot;<br>        ]<br>        Resource = &quot;*&quot;<br>      }<br>    ]<br>  })<br>}</pre><p>This will ensure no permission issues when invoking the Amazon Titan model.</p><p>After this we extended the Lambda’s environment configuration within <strong>Terraform</strong> to make it fully dynamic and environment-aware. Key variables such as <strong>S3_BUCKET</strong>, <strong>SNS_TOPIC_ARN</strong>, and <strong>ENV</strong> ensure that each deployment automatically connects to the correct resources for its workspace. Parameters like <strong>MODEL_ID</strong>, <strong>SUMMARY_TOP_N</strong>, and <strong>SUMMARY_RECENCY_WINDOW</strong> give the AI-Insights Lambda precise control over which Bedrock model to use and how much historical data to summarize. This modular setup allows the function to adapt seamlessly across staging and production without any code changes, maintaining flexibility and consistency in deployment.</p><p>Then we altered our Lambda function to actually invoke the <strong>Amazon Bedrock service</strong>, leveraging the <strong>Amazon Titan</strong> model for natural-language summarization. Instead of simply logging or returning raw JSON data, the function now reads recent cost reports from S3, constructs a contextual prompt describing idle resources and potential savings, and sends it directly to Bedrock’s <strong>text-generation API</strong>. The Titan model processes this input and produces a concise, executive-style summary highlighting key findings and recommended actions. This enhancement effectively adds an intelligence layer to the pipeline — turning raw operational data into clear, AI-driven insights that can be shared with anyone.</p><p>Of course the script is stored in an image on our ECR respository but the code itself has the following updates:</p><pre>def lambda_handler(event, context):<br>    timestamp = datetime.datetime.utcnow().isoformat()<br><br>    # Get latest reports from S3<br>    prefix = f&quot;{ENV}/reports/&quot;<br>    objects = s3.list_objects_v2(Bucket=S3_BUCKET, Prefix=prefix)<br>    if &quot;Contents&quot; not in objects:<br>        return {&quot;status&quot;: &quot;no reports found&quot;}<br><br>    recent_keys = sorted([o[&quot;Key&quot;] for o in objects[&quot;Contents&quot;]])[-SUMMARY_TOP_N:]<br>    summaries = []<br>    for key in recent_keys:<br>        obj = s3.get_object(Bucket=S3_BUCKET, Key=key)<br>        summaries.append(obj[&quot;Body&quot;].read().decode(&quot;utf-8&quot;))<br><br>    # Build prompt<br>    prompt = (<br>        &quot;You are a cloud cost optimization assistant.\n&quot;<br>        &quot;Summarize these recent scan results for executives.\n&quot;<br>        &quot;Focus on key idle resources, cost savings, and recommended next actions.\n\n&quot;<br>        f&quot;{summaries}\n&quot;<br>        &quot;Write the response in under 250 words, bullet-first.&quot;<br>    )<br><br>    # Call Bedrock model<br>    response = bedrock.invoke_model(<br>        modelId=MODEL_ID,<br>        contentType=&quot;application/json&quot;,<br>        accept=&quot;application/json&quot;,<br>        body=json.dumps({&quot;inputText&quot;: prompt})<br>    )<br><br>    result = json.loads(response[&quot;body&quot;].read())<br>    summary_text = result.get(&quot;results&quot;, [{}])[0].get(&quot;outputText&quot;, &quot;[No summary returned]&quot;)<br><br>    # Store to S3<br>    key = f&quot;{ENV}/ai_summaries/summary_{timestamp}.json&quot;<br>    s3.put_object(<br>        Bucket=S3_BUCKET,<br>        Key=key,<br>        Body=json.dumps({&quot;summary&quot;: summary_text}, indent=2)<br>    )<br><br>    # Publish to SNS<br>    sns.publish(<br>        TopicArn=SNS_TOPIC_ARN,<br>        Message=summary_text,<br>        Subject=f&quot;AI Summary ({ENV})&quot;<br>    )<br><br>    return {&quot;status&quot;: &quot;summary generated&quot;, &quot;key&quot;: key}</pre><p>After this rebuilding our docker image with he new code and pushing to the the ECR repository is the next step:</p><pre>docker build -t ai-insights-lambda .<br>docker tag ai-insights-lambda:latest 982081047984.dkr.ecr.us-east-1.amazonaws.com/ai-insights-lambda:latest<br>aws ecr get-login-password --region us-east-1 | docker login --username AWS --password-stdin 982081047984.dkr.ecr.us-east-1.amazonaws.com<br>docker push 982081047984.dkr.ecr.us-east-1.amazonaws.com/ai-insights-lambda:latest</pre><ul><li>We will build the image and assign a name to it.</li><li>Then we will tag the local with the remote to setup the connection</li><li>After that we will login to our remote ECR repository (if we have been timed out)</li><li>And finally we will push the script to the repository</li></ul><p>And then with everything in place we know that our script is up to date and therefore our container will allow the lambda function to invoke it properly. We can test via:</p><pre>aws lambda invoke --function-name ai-insights-lambda-staging out.json</pre><p>If we get a success output log, that means we have done everything right.</p><p>In this phase, we containerized the updated <strong>AI-Insights Lambda</strong> and deployed it using <strong>Amazon ECR</strong> as the source for its runtime image. The Docker image was rebuilt with the latest Python code, tagged to match the ECR repository, and pushed securely after authenticating with AWS. We temporarily disabled <strong>Docker BuildKit</strong> to prevent schema compatibility issues with AWS Lambda’s image format. Finally, by reapplying <strong>Terraform</strong>, the Lambda function was seamlessly redeployed with the new container, completing the integration of the AI-powered insights workflow.</p><h3><strong>Summary</strong></h3><p>This project demonstrates a complete <strong>end-to-end DevOps workflow</strong> that combines Infrastructure as Code (IaC), containerized serverless functions, continuous integration and deployment (CI/CD), and intelligent automation through <strong>AWS Bedrock</strong>. Using <strong>Terraform</strong>, we provisioned the foundational cloud infrastructure — including <strong>S3</strong>, <strong>DynamoDB</strong>, <strong>IAM</strong>, and <strong>Lambda</strong> — ensuring that all components were reproducible, version-controlled, and environment-aware. Each Lambda function was packaged into <strong>Docker images</strong>, pushed to <strong>Amazon ECR</strong>, and deployed seamlessly into AWS Lambda using Terraform, achieving consistent and scalable deployments.</p><p>On the serverless side, three Lambda functions orchestrate the system’s logic: the <strong>Scanner Lambda</strong> identifies idle or underutilized AWS resources and records them; the <strong>Action Lambda</strong> acts on those findings by performing cost-saving measures such as stopping EC2 instances; and the <strong>AI Insights Lambda</strong> leverages <strong>Amazon Bedrock</strong> to analyze recent reports and generate executive-level summaries stored in S3 and broadcast via <strong>SNS</strong>. These components form a self-sustaining feedback loop — detecting, acting, and reporting on cost optimization opportunities automatically.</p><p>Finally, a <strong>CI/CD pipeline</strong> built with <strong>GitHub Actions</strong> automates every build and deployment step, from Docker image creation to Terraform updates. Together, these layers demonstrate how modern DevOps practices, when combined with <strong>serverless computing and AI integration</strong>, can deliver intelligent cloud automation. The project not only optimizes AWS resource usage but also embodies the larger goal of building <strong>self-managing, insight-driven cloud architectures</strong> — scalable, observable, and intelligent from code to execution.</p><p>*Drop this article a clap if you enjoyed the read*</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=169a871a0311" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Full IaC DevOps Pipeline: S3 + CloudFront + GitHub Actions for a personal portfolio]]></title>
            <link>https://medium.com/@abdulshaikhr99/full-iac-devops-pipeline-s3-cloudfront-github-actions-for-a-personal-portfolio-68e6ca040a44?source=rss-c47a96f96b0a------2</link>
            <guid isPermaLink="false">https://medium.com/p/68e6ca040a44</guid>
            <category><![CDATA[cloud-computing]]></category>
            <category><![CDATA[azure]]></category>
            <category><![CDATA[aws]]></category>
            <category><![CDATA[github-actions]]></category>
            <category><![CDATA[devops]]></category>
            <dc:creator><![CDATA[Abdul Rahim Shaikh]]></dc:creator>
            <pubDate>Mon, 13 Oct 2025 21:02:54 GMT</pubDate>
            <atom:updated>2025-10-13T21:02:54.551Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*pDg7fdJFSSYT-3Aairiy9Q.png" /></figure><p>Every developer eventually reaches that moment when they want to take their portfolio to the next level, not just host it, but <strong>deploy it like a real production application</strong>. I wanted my React portfolio to reflect the same DevOps principles I use in real projects: Infrastructure as Code, automated CI/CD, and scalable cloud hosting. So I decided to build a <strong>fully automated AWS pipeline</strong> using Terraform, GitHub Action, S3, and CloudFront, a setup that deploys my site automatically every time I push to GitHub. In this article, I’ll walk you through how I built it step-by-step, from provisioning infrastructure with Terraform to setting up continuous delivery using AWS-native services.</p><h3><strong>Phase 1 — Project Setup</strong></h3><p>We’ll start off by creating our Terraform project, this is where all of our infrastructure will be defined as code. Instead of manually setting up services in the AWS console, Terraform allows us to describe each resource (like S3 buckets, CloudFront distributions, and IAM roles) in configuration files that can be version-controlled and reused. This approach not only keeps the setup consistent across environments but also makes it easy to automate deployments or rebuild the entire stack with a single command. In this phase, we’ll initialize our Terraform workspace, connect it to AWS through the CLI, and prepare the core project structure that will serve as the foundation for our automated pipeline.</p><p>We will have this project setup:</p><pre>terraform/<br> ├── cloudfront.tf<br> ├── iam.tf<br> ├── main.tf<br> ├── outputs.tf<br> ├── provider.tf<br> ├── s3.tf<br> └── variables.tf</pre><p>Our code setup will look like the following:</p><p>provider.tf</p><pre>terraform {<br>  required_version = &quot;&gt;= 1.6.0&quot;<br>  required_providers {<br>    aws = {<br>      source  = &quot;hashicorp/aws&quot;<br>      version = &quot;~&gt; 5.0&quot;<br>    }<br>  }<br>}<br><br>provider &quot;aws&quot; {<br>  region = var.aws_region<br>}</pre><p>variables.tf</p><pre>variable &quot;aws_region&quot; {<br>  description = &quot;AWS region for all resources&quot;<br>  type        = string<br>  default     = &quot;us-east-1&quot;<br>}</pre><p>We have setup our basic boilerplate code for the version, provider and region.</p><p>After doing a terraform init we should see the success message:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/725/1*ub_kpgUCSQBkTYF4x5MgSg.png" /></figure><p>For this project we are keeping our terraform source code and application source code in separate repositories to replicate enterprise level scenarios of having separate pipelines.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*9w_wBe_mx8oOktzNf5nGow.png" /></figure><p>Once our terraform code is on a remote repo, we can move onto the next step.</p><h3>Phase 2— Terraform Setup</h3><p>Now we move into Phase 2: Infrastructure Provisioning, where we define the foundation of our hosting environment using Terraform. In this stage, we set up the S3 bucket that will serve our built React app, configure CloudFront as a global content delivery network (CDN) for faster and more secure distribution. By defining these resources in Terraform, we ensure our infrastructure is consistent, version-controlled, and easily reproducible. Once we run terraform init, terraform plan, and terraform apply, Terraform provisions the stack and outputs a CloudFront domain, your first live endpoint hosted entirely on AWS. At this point, you’ll have a working S3 + CloudFront setup and a CloudFront URL serving your static site globally.</p><p>We can start adding to our terraform infra by creating a couple of new files:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/219/1*64E3Bj8pc8v2B8RzyeeraQ.png" /></figure><p>For now these are the file names with the resources we will have in our infra.</p><p>Once we include the code for our:</p><ul><li>S3 Bucket</li><li>CloudFront distribution</li><li>IAM user and role</li></ul><p>We will have our Terraform configuration in place we will have the infrastructure for the portfolio app, and we did it entirely through code. At the core of this setup is an <strong>Amazon S3 bucket</strong>, which stores all the static assets from our React build: the HTML, CSS, and JavaScript files. To keep things secure, the bucket blocks all public access, meaning no one can directly reach it , not even by guessing the URL.</p><p>Instead, access is funneled through <strong>Amazon CloudFront</strong>, AWS’s content delivery network (CDN). CloudFront acts as the public face of our app, caching files at edge locations across the world and delivering them quickly to users no matter where they are. Behind the scenes, we use an <strong>Origin Access Identity (OAI)</strong> , a special credential that lets only CloudFront read from our private S3 bucket. This gives us a perfect balance of speed and security, with HTTPS support out of the box.</p><p>Finally, Terraform outputs key values like the <strong>S3 bucket name</strong>, <strong>CloudFront domain</strong>, and <strong>distribution ID</strong>. These will come in handy in the next phase when we connect our CI/CD pipeline, allowing future deployments from GitHub Actions to automatically upload new builds and trigger CloudFront cache invalidations.</p><p>In other words, we’ve now established a <strong>fully automated, production-ready foundation</strong>: Terraform provisions the resources, S3 handles storage, CloudFront ensures global performance, and our upcoming pipeline will take care of deployment.</p><h3>Phase 3— CI/CD Automation with GitHub Actions</h3><p>In this phase, we bring automation into the picture by connecting our <strong>GitHub repository</strong> directly to the AWS infrastructure we just built. The goal here is to eliminate manual deployments entirely, every time we push new code to the main branch, the application will automatically build, upload to the S3 bucket, and refresh across CloudFront’s global network. This is powered by <strong>GitHub Actions</strong>, which will handle the continuous integration and deployment (CI/CD) process. With this setup, updates to your portfolio become seamless, you focus on writing code, and AWS takes care of building, deploying, and delivering it to users worldwide.</p><p>First we’ll start off by creating an IAM user which will be used for our GitHub Actions pipeline.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*r7p7bDjdCTWv17tN4iPQBw.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*NMXFKXYtT0S3J7l2dGHkxg.png" /></figure><p>We will attach this inline policy:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/437/1*tScUX91Sgxr7-BRIpAi_sQ.png" /></figure><p>Attach this policy to our newly created IAM user and we should have:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*mwZh0ag_cCdSnjsoNhguKw.png" /></figure><p>After this we can create and record an access key and secret access keys of this user and create repository secrets which will correspond to the access key and secret access key in our GitHub application code:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/967/1*oGoqn50kVJ-2AyTfsFqSzA.png" /></figure><p>Doing the above step will allow GitHub to build and authenticate a connection to AWS to validate the user being used.</p><p>Now let’s add our .github/workflows/deploy.yaml file, which will define and orchestrate our deployment pipeline.</p><pre>name: Deploy Portfolio to AWS<br><br>on:<br>  workflow_dispatch<br><br>jobs:<br>  deploy:<br>    runs-on: ubuntu-latest<br><br>    steps:<br>      - name: Checkout repository<br>        uses: actions/checkout@v4<br><br>      - name: Setup Node.js<br>        uses: actions/setup-node@v4<br>        with:<br>          node-version: 18<br><br>      - name: Install dependencies<br>        run: npm ci<br><br>      - name: Build project<br>        run: npm run build<br><br>      - name: Configure AWS credentials<br>        uses: aws-actions/configure-aws-credentials@v4<br>        with:<br>          aws-access-key-id: ${{ secrets.AWS_ACCESS_KEY_ID }}<br>          aws-secret-access-key: ${{ secrets.AWS_SECRET_ACCESS_KEY }}<br>          aws-region: us-east-1<br><br>      - name: Deploy to S3<br>        run: aws s3 sync dist/ s3://abdul-portfolio-hosting --delete<br><br>      - name: Invalidate CloudFront cache<br>        run: |<br>          CLOUDFRONT_ID=$(aws cloudfront list-distributions --query &quot;DistributionList.Items[?Comment==&#39;CloudFront distribution for Abdul Shaikh portfolio&#39;].Id&quot; --output text)<br>          aws cloudfront create-invalidation --distribution-id $CLOUDFRONT_ID --paths &quot;/*&quot;</pre><p>We can use the on: workflow dispatch to trigger the workflow manually in the GitHub UI instead of having it run automatically for just any simple push to the repo:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*HhORA2rlvcsCTu9FD52N9A.png" /></figure><p>The “Run workflow” option is what we can use to trigger a workflow.</p><p>In this case we are getting failed runs because of some react issues. The react-tilt library we’re using is very old, it only supports React 15 or 16, but the app uses React 18, so npm can’t resolve the dependency tree.</p><p>so we will replace:</p><pre>- name: Install dependencies<br>  run: npm ci </pre><p>to</p><pre>- name: Install dependencies<br>  run: npm ci --legacy-peer-deps</pre><p>Once we do this we should have a successful run</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Lqr0P5ZW79m9fUms8bVTNQ.png" /></figure><p>After we trigger the workflow, if everything checks out, we should see a successful deployment run in GitHub Actions.</p><p>We’ve now introduced <strong>GitHub Actions</strong>, a powerful automation tool that enables <strong>Continuous Integration and Continuous Deployment (CI/CD)</strong> directly from your GitHub repository. It eliminates the need for manual deployments by automatically building, testing, and deploying your application whenever you push new code. In our case, GitHub Actions connects the portfolio’s source code to the AWS infrastructure provisioned with Terraform, deploying the latest build to the S3 hosting bucket and refreshing CloudFront’s global cache. This setup ensures that every update is delivered automatically and consistently, allowing you to focus solely on development while GitHub Actions handles the entire deployment process from start to finish.</p><h3>Recap</h3><p>Building this automated portfolio deployment pipeline was more than just a DevOps exercise, it was about creating a <strong>living, self-updating representation</strong> of my work. Instead of manually rebuilding and uploading my site each time, I now have a fully automated system that handles everything from code commits to production deployment. Every push to GitHub triggers a clean, repeatable process powered by <strong>Terraform</strong>, <strong>AWS GitHub Actions</strong>, seamlessly updating my site on <strong>S3</strong> and distributing it globally through <strong>CloudFront</strong>.</p><p>Having a <strong>personal portfolio</strong> isn’t just about showcasing projects, it’s about showing how you think as an engineer. In my case, I wanted mine to reflect the principles I apply professionally: <strong>automation, scalability, and infrastructure as code</strong>. The tools I used — AWS for the cloud backbone, Terraform for provisioning, and CI/CD pipelines for continuous delivery, all tie into that mindset.</p><p>This project demonstrates how a simple React app can evolve into a <strong>production-grade, globally distributed application</strong> managed entirely through DevOps practices. It’s a testament to how even personal projects can (and should) be built with the same rigor and automation as enterprise systems, because at the end of the day, your portfolio isn’t just what people see, it’s how you built it that truly defines you as a developer.</p><p>GitHub repo: <a href="https://github.com/AbdulRahimShaikh1999/portfolio">https://github.com/AbdulRahimShaikh1999/portfolio</a></p><p>Hosted site: <a href="https://tinyurl.com/25x5xbf6">https://tinyurl.com/25x5xbf6</a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=68e6ca040a44" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[CI/CD for Prompt Engineering: Deploying Bedrock-Generated Content to S3 with GitHub Actions]]></title>
            <link>https://medium.com/@abdulshaikhr99/ci-cd-for-prompt-engineering-deploying-bedrock-generated-content-to-s3-with-github-actions-2a615bf7dc36?source=rss-c47a96f96b0a------2</link>
            <guid isPermaLink="false">https://medium.com/p/2a615bf7dc36</guid>
            <category><![CDATA[aws]]></category>
            <category><![CDATA[cloud]]></category>
            <category><![CDATA[serverless]]></category>
            <category><![CDATA[ci-cd-pipeline]]></category>
            <category><![CDATA[devops]]></category>
            <dc:creator><![CDATA[Abdul Rahim Shaikh]]></dc:creator>
            <pubDate>Wed, 25 Jun 2025 04:55:47 GMT</pubDate>
            <atom:updated>2025-06-25T04:55:47.273Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*JGXDZaIX-uPeBVb4q6kN6w.png" /></figure><p>As AI-generated content becomes a core asset in digital education, operationalizing prompt engineering workflows has become critical. This proof of concept (POC) outlines the implementation of a fully automated content deployment pipeline for Pixel Learning Co., an education technology startup leveraging foundation models to deliver onboarding messages, learning tips, and course summaries at scale.</p><p>The solution integrates Amazon Bedrock for real-time inference using the Claude Sonnet model and GitHub Actions for CI/CD orchestration. Prompt templates and variable configurations are stored and version-controlled in GitHub, with content automatically rendered, reviewed, and published to static S3-hosted websites. Distinct beta and production workflows allow for safe staging and controlled promotion of generated content via structured branch-based deployments.</p><p>By combining Bedrock’s serverless inference capabilities with GitHub-native automation, this approach offers a scalable, developer-friendly system for managing content generation with reproducibility, traceability, and minimal operational overhead.</p><p>Let’s get started</p><h3><strong>1. Create GitHub Repo</strong></h3><p>The first step is to create a GitHub Repository that we can push our code to:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*uI5ziNXVT2X4JCGWV2Rjwg.png" /></figure><p>This repository automates the end-to-end workflow of generating content using Amazon Bedrock and deploying it to static S3-hosted websites. By combining prompt templates with structured variables, the system dynamically renders prompts, invokes real-time Bedrock inference, and uploads the generated outputs — such as onboarding messages or course summaries — as .html or .md files to S3. The entire pipeline is orchestrated through GitHub Actions, enabling seamless CI/CD for prompt engineering with environment-specific deployment (beta vs prod) driven by pull requests and merges.</p><h3><strong>2. Set Up Local Project Structure</strong></h3><p>Now let’s setup our local project template with the following structure:</p><pre>.<br>├── prompt_templates/<br>│   └── welcome_email.txt<br>├── prompts/<br>│   └── welcome_prompt.json<br>├── outputs/<br>├── .github/<br>│   └── workflows/<br>│       ├── on_push.yaml<br>│       └── on_merge_with_approval.yaml<br>└── process_prompt.py</pre><p>Now our IDE will look like the following and is setup properly for our development.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/389/1*A0F0ZlE2hDVCIsVLE9GXHg.png" /></figure><p>This project uses a clean structure to manage prompt generation and deployment. Templates live in prompt_templates/, with variable data in prompts/. Generated outputs are saved to outputs/ before being uploaded to S3. CI/CD workflows in .github/workflows/ automate deployment to beta and prod environments, while process_prompt.py handles rendering, Bedrock invocation, and S3 uploads.</p><h3><strong>3. Write the Prompt Template &amp; Config</strong></h3><p>We will add these following values into our files. This step defines the <em>structure</em> and <em>content</em> of what will be generated by Amazon Bedrock.</p><pre>Write a friendly welcome message for {{ name }} who just joined the {{ course_name }} course. Their instructor is {{ instructor_name }}.</pre><pre>{<br>  &quot;name&quot;: &quot;Jordan&quot;,<br>  &quot;course_name&quot;: &quot;AI Fundamentals&quot;,<br>  &quot;instructor_name&quot;: &quot;Dr. Lin&quot;<br>}</pre><p>Here we created a reusable prompt template with placeholder variables (like {{ name }} and {{ course_name }}) using Jinja2 (which will be implemented in the process_prompt.py script) , and a corresponding JSON file that holds real values to fill in those placeholders. This setup allows us to dynamically personalize prompts without hardcoding anything. For example, we store the template in prompt_templates/ and the variables in prompts/, making it easy to reuse the same structure for different users or contexts. Later in the script, these files are loaded and combined to generate a complete, customized prompt for Amazon Bedrock to process.</p><h3><strong>4. Creating the Processing Script (</strong><strong>process_prompt.py)</strong></h3><p>The Python script we will create acts as the engine of our automation pipeline. It takes a <strong>prompt template</strong> and a <strong>JSON config file</strong> (containing dynamic user info like name, course, and instructor), and merges them using Jinja2 to generate a final, personalized prompt. For example, a welcome message for a student named Jordan in the AI Fundamentals course taught by Dr. Lin. Once rendered, the prompt is sent to <strong>Amazon Bedrock</strong>, where the Claude 3 Sonnet model generates a full response (e.g., a friendly onboarding email). This call is made securely via the boto3 SDK, using AWS credentials that are either stored locally or injected through GitHub Actions.</p><p>After receiving the AI-generated message from Bedrock, the script saves the output as an .html file inside an outputs/ folder. The filename reflects the deployment environment (beta or prod) and the user’s name, ensuring it&#39;s easy to organize and distinguish. It then determines the correct <strong>Amazon S3 bucket</strong> to use by checking an environment variable (DEPLOY_ENV) and selecting the corresponding secret (S3_BUCKET_BETA or S3_BUCKET_PROD). Finally, the HTML file is uploaded to the right location in the S3 bucket — under a key like beta/outputs/beta_welcome_jordan.html. This automation makes it easy to preview or publish content dynamically, as part of a larger CI/CD pipeline.</p><p>Our script will be the following:</p><pre>import os<br>import json<br>from jinja2 import Template<br>import boto3<br><br># === Step 1: Load template and variables ===<br>TEMPLATE_PATH = &quot;prompt_templates/welcome_email.txt&quot;<br>CONFIG_PATH = &quot;prompts/welcome_prompt.json&quot;<br><br>with open(TEMPLATE_PATH, &quot;r&quot;) as f:<br>    template_str = f.read()<br><br>with open(CONFIG_PATH, &quot;r&quot;) as f:<br>    variables = json.load(f)<br><br>template = Template(template_str)<br>final_prompt = template.render(variables)<br><br>print(&quot;Rendered Prompt:&quot;)<br>print(final_prompt)<br><br># === Step 2: Call Claude via Bedrock ===<br>json_body = json.dumps({<br>    &quot;anthropic_version&quot;: &quot;bedrock-2023-05-31&quot;,<br>    &quot;max_tokens&quot;: 1024,<br>    &quot;messages&quot;: [<br>        {<br>            &quot;role&quot;: &quot;user&quot;,<br>            &quot;content&quot;: f&quot;Human: {final_prompt}&quot;<br>        }<br>    ]<br>})<br><br>bedrock = boto3.client(&quot;bedrock-runtime&quot;, region_name=&quot;us-east-1&quot;)<br><br>response = bedrock.invoke_model(<br>    modelId=&quot;anthropic.claude-3-sonnet-20240229-v1:0&quot;,<br>    body=json_body,<br>    contentType=&quot;application/json&quot;,<br>    accept=&quot;application/json&quot;<br>)<br><br>response_body = json.loads(response[&quot;body&quot;].read())<br>bedrock_output = response_body[&quot;content&quot;][0][&quot;text&quot;]<br><br>print(&quot;\n🧠 Bedrock Response:&quot;)<br>print(bedrock_output)<br><br># === Step 3: Save response and upload to S3 ===<br>env = os.getenv(&quot;DEPLOY_ENV&quot;, &quot;beta&quot;)<br><br>bucket_name = os.getenv(&quot;S3_BUCKET_BETA&quot;) if env == &quot;beta&quot; else os.getenv(&quot;S3_BUCKET_PROD&quot;)<br><br>filename = f&quot;{env}_welcome_{variables[&#39;name&#39;].lower()}.html&quot;<br>output_path = f&quot;outputs/{filename}&quot;<br>s3_key = f&quot;{env}/outputs/{filename}&quot;<br><br>os.makedirs(&quot;outputs&quot;, exist_ok=True)<br><br>with open(output_path, &quot;w&quot;, encoding=&quot;utf-8&quot;) as f:<br>    f.write(bedrock_output)<br><br>print(f&quot;✅ Saved to {output_path}&quot;)<br>print(f&quot;🔍 Environment: {env}&quot;)<br>print(f&quot;📁 Filename: {filename}&quot;)<br>print(f&quot;🪣 S3 Bucket: {bucket_name}&quot;)<br>print(f&quot;🗂️ S3 Key: {s3_key}&quot;)<br><br>s3 = boto3.client(&quot;s3&quot;)<br>s3.upload_file(Filename=output_path, Bucket=bucket_name, Key=s3_key)<br><br>print(f&quot;📤 Uploaded to s3://{bucket_name}/{s3_key}&quot;)</pre><p>We will also need to create an S3 bucket to ensure the response gets uploaded to our S3 bucket:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*pTNyve4XGpYoUSVd0OHfRg.png" /></figure><p>when we run our script we should see our S3 bucket get populated with the following:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*g4GapEoT9ZeTvf8cKbcyjg.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*89hEemeqvwVmfh72S_UHRQ.png" /></figure><p>This means our script ran successfully.</p><p>Note: We need to ensure that we have a user created in IAM with appropriate policies:</p><p><a href="https://982081047984-buwpsdjh.us-east-1.console.aws.amazon.com/iam/home?region=us-east-1#/users/details/bedrock-prompt-user/editPolicy/invokeModelPolicy?step=addPermissions">invokeModelPolicy</a> policy:</p><pre>{<br> &quot;Version&quot;: &quot;2012-10-17&quot;,<br> &quot;Statement&quot;: [<br>  {<br>   &quot;Effect&quot;: &quot;Allow&quot;,<br>   &quot;Action&quot;: &quot;bedrock:InvokeModel&quot;,<br>   &quot;Resource&quot;: &quot;arn:aws:bedrock:us-east-1::foundation-model/anthropic.claude-3-sonnet-20240229-v1:0&quot;<br>  }<br> ]<br>}</pre><p><a href="https://982081047984-buwpsdjh.us-east-1.console.aws.amazon.com/iam/home?region=us-east-1#/policies/details/arn%3Aaws%3Aiam%3A%3A982081047984%3Apolicy%2Fs3-put-object-policy">s3-put-object-policy</a> policy:</p><pre>{<br>    &quot;Version&quot;: &quot;2012-10-17&quot;,<br>    &quot;Statement&quot;: [<br>        {<br>            &quot;Effect&quot;: &quot;Allow&quot;,<br>            &quot;Action&quot;: [<br>                &quot;s3:PutObject&quot;<br>            ],<br>            &quot;Resource&quot;: &quot;arn:aws:s3:::bedrock-prompt-deployment-beta-abdul/beta/outputs/*&quot;<br>        }<br>    ]<br>}</pre><p>After all this, we should have a successful flow of generating a prompt using dynamic variables, invoking the Claude model on Amazon Bedrock, saving the response as an HTML file, and uploading that file to the correct path in our S3 bucket — all automated via a script with proper IAM permissions in place.</p><h3><strong>5. Build the GitHub Actions Workflow</strong></h3><p>The .github/workflows/ directory contains GitHub Actions that automate the entire prompt deployment lifecycle. The on_push.yaml workflow triggers on pushes to feature branches, running the script and uploading outputs to the beta/ folder in S3 for staging and review. The on_merge_with_approval.yaml workflow activates when pull requests are merged into main, sending final outputs to the prod/ S3 prefix. This setup ensures environment-specific behavior, enabling a clean separation between preview and production content—all without needing to access the AWS Console.</p><p>We can create a new branch locally (so that when we push to the main branch the GitHub actions workflow is triggered).</p><pre>git checkout -b beta-branch<br>git add .<br>git commit -m &quot;added beta branch&quot;<br>git push --set-upstream origin beta-branch</pre><p>We will set our secrets for our repository with the following values:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/969/1*lBed3Cq9fyBqPmVjyQYG8Q.png" /></figure><p>The AWS_ACCESS_KEY_ID is our user’s AWS_ACCESS_KEY_ID.</p><p>The AWS_SECRET_ACCESS_KEY is our user’s AWS_SECRET_ACCESS_KEY</p><p>The S3_BUCKET_BETA is our S3 bucket name (beta)</p><h3><strong>6. Creating the </strong><strong>merge approval workflow</strong></h3><p>We’ll add the following workflow instructions into our on_merge_with_approval.yaml file:</p><pre>name: On Merge to Main with Approval<br><br>on:<br>  push:<br>    branches:<br>      - main<br><br>jobs:<br>  deploy-to-prod:<br>    runs-on: ubuntu-latest<br>    environment:<br>      name: production  <br>    env:<br>    env:<br>      S3_BUCKET_PROD: ${{ secrets.S3_BUCKET_PROD }}<br>      AWS_ACCESS_KEY_ID: ${{ secrets.AWS_ACCESS_KEY_ID }}<br>      AWS_SECRET_ACCESS_KEY: ${{ secrets.AWS_SECRET_ACCESS_KEY }}<br>      DEPLOY_ENV: prod<br>      S3_BUCKET_PROD: ${{ secrets.S3_BUCKET_PROD }}<br>    steps:<br>      - name: Checkout code<br>        uses: actions/checkout@v3<br><br>      - name: Set up Python<br>        uses: actions/setup-python@v4<br>        with:<br>          python-version: 3.11<br><br>      - name: Install dependencies<br>        run: pip install boto3 jinja2<br><br>      - name: Run Bedrock prompt and upload<br>        run: python process_prompt.py</pre><p>We can also set an additional secret:</p><p>S3_BUCKET_PROD</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/992/1*axwA3bpP4UzBX6yZCUKrOQ.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*S9Yebm0WkSOWFRalPSJKpA.png" /></figure><p>After merging to the main branch, GitHub Actions automatically triggered the on_merge_with_approval workflow. This workflow included a manual approval step (to simulate stakeholder sign-off) followed by a production deployment. The deployment executed the same process_prompt.py script, but this time with environment variables configured for the production environment. The generated response was saved in the outputs/ directory and successfully uploaded to the production S3 bucket. This confirms the full end-to-end flow of our Bedrock content generation pipeline—from dynamic prompt templating, through AI generation, to automated deployment in the cloud.</p><p>With this final workflow successfully deployed, we’ve validated our ability to automate production-ready Bedrock interactions through a controlled, CI/CD-approved pipeline. The manual approval step gave us an extra layer of review before releasing to the cloud, simulating real-world deployment safeguards.</p><h3><strong>7. Conclusion</strong></h3><p>This project showcased a complete, cloud-native proof of concept for automating generative AI content workflows. By combining Jinja2 for dynamic prompt templating, Amazon Bedrock for powerful language generation, and GitHub Actions for CI/CD automation, we built a pipeline that transforms user-defined input into AI-generated outputs and deploys them to an S3 bucket with environment-specific control. We simulated both staging and production environments, integrated IAM-secured API calls, and included human-in-the-loop approval logic for safer releases. Altogether, this project demonstrates how modern teams can operationalize generative AI as part of a scalable, secure, and automated deployment pipeline.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=2a615bf7dc36" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Voice to Multilingual Reach: Automating Audio Localization with AWS and GitHub Actions]]></title>
            <link>https://medium.com/@abdulshaikhr99/voice-to-multilingual-reach-automating-audio-localization-with-aws-and-github-actions-4b283e0a52c5?source=rss-c47a96f96b0a------2</link>
            <guid isPermaLink="false">https://medium.com/p/4b283e0a52c5</guid>
            <category><![CDATA[software-development]]></category>
            <category><![CDATA[devops]]></category>
            <category><![CDATA[amazon]]></category>
            <category><![CDATA[ci-cd-pipeline]]></category>
            <category><![CDATA[cloud]]></category>
            <dc:creator><![CDATA[Abdul Rahim Shaikh]]></dc:creator>
            <pubDate>Fri, 13 Jun 2025 22:35:21 GMT</pubDate>
            <atom:updated>2025-06-13T22:35:21.318Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*dUC4KWD6wWZuZJCNxGTDqw.png" /></figure><p>In this proof of concept, our team at Pixel Learning Co. set out to streamline how educational audio content is localized for global learners. We built a multilingual audio transformation pipeline using Amazon’s AI services — Transcribe, Translate, and Polly — integrated seamlessly into our GitHub-based workflow. The goal? Turn a single English audio file into high-quality, localized voice content in multiple languages with zero manual intervention. This project not only reduces repetitive effort for content teams but also sets the stage for scalable, accessible, and inclusive learning experiences powered by cloud-native automation.</p><p>Let’s get started</p><h3><strong>1: GitHub Repo Creation</strong></h3><p>To kick things off, we’ll create a dedicated GitHub repository to house our audio processing pipeline. This repo will serve as the central hub for managing input audio files, storing our automation scripts, and orchestrating CI/CD workflows. By leveraging GitHub Actions, we can ensure that every audio file pushed into the repo automatically triggers our end-to-end pipeline — from transcription to translation to speech synthesis — without requiring any manual steps. This setup not only keeps the process consistent and version-controlled but also empowers our content team to manage everything through a familiar GitHub interface.</p><p>We can create a remote repo on GitHub, create a local one on our machine and then push to remote:</p><pre>echo &quot;# Multilingual-Amazon_transcribe-pipeline&quot; &gt;&gt; README.md<br>git init<br>git add README.md<br>git commit -m &quot;first commit&quot;<br>git branch -M main<br>git remote add origin git@github.com:AbdulRahimShaikh1999/Multilingual-Amazon_transcribe-pipeline.git<br>git push -u origin main</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*VJSix1lBf36vYtWL_MvyzA.png" /></figure><h3><strong>2: Setup Folder Structure</strong></h3><p>On our local repo we will setup the following folder structure:</p><pre>multilingual-audio-pipeline/<br>├── audio_inputs/<br>│   └── lesson1.mp3<br>├── scripts/<br>│   └── process_audio.py<br>├── .github/<br>│   └── workflows/<br>│       ├── on_pull_request.yml<br>│       └── on_merge.yml<br>├── requirements.txt<br>├── README.md</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/383/1*id4gMl5K5_9sBZg5luBwJA.png" /></figure><p>Once we have our directory structure ready we can create a new branch (“dev”) and push to our remote repo.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*homFCjEhpysxvJvpcevBSQ.png" /></figure><p>For now we won’t create a PR for a merge, we can do that later.</p><h3><strong>3: Configure GitHub Secrets for AWS</strong></h3><p>To securely manage access to AWS services without hardcoding sensitive information, we use GitHub Secrets to store our credentials — specifically, AWS_ACCESS_KEY_ID, AWS_SECRET_ACCESS_KEY, AWS_REGION, and S3_BUCKET. These secrets are encrypted and injected into the GitHub Actions workflow at runtime, ensuring credentials remain secure and out of source control. To follow the principle of least privilege, we also created a custom IAM policy attached to the AWS user running the pipeline. This policy grants only the necessary permissions to access Amazon Transcribe, Translate, Polly, and limited read/write access to specific S3 prefixes. This approach balances security with functionality, enabling the GitHub Actions workflow to interact with AWS safely and efficiently.</p><p>Let’s setup the necessary secrets.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Osl-F96QKausrISxEI2O6A.png" /></figure><p>On the left hand side of general settings bar in our repo we will see an option for “secrets and variables” &gt;&gt; “Actions”</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/400/1*FDFkbCbAVfQehKBZmC5Ogg.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/985/1*f_SBCD8gii_Fddc_AHXBfA.png" /></figure><p>For this secret we can input our S3 Bucket name, for our AWS_REGION we can input whatever region we want to work out of and for AWS_ACCESS_KEY_ID, AWS_SECRET_ACCESS_KEY we can input any correct and current combination of an AWS access key and secret access key.</p><p>Once we’ve inputted all the secrets, we’ll have:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/993/1*BvgIrjGoXrf59uIATgtTgg.png" /></figure><h3><strong>Step 4: Write GitHub Actions Workflows</strong></h3><p>Now, we will implement a CI/CD workflow using GitHub Actions to automate the multilingual audio processing pipeline. When a pull request is opened, the workflow is triggered in a beta environment, where all outputs — transcripts, translations, and synthesized audio — are uploaded to the beta/ prefix in our S3 bucket. This allows the team to validate the transformation steps before anything goes live. Once the PR is merged into the main branch, the same pipeline runs again, this time pushing results to the prod/ prefix for production-ready use. This separation between beta and prod environments ensures that every step in the pipeline can be tested safely before release, aligning with best practices in automated and reliable content delivery.</p><p>We will add to our yaml files ( on_pull_request.yml and on_merge.yml ) soon. But first we can create our python script.</p><h3><strong>5: Add Python Script</strong></h3><p>For our process_audio.py file we can have the following script:</p><pre>import os<br>import boto3<br>import time<br>from pathlib import Path<br>import requests<br><br># Load env variables<br>AWS_REGION = os.environ.get(&#39;AWS_REGION&#39;, &#39;us-east-1&#39;)  <br>S3_BUCKET = os.environ.get(&#39;S3_BUCKET&#39;)<br><br>TARGET_LANGUAGE = &#39;es&#39;  # Spanish<br><br># Initialize AWS clients<br>s3 = boto3.client(&#39;s3&#39;, region_name=AWS_REGION)<br>transcribe = boto3.client(&#39;transcribe&#39;, region_name=AWS_REGION)<br>translate = boto3.client(&#39;translate&#39;, region_name=AWS_REGION)<br>polly = boto3.client(&#39;polly&#39;, region_name=AWS_REGION)<br><br># Determine if we&#39;re in beta or prod<br>def get_env_prefix():<br>    ref = os.getenv(&#39;GITHUB_REF&#39;, &#39;&#39;)<br>    return &#39;prod&#39; if ref == &#39;refs/heads/main&#39; else &#39;beta&#39;<br><br># Upload a file to S3<br>def upload_to_s3(file_path, s3_key):<br>    s3.upload_file(file_path, S3_BUCKET, s3_key)<br>    print(f&quot;Uploaded to s3://{S3_BUCKET}/{s3_key}&quot;)<br><br># Start Transcribe job<br>def transcribe_audio(job_name, s3_uri):<br>    transcribe.start_transcription_job(<br>        TranscriptionJobName=job_name,<br>        Media={&#39;MediaFileUri&#39;: s3_uri},<br>        MediaFormat=&#39;mp3&#39;,<br>        LanguageCode=&#39;en-US&#39;<br>    )<br><br>    print(&quot;Waiting for transcription to complete...&quot;)<br>    while True:<br>        status = transcribe.get_transcription_job(TranscriptionJobName=job_name)<br>        state = status[&#39;TranscriptionJob&#39;][&#39;TranscriptionJobStatus&#39;]<br>        if state in [&#39;COMPLETED&#39;, &#39;FAILED&#39;]:<br>            break<br>        time.sleep(5)<br><br>    if state == &#39;FAILED&#39;:<br>        raise Exception(&quot;Transcription failed.&quot;)<br><br>    transcript_url = status[&#39;TranscriptionJob&#39;][&#39;Transcript&#39;][&#39;TranscriptFileUri&#39;]<br>    return transcript_url<br><br># Download and extract text from transcript<br>def get_transcript_text(transcript_url):<br>    response = requests.get(transcript_url)<br>    return response.json()[&#39;results&#39;][&#39;transcripts&#39;][0][&#39;transcript&#39;]<br><br># Translate English to target language<br>def translate_text(text, target_language):<br>    result = translate.translate_text(Text=text, SourceLanguageCode=&#39;en&#39;, TargetLanguageCode=target_language)<br>    return result[&#39;TranslatedText&#39;]<br><br># Synthesize translated speech with Polly<br>def synthesize_speech(text, lang_code, output_path):<br>    voice_id = &#39;Lucia&#39; if lang_code == &#39;es&#39; else &#39;Joanna&#39;  # Adjust as needed<br>    response = polly.synthesize_speech(Text=text, OutputFormat=&#39;mp3&#39;, VoiceId=voice_id)<br>    with open(output_path, &#39;wb&#39;) as f:<br>        f.write(response[&#39;AudioStream&#39;].read())<br><br>def main():<br>    prefix = get_env_prefix()<br>    audio_folder = Path(&quot;audio_inputs&quot;)<br>    for file in audio_folder.glob(&quot;*.mp3&quot;):<br>        base_name = file.stem<br>        s3_input_key = f&quot;{prefix}/audio_inputs/{file.name}&quot;<br>        upload_to_s3(str(file), s3_input_key)<br><br>        s3_uri = f&quot;s3://{S3_BUCKET}/{s3_input_key}&quot;<br>        job_name = f&quot;{prefix}-{base_name}-{int(time.time())}&quot;<br>        transcript_url = transcribe_audio(job_name, s3_uri)<br>        transcript = get_transcript_text(transcript_url)<br><br>        # Upload transcript<br>        transcript_key = f&quot;{prefix}/transcripts/{base_name}.txt&quot;<br>        s3.put_object(Bucket=S3_BUCKET, Key=transcript_key, Body=transcript.encode(&#39;utf-8&#39;))<br>        print(f&quot;Uploaded transcript: {transcript_key}&quot;)<br><br>        # Translate<br>        translated_text = translate_text(transcript, TARGET_LANGUAGE)<br>        translation_key = f&quot;{prefix}/translations/{base_name}_{TARGET_LANGUAGE}.txt&quot;<br>        s3.put_object(Bucket=S3_BUCKET, Key=translation_key, Body=translated_text.encode(&#39;utf-8&#39;))<br>        print(f&quot;Uploaded translation: {translation_key}&quot;)<br><br>        # Synthesize and upload audio<br>        local_audio_file = f&quot;{base_name}_{TARGET_LANGUAGE}.mp3&quot;<br>        synthesize_speech(translated_text, TARGET_LANGUAGE, local_audio_file)<br>        audio_output_key = f&quot;{prefix}/audio_outputs/{local_audio_file}&quot;<br>        upload_to_s3(local_audio_file, audio_output_key)<br><br>if __name__ == &quot;__main__&quot;:<br>    main()</pre><p>This script might seem daunting at first glance, but what it’s really doing is orchestrating a simple yet powerful pipeline using AWS services. Here’s a breakdown of what happens behind the scenes:</p><ul><li><strong>Loads AWS credentials and environment variables</strong> (like region and S3 bucket) so the script knows where to send data.</li><li><strong>Determines if the deployment is for </strong><strong>beta or </strong><strong>prod</strong> by checking the GitHub branch it&#39;s triggered from.</li><li><strong>Finds all </strong><strong>.mp3 files in the </strong><strong>audio_inputs/ folder</strong>, allowing it to process one or multiple files at once.</li><li><strong>Uploads each </strong><strong>.mp3 to the S3 bucket</strong> under a structured prefix (beta/ or prod/), creating a clean separation between staging and production environments.</li><li><strong>Starts an Amazon Transcribe job</strong> to convert the uploaded audio into English text.</li><li><strong>Waits for transcription to complete</strong>, then retrieves the transcript from a public URL.</li><li><strong>Translates the transcript from English to Spanish</strong> using Amazon Translate.</li><li><strong>Synthesizes the translated text into Spanish audio</strong> using Amazon Polly’s neural voices.</li><li><strong>Uploads the transcript, translation, and final audio file back to S3</strong>, neatly organized in respective folders for transcripts, translations, and audio outputs.</li></ul><p>To test if the script is working correctly, we can simply run</p><pre>python scripts/process_audio.py</pre><p>Getting a folder beta/ with:</p><p><a href="https://982081047984-buwpsdjh.us-east-1.console.aws.amazon.com/s3/buckets/multilingual-amazon-transcribe-pipeline-bucket?region=us-east-1&amp;bucketType=general&amp;prefix=beta/audio_inputs/&amp;showversions=false">audio_inputs/</a></p><p><a href="https://982081047984-buwpsdjh.us-east-1.console.aws.amazon.com/s3/buckets/multilingual-amazon-transcribe-pipeline-bucket?region=us-east-1&amp;bucketType=general&amp;prefix=beta/audio_outputs/&amp;showversions=false">audio_outputs/</a></p><p><a href="https://982081047984-buwpsdjh.us-east-1.console.aws.amazon.com/s3/buckets/multilingual-amazon-transcribe-pipeline-bucket?region=us-east-1&amp;bucketType=general&amp;prefix=beta/transcripts/&amp;showversions=false">transcripts/</a></p><p><a href="https://982081047984-buwpsdjh.us-east-1.console.aws.amazon.com/s3/buckets/multilingual-amazon-transcribe-pipeline-bucket?region=us-east-1&amp;bucketType=general&amp;prefix=beta/translations/&amp;showversions=false">translations/</a></p><p>is what we should be looking out for on our S3 bucket.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*MM19RIWgy2K26vvntUmNDg.png" /></figure><h3><strong>6: Add requirements.txt</strong></h3><p>The simplest step in this POC.</p><p>We will add the following requirements in the requirements.txt file:</p><pre>boto3<br>requests</pre><p>The requirements.txt file lists all the Python libraries needed to run the audio processing script. Including this file ensures that GitHub Actions can install the correct dependencies during workflow execution. This guarantees that the runtime environment matches local development, reducing errors caused by missing or incompatible packages.</p><h3><strong>7: Push to GitHub and Test the Beta Workflow</strong></h3><p>After creating a beta branch, making a change, and opening a pull request to main, the on_pull_request.yml workflow was automatically triggered. In the GitHub Actions tab, we can see the CI/CD pipeline in action—processing the newly added audio file, running transcription, translation, and speech synthesis, and uploading the results to the beta/ S3 prefix for review. This confirms that our automated test environment is functioning as expected.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*oLRwycp-FkLsCn-nWPNylQ.png" /></figure><p>When we merge our beta branch to Main we’ll see another GitHub Actions workflow run — if everything was done correctly, it should be a successful run.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*xARJSfeN1vocVFRzGHxbrQ.png" /></figure><p>Once the workflow runs successfully, each output type is written to a clearly structured folder in your S3 bucket under either the beta/ or prod/ prefix. These include subfolders for audio_inputs, transcripts, translations, and audio_outputs, making it easy to trace each stage of the pipeline. Screenshots of these folders in the S3 console can visually confirm that the pipeline executed as expected.</p><p>After completing all the steps, we’ve successfully built a fully automated, end-to-end multilingual audio transformation pipeline that integrates GitHub Actions with AWS services like Transcribe, Translate, and Polly. This system allows educational audio content to be transcribed, translated, and re-voiced in multiple languages automatically, with all outputs organized in S3 under structured beta/ and prod/ environments—laying the groundwork for scalable, international-ready content delivery.</p><p>This project implements a multilingual audio processing pipeline designed for scale and automation using AWS AI services and GitHub Actions. The goal was to support a use case where educational audio content — such as instructor-recorded lessons — is automatically transcribed, translated into other languages, and re-generated as speech for international learners. By leveraging Amazon Transcribe, Translate, and Polly, this pipeline eliminates manual intervention while ensuring consistency, accuracy, and timely delivery of multilingual content. All artifacts — transcripts, translations, and synthesized audio — are stored in structured S3 folders for easy access and organization.</p><p>The pipeline is fully orchestrated using GitHub Actions, enabling continuous integration and delivery (CI/CD) based on the state of the code repository. A pull request triggers a staging run that uploads outputs to a beta/ prefix, while merging to the main branch triggers a production deployment to the prod/ prefix. This clear separation between staging and production environments mirrors real-world DevOps practices and allows teams to safely test and promote changes. CI/CD in this context ensures that any update—whether it&#39;s a new audio file or a code enhancement—triggers a consistent, automated transformation process from raw input to multilingual-ready output.</p><p>This project reflects the broader value of CI/CD pipelines in cloud-based environments: they reduce human error, accelerate content delivery, and enforce a structured release process. For companies building globally accessible platforms, integrating automation with AI services creates a scalable solution that adapts to growing content demands and diverse user bases. It also demonstrates how developer-friendly tools like GitHub Actions can serve as a control plane for powerful backend workflows, enabling cross-functional teams to contribute to and maintain complex pipelines without needing direct AWS console access.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=4b283e0a52c5" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[POC: Serverless Text-to-Speech with Amazon Polly, S3, and GitHub Actions]]></title>
            <link>https://medium.com/@abdulshaikhr99/pixel-learning-co-poc-serverless-text-to-speech-with-amazon-polly-s3-and-github-actions-ab1dd28ec495?source=rss-c47a96f96b0a------2</link>
            <guid isPermaLink="false">https://medium.com/p/ab1dd28ec495</guid>
            <category><![CDATA[boto3]]></category>
            <category><![CDATA[devops]]></category>
            <category><![CDATA[github-actions]]></category>
            <category><![CDATA[cloud-computing]]></category>
            <category><![CDATA[aws]]></category>
            <dc:creator><![CDATA[Abdul Rahim Shaikh]]></dc:creator>
            <pubDate>Mon, 02 Jun 2025 12:02:30 GMT</pubDate>
            <atom:updated>2025-10-06T22:45:52.923Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*hyT5X2zm1JwtbeQnFFlvjA.png" /></figure><p>This POC showcases how Pixel Learning Co. can seamlessly transform its GitHub-stored course content into high-quality audio using Amazon Polly’s neural TTS capabilities, while securely storing these audio files in Amazon S3. By integrating this conversion workflow directly into GitHub Actions, every pull request and merge automatically triggers the Python-based text-to-speech pipeline, ensuring separate beta and production-ready audio versions. The POC leverages fully managed AWS services — Polly for instant speech synthesis and S3 for scalable storage — combined with the automation of GitHub Actions, resulting in a serverless, cost-effective solution that boosts accessibility and streamlines content delivery.</p><h3><strong>1: Goal</strong></h3><p>The goal of this proof-of-concept (POC) is to establish a lightweight, fully serverless CI/CD pipeline that converts text files into audio using Amazon Polly, and automatically uploads the generated audio to Amazon S3. This setup leverages GitHub Actions for workflow automation, requiring no server management throughout the process. The pipeline automatically triggers on pull request (PR) and merge events, ensuring seamless audio conversion and deployment.</p><p><strong>Key Objectives:</strong></p><ul><li><strong>Amazon Polly Integration</strong>: Use Amazon Polly to convert text content into high-quality audio files.</li><li><strong>S3 Deployment</strong>: Upload the generated audio files directly to an Amazon S3 bucket, enabling scalable and reliable storage.</li><li><strong>GitHub Actions Automation</strong>: Automate the entire process (conversion, upload) in response to PR and merge events, ensuring that every content update is reflected in the audio outputs.</li><li><strong>Serverless Architecture</strong>: Build using only managed AWS services (Amazon Polly, S3, GitHub Actions) with no EC2 instances or servers to maintain.</li></ul><p>By achieving these objectives, the POC will provide a scalable, low-maintenance workflow that streamlines the audio conversion of textual content.</p><p>Let’s get started.</p><h3><strong>2: Local Script Development</strong></h3><p>We’ll create2 files called speech.txt and synthesize.py</p><p>And add the following content:</p><pre>Hello! Welcome to Pixel Learning Co.</pre><pre>import boto3<br>import os<br><br># Read text from file<br>with open(&#39;speech.txt&#39;, &#39;r&#39;) as file:<br>    text = file.read()<br><br># Initialize Polly client<br>polly_client = boto3.client(&#39;polly&#39;, region_name=&#39;us-east-1&#39;)<br><br># Synthesize speech<br>response = polly_client.synthesize_speech(<br>    Text=text,<br>    OutputFormat=&#39;mp3&#39;,<br>    VoiceId=&#39;Joanna&#39;<br>)<br><br># Save the audio stream locally<br>with open(&#39;output.mp3&#39;, &#39;wb&#39;) as file:<br>    file.write(response[&#39;AudioStream&#39;].read())<br><br>print(&quot;Audio synthesis complete. File saved as output.mp3.&quot;)<br><br># Upload to S3<br>s3_client = boto3.client(&#39;s3&#39;, region_name=&#39;us-east-1&#39;)<br>bucket_name = os.environ[&#39;S3_BUCKET&#39;]<br>s3_key = &#39;polly-audio/output.mp3&#39;<br><br># Adding a comment<br><br>s3_client.upload_file(&#39;output.mp3&#39;, bucket_name, s3_key)<br>print(f&quot;Uploaded to s3://{bucket_name}/{s3_key}&quot;)</pre><p>This Python script automates the process of converting text into speech using Amazon Polly and storing the resulting audio in an Amazon S3 bucket. It begins by reading text content from a file called speech.txt into memory. Using the boto3 library, it then initializes a Polly client in the us-east-1 region, enabling it to send a request to Amazon Polly to synthesize speech from the text. The Polly service generates an audio stream in MP3 format using the Joanna voice.</p><p>After the audio stream is created, the script saves it locally as output.mp3, ensuring that the generated audio file is accessible on the local machine. Once the file has been successfully created, the script sets up an S3 client, again in the us-east-1 region, and retrieves the name of the target S3 bucket from an environment variable called S3_BUCKET. It then defines the destination path within the bucket and uploads the audio file to S3. Throughout this process, the script provides log messages to indicate when the audio synthesis and S3 upload are complete, ensuring clarity and transparency in each step of the workflow.</p><h3><strong>3: Local Testing</strong></h3><p>Install dependencies:</p><pre>pip install boto3</pre><p>Create S3 bucket with the following configurations:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Qd4rPPQVftl0KmOGhmD6zg.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*382VCcPmBRKgVgP6GZqr-Q.png" /></figure><p>Export AWS credentials in our environment:</p><pre>export AWS_ACCESS_KEY_ID=your-access-key<br>export AWS_SECRET_ACCESS_KEY=your-secret-key<br>export S3_BUCKET=your-s3-bucket</pre><p>Run the script:</p><pre>python3 synthesize.py</pre><p>After running our python script, we should see an audio file pop up, listening to it will confirm that the content in our speech.txt file is using Amazon Polly to generate the audio of it:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/213/1*XTvuWR6BS5rn9goWFPTORw.png" /></figure><p>The file is also uploaded onto our S3 bucket as an object:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*gCF7avxeazMPPKeC1XMaiw.png" /></figure><h3><strong>4: Create GitHub Repo &amp; Push Files</strong></h3><p>After creating our Github repo, we can push the files from our IDE onto a remote repo.</p><pre>git init<br>git add README.md<br>git commit -m &quot;first commit&quot;<br>git branch -M main<br>git remote add origin &lt;repo-address&gt;<br>git push -u origin main</pre><h3><strong>5: Configure GitHub Secrets</strong></h3><p>Now we’ll store sensitive values in GitHub Secrets by going to <strong>Settings → Secrets and variables → Actions</strong>, and add AWS_ACCESS_KEY_ID, AWS_SECRET_ACCESS_KEY, and S3_BUCKET. These will be securely referenced by the workflow for AWS authentication and S3 uploads.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*_KWL8yPcp_aGNp4dOmNwzA.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/401/1*8MIY2MaiVB9_rJD2plpLHw.png" /></figure><p>We can enter a secret for each of the values:</p><p>AWS_ACCESS_KEY_ID, AWS_SECRET_ACCESS_KEY,S3_BUCKET</p><p>For example:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1003/1*mNlImWecKIMLjHx17hREgA.png" /></figure><h3><strong>6: Create GitHub Actions Workflow Files</strong></h3><p>Now we’ll create two GitHub Actions workflow files to automate the audio conversion and upload process. In your local project, create a .github/workflows directory if it doesn&#39;t exist. Inside it, create on_pull_request.yml to trigger the workflow for pull request events, converting the text to audio and uploading it as beta.mp3. Similarly, create on_merge.yml to trigger on push events to the main branch, generating prod.mp3 for production-ready audio. Both workflows run on ubuntu-latest, install Python dependencies (boto3), and securely reference your GitHub Secrets to interact with AWS services without exposing sensitive credentials.</p><p>In our local project, we can create:</p><pre>mkdir -p .github/workflows</pre><p>Then create the beta workflow file at:</p><p>.github/workflows/on_pull_request.yml</p><p>with the following content:</p><pre>name: Polly Audio - Beta<br><br>on:<br>  pull_request:<br>    branches:<br>      - main<br><br>jobs:<br>  beta_audio:<br>    runs-on: ubuntu-latest<br><br>    steps:<br>      - name: Checkout code<br>        uses: actions/checkout@v4<br><br>      - name: Set up Python<br>        uses: actions/setup-python@v5<br>        with:<br>          python-version: &#39;3.x&#39;<br><br>      - name: Install dependencies<br>        run: pip install boto3<br><br>      - name: Run synthesize script<br>        env:<br>          AWS_ACCESS_KEY_ID: ${{ secrets.AWS_ACCESS_KEY_ID }}<br>          AWS_SECRET_ACCESS_KEY: ${{ secrets.AWS_SECRET_ACCESS_KEY }}<br>          S3_BUCKET: ${{ secrets.S3_BUCKET }}<br>        run: |<br>          echo &quot;Overriding S3 key to beta.mp3&quot;<br>          sed -i &quot;s|polly-audio/output.mp3|polly-audio/beta.mp3|g&quot; synthesize.py<br>          python3 synthesize.py</pre><p>save and exit it.</p><p>The same can be done for our Prod workflow.</p><pre>nano .github/workflows/on_merge.yml</pre><pre>name: Polly Audio - Prod<br><br>on:<br>  push:<br>    branches:<br>      - main<br><br>jobs:<br>  prod_audio:<br>    runs-on: ubuntu-latest<br><br>    steps:<br>      - name: Checkout code<br>        uses: actions/checkout@v4<br><br>      - name: Set up Python<br>        uses: actions/setup-python@v5<br>        with:<br>          python-version: &#39;3.x&#39;<br><br>      - name: Install dependencies<br>        run: pip install boto3<br><br>      - name: Run synthesize script<br>        env:<br>          AWS_ACCESS_KEY_ID: ${{ secrets.AWS_ACCESS_KEY_ID }}<br>          AWS_SECRET_ACCESS_KEY: ${{ secrets.AWS_SECRET_ACCESS_KEY }}<br>          S3_BUCKET: ${{ secrets.S3_BUCKET }}<br>        run: |<br>          echo &quot;Overriding S3 key to prod.mp3&quot;<br>          sed -i &quot;s|polly-audio/output.mp3|polly-audio/prod.mp3|g&quot; synthesize.py<br>          python3 synthesize.py</pre><p>save and exit.</p><h3><strong>7: Commit and Push the Workflow Files</strong></h3><pre>git add .github/workflows/on_pull_request.yml .github/workflows/on_merge.yml<br>git commit -m &quot;Add GitHub Actions workflows for beta and prod&quot;<br>git push</pre><h3><strong>8: Test the Workflows</strong></h3><p>Let’s test our workflow:</p><p>creating a new branch and then making any minor change in our source code, we can confirm if things are working properly.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/592/1*5A7qA3XbKHfbXeWpLfcUDA.png" /></figure><pre>git checkout -b test-pr-branch</pre><pre>git add .<br>git commit -m &quot;Test: Trigger beta workflow&quot;<br>git push origin test-pr-branch</pre><p>once we commit, we’ll see an option to create a pull request:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*chC7r7b6lirIz6B8tFWwPQ.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*pp0XiHMzOGy9405rUafX6g.png" /></figure><p>Approving the PR:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*xTtLi8GwLbi9I99ryuRZhg.png" /></figure><p>Once we approve the pull request we will see our updated S3 Bucket:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*flu2naKjcntINIkUxRsRww.png" /></figure><h3><strong>Summary</strong></h3><p>In this POC, we built a fully serverless audio conversion pipeline leveraging Amazon Polly, Amazon S3, and GitHub Actions to automate the process of transforming text into high-quality audio and storing it securely in the cloud. By configuring GitHub Secrets to manage AWS credentials and S3 bucket names, setting up workflow files for both beta and production audio deployments, and writing a Python script to interact with the AWS services using boto3, we&#39;ve ensured a streamlined, secure, and scalable approach to content delivery. This end-to-end pipeline empowers us to automate audio conversion and deployment seamlessly, without any manual intervention or infrastructure to maintain.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=ab1dd28ec495" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Scaling Vitality Sphere: S3 Migration, User Tracking, and CI/CD Automation — Serverless…]]></title>
            <link>https://medium.com/@abdulshaikhr99/scaling-vitality-sphere-s3-migration-user-tracking-and-ci-cd-automation-80abab04d723?source=rss-c47a96f96b0a------2</link>
            <guid isPermaLink="false">https://medium.com/p/80abab04d723</guid>
            <category><![CDATA[lambda]]></category>
            <category><![CDATA[serverless]]></category>
            <category><![CDATA[aws]]></category>
            <category><![CDATA[cloud-computing]]></category>
            <category><![CDATA[devops]]></category>
            <dc:creator><![CDATA[Abdul Rahim Shaikh]]></dc:creator>
            <pubDate>Mon, 26 May 2025 12:02:31 GMT</pubDate>
            <atom:updated>2025-06-16T23:52:37.767Z</atom:updated>
            <content:encoded><![CDATA[<h3>Scaling Vitality Sphere: S3 Migration, User Tracking, and CI/CD Automation — Serverless architecture</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/668/1*1tIylWn6a-Vz4h-dcKlreg.jpeg" /></figure><p>Vitality Sphere, a health and wellness company, recognized the importance of optimizing its website and infrastructure to enhance user experience, reduce operational costs, and stay competitive in the ever-evolving digital landscape. To address these needs, the company migrated its website from an EC2-hosted solution to a more cost-effective and scalable static website hosted on Amazon S3. Additionally, an API was implemented to track user interactions and store visit data in a DynamoDB table, providing valuable insights into user behavior. To streamline operations, a CI/CD pipeline using AWS services was introduced to automate the build and deployment process, enabling faster and more efficient updates. This documentation outlines the steps taken to achieve these goals, addressing the technical challenges faced, solutions implemented, and best practices followed, ultimately providing a scalable, secure, and efficient infrastructure to support Vitality Sphere’s growth in the health and wellness industry.</p><p>Let’s get started:</p><h3><strong>Part 1: CI/CD Base</strong></h3><p>To set up the CodeBuild part of the project, the first step was to connect it to the GitHub repository where the source code for the website is hosted.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*mooxUngfOcPC35_1XB5ItA.png" /></figure><p>This was done by linking the repository to AWS CodeBuild, ensuring that each change made to the repository would trigger an automatic build. After the connection was established, the build configuration file (buildspec.yml) was created, which defines the phases of the build, such as installation, build, and post-build steps.</p><p>To verify that the build would trigger automatically, a test file was manually added to the GitHub repository, simulating a code update. This allowed the CodeBuild project to automatically detect the change, start the build process, and confirm that the pipeline was functioning correctly. The build was then tested for success, ensuring the process was streamlined and fully integrated into the deployment workflow.</p><p>Below is an image of the build project being created to ensure that each time there is a push to the GitHub repository, an automatic build is triggered:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*axRVlkKGBLBPW2TeT931yg.png" /></figure><p>Now the next step is to set up the orchestration:</p><p>To automate the deployment process, a CodePipeline was created that connects to the GitHub repository, triggering a build every time code is pushed.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*CWb5HVzChCRztnTSZwBe8A.png" /></figure><p>The pipeline currently includes only the source and build stages, which ensures that the website is always packaged correctly with each code update. While the deploy stage has not yet been added, this setup mirrors the structure of real-world CI/CD pipelines used in industry, albeit on a smaller scale. This approach ensures that the build process is automated and reliable, laying the foundation for future pipeline expansions.</p><p>The pipeline is starting simple with just the source and build stages, but it will grow into a larger CI/CD setup. A deploy stage will soon be added to push files to an S3 bucket and include a manual approval step for production. Later, Lambda functions and an API Gateway will be built for user tracking. These won’t live in the pipeline itself but will integrate with the architecture — reflecting how real systems evolve in layers of automation and infrastructure.</p><p>Now we will add our necessary file to our GitHub repo and this will automatically trigger our pipeline to build and ultimately deploy to our S3 bucket:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/877/1*ZAWRh6evY0JzA2SNyRrPeg.png" /></figure><p>All relevant files relating to the web application will be pushed to GitHub:</p><pre>git add .<br>git commit -m &lt;message&gt;<br>git push</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*3zlRQRXG4MIMA5UWSz7pHw.png" /></figure><p>Once this happens, permission errors may occur in the pipeline.</p><p>To resolve these permission errors, we want to attach the appropriate IAM policy to ensure AWS CodePipeline is allowed to upload objects to S3.</p><p>We can attach the AmazonS3FullAccess policy to the pipeline role that was created earlier.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*X03h8B5Tr7GZc-BfZkjW3A.png" /></figure><p>Once this is done, the pipeline will be successful and the following output will be displayed:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*k55GdOVQLKWF0IgYp_fJbg.png" /></figure><p>In this section, the foundational CI/CD pipeline was successfully established by connecting AWS CodeBuild to the GitHub repository, ensuring that each code push automatically triggers a build and correctly packages the website. The pipeline currently includes the source and build stages, setting the stage for future automation. While the deploy stage hasn’t been added yet, this simple setup mirrors industry practices and will be expanded to include deployment to an S3 bucket and a manual approval step for production. The architecture will continue to evolve, incorporating Lambda functions and an API Gateway for user tracking, further reflecting how real-world systems scale over time. To ensure the pipeline operates without errors, the necessary IAM policies were applied to grant CodePipeline the required permissions for uploading objects to S3, resulting in a successful build and pipeline operation. This initial setup provides a solid foundation for further automation and expansion as the project progresses.</p><h3><strong>Part 2: Production Hosting Environment with S3 and CloudFront</strong></h3><p>The production hosting environment for the Vitality Sphere website was established using Amazon S3 and CloudFront to create a scalable, cost-effective, and high-performance solution. By migrating the website’s content to an S3 bucket configured for static website hosting, the site leverages S3’s durability and scalability, while CloudFront accelerates content delivery through a globally distributed network of edge locations. This approach reduces latency and improves load times for users worldwide, ensuring the website can efficiently handle increased traffic while maintaining low operational costs. Additionally, Route 53 was used to point the domain to the CloudFront distribution, further enhancing availability and performance for a seamless user experience. This infrastructure setup is essential for supporting the company’s continued growth and ensuring a reliable, efficient online presence.</p><p>An S3 bucket will be created for the production environment, similar to the previous steps.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*f4W_X6_Jg6-944EkbgW5wA.png" /></figure><p>The settings and configurations will follow standard practices, and static website hosting will be enabled.</p><p>Next, a CloudFront distribution will be connected to this bucket, allowing public users to access the application through CloudFront rather than directly through S3.</p><p>When connecting the S3 bucket to the CloudFront distribution, permission and access issues may arise, such as those shown below. These issues can be caused by factors like using the wrong S3 origin endpoint (for example, the static website endpoint instead of the REST endpoint), missing or misconfigured Origin Access Control (OAC), incorrect or incomplete bucket policies, missing default root object, or simply the files not being present in the bucket.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*tYewI2MxNxytL8CTPWodiA.png" /></figure><p>But after getting them resolved we will have our CloudFront distribution serving our application, instead of our S3 bucket directly:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*0p2e8EsMUtJvJReUmz1cCw.png" /></figure><p>At this point, a DNS record can also be created using Route 53 to provide the site with a clean, branded URL. However, this step will be skipped for now and revisited later, as it is not essential for the application’s core functionality and infrastructure.</p><p>At this stage, the production hosting environment for the Vitality Sphere website has been successfully established using S3 and CloudFront. By migrating the website’s content to S3 and configuring CloudFront for content delivery, the site now benefits from improved scalability, performance, and reduced latency for users worldwide. With Route 53 directing the domain to the CloudFront distribution, the website is now hosted in a highly available and cost-effective environment. This setup ensures that the production site is optimized for both performance and reliability, supporting Vitality Sphere’s growth and online presence.</p><h3><strong>Part 3: User Interaction Tracking with API Gateway and Lambda</strong></h3><p>The serverless backend will be implemented to track user page visits using API Gateway, Lambda, and DynamoDB. API Gateway will expose a public endpoint that triggers a Python-based Lambda function on each page load. The function captures the current timestamp and page path, then stores the data in a DynamoDB table using the timestamp as the partition key and the page as the sort key. A small JavaScript snippet on the frontend will call this endpoint automatically, enabling lightweight, scalable tracking of user activity without the need to manage any servers.</p><p>Let’s begin with the DynamoDB table. It will be a simple setup with a partition key and a sort key:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*BmFclQxI9qvMO5AYmkhgiQ.png" /></figure><ul><li>The timestamp will uniquely identify each visit.</li><li>The page field will help group visits by page for analysis.</li></ul><p>Now, let’s move on to the Lambda function.</p><p>This Lambda function will record each user visit to the website by storing the page path and the exact timestamp of the visit in the DynamoDB table. It will receive a JSON payload via an HTTP POST request containing the page the user visited. The function will automatically generate the current UTC timestamp and write both values as a new item in the VisitorMetrics table.</p><p>The following Python script will be used for the Lambda function logic:</p><pre>import json<br>import boto3<br>from datetime import datetime<br><br>dynamodb = boto3.resource(&#39;dynamodb&#39;)<br>table = dynamodb.Table(&#39;VisitorMetrics&#39;)<br><br>def lambda_handler(event, context):<br>    try:<br>        body = json.loads(event[&#39;body&#39;])<br>        page = body.get(&#39;page&#39;, &#39;unknown&#39;)<br>        timestamp = datetime.utcnow().isoformat()<br><br>        table.put_item(<br>            Item={<br>                &#39;timestamp&#39;: timestamp,<br>                &#39;page&#39;: page<br>            }<br>        )<br><br>        return {<br>            &#39;statusCode&#39;: 200,<br>            &#39;body&#39;: json.dumps({&#39;message&#39;: &#39;Visit recorded&#39;})<br>        }<br><br>    except Exception as e:<br>        return {<br>            &#39;statusCode&#39;: 500,<br>            &#39;body&#39;: json.dumps({&#39;error&#39;: str(e)})<br>        }</pre><p>Briefly our function works in the following way:</p><p>1. Connect to the DynamoDB table named “VisitorMetrics”</p><p>2. When the Lambda function is triggered:<br> a. Parse the incoming HTTP request body as JSON<br> b. Extract the “page” value from the request (or use “unknown” if missing)<br> c. Generate the current UTC timestamp</p><p>3. Write a new item into DynamoDB with:<br> — Partition key: timestamp<br> — Sort key: page</p><p>4. If successful:<br> — Return a 200 status with a “Visit recorded” message</p><p>5. If there’s an error:<br> — Return a 500 status with the error message</p><p>Let’s also push our code Python script to GitHub with:</p><pre>git add .<br>git commit -m &quot;added lambda function&quot;<br>git push</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*XcuOIKT26dvuzyvNsJSHSw.png" /></figure><p>It is important to ensure that the Lambda function has permission to write to the DynamoDB table by attaching the appropriate IAM policies.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Xj_dcdKNyP6l44f6SqTROA.png" /></figure><p>A test event can be created to ensure the Lambda function works as expected:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*7VG9l6FTkKolZ3ztoiwEGA.png" /></figure><p>If everything is configured correctly, a 200 response will be received:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*dHbtE41McERZZUzubwkQMg.png" /></figure><p>The function will then successfully write data to the DynamoDB table:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*C9MGxZUq1AadMmYfuHTW1A.png" /></figure><p>With the Lambda implementation complete, the next step is to move on to the API Gateway portion.</p><p>The Lambda function will be exposed to the public using Amazon API Gateway. This will enable the website frontend to make HTTP POST requests to record user visits (page and timestamp) in DynamoDB.</p><p>An HTTP API will be created with a single route, /metrics, integrated with the previously built TrackVisitorFunction Lambda. Once deployed, the API will provide a public endpoint URL that the frontend can call.</p><p>This step connects the backend logic (Lambda) to the outside world through a secure HTTP endpoint. With API Gateway as the interface, requests from the website will be routed directly to the Lambda function. This represents a standard serverless pattern for building lightweight, scalable APIs without maintaining backend servers.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*tDhxPMxY68IO3Z7-AaRkbA.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*5AA4glPrvDvYktKROmburA.png" /><figcaption>Our invoke URL is available</figcaption></figure><p>Once the invoke URL is available, it can be tested using curl:</p><pre>curl -X POST https://vvgkfs7nn4.execute-api.us-east-1.amazonaws.com/prod/metrics -H &quot;Content-Type: application/json&quot; -d &#39;{&quot;page&quot;: &quot;/contact&quot;}&#39;</pre><p>If the API is functioning correctly, the expected response will be:</p><pre>{&quot;message&quot;: &quot;Visit recorded&quot;}</pre><p>This process was not flawless from the start — there’s a story and lessons behind it:</p><h3>🛠A Real-World Debugging Moment: The CORS Asterisk That Broke Everything</h3><p>While integrating the frontend with API Gateway and Lambda, a frustrating CORS issue was encountered:</p><blockquote>“No ‘Access-Control-Allow-Origin’ header is present on the requested resource.”</blockquote><p>The API Gateway was set up correctly, the Lambda function was returning the necessary CORS headers, and the site was deploying without errors. However, the browser continued to block the request. After repeatedly reviewing the configuration, a subtle oversight was discovered: in the API Gateway’s CORS settings, the <strong>* wildcard had been typed into the Access-Control-Allow-Origin field, but the “Add” button had never been clicked.</strong></p><p>That single missed UI action silently prevented the header from being applied. Once “Add” was clicked and the API was redeployed, everything worked as expected — no more CORS errors, and data began flowing into DynamoDB.</p><blockquote>This served as a perfect reminder that even experienced engineers can make small configuration mistakes, and often it’s these simple, overlooked details that cause the biggest issues.</blockquote><p>This experience reinforced a key engineering principle: debugging isn’t about knowing everything — it’s about staying curious and methodically testing every layer of the stack.</p><h3><strong>Part 4 — CI/CD to Prod + Manual Approval</strong></h3><p>After verifying the Dev site and serverless backend, the deployment pipeline was elevated by adding a manual approval step and production deployment automation. The goal was to ensure every change is reviewed before reaching production, aligning with real-world deployment practices.</p><p>New stages were added to the existing AWS CodePipeline pipeline: a Manual Approval stage and a DeployToProd stage.</p><p>The original pipeline had only one deploy stage, which targeted the Dev environment. For clarity, the existing Deploy stage was deleted and recreated with the same configuration under a new name: DeployToDev.</p><p>Here is the configuration for the DeployToDev stage:</p><ul><li><strong>Action provider:</strong> Amazon S3</li><li><strong>Input artifact:</strong> BuildArtifact</li><li><strong>S3 bucket:</strong> vitality-dev-site</li><li><strong>S3 object key:</strong> /</li><li><strong>Extract file before deploy:</strong> Unchecked</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*fARfXuS0eU1hrqletCblVA.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*lOq5nXyj9G5umqIrDcQQqg.png" /></figure><p>With the Dev deployment explicitly defined, a new stage — ManualApproval — was inserted right after it.</p><p>The built-in Manual Approval action in CodePipeline was used for this step:</p><ul><li>No artifacts are needed</li><li>An optional SNS topic can be set up for email notifications</li><li>This step pauses the pipeline and waits for human approval</li></ul><p>This introduces a clear review checkpoint between the Dev and Prod deployments.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*UPpMXNVo0gLpI4nnmqpyFQ.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*SFrB1XdXid1TmeR4f7iJ4Q.png" /></figure><p>Following the approval gate, the final stage — DeployToProd — was added.</p><p>This stage uses the same build artifact from earlier stages to deploy the exact same version to the production bucket.</p><p>Here is the configuration for the DeployToProd stage:</p><ul><li><strong>Action provider:</strong> Amazon S3</li><li><strong>Input artifact:</strong> BuildArtifact</li><li><strong>S3 bucket:</strong> vitality-prod-site</li><li><strong>S3 object key:</strong> /</li><li><strong>Extract file before deploy:</strong> Unchecked</li></ul><p>With this configuration, the pipeline ends with a clean production deployment after human review.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*ytGVgefxN5DhX1LVpj7XKQ.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*LIVb1vas_ozvHVb7KrTyig.png" /></figure><p>To test the updated pipeline, a small change in the code — such as adding a comment — can be committed.</p><p>The updated pipeline should appear as follows:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*eL1Df382-uJPGffk5N4rtg.png" /></figure><p>Including the manual approval stage:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/574/1*MQVYUwjiB3-d1TZRkAXnDQ.png" /></figure><p>Once the deployment is approved, the Dev environment’s code will be deployed to the Prod environment’s S3 bucket.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*vRk19_GHBBPl5GPzqBU2vg.png" /></figure><p>As a result, the CloudFront distribution connected to the Prod S3 bucket will serve the updated environment.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*fl4LDODcWHXdBsMDZ0tF1g.png" /></figure><p>Any changes made to the source code — such as removing the logo — should now be visible on the production site.</p><p>However, there may be issues with the CloudFront distribution not updating immediately, even if the bucket has the latest files. This can be easily resolved by creating an invalidation in the CloudFront console.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*kIy5-sD2QHwhAE3W_ZjTZA.png" /></figure><p>CloudFront caches content from the S3 origin at edge locations globally. When new files are deployed to S3, CloudFront does not automatically fetch them right away.</p><ul><li>Instead, it continues serving the cached version until either:</li><li>The file’s TTL (time-to-live) expires, or</li><li>A manual cache invalidation is performed</li></ul><p>CloudFront is designed for performance and cost efficiency:</p><ul><li>It reduces the load on the S3 bucket</li><li>It delivers faster response times by serving from nearby edge locations</li><li>However, this also means that stale content can persist unless explicitly refreshed</li></ul><p>With these additions, the pipeline now mirrors a real-world deployment workflow. Changes are automatically pushed to the Dev environment, held for manual review, and then safely promoted to production via an approval gate. After confirming that the updated files reached the Prod S3 bucket, the end-user experience was validated by testing the CloudFront distribution. Because CloudFront caches content by default, a manual cache invalidation was performed to ensure the latest site version was served. This completed the CI/CD flow, resulting in a fully functional, promotion-gated pipeline that deploys from GitHub all the way to a globally distributed production environment.</p><h3>Part 5— IAM Roles Lockdown (principle of least privilege)</h3><p>The focus was now on securing Vitality Sphere’s production environment by locking down the S3 bucket to only allow access through CloudFront using Origin Access Control (OAC). Optional security measures were also considered for the development S3 bucket. Additionally, IAM roles for Lambda, CodeBuild, and CodePipeline were reviewed and restricted to enforce least privilege, ensuring the infrastructure follows security best practices.</p><p>In the IAM console, navigate to “Roles.”</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/345/1*s2Omp9gbWQnDOCMuuCMMzQ.png" /></figure><p>Search for the CodePipeline roles.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*hIBHFvdAsdAt4-V8GVvObQ.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*xfcmYT7cYGaCnHdLAjEy5w.png" /></figure><p>The AmazonS3FullAccess policy (an AWS managed policy) is too permissive and allows more access than necessary.</p><p>AWS managed policies are typically too permissive because they are designed for general use cases. In contrast, customer-managed policies (created by us) are more specific and tailored to the application’s needs. AWS managed policies are useful for testing, but customer-managed policies should be implemented for production environments.</p><p>A more restrictive policy can be used for the role instead of S3FullAccess:</p><pre>{<br>  &quot;Version&quot;: &quot;2012-10-17&quot;,<br>  &quot;Statement&quot;: [<br>    {<br>      &quot;Sid&quot;: &quot;DevDeployAccess&quot;,<br>      &quot;Effect&quot;: &quot;Allow&quot;,<br>      &quot;Action&quot;: [<br>        &quot;s3:PutObject&quot;,<br>        &quot;s3:PutObjectAcl&quot;<br>      ],<br>      &quot;Resource&quot;: &quot;arn:aws:s3:::&lt;your-bucket&gt;/*&quot;<br>    },<br>    {<br>      &quot;Sid&quot;: &quot;ProdDeployAccess&quot;,<br>      &quot;Effect&quot;: &quot;Allow&quot;,<br>      &quot;Action&quot;: [<br>        &quot;s3:PutObject&quot;,<br>        &quot;s3:PutObjectAcl&quot;<br>      ],<br>      &quot;Resource&quot;: &quot;arn:aws:s3:::&lt;your-bucket&gt;/*&quot;<br>    }<br>  ]<br>}</pre><p>This policy grants only the necessary permissions, enforcing least privilege compared to the broader AWS managed policy.</p><p>After detaching the old permissive policy, the new restrictive policy can be attached.</p><p>The same approach can be applied to the Lambda function.</p><ul><li>The CodeBuild roles are already following the principle of least privilege because there are no AWS managed policies attached, and the existing policies were created and scoped specifically for their intended use case.</li></ul><p>For the Lambda function, any excessively permissive policies — such as DynamoDBFullAccess — can be removed and replaced with the following policy:</p><pre>{<br>    &quot;Version&quot;: &quot;2012-10-17&quot;,<br>    &quot;Statement&quot;: [<br>        {<br>            &quot;Effect&quot;: &quot;Allow&quot;,<br>            &quot;Action&quot;: &quot;logs:CreateLogGroup&quot;,<br>            &quot;Resource&quot;: &quot;arn:aws:logs:us-east-1:982081047984:*&quot;<br>        },<br>        {<br>            &quot;Effect&quot;: &quot;Allow&quot;,<br>            &quot;Action&quot;: [<br>                &quot;logs:CreateLogStream&quot;,<br>                &quot;logs:PutLogEvents&quot;<br>            ],<br>            &quot;Resource&quot;: [<br>                &quot;arn:aws:logs:us-east-1:982081047984:log-group:/aws/lambda/TrackVisitorFunction:*&quot;<br>            ]<br>        },<br>        {<br>            &quot;Effect&quot;: &quot;Allow&quot;,<br>            &quot;Action&quot;: &quot;dynamodb:PutItem&quot;,<br>            &quot;Resource&quot;: &quot;arn:aws:dynamodb:us-east-1:982081047984:table/VisitorMetrics&quot;<br>        }<br>    ]<br>}</pre><p>This policy grants the Lambda function permission to create log groups and log streams and to write logs to CloudWatch for monitoring. It also allows the function to insert (PutItem) new visitor activity records into the VisitorMetrics DynamoDB table. The policy does not grant any other permissions, enforcing least privilege for security.</p><p>The pipeline can be tested to ensure that nothing is broken and the new permissions are working as intended.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*FjlU53j4OWJVvm3UK9UL8g.png" /></figure><p>A simple code change can be pushed to the GitHub repo to validate that the pipeline works correctly.</p><p>The Lambda function will continue writing items to the DynamoDB table, confirming that the two new policies are functioning as expected.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*i4SAfCrsAJX_tIMAemHlQg.png" /></figure><p>Once everything is confirmed to be working and nothing is broken, the process can confidently move forward with final deployment to production.</p><p>In this section, the focus was on securing Vitality Sphere’s production environment by locking down the S3 bucket to only allow access through CloudFront using Origin Access Control (OAC). Optional security measures were also considered for the development S3 bucket. Additionally, IAM roles for Lambda, CodeBuild, and CodePipeline were reviewed and restricted to enforce least privilege, ensuring the infrastructure follows security best practices.</p><h3><strong>Summary: Wrapping it all up</strong></h3><p>This project served as a real-world case study in migrating a static website from traditional EC2 hosting to a fully serverless architecture on AWS. All static assets were moved to S3 buckets, with CloudFront configured as a CDN for global content delivery. The EC2 backend was replaced by a Lambda function that logs page visits to DynamoDB through an API Gateway endpoint. The entire deployment process was encapsulated within a CI/CD pipeline, featuring automated builds and manual approval steps to ensure safe promotion of updates to production. Each step was rigorously tested and secured with IAM roles and least-privilege policies, ensuring that only the minimal necessary permissions were granted.</p><p>The value of this serverless approach lies in its scalability, cost-effectiveness, and ease of maintenance. Static assets are served directly from S3, while dynamic interactions are handled by Lambda. This architecture automatically adjusts to traffic spikes without manual intervention or the need to manage EC2 instances, eliminating much of the operational overhead typically associated with web hosting. As a result, the focus can shift toward building features and optimizing the user experience. Serverless architectures like this one demonstrate how modern cloud services can deliver high availability and performance at a fraction of the cost of traditional server-based solutions.</p><p>More broadly, this project reflects the ongoing trend toward cloud computing and the shift away from monolithic, self-managed infrastructure. As businesses increasingly adopt cloud-native patterns, the focus shifts from managing infrastructure to delivering value to end users. By leveraging managed services like S3, Lambda, and DynamoDB, organizations can achieve rapid deployment cycles, secure operations, and inherent resilience. This case study captures a snapshot of that evolution — and stands as a testament to how serverless can empower teams to build scalable, secure, and cost-efficient applications in today’s digital economy.</p><p>This project was a real-world case study in migrating a static website from traditional EC2 hosting to a fully serverless architecture on AWS. We moved all static assets to S3 buckets, configured CloudFront as a CDN for global delivery, and replaced the EC2 backend with a Lambda function that logs page visits to DynamoDB via an API Gateway endpoint. We then wrapped the entire deployment process in a CI/CD pipeline, complete with automated builds and manual approval steps to ensure safe promotion of updates to production. Each step was tested and secured using IAM roles and least-privilege policies, ensuring that only the minimal necessary permissions were granted.</p><p>The value of this serverless approach lies in its scalability, cost-effectiveness, and ease of maintenance. With static assets served directly from S3 and dynamic interactions handled by Lambda, the infrastructure automatically adjusts to traffic spikes without any manual intervention or the need to maintain EC2 instances. This eliminates much of the operational overhead typically associated with web hosting, allowing the focus to shift towards building features and optimizing user experience. Serverless architectures like this one exemplify how modern cloud services can deliver high availability and performance at a fraction of the cost of traditional server-based stacks.</p><p>More broadly, this project reflects the ongoing trend toward cloud computing and the shift away from monolithic, self-managed infrastructure. As businesses increasingly adopt cloud-native patterns, the focus is less on managing infrastructure and more on delivering value to end users. By leveraging managed services like S3, Lambda, and DynamoDB, organizations can achieve rapid deployment cycles, secure operations, and resilience by default. This case study is a snapshot of that evolution — and a testament to how serverless can empower teams to build scalable, secure, and cost-efficient applications in today’s digital economy.</p><h3><strong>Links:</strong></h3><p>Site: <a href="https://d9jffr7jmb115.cloudfront.net/about.html">https://d9jffr7jmb115.cloudfront.net/about.html</a></p><p>GitHub Repo: <a href="https://github.com/AbdulRahimShaikh1999/VitalityWebsite">https://github.com/AbdulRahimShaikh1999/VitalityWebsite</a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=80abab04d723" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Using AWS Lambda to manage EC2 Instances]]></title>
            <link>https://medium.com/@abdulshaikhr99/using-aws-lambda-top-manage-ec2-instances-d2c352148f49?source=rss-c47a96f96b0a------2</link>
            <guid isPermaLink="false">https://medium.com/p/d2c352148f49</guid>
            <category><![CDATA[lambda]]></category>
            <category><![CDATA[python]]></category>
            <category><![CDATA[automation]]></category>
            <category><![CDATA[devops]]></category>
            <category><![CDATA[aws]]></category>
            <dc:creator><![CDATA[Abdul Rahim Shaikh]]></dc:creator>
            <pubDate>Thu, 15 May 2025 14:01:15 GMT</pubDate>
            <atom:updated>2025-10-06T22:44:15.882Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*GXiPI65iC44DYeFZn7l8qw.png" /></figure><p>In this project, I built an AWS Lambda function using Python and Boto3 to automatically stop EC2 instances after hours to reduce unnecessary cloud costs. The goal is to help a DevOps team manage spending by shutting down development instances nightly via a scheduled trigger. The Lambda targets only EC2 instances tagged with Environment: Dev and can optionally be extended with an API Gateway to stop specific instances based on tag query parameters.</p><h3><strong>Step 1: Setup</strong></h3><p>Now we’ll set up the basic structure for the project. This includes creating a project directory, initializing a Python script for the Lambda function, and setting up a GitHub repository to track our progress. The core script will use the Boto3 library to interact with the EC2 service and stop instances as needed. Once the initial structure is ready, we’ll push the project to GitHub so it’s easy to reference and share throughout the documentation.</p><pre>git add .<br>git commit -m &quot;Initialize project structure and script file&quot;</pre><p>We will create the remote branch later when there is more substance to push (once we create our script(s)).</p><h3><strong>Step 2: Creating our EC2 instances stop script</strong></h3><p>With the project structure in place, we can now create the Lambda function script. The code is organized into clear blocks to keep it readable and maintainable.</p><p>The first block initializes our environment by importing the boto3 library and creating an EC2 client. This client allows the Lambda function to interact with the EC2 service:</p><pre>import boto3<br><br>def lambda_handler(event, context):<br>    ec2 = boto3.client(&#39;ec2&#39;)</pre><p>The second block queries AWS for all EC2 instances that are currently in the running state. This is done using the describe_instances call with a filter applied:</p><pre>    response = ec2.describe_instances(<br>        Filters=[{&#39;Name&#39;: &#39;instance-state-name&#39;, &#39;Values&#39;: [&#39;running&#39;]}]<br>    )</pre><p>The third block processes the response by extracting the instance IDs of all running instances. It loops through the reservations and instances returned in the response and collects their IDs:</p><pre>    instances_to_stop = []<br>    for reservation in response[&#39;Reservations&#39;]:<br>        for instance in reservation[&#39;Instances&#39;]:<br>            instances_to_stop.append(instance[&#39;InstanceId&#39;])</pre><p>The final block checks if there are any running instances to stop. If so, it calls stop_instances on the collected IDs and logs which instances were stopped. If none are found, it logs that as well:</p><pre>    if instances_to_stop:<br>        ec2.stop_instances(InstanceIds=instances_to_stop)<br>        print(f&quot;Stopped instances: {instances_to_stop}&quot;)<br>    else:<br>        print(&quot;No running instances found.&quot;)</pre><p>Combining it all we have:</p><pre>import boto3<br><br>def lambda_handler(event, context):<br>    ec2 = boto3.client(&#39;ec2&#39;)<br>    response = ec2.describe_instances(<br>        Filters=[{&#39;Name&#39;: &#39;instance-state-name&#39;, &#39;Values&#39;: [&#39;running&#39;]}]<br>    )<br><br>    instances_to_stop = []<br>    for reservation in response[&#39;Reservations&#39;]:<br>        for instance in reservation[&#39;Instances&#39;]:<br>            instances_to_stop.append(instance[&#39;InstanceId&#39;])<br><br>    if instances_to_stop:<br>        ec2.stop_instances(InstanceIds=instances_to_stop)<br>        print(f&quot;Stopped instances: {instances_to_stop}&quot;)<br>    else:<br>        print(&quot;No running instances found.&quot;)</pre><p>This is the core script for our Lambda function. While it’s relatively simple, its impact is practical and meaningful — it automates a routine task that can significantly reduce unnecessary EC2 costs. By stopping all running instances after hours, the script helps enforce cost-saving policies without relying on manual oversight, making it a valuable addition to any development environment.</p><p>let’s now push to GitHub:</p><p>We have already added and committed, now we will just push to remote and our code will be backed up on our remote repo.</p><pre>git branch -M main<br>git remote add origin https://github.com/AbdulRahimShaikh1999/AWSLambdaProject.git<br>git push -u origin main</pre><p>Once we see our code remotely we can move onto the next step.</p><h3><strong>Step 3: Create and Deploy the Lambda</strong></h3><p>We will zip our file so we can upload it to AWS Lambda.</p><p>Then we will create our Lambda function.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*OicZIzCq_khyYBmcNoWbMg.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*gmAyRCdzbIRRFNYihiRQiA.png" /></figure><p>We can click “Deploy” and then test via the “test” tab next to “Code”</p><p>Once our code successfully and returns an API response of something like</p><p>“Instances stopped” or “No Instances found”</p><p>Our Lambda function is now successfully stopping EC2 instances as expected, automating a key task in managing cloud costs. AWS Lambda is a serverless compute service that runs code in response to events without the need to manage infrastructure. In this project, it executes daily to shut down unused instances, showing how serverless tools support modern, event-driven development with minimal overhead.</p><h3><strong>Step 4: Schedule a task with EventBridge</strong></h3><p>Now we’ll schedule a trigger to run this Lambda function automatically every day at 7 PM. This ensures that any development EC2 instances left running after work hours are shut down consistently without manual intervention. We’ll use Amazon EventBridge to define the schedule and link it to our Lambda function, enabling reliable, time-based execution.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/843/1*p4jrUmxbiwxyzAONZU92kw.png" /></figure><p>Setting this cron based schedule will mean the rule will trigger at 7pm EDT (11PM UDT) everyday.</p><h3>Step 5: GitHub Commit</h3><p>To wrap up the basic part of the project, we make a commit to GitHub with all the working code and configurations. While this isn’t a traditional software development project, version control is still essential. Committing your work ensures that your code is backed up, documented, and easy to revisit or extend in the future. It also promotes good engineering habits, especially when building automation and infrastructure-as-code solutions in cloud environments.</p><pre>git add .<br>git commit -m &quot;Deployed Lambda and scheduled with EventBridge for daily EC2 stop&quot;<br>git push</pre><h3><strong>Part 2 — Step 1: Filtering Instances by Environment Tag: Protecting Production</strong></h3><p>To prevent accidental shutdown of production resources, we’ll enhance our describe_instances() call by adding a filter that targets only instances with the tag Environment: Dev. This ensures that the Lambda function only identifies and stops development instances, leaving production workloads untouched. Tag-based filtering is a simple but effective way to apply environment-specific automation and maintain safe operational boundaries.</p><p>We will add a small little filter to our Filters list:</p><pre>{&#39;Name&#39;: &#39;tag:Environment&#39;, &#39;Values&#39;: [&#39;Dev&#39;]}</pre><p>Now our script will look like:</p><pre>import boto3<br><br>def lambda_handler(event, context):<br>    ec2 = boto3.client(&#39;ec2&#39;)<br>    response = ec2.describe_instances(<br>        Filters=[{&#39;Name&#39;: &#39;instance-state-name&#39;, &#39;Values&#39;: [&#39;running&#39;]}, <br>                 {&#39;Name&#39;: &#39;tag:Environment&#39;, &#39;Values&#39;: [&#39;Dev&#39;]}<br>]<br>    )<br><br>    instances_to_stop = []<br>    for reservation in response[&#39;Reservations&#39;]:<br>        for instance in reservation[&#39;Instances&#39;]:<br>            instances_to_stop.append(instance[&#39;InstanceId&#39;])<br><br>    if instances_to_stop:<br>        ec2.stop_instances(InstanceIds=instances_to_stop)<br>        print(f&quot;Stopped instances: {instances_to_stop}&quot;)<br>    else:<br>        print(&quot;No running instances found.&quot;)</pre><p>We can also test this using some dummy instances adding some to have a Dev tag and others to have a Prod tag.</p><p>We can do this by creating some instances, following the same procedure as always and then assigning some Name and Value tags:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*UaDCCyedbCvQJqKA_I2X7Q.png" /></figure><p>Once created we will follow the same steps to update our Lambda function (zipping and deploying to the AWS Lambda console).</p><p>Once we upload the zipped file to lambda, deploy the code and run it we will see that only the instances with the appropritate tags were stopped.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*T6G9jET-JDX9dnjynP5NWA.png" /></figure><p>The instance that is still running has the {Environment : Prod}tag. This means our new script was successful.</p><p>Our EventBridge trigger will continue to run as specified before (7pm EDT).</p><p>We’ve now updated our Lambda function to include an additional filter in the describe_instances() API call: {&#39;Name&#39;: &#39;tag:Environment&#39;, &#39;Values&#39;: [&#39;Dev&#39;]}. This ensures the function only targets EC2 instances that are both in a running state and explicitly tagged as part of the development environment. By using this tag-based filtering, we prevent accidental termination of production resources and introduce safer automation boundaries. This Lambda implementation reflects a DevOps-centric pattern, where serverless compute is used not to support application-level logic, but to programmatically manage cloud infrastructure — in this case, enforcing cost controls through scheduled instance shutdowns.</p><h3>Part 2 — Step 2: Dynamic Tag-Based EC2 Control with API Gateway and Lambda</h3><p>In the final stage of this project, we extend our Lambda function by connecting it to an HTTP API using Amazon API Gateway. This allows engineers to dynamically trigger EC2 shutdowns by passing specific tag key-value pairs via URL query parameters. For example, a call to /stop?key=Release&amp;value=2 will stop all running instances tagged with Release=2. This transforms our cost-saving automation into a flexible, on-demand API service — a common pattern in internal DevOps tooling, where infrastructure operations are safely exposed through controlled endpoints.</p><p>We will navigate to API Gateway in our Management console and select HTTP API</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1020/1*pH-xO1tf9He1_XlTAjg4NQ.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*4rRzqzH8rBByHWtg12ctuw.png" /></figure><p>After entering in the basic details we can test our API by pasting our invoke URL into the browser:</p><pre>&lt;invoke url&gt;/&lt;route path defined in API Gateway&gt;/&lt;query parameters for Lambda&gt;</pre><p>In our particular case:</p><pre>https://894hof5967.execute-api.us-east-1.amazonaws.com/stop-tagged-instances?key=Release&amp;value=2</pre><p>If we paste into the browser and see a response such as:</p><p>“Stopped instances: [‘i-0dc2a828b9b9ca4f0’, ‘i-0348e730bb516d6f6’]”</p><p>If we paste into the browser and see a response such as:<br> <strong>“Stopped instances: [‘i-0dc2a828b9b9ca4f0’, ‘i-0348e730bb516d6f6’]”</strong><br> that means the Lambda function successfully identified and stopped all <strong>running EC2 instances</strong> that had the tag <strong>Release=2</strong>, as specified by the query parameters passed through the API Gateway URL.</p><p>Extending beyond this we can add additional logic to our lambda function to manipulate instances based on whatever criteria we like, tags, instance state, size etc.</p><p>This project began with a simple goal: reduce AWS costs by automatically stopping EC2 instances after hours. Using AWS Lambda and Python, we started with a <strong>foundational script</strong> that stopped <em>all</em> running instances — functional, but indiscriminate.</p><p>Then we introduced safety and intent: only instances tagged with Environment: Dev would be stopped. This showed how tagging and IAM best practices help prevent costly mistakes in real-world cloud operations.</p><p>After that we elevated the Lambda function into a fully controlled service. By integrating <strong>API Gateway</strong>, we allowed team members to stop specific groups of instances dynamically via query parameters, e.g., Release=2. Now, the logic is accessible, auditable, secure, and scalable — all without managing a single server.</p><h3><strong>Big Picture: Why This Matters</strong></h3><p>In the traditional IT world, scripting automation often meant managing servers, crons, firewalls, and deployment pipelines. But in the cloud, infrastructure becomes <em>programmable</em>, <em>event-driven</em>, and <em>ephemeral</em>. You write code — the cloud handles the rest.</p><p>This project demonstrates the <strong>power of cloud-native thinking</strong>:</p><ul><li>Build automation that runs only when needed.</li><li>Trigger infrastructure actions with simple API calls.</li><li>Replace always-on machines with on-demand functions.</li></ul><p>In a world where speed, reliability, and cost-efficiency matter, cloud engineers are the architects of invisible infrastructure. With just a few lines of Python, you wrote logic that saved money, scaled safely, and responded to dynamic input — all without touching a server.</p><p>That’s not just automation. That’s cloud engineering.</p><p>GitHub Link: <a href="https://github.com/AbdulRahimShaikh1999/AWSLambdaProject">https://github.com/AbdulRahimShaikh1999/AWSLambdaProject</a></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/480/0*Sb7QwVKAo4N_7KEt.gif" /></figure><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=d2c352148f49" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Deploying a Two-Tier Web Architecture on AWS (EC2, RDS and Joomla )]]></title>
            <link>https://medium.com/@abdulshaikhr99/deploying-a-two-tier-web-architecture-on-aws-ec2-rds-and-joomla-8a287c6ec62a?source=rss-c47a96f96b0a------2</link>
            <guid isPermaLink="false">https://medium.com/p/8a287c6ec62a</guid>
            <category><![CDATA[infrastructure-as-code]]></category>
            <category><![CDATA[cloud-computing]]></category>
            <category><![CDATA[cloud-services]]></category>
            <category><![CDATA[devops]]></category>
            <category><![CDATA[aws]]></category>
            <dc:creator><![CDATA[Abdul Rahim Shaikh]]></dc:creator>
            <pubDate>Thu, 24 Apr 2025 03:14:25 GMT</pubDate>
            <atom:updated>2025-04-24T03:14:25.739Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*UPiZ9_pSF95r6jlt6EHotw.jpeg" /></figure><p>In this project, We will deploy a fully functional two-tier architecture using <strong>AWS services</strong>, where the <strong>web application</strong> (Joomla CMS) runs on an EC2 instance in a <strong>public subnet</strong>, and the <strong>database</strong> is hosted securely on an RDS MySQL instance in a <strong>private subnet</strong>.</p><p>This reflects a common production architecture used across the industry for separating concerns between frontend and backend, improving <strong>security</strong>, <strong>scalability</strong>, and <strong>resilience</strong>.</p><p>While the technologies used were straightforward, the real value of the project was learning how to:</p><p>Design and implement <strong>secure network segmentation</strong> using public/private subnets</p><p>Set up <strong>least-privilege access controls</strong> using Security Groups</p><p>Understand <strong>how infrastructure components interact</strong> in a real cloud environment</p><h3>High-Level Architecture</h3><p><strong>Services Used</strong>:</p><ul><li>VPC (Custom)</li><li>Subnets (Public &amp; Private)</li><li>EC2 (Ubuntu with Joomla)</li><li>RDS (MySQL)</li><li>Security Groups</li><li>Internet Gateway + Route Tables</li></ul><pre>[ User ] <br>   ↓<br>[ EC2 Instance (Joomla CMS) ]  ← HTTP (port 80)<br>   ↓<br>[ RDS (MySQL Database) ] ← MySQL (port 3306)</pre><ul><li>EC2 in public subnet → accessible via browser</li><li>RDS in private subnet → <strong>not accessible publicly</strong></li><li>Access between them is <strong>explicitly allowed</strong> through security groups</li></ul><h3><strong>Step 1: Create a Custom VPC</strong></h3><p>Creating a custom VPC gives you complete control over your cloud network. Just like having your own private data center. This is the foundation of everything in AWS. Instead of relying on default networks (which are fine for testing but unsuitable for production), custom VPCs let you define exactly how traffic flows, which resources are exposed to the internet, and how systems communicate with each other. It’s a must-have in any real-world cloud deployment.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*3oXDv6gEkd2LvfgeT7vnjQ.png" /></figure><p>Above are the key details we will input for now, the rest can stay as default.</p><h3><strong>Step 2: Create Public and Private Subnets</strong></h3><p>Subnets divide your VPC into segments. <strong>Public subnets</strong> are for resources that need internet access — like a web server. <strong>Private subnets</strong> are isolated and reserved for internal components — like databases. This mirrors real-world architectures where public exposure is minimized and critical data is isolated for security.</p><p>Spanning across <strong>two Availability Zones</strong> gives your system <strong>fault tolerance</strong>: if one AZ goes down, the other can still operate.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*jfE-wDnFjGiCjLTL_Lup5Q.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*kVVc3tdr_fKpo_FGCOw3GA.png" /><figcaption>We will create 4 subnets. all with similar settings, 2 will be public, and 2 will be private</figcaption></figure><h3><strong>Step 3: Set Up Internet Gateway + Public Routing</strong></h3><p>These following steps are what make any <strong>subnet truly “public”</strong> — not just its name.</p><p>By default, subnets in AWS are isolated. To allow instances in a subnet to connect to the internet (and receive connections from users), you must:</p><ul><li>Attach an <strong>Internet Gateway</strong> to the VPC</li><li>Explicitly create a <strong>route</strong> to 0.0.0.0/0 (all traffic)</li><li>Assign that route to the correct subnets</li></ul><p>This design reflects real-world cloud architecture, where only <strong>specific subnets are granted external access</strong> — typically for things like web servers or NAT gateways — while everything else remains private and secure.</p><p>Associating the public route table with only PublicSubnet1 and PublicSubnet2 ensures that <strong>any EC2 instance</strong> launched there can reach the internet (for updates, downloading Joomla, or serving public content).</p><p>Your private subnets (which we’ll use for the database) will <strong>not have this route</strong>, so they’ll be safe from internet exposure — enforcing <strong>security by design</strong> and <strong>principle of least privilege</strong>.</p><p>Below are a couple of key steps to make the subnets from earlier public</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*2U9jScznqB3UguBpzKWkAA.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*qXSdk6jfQQfTkqB4OlvD0Q.png" /><figcaption>Attach the IGW to our Joomla VPC</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*mxZid1HIEbS_Wke0OV33KQ.png" /><figcaption>Edit the routes</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*yKGSa-bwm8CYUV9ReU4xOg.png" /><figcaption>“Traffic going anywhere will go through our IGW”</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*YnzI0cEJzqphKK1aFzxzMw.png" /><figcaption>Which subnets will actually have the “public” capability</figcaption></figure><h3><strong>Step 4: Launch an EC2 Instance for the Web Tier (Joomla)</strong></h3><p>This EC2 instance represents the <strong>web tier</strong> of our architecture — the part of the system users interact with directly.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*KF8ihxujTtcQ9qQ55zISXg.png" /><figcaption>For this architecture we will choose ubuntu as the AMI</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*uzXQtPXNRqCxdq4nKMilVg.png" /><figcaption>We will choose our VPC (JoomlaVPC) and one of the subnets named Public</figcaption></figure><p>Once the Instance is created, let’s SSH into it and run a couple of commands</p><p>Install Joomla prerequisites (Apache, PHP, MySQL client):</p><pre>sudo apt update &amp;&amp; sudo apt install apache2 php php-mysql libapache2-mod-php unzip wget -y</pre><p>Download and setup Joomla:</p><pre>cd /tmp<br>wget https://downloads.joomla.org/cms/joomla4/4-4-2/Joomla_4-4-2-Stable-Full_Package.zip<br>sudo unzip Joomla_4-4-2-Stable-Full_Package.zip -d /var/www/html<br>sudo chown -R www-data:www-data /var/www/html<br>sudo chmod -R 755 /var/www/html</pre><p>Restart the Apache web server:</p><pre>sudo systemctl restart apache2</pre><p>After these installations, go to your public IP and you will see this page:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*y1WDx_Vr6nQHFi4qbcnKFQ.png" /><figcaption>Joomla installer page</figcaption></figure><p>Now that the web tier of our architecture is up and running let’s summarize it:</p><p>To serve the frontend portion of this two-tier architecture, we launched an <strong>Ubuntu-based EC2 instance</strong> into one of the public subnets of my custom VPC. This instance was associated with an <strong>Elastic IP address</strong> to ensure persistent access from the internet. We configured the instance’s <strong>security group</strong> to allow HTTP traffic (port 80) from anywhere and SSH access (port 22) restricted to our own IP — aligning with common best practices for public-facing web servers.</p><p>On the EC2 instance, we installed and configured <strong>Apache</strong>, <strong>PHP</strong>, and all required PHP extensions needed to support Joomla. we then downloaded and extracted the Joomla installation package into /var/www/html, adjusted the file permissions for Apache (www-data), and restarted the web server. Once the setup was complete, we accessed the Joomla installation interface through the browser using the Elastic IP address, confirming the web tier was fully operational and ready to connect to the backend database.</p><p>This phase represents the <strong>presentation layer</strong> of the application — the part end users directly interact with. It also mirrors how real-world organizations host web applications in the cloud, where <strong>public-facing EC2 instances</strong> act as entry points into larger, distributed systems.</p><h3><strong>Step 5: Database Tier Setup</strong></h3><p>Let’s setup the database tier in our architecture sitting with the DB subnet groups:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*k6vXNQmVE3woM-Z_KatXPA.png" /><figcaption>we will create a DB subnet group which will allow the subnets to communicate with RDS</figcaption></figure><p>Creating an EC2 security group directed towards MYSQL/Aurora on port 3306 will ensure only our Joomla web server can access the database we will create.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*pfQhhW9MK697YeVC4dPXRQ.png" /></figure><p>Creating a Database is simple as well (we will use the following configurations):</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*7UMIdpn36mNYbH2EN4Wifw.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*YOox_1RcloUjGozBxDY1Kg.png" /></figure><h3>Step 6: Database Tier Setup</h3><p>With both the web and database tiers fully provisioned, the final step was to connect the Joomla web server to the backend database and complete the installation.</p><p>Using the <strong>public Elastic IP</strong> of the EC2 instance, I accessed the Joomla web installer in the browser. On the database configuration screen, I provided the connection details for the MySQL RDS instance hosted in the private subnet. These included:</p><ul><li><strong>Database Type</strong>: MySQLi</li><li><strong>Host Name</strong>: The private endpoint of the RDS instance</li><li><strong>Username</strong>: The master username defined during RDS creation</li><li><strong>Password</strong>: The matching master password</li><li><strong>Database Name</strong>: The name I assigned during RDS setup (e.g., joomla)</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/800/1*XvnrULhf3Bva3g6xUUgCpA.png" /></figure><p>Now we will enter the credentials that we used to set up the database earlier</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/636/1*Un-DeS3NFHtIqxhpX59Sng.png" /><figcaption>The “database-1.cluster-cj0ecqgyudvg.us-east-1.rds.amazonaws.com” is our RDS endpoint which can be viewed on the “Aurora and RDS” console page.</figcaption></figure><p>After clicking install we will get this message:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/648/1*VlJsdkrJj-Lyx4cC5WDLZw.png" /></figure><p>This is good and means that Joomla wants to verify that you own the site, by prompting you delete a file it has created.</p><p>We will:</p><ul><li>SSH into our EC2 instance</li><li>Run “sudo rm /var/www/html/installation/_Joomla*.txt”</li><li>(This deletes the ownership check file Joomla created — the filename will look something like _Joomla6YGE1OTkN44pIhW2BkYrT.txt)</li><li>Refresh the <strong>Joomla installation screen</strong> in the browser</li><li>Click <strong>Install Joomla</strong></li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*9Ppi68lVI6wP4t7hJwJAhQ.png" /></figure><p>Upon submitting this information, Joomla successfully connected to the database and completed the installation process. This confirmed that <strong>secure connectivity between the web tier and database tier was working as intended</strong> — with traffic flowing privately inside the VPC.</p><p>Once installed, Joomla will launch its admin interface and public site, providing a fully functional content management system hosted entirely within a custom, two-tier AWS architecture.</p><h3><strong>Taking it a step further: adding some complexity</strong></h3><p>After successfully getting a functional Joomla application running on AWS using a two-tier architecture, I decided to enhance the setup by bringing in some real-world complexity. While a single EC2 instance connected to a private RDS database works fine for basic scenarios, most production environments demand more — particularly around <strong>availability, scalability, and automation</strong>.</p><p>To reflect this, We began expanding the architecture to make it more <strong>resilient and fault-tolerant</strong>. This next stage includes adding a second web server in a separate Availability Zone, automating the application setup with scripts, and introducing an Application Load Balancer to evenly distribute traffic. These changes not only improve performance and reliability but also bring the architecture closer to what modern cloud-native systems look like in real-world deployments.</p><h3><strong>Step 7: Launch a Second EC2 Instance in a Different Availability Zone</strong></h3><p>We will have similar settings as compared to the first instance we launched with one key difference.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*LvMwxaRDXdWTlrXDgPdc5w.png" /></figure><p>The public subnet will be different as compared to the first instance</p><p>To avoid errors and constant back and forth we will manually install Joomla after we SSH into our instance instead of provisioning a script for it upon startup.</p><p>We will run these following commands in that exact order.</p><pre># 1. Update the System<br>sudo apt update -y &amp;&amp; sudo apt upgrade -y<br><br># 2. Install Apache, PHP, and Required Extensions<br>sudo apt install -y apache2 php php-mysql php-xml php-mbstring php-intl unzip wget libapache2-mod-php<br><br># 3. Download Joomla<br>cd /tmp<br>sudo wget https://downloads.joomla.org/cms/joomla4/4-4-2/Joomla_4-4-2-Stable-Full_Package.zip?format=zip -O joomla.zip<br><br># 4. Unzip Joomla Into Web Root<br>sudo rm -rf /var/www/html/*<br>sudo unzip joomla.zip -d /var/www/html<br><br># 5. Set Permissions<br>sudo chown -R www-data:www-data /var/www/html<br>sudo chmod -R 755 /var/www/html<br><br># 6. Create Required Joomla Folders<br>sudo mkdir -p /var/www/html/logs<br>sudo mkdir -p /var/www/html/tmp<br>sudo touch /var/www/html/logs/index.html<br>sudo touch /var/www/html/tmp/index.html<br><br># 7. Enable Apache Rewrite Module<br>sudo a2enmod rewrite<br>sudo systemctl restart apache2<br><br># 8. Test PHP (optional)<br>echo &quot;&lt;?php phpinfo(); ?&gt;&quot; | sudo tee /var/www/html/test.php<br><br># Now go to: http://&lt;your-ec2-public-ip&gt; and the Joomla installation page will appear</pre><p>After this we will be able to view the Joomla installation page on our instance IP and follow the same procedure as we did the first time setting up our instance.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*5gyPqsAkw7JZ9ylN7Pp3Jw.png" /></figure><h3>Step 8: Set up and ALB for traffic distribution</h3><p>We will create an Application Load Balancer to split traffic between the previously created EC2 instance and the recently created one.</p><p>Some key configurations such as using the public subnets that our instances reside in and creating a target group with both of our instances will be needed.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*8jr77aofRTWm640r-GlexQ.png" /></figure><p>To confirm our ALB is working, we can use the DNS name url in the load balancers description and reload the page and ensure the page is slightly different each time (this means the ALB is rotating between the two IPs)</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*pVymlLMmjl9q5lI1istQuw.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*V1xR7RpsWeM0zIHA-p9wwg.png" /></figure><p>To improve scalability and fault tolerance, we implemented an <strong>Application Load Balancer (ALB)</strong> in front of two EC2 instances running Joomla. This step marks a transition from a simple web setup to a more <strong>production-grade infrastructure</strong> with built-in <strong>high availability and load distribution</strong>.</p><p>By distributing incoming HTTP traffic across two Joomla EC2 instances (each in a separate Availability Zone), the ALB ensures that user requests are handled reliably — even if one instance fails. This approach aligns with real-world patterns used in scalable web applications.</p><h3>Step 9: Controlling traffic for our instances</h3><p>Let’s ensure that our users can only access our application from our ALB, not the security groups directly.</p><p>First we can create a security group and then attach it to our ALB</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*ZxhfUVRQI_IPLqmj-VN1IQ.png" /></figure><p>Once created we can attach that security group to our ALB</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*XNPoCVHvgAJGUVzK5IJ1uw.png" /></figure><p>Now we’ll ensure that our instances only accept traffic from our ALB, so that public users from any IP cannot access our application directly.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*MN5O_Gayk6XFoDzFZ3urWg.png" /></figure><p>We can test this by trying to access one or both of our instances directly via their IP and we should get a connection timeout.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*VbZSKMrtJCtTlotFpihaYg.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/proxy/1*V1xR7RpsWeM0zIHA-p9wwg.png" /><figcaption>The ALB being publicly accessible</figcaption></figure><p>So now the traffic flow is: Users (internet) → ALB → EC2 instances.</p><p>To tighten the security of our two-tier architecture, we configured the EC2 instances to <strong>only accept traffic from the Application Load Balancer (ALB)</strong>. This was done by modifying the EC2 security group to allow inbound HTTP traffic exclusively from the ALB’s security group, rather than from the open internet. As a result, users can only access the Joomla application by going through the ALB, which acts as a controlled entry point, ensuring proper load distribution and preventing direct access to the backend instances. This step enhances both <strong>security</strong> and <strong>high availability</strong> — key goals of our architecture.</p><h3><strong>Wrapping It All Up: Scaling, Availability, and Security in Action</strong></h3><p>By adding a second EC2 instance and placing both behind an Application Load Balancer (ALB), I extended this project beyond basic functionality into real-world infrastructure design. The ALB acts as the entry point for all public traffic, distributing incoming requests evenly across multiple Joomla web servers — dramatically improving both <strong>availability</strong> and <strong>resilience</strong>.</p><p>I also implemented <strong>network-layer security</strong> by tightening the EC2 instance rules. Instead of allowing the world to connect directly to our instances, I configured them to only accept traffic from the ALB. This aligns with best practices in cloud security and represents a key design decision: users should never hit the web server directly — the ALB becomes the controlled front door.</p><p>This phase of the project gave real hands-on experience with:</p><ul><li>Auto-scaling architecture patterns</li><li>Application Load Balancers and target groups</li><li>Network isolation through Security Groups and subnet planning</li><li>Secure routing of traffic across tiers</li></ul><p>This project stands out as the most comprehensive, real-world architecture I’ve built to date. It reflects a professional-grade deployment pattern used across the industry — a fully segmented two-tier application environment with decoupled frontend and backend layers, secured network boundaries, and load-balanced infrastructure for high availability. From configuring a custom VPC and deploying a CMS on EC2 to setting up a MySQL RDS instance in a private subnet and distributing traffic through an Application Load Balancer, this project mirrors the kinds of infrastructure challenges tackled by real DevOps and Cloud Engineers in production environments. It’s not just about using AWS services — it’s about designing resilient systems that scale, stay secure, and deliver consistent performance.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/480/1*BH1xoKvDttZ-axeuJJffjA.gif" /></figure><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=8a287c6ec62a" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Using IaC to Deploy Scalable Infrastructure on AWS with CloudFormation]]></title>
            <link>https://medium.com/@abdulshaikhr99/using-iac-to-deploy-scalable-infrastructure-on-aws-with-cloudformation-fe44a5513528?source=rss-c47a96f96b0a------2</link>
            <guid isPermaLink="false">https://medium.com/p/fe44a5513528</guid>
            <category><![CDATA[infrastructure-as-code]]></category>
            <category><![CDATA[cloud]]></category>
            <category><![CDATA[networking]]></category>
            <category><![CDATA[application-development]]></category>
            <category><![CDATA[devops]]></category>
            <dc:creator><![CDATA[Abdul Rahim Shaikh]]></dc:creator>
            <pubDate>Mon, 17 Mar 2025 16:01:19 GMT</pubDate>
            <atom:updated>2025-03-17T16:01:19.794Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/942/1*-eMbumwg-ya2ZlMLKH1L7A.jpeg" /></figure><p>In this project, we illustrate the deployment of a robust, scalable, and automated infrastructure for Aurora Digital, a rapidly growing fictional online retailer specializing in luxury home goods. To effectively manage growing customer traffic and to ensure seamless experiences during major sales events, Aurora Digital requires a resilient infrastructure. We accomplish this using AWS CloudFormation and Infrastructure as Code (IaC).</p><h3>Infrastructure as Code (IaC)</h3><p>Infrastructure as Code (IaC) is the practice of managing and provisioning technology stacks through software and code rather than manual processes. It allows for automation, scalability, consistency, and collaboration among team members. IaC enables version control, reduces errors, and facilitates rapid and repeatable deployments.</p><p>AWS CloudFormation is a powerful IaC service provided by Amazon Web Services that allows users to describe and provision AWS infrastructure resources using YAML or JSON templates. This project leverages CloudFormation’s YAML syntax for readability and ease of management.</p><h3>Why Use AWS CloudFormation and YAML?</h3><ul><li><strong>Automation:</strong> Automatically provision and manage complex AWS environments with minimal manual intervention.</li><li><strong>Consistency:</strong> Deploy repeatable and consistent environments for development, staging, and production.</li><li><strong>Version Control:</strong> Easily track and revert infrastructure changes using version control systems like Git.</li><li><strong>Scalability:</strong> Quickly scale infrastructure in response to demand changes using Auto Scaling Groups (ASGs).</li></ul><h3>Project Objectives</h3><p>For Aurora Digital, the infrastructure should meet the following objectives:</p><ul><li><strong>Automation:</strong> Fully automated provisioning of infrastructure resources.</li><li><strong>Scalability:</strong> Dynamic scaling through an Auto Scaling Group.</li><li><strong>High Availability:</strong> Instances distributed across multiple subnets and availability zones.</li><li><strong>Security:</strong> Controlled network access through security groups.</li></ul><p>Let’s walk through the deployment process for this architecture.</p><h4>Step 1: Define Infrastructure Requirements</h4><p>We first clearly define the architecture needed for Aurora Digital:</p><ul><li>An Auto Scaling Group that dynamically adjusts EC2 instances based on demand.</li><li>Apache Web Server installed automatically on each instance.</li><li>Deployment in at least three AWS public subnets for fault tolerance and high availability.</li><li>Security groups configured to allow HTTP and SSH traffic appropriately.</li></ul><p>All of this will go into our YAML file as our template which we will add to shortly.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*RbDrQ7jEAYyWe0uP1cl72w.png" /></figure><p>We begin by navigating to CloudFormation in the AWS management console and creating a stack with an existing template.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*adCaMQcqqUmv94x2ZEx8kw.png" /></figure><p>We can choose between YAML or JSON templates. In this documentation we decided to go with YAML since it is more human-friendly and better for provisioning resources in AWS as compared to JSON.</p><p>While this isn’t a YAML crash course, we can go over the code used for our template on a high level and what key parts mean.</p><h3><strong>YAML template:</strong></h3><pre>AWSTemplateFormatVersion: &#39;2010-09-09&#39;<br>Description: &gt;<br>  Advanced CloudFormation template using Launch Templates to deploy an Auto Scaling Group (ASG)<br>  with Apache web servers on t2.micro instances. The ASG spans at least three public subnets,<br>  with a minimum of 2 instances and a maximum of 5 instances. Each instance installs Apache<br>  and serves a custom web page via UserData.<br><br>Parameters:<br>  VpcId:<br>    Description: &quot;The VPC ID where resources will be deployed&quot;<br>    Type: AWS::EC2::VPC::Id<br>  SubnetIds:<br>    Description: &quot;List of at least 3 public subnet IDs where the ASG instances will be launched&quot;<br>    Type: List&lt;AWS::EC2::Subnet::Id&gt;<br>  InstanceType:<br>    Description: &quot;EC2 instance type&quot;<br>    Type: String<br>    Default: &quot;t2.micro&quot;<br>    AllowedValues:<br>      - t2.micro<br>      - t2.small<br>      - t2.medium<br>  KeyName:<br>    Description: &quot;Name of an existing EC2 KeyPair for SSH access&quot;<br>    Type: AWS::EC2::KeyPair::KeyName<br>  AllowedIP:<br>    Description: &quot;IP address allowed for SSH access (CIDR notation)&quot;<br>    Type: String<br>    Default: &quot;0.0.0.0/0&quot;  # For production, consider limiting access to your IP<br>  LatestAmiId:<br>    Description: &quot;Amazon Linux 2 AMI from SSM Parameter Store&quot;<br>    Type: AWS::SSM::Parameter::Value&lt;AWS::EC2::Image::Id&gt;<br>    Default: &quot;/aws/service/ami-amazon-linux-latest/amzn2-ami-hvm-x86_64-gp2&quot;<br><br>Resources:<br>  WebServerSecurityGroup:<br>    Type: AWS::EC2::SecurityGroup<br>    Properties:<br>      GroupDescription: &quot;Allow HTTP (port 80) and SSH (port 22) access&quot;<br>      VpcId: !Ref VpcId<br>      SecurityGroupIngress:<br>        - IpProtocol: tcp<br>          FromPort: 22<br>          ToPort: 22<br>          CidrIp: !Ref AllowedIP<br>        - IpProtocol: tcp<br>          FromPort: 80<br>          ToPort: 80<br>          CidrIp: &quot;0.0.0.0/0&quot;<br><br>  WebServerLaunchTemplate:<br>    Type: AWS::EC2::LaunchTemplate<br>    Properties:<br>      LaunchTemplateName: &quot;WebServerLaunchTemplate&quot;<br>      LaunchTemplateData:<br>        ImageId: !Ref LatestAmiId<br>        InstanceType: !Ref InstanceType<br>        KeyName: !Ref KeyName<br>        SecurityGroupIds:<br>          - !Ref WebServerSecurityGroup<br>        UserData:<br>          Fn::Base64: !Sub |<br>            #!/bin/bash<br>            yum update -y<br>            yum install -y httpd<br>            systemctl start httpd<br>            systemctl enable httpd<br>            echo &quot;&lt;html&gt;&lt;body&gt;&lt;h1&gt;Welcome to the Advanced ASG Web Server&lt;/h1&gt;<br>            &lt;p&gt;This is a custom web page served by an auto scaling group using Launch Templates.&lt;/p&gt;&lt;/body&gt;&lt;/html&gt;&quot; &gt; /var/www/html/index.html<br><br>  WebServerASG:<br>    Type: AWS::AutoScaling::AutoScalingGroup<br>    Properties:<br>      VPCZoneIdentifier: !Ref SubnetIds<br>      MinSize: &quot;2&quot;<br>      MaxSize: &quot;5&quot;<br>      DesiredCapacity: &quot;2&quot;<br>      HealthCheckType: EC2<br>      HealthCheckGracePeriod: 300<br>      LaunchTemplate:<br>        LaunchTemplateId: !Ref WebServerLaunchTemplate<br>        Version: !GetAtt WebServerLaunchTemplate.LatestVersionNumber<br><br>Outputs:<br>  ASGName:<br>    Description: &quot;Name of the Auto Scaling Group&quot;<br>    Value: !Ref WebServerASG</pre><p>The above CloudFormation template automates the creation of AWS infrastructure for Aurora Digital. It deploys an Auto Scaling Group (ASG) managing EC2 instances configured as Apache web servers, providing scalability, reliability, and responsiveness to varying traffic.</p><h3>Parameters:</h3><p>These parameters allow configuration and flexibility when deploying the stack:</p><pre>Parameters:<br>  VpcId:<br>    Type: AWS::EC2::VPC::Id<br>    Description: &quot;Aurora Digital&#39;s VPC for resource deployment.&quot;<br>  SubnetIds:<br>    Description: &quot;At least 3 public subnet IDs for the Auto Scaling Group.&quot;<br>    Type: List&lt;AWS::EC2::Subnet::Id&gt;<br>  InstanceType:<br>    Type: String<br>    Default: t2.micro<br>    AllowedValues:<br>      - t2.micro<br>      - t2.small<br>      - t2.medium<br>  KeyName:<br>    Type: AWS::EC2::KeyPair::KeyName<br>  AllowedIP:<br>    Type: String<br>    Default: 0.0.0.0/0<br>  LatestAmiId:<br>    Type: AWS::SSM::Parameter::Value&lt;AWS::EC2::Image::Id&gt;<br>    Default: /aws/service/ami-amazon-linux-latest/amzn2-ami-hvm-x86_64-gp2</pre><h3>Resources</h3><p><strong>WebServerSecurityGroup</strong> Defines network access control allowing SSH and HTTP traffic:</p><pre>WebServerSecurityGroup:<br>  Type: AWS::EC2::SecurityGroup<br>  Properties:<br>    GroupDescription: Allow HTTP and SSH access for Aurora Digital<br>    VpcId: !Ref VpcId<br>    SecurityGroupIngress:<br>      - IpProtocol: tcp<br>        FromPort: 22<br>        ToPort: 22<br>        CidrIp: 0.0.0.0/0<br>      - IpProtocol: tcp<br>        FromPort: 80<br>        ToPort: 80<br>        CidrIp: 0.0.0.0/0</pre><p><strong>WebServerLaunchTemplate</strong> Defines EC2 instance configurations including automatic Apache setup:</p><pre>WebServerLaunchTemplate:<br>  Type: AWS::EC2::LaunchTemplate<br>  Properties:<br>    LaunchTemplateData:<br>      ImageId: !Ref LatestAmiId<br>      InstanceType: !Ref InstanceType<br>      KeyName: !Ref KeyName<br>      SecurityGroupIds:<br>        - !Ref WebServerSecurityGroup<br>      UserData:<br>        Fn::Base64: !Sub |<br>          #!/bin/bash<br>          yum update -y<br>          yum install -y httpd<br>          systemctl start httpd<br>          systemctl enable httpd<br>          echo &quot;&lt;html&gt;&lt;body&gt;&lt;h1&gt;Welcome to Aurora Digital&#39;s Auto Scaling Web Server&lt;/h1&gt;&lt;p&gt;Ensuring reliability and scalability for our customers.&lt;/p&gt;&lt;/body&gt;&lt;/html&gt;&quot; &gt; /var/www/html/index.html</pre><p><strong>WebServerASG</strong> (Auto Scaling Group) Automatically manages EC2 instances, ensuring optimal availability:</p><pre>WebServerASG:<br>  Type: AWS::AutoScaling::AutoScalingGroup<br>  Properties:<br>    VPCZoneIdentifier: !Ref SubnetIds<br>    LaunchTemplate:<br>      LaunchTemplateId: !Ref WebServerLaunchTemplate<br>      Version: !GetAtt WebServerLaunchTemplate.LatestVersionNumber<br>    MinSize: &#39;2&#39;<br>    MaxSize: &#39;5&#39;<br>    DesiredCapacity: &#39;2&#39;<br>    HealthCheckType: EC2<br>    HealthCheckGracePeriod: 300</pre><p><strong>Outputs</strong> Provides reference information for created resources:</p><pre>Outputs:<br>  ASGName:<br>    Description: Name of the Auto Scaling Group<br>    Value: !Ref WebServerASG</pre><p>The above YAML code will provide for a fully ready-to-go template we can use in the template section.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Oy5DZvx4qZc38HuSzt80QQ.png" /></figure><h3><strong>Step 2: Deploy Infrastructure</strong></h3><p>Once we have taken care of the the template section, we can specify all the details as shown below and click “Next”.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*CTmSNBlBdZ3AGNzQR728VA.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*6s6puxrdZzv4zI7sKYQ8qg.png" /></figure><ul><li>Allowed IP is set to everywhere (0.0.0.0/0) for demo purposes.</li><li>KeyName must exist prior to selecting an option.</li><li>A VPC must exist in your management console to be able to select it in this section.</li></ul><h3><strong>Validate Deployed Infrastructure</strong></h3><p>Once we have successfully deployed our stack, we can check to see if our specified resources have been provisioned.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/986/1*dpcnYvCeC3yFJfONajlzig.png" /></figure><p><strong>Check for Instances creation:</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*5ZyJGM9QUWRF3esTWydO4Q.png" /></figure><p><strong>Check for ASG creation:</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*dsWPj9eQUNQZMiqV59VzyA.png" /></figure><p>Check EC2 instances to see if they are serving our Apache web page:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/802/1*IDWUZ8g7g3AdtOgHZx-aXg.png" /></figure><p>We can also test if our ASG is working correctly by rendering instances as unhealthy and ensuring the minimum of 2 instances is created.</p><h3>Putting It All Together</h3><p>By combining AWS CloudFormation with Infrastructure as Code practices, we have successfully created a scalable, highly available, and secure web architecture. We began by clearly defining our infrastructure needs, setting parameters for flexibility, and using best practices to create an effective YAML template. The template automates the provisioning of Apache web servers, manages instance scaling through an Auto Scaling Group, and ensures robust security with well-defined Security Groups.</p><p>Through our YAML template, we emphasized scalability by dynamically managing EC2 instances across multiple subnets, ensuring reliable web hosting. Automation significantly reduced manual interventions, reducing errors, and streamlining deployment processes.</p><p>This entire approach demonstrates a powerful method for rapidly deploying and managing complex infrastructure, enabling Aurora Digital to seamlessly scale with customer demand.</p><h3>Conclusion</h3><p>Leveraging AWS CloudFormation and Infrastructure as Code has proven immensely effective for building resilient, secure, and scalable cloud solutions. This project was about key skills crucial for modern cloud engineers: automated deployment, infrastructure scalability, and robust security practices. By employing automation, we have reduced human error, improved efficiency, and significantly accelerated the delivery of infrastructure solutions.</p><p>Looking ahead, the skills and methods demonstrated in this project represent more than technical capability — they embody the future of digital innovation. Embracing automation and scalable architectures enables organizations like Aurora Digital not just to meet today’s demands but to anticipate and effortlessly adapt to tomorrow’s challenges. This approach empowers companies to innovate faster, respond proactively to market changes, and maintain a competitive advantage in a rapidly evolving digital landscape.</p><p>Ultimately, adopting Infrastructure as Code practices and cloud automation equips teams with the agility, efficiency, and reliability essential for thriving in our ever-accelerating digital world. Through this project, we’ve demonstrated how innovative approaches to infrastructure can unlock immense potential, laying the foundation for growth, excellence, and sustained success.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/450/1*rvNKcUWWimyCFeT25clEjw.gif" /></figure><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=fe44a5513528" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Creating A High Availability Two-Tier App On AWS]]></title>
            <link>https://medium.com/@abdulshaikhr99/creating-a-high-availability-two-tier-app-on-aws-7a0ea35758dd?source=rss-c47a96f96b0a------2</link>
            <guid isPermaLink="false">https://medium.com/p/7a0ea35758dd</guid>
            <category><![CDATA[cloud-computing]]></category>
            <category><![CDATA[networking]]></category>
            <category><![CDATA[cloud-engineering]]></category>
            <category><![CDATA[virtual-private-cloud]]></category>
            <category><![CDATA[devops]]></category>
            <dc:creator><![CDATA[Abdul Rahim Shaikh]]></dc:creator>
            <pubDate>Mon, 10 Mar 2025 12:02:33 GMT</pubDate>
            <atom:updated>2025-03-10T12:02:33.954Z</atom:updated>
            <content:encoded><![CDATA[<p>A practical guide to deploying a secure, scalable AWS solution using a VPC, Auto Scaling Groups, and Application Load Balancer.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/845/1*9abokUD6qDqK-AK_86n1ow.png" /></figure><h3>Introduction</h3><p>In this walkthrough, we demonstrate step-by-step how a secure, scalable, and highly available two-tier web architecture on AWS is built. This project is based on a scenario for a fictional bank, Level-Up Bank, aiming to transition from traditional on-premises infrastructure to a cloud-based solution. The primary reasons for this migration include improved scalability to handle fluctuating customer demand, enhanced reliability and uptime using AWS’s multiple Availability Zones, strengthened security through AWS-managed services, cost-efficiency via pay-as-you-go pricing, and increased flexibility to rapidly deploy and test new services. Specifically, this guide covers creating a custom VPC, public subnets, security groups, EC2 instances with Apache web servers, an Auto Scaling Group, and an Application Load Balancer (ALB).</p><h3>Step 1: Creating the Network Foundation (VPC)</h3><p>To establish a private and secure network environment for Level-Up Bank, we start by setting up a Virtual Private Cloud (VPC). This VPC functions as a logically isolated section of the AWS cloud, providing control over network configurations, IP address ranges, and internet connectivity.</p><figure><img alt="" src="https://cdn-images-1.medium.com/proxy/1*gtzK5ArdfldxwelFnqzpRw.png" /></figure><p>The CIDR block is set as 10.10.0.0/16, allowing for a large number of IP addresses to be assigned within our network. While most settings remain default, choosing a meaningful name like LevelUpBank-VPC ensures clarity in architecture and consistency within the organization. The VPC is the backbone of our infrastructure, housing all network resources and facilitating internal communication between service</p><h3>Step 2: Establishing High Availability (Public Subnets)</h3><p>Within the VPC, public subnets are created across multiple AWS Availability Zones (AZs) to ensure high availability and fault tolerance. Each public subnet allows instances to connect directly to the internet, enabling redundancy and minimizing downtime if an Availability Zone experiences issues.</p><p>Subnets function as logical partitions within our VPC, helping to isolate resources based on business needs, security policies, and scalability requirements. We deploy three public subnets within the us-east-1 region, distributed across different AZs (us-east-1a, us-east-1b, and us-east-1c).</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*lEvuBZcZ9_guMUvgcEvduQ.png" /></figure><p>The CIDR allocations for our subnets are:</p><ul><li>10.10.1.0/24 (Public Subnet 1 - AZ: us-east-1a)</li><li>10.10.2.0/24 (Public Subnet 2 - AZ: us-east-1b)</li><li>10.10.3.0/24 (Public Subnet 3 - AZ: us-east-1c)</li></ul><p>By distributing instances across multiple subnets in different AZs, we ensure resilience in case of infrastructure failure in one AZ.</p><h3>Step 3: Providing Internet Access (Internet Gateway)</h3><p>To allow external traffic to reach our instances within the public subnets, an Internet Gateway (IGW) is attached to the VPC. The IGW enables communication between resources inside the VPC and the internet, which is essential for web-facing applications like the one being built for Level-Up Bank.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*OFwnYt7SExdsfXdXdszuTg.png" /></figure><p>Once created, the IGW is attached to LevelUpBank-VPC. This means:</p><ul><li>Any instance within the public subnets can communicate externally.</li><li>A route is added to the VPC’s route table to direct outbound traffic (0.0.0.0/0) through the IGW.</li><li>Resources requiring private networking remain isolated unless explicitly configured otherwise.</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*_ajoxS6VmKPobJJs9TLa0Q.png" /></figure><h3>Step 4: Securing the Infrastructure (Security Groups)</h3><p>Security groups are implemented to define and restrict the types of traffic that can reach the application infrastructure. They act as virtual firewalls at the instance level, controlling inbound and outbound traffic.</p><ul><li><strong>Application Load Balancer Security Group (ALB-SG)</strong>: Allows inbound HTTP requests (port 80) from anywhere on the internet, ensuring public access to the application.</li><li><strong>Web Server Security Group (WebServer-SG)</strong>: Restricts inbound HTTP traffic to only allow connections from ALB-SG. This prevents direct public access to the EC2 instances, significantly improving security.</li></ul><p>These configurations enforce <strong>the principle of least privilege</strong>, ensuring that only necessary traffic is allowed while blocking unauthorized access.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*orakg0A9N6Y7aQc_ZZebcA.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/proxy/1*I8G0cV9xN26dSptU_vm1bg.png" /><figcaption>SSH permission does not need to be enabled unless we would like to access our instance from the terminal</figcaption></figure><h3>Step 5: Automating the Infrastructure (EC2 Launch Template)</h3><p>To simplify and standardize the deployment of web servers, an EC2 Launch Template is created. This template defines:</p><ul><li>Instance type: t2.micro (cost-effective for this deployment).</li><li>Operating system: Amazon Linux 2.</li><li>Security group: WebServer-SG.</li><li>User-data script: Automates Apache installation and configuration to serve our web application upon instance launch.</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/proxy/1*112HlNGlaSPz60ig6_hiFw.png" /></figure><p>This automation ensures consistency, eliminates manual setup, and allows instances to be rapidly deployed or replaced as needed.</p><h3>Step 6: Enabling Scalability (Auto Scaling Group)</h3><p>An Auto Scaling Group (ASG) ensures application availability and scalability by dynamically launching instances based on demand. The ASG:</p><ul><li>Maintains a <strong>minimum of 2 instances</strong> for redundancy.</li><li>Can scale up to <strong>5 instances</strong> if needed.</li><li>Spreads instances across our three public subnets for high availability.</li></ul><p>This setup ensures load distribution and resilience, automatically adjusting capacity to maintain performance.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*rPHHGoPHnkU0S8vZUaTH_w.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*jjzawwR1fBeIN5JjG_9tIQ.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*H2dWKKcMnpgZqLHXVlzAzg.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*7WlUUMnZ3NbcYGJ_ntb4yg.png" /></figure><h3>Step 7: Distributing Traffic And Decoupling (Application Load Balancer)</h3><p>An Application Load Balancer (ALB) is deployed to evenly distribute incoming HTTP requests to the web servers within the Auto Scaling Group. The ALB continuously monitors the health of the instances and ensures traffic is only routed to healthy servers, maintaining a responsive and robust user experience.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*1VVbnnBC8Vpnx8ASTMl4Bg.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*8g2_5dM2duNPxpDLHMLr4g.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*77cmIs4R241B1T5s6MHMwQ.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/proxy/1*U6G4NoNZXgRK4UbMv2bKKw.png" /></figure><p>By decoupling traffic distribution from individual instances, the ALB ensures:</p><ul><li>No single instance becomes a bottleneck.</li><li>If an instance fails, traffic is redirected to the remaining healthy instances.</li><li>The application can handle fluctuations in traffic without manual intervention.</li></ul><h3>Step 8: Integration of ASG and ALB</h3><p>The Auto Scaling Group is integrated with the ALB, automatically registering new EC2 instances with the load balancer and removing unhealthy instances. This seamless integration further enhances the availability and reliability of the application.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*pE0ID1ul3eiYDbX4xuw-HQ.png" /></figure><p>Our EC2 instances are <strong>registered as healthy targets in the ALB Target Group</strong>, meaning:</p><ul><li>Traffic is evenly distributed across available instances.</li><li>Instances can be replaced automatically in case of failure.</li><li>The system can scale without affecting performance or availability.</li></ul><h3>Putting it all together</h3><p>The final architecture is validated by accessing the ALB URL through a web browser, confirming that traffic is appropriately distributed, and the custom Apache web page (“Welcome to Level-Up Bank Webserver!”) is accessible.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/735/1*hePHkpQq1a48AT0KVHZgcQ.png" /></figure><p>We can also ensure our application is working properly in the command line by running the following command:</p><pre>curl -I http://&lt;ALB-DNS-Name&gt;</pre><p>For us it will be:</p><pre>curl -I http://webserver-alb-2041100551.us-east-1.elb.amazonaws.com/</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*M7lUblQOuuOIqZJFPEDAyw.png" /></figure><p>We get a 200 success status code which means everything is working like a well-oiled machine!</p><ul><li>For users on windows using PowerShell for this confirmation, ensure there are quotes around the http address otherwise the shell won’t be able to interpret the command accurately.</li></ul><h3>Looking Beyond: The Power of Cloud Innovation</h3><p>With this project, we’ve not only built a robust, scalable, and secure web infrastructure, but we’ve also unlocked a glimpse into the future of cloud computing. AWS empowers organizations to break free from the limitations of traditional infrastructure, offering agility, resilience, and automation at an unprecedented scale. The ability to dynamically scale resources, optimize costs, and enhance security without excessive operational overhead is transforming the way businesses operate in the digital age.</p><p>But this is just the beginning. The cloud is continuously evolving, with advancements like serverless computing, Infrastructure as Code (IaC), and AI-driven optimizations pushing the boundaries of efficiency and innovation. Organizations that embrace these technologies gain a competitive edge, ensuring their applications remain resilient, scalable, and adaptable to ever-changing demands. The flexibility of cloud-native architectures allows businesses to innovate faster, deploy globally, and maintain high availability with minimal human intervention.</p><p>For engineers, architects, and cloud enthusiasts, mastering cloud infrastructure isn’t just about technical skills — it’s about adopting a new mindset. The cloud isn’t just a tool; it’s a gateway to endless possibilities. Whether you’re refining architectures, optimizing costs, or pushing the boundaries of scalability, the only limit is how far you’re willing to go. The future belongs to those who can harness the full potential of the cloud. Now, the question is — what will you build next? 🚀</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/480/1*n1UFTEsHdDWRZNoTi4aTDQ.gif" /></figure><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=7a0ea35758dd" width="1" height="1" alt="">]]></content:encoded>
        </item>
    </channel>
</rss>