<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Kevin Kiruri on Medium]]></title>
        <description><![CDATA[Stories by Kevin Kiruri on Medium]]></description>
        <link>https://medium.com/@kevinkiruri?source=rss-4f39f6231241------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/1*dmbNkD5D-u45r44go_cf0g.png</url>
            <title>Stories by Kevin Kiruri on Medium</title>
            <link>https://medium.com/@kevinkiruri?source=rss-4f39f6231241------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Sat, 21 Nov 2026 15:20:20 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@kevinkiruri/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[Building a Serverless Sales Analytics Pipeline Using S3, Glue, Athena and Lambda]]></title>
            <link>https://kevinkiruri.medium.com/building-a-serverless-sales-analytics-pipeline-using-s3-glue-athena-and-lambda-96efbb82929c?source=rss-4f39f6231241------2</link>
            <guid isPermaLink="false">https://medium.com/p/96efbb82929c</guid>
            <category><![CDATA[aws]]></category>
            <category><![CDATA[aws-s3]]></category>
            <category><![CDATA[serverless]]></category>
            <category><![CDATA[aws-athena]]></category>
            <category><![CDATA[aws-glue]]></category>
            <dc:creator><![CDATA[Kevin Kiruri]]></dc:creator>
            <pubDate>Mon, 02 Nov 2026 11:57:37 GMT</pubDate>
            <atom:updated>2026-11-02T11:59:02.246Z</atom:updated>
            <content:encoded><![CDATA[<h3>Introduction</h3><p>Analysing sales data manually can take a lot of time, especially for retail businesses with many sales transactions and across different branches. Business owners need to know how much they are selling, which products sell best, and how each branch is performing. However, finding these insights from raw sales files can involve a lot of manual work.</p><p>For this project, I wanted to explore how AWS could be used to automate this process. I built a serverless retail sales analytics platform using Amazon S3, AWS Glue, Amazon Athena, and AWS Lambda.</p><p>The solution stores sales data in Amazon S3, uses AWS Glue to organise and catalogue it, and Amazon Athena to run SQL queries for analysing sales performance. I then used AWS Lambda to automate the process so that whenever a new CSV file is uploaded, the analytics queries are triggered automatically, and the results are saved back to S3.</p><p>In this blog, I’ll walk through how I built the solution and how these AWS services work together to automate sales data analysis.</p><h3>Architecture Overview</h3><p>The solution uses a serverless, event-driven architecture built using five main AWS services: Amazon S3, AWS Glue, Amazon Athena, IAM, and AWS Lambda.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*GptJyTsIT3hN7cCYId7zUw.png" /><figcaption>Figure 1: Architecture of the serverless sales analytics pipeline on AWS.</figcaption></figure><p>The automation starts when a sales CSV file is uploaded to an S3 bucket. An S3 event notification detects the new file and triggers a Lambda function. The function submits predefined SQL queries to Amazon Athena.</p><p>Athena uses the table metadata stored in the AWS Glue Data Catalog to understand the structure and location of the sales data. It then reads the data from S3, runs the queries, and saves the results in a separate folder in the same S3 bucket.</p><p>This means that whenever a new sales CSV file is uploaded to the specified folder, the analysis process can run automatically without me having to submit each query manually.</p><h3>AWS Services Used</h3><p>I used five AWS services to build the sales analytics pipeline. Each service had a specific role in storing, organising, analysing, and automating the sales data.</p><p><strong>1. Amazon S3</strong></p><p>Amazon S3 is a fully managed object storage service. I used it to store the raw retail sales CSV files and the results generated by Athena.</p><p><strong>2. AWS Glue</strong></p><p>AWS Glue is a serverless data integration service. I used it’s crawler to identify the schema of the sales data and create metadata in the Glue Data Catalog.</p><p><strong>3. Amazon Athena</strong></p><p>Amazon Athena is a serverless query service that allows users to analyse data directly in Amazon S3 using SQL. I used it to run queries and generate sales insights.</p><p><strong>4. AWS Lambda</strong></p><p>AWS Lambda is a serverless compute service that runs code in response to events without requiring users to manage servers. I used it to automatically submit Athena queries whenever a new sales CSV file was uploaded to S3.</p><p><strong>5. AWS IAM</strong></p><p>AWS Identity and Access Management (IAM) is used to manage access to AWS resources. I used IAM roles and policies to give the Lambda function the permissions it needed to interact with other AWS services.</p><h3>Project Implementation</h3><ol><li><strong>Creating the S3 Bucket</strong></li></ol><p>I started by creating an S3 bucket named <strong>sales-analytics-app-2026</strong>. Inside the bucket, I created two folders to organise the project files:</p><ul><li><strong>raw-sales-data/</strong> — to store the uploaded retail sales CSV files.</li><li><strong>athena-results/</strong> — to store the results generated by Amazon Athena.</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*7pD2lKjkWr0WH1oK1FYgyg.png" /><figcaption><em>Figure 2: S3 bucket showing the raw sales data and Athena query results folders.</em></figcaption></figure><p>I then uploaded the retail sales dataset into the <strong>raw-sales-data/</strong> folder.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/716/1*9OFvPVfIIdkuR7P5UoGkYA.png" /><figcaption>Figure 3: Preview of the retail sales dataset showing the transaction records and columns used for analysis.</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*06bJ2zfNeoC1Oy5hoeTKNQ.png" /><figcaption><em>Figure 4: U</em>ploaded the retail sales dataset into the raw-sales-data/.</figcaption></figure><p><strong>2. Setting Up AWS Glue</strong></p><p>Next, I set up AWS Glue to identify the structure of the sales dataset. I created a database named <strong>sales_analytics_db</strong> and a crawler named <strong>sales-analytics-crawler</strong>.</p><p>I then configured the crawler to scan the <strong>raw-sales-data/</strong> folder in my S3 bucket. After running it, AWS Glue identified the columns and data types in the CSV file and created a table named <strong>raw_sales_data</strong> in the Glue Data Catalog. This made the sales data available for querying in Amazon Athena.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*6aCuFGM5BKq2GXg6o_h5Xg.png" /><figcaption><em>Figure 5: AWS Glue Data Catalog showing the sales_analytics_db database and raw_sales_data table.</em></figcaption></figure><p><strong>3. Running SQL Queries in Amazon Athena</strong></p><p>After AWS Glue created the Data Catalog table, I moved to Amazon Athena to analyse the sales data.</p><p>I selected the <strong>sales_analytics_db</strong> database and used the <strong>raw_sales_data</strong> table to write and run five SQL queries. I ran the queries manually to test the data and confirm that Athena could return the expected results.</p><p>The queries focused on answering common retail business questions:</p><ul><li><strong>Sales Summary:</strong> What was the total revenue, number of transactions, units sold and average transactions?</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*alSRB2WuulaF-bDsbFo5GA.png" /><figcaption>Figure 6: Amazon Athena showing the sales summary query and its results.</figcaption></figure><ul><li><strong>Revenue by Branch:</strong> Which branches generated the most revenue?</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*EZ90-2sVKPBgR8H0D3qSwQ.png" /><figcaption>Figure 7: Revenue generated by each retail branch.</figcaption></figure><ul><li><strong>Revenue by Category:</strong> Which product categories contributed the most to sales?</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*5EptfE0MdUCQ-LTgOLTMPA.png" /><figcaption>Figure 8: Sales performance grouped by product category.</figcaption></figure><ul><li><strong>Monthly Sales Trend:</strong> How did sales performance change from month to month?</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*t8uaG1OT2-oZvTcQ_wTn-g.png" /><figcaption>Figure 9: Monthly revenue and transaction trends from January to June 2026.</figcaption></figure><ul><li><strong>Top Five Products:</strong> Which products generated the highest revenue?</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*yMnlC-lt7ucaShfUD8GfmA.png" /><figcaption>Figure 10: The five products with the highest total revenue.</figcaption></figure><p>All five queries executed successfully, and Athena saved their results in the <strong>athena-results/</strong> folder in my S3 bucket.</p><p>This led me to the next part of the project: using AWS Lambda to automate the process. The idea was to allow the business owner to upload a new sales CSV file and have the queries run automatically without needing to interact with SQL.</p><p><strong>4. Setting Up IAM Permissions</strong></p><p>Before creating the Lambda function, I needed to make sure it had the right permissions to run the analytics queries. I created an IAM execution role named <strong>LambdaSalesAnalyticsExecutionRole</strong> and attached the <strong>AWSLambdaBasicExecutionRole</strong> policy to allow the function to write logs to Amazon CloudWatch.</p><p>I also added a custom policy called <strong>SalesAnalyticsLambdaAccessPolicy</strong>. This provided the permissions needed to submit queries to Athena, access the Glue Data Catalog and write query results to the S3 bucket.</p><p>With the execution role ready, the Lambda function had the access it needed to submit queries and save their results.</p><p><strong>5. Creating the Lambda Function</strong></p><p>After setting up the IAM permissions, I created a Lambda function named <strong>SalesAnalyticsAutomationFunction</strong>.</p><p>I added environment variables to provide the function with the details it needed to run the queries:</p><ul><li><strong>DATABASE_NAME:</strong> sales_analytics_db</li><li><strong>ATHENA_WORKGROUP:</strong> primary</li><li><strong>ATHENA_OUTPUT:</strong> s3://sales-analytics-app-2026/athena-results/</li></ul><p>The function purpose was to submit five predefined SQL queries to Amazon Athena. These queries covered the sales summary, revenue by branch, revenue by category, monthly sales trend and top five products.</p><pre>import json<br>import os<br>import boto3<br>import urllib.parse<br><br># AWS clients<br>athena = boto3.client(&quot;athena&quot;)<br><br># Environment variables<br>DATABASE_NAME = os.environ[&quot;DATABASE_NAME&quot;]<br>ATHENA_WORKGROUP = os.environ[&quot;ATHENA_WORKGROUP&quot;]<br>ATHENA_OUTPUT = os.environ[&quot;ATHENA_OUTPUT&quot;]<br><br># Five sales analytics queries<br>QUERIES = {<br>    &quot;sales_summary&quot;: &quot;&quot;&quot;<br>        SELECT<br>            SUM(total_amount) AS total_revenue,<br>            COUNT(DISTINCT transaction_id) AS total_transactions,<br>            SUM(quantity) AS total_units_sold,<br>            AVG(total_amount) AS average_transaction_value<br>        FROM raw_sales_data<br>    &quot;&quot;&quot;,<br><br>    &quot;revenue_by_branch&quot;: &quot;&quot;&quot;<br>        SELECT<br>            branch,<br>            SUM(total_amount) AS total_revenue,<br>            COUNT(DISTINCT transaction_id) AS total_transactions,<br>            SUM(quantity) AS total_units_sold<br>        FROM raw_sales_data<br>        GROUP BY branch<br>        ORDER BY total_revenue DESC<br>    &quot;&quot;&quot;,<br><br>    &quot;revenue_by_category&quot;: &quot;&quot;&quot;<br>        SELECT<br>            category,<br>            SUM(total_amount) AS total_revenue,<br>            SUM(quantity) AS total_units_sold,<br>            COUNT(DISTINCT transaction_id) AS total_transactions<br>        FROM raw_sales_data<br>        GROUP BY category<br>        ORDER BY total_revenue DESC<br>    &quot;&quot;&quot;,<br><br>    &quot;monthly_sales_trend&quot;: &quot;&quot;&quot;<br>        SELECT<br>            date_format(<br>                date_parse(transaction_date, &#39;%Y-%m-%d&#39;),<br>                &#39;%Y-%m&#39;<br>            ) AS sales_month,<br>            SUM(total_amount) AS total_revenue,<br>            COUNT(DISTINCT transaction_id) AS total_transactions,<br>            SUM(quantity) AS total_units_sold<br>        FROM raw_sales_data<br>        GROUP BY<br>            date_format(<br>                date_parse(transaction_date, &#39;%Y-%m-%d&#39;),<br>                &#39;%Y-%m&#39;<br>            )<br>        ORDER BY sales_month<br>    &quot;&quot;&quot;,<br><br>    &quot;top_five_products&quot;: &quot;&quot;&quot;<br>        SELECT<br>            product_name,<br>            SUM(total_amount) AS total_revenue,<br>            SUM(quantity) AS total_units_sold,<br>            COUNT(DISTINCT transaction_id) AS total_transactions<br>        FROM raw_sales_data<br>        GROUP BY product_name<br>        ORDER BY total_revenue DESC<br>        LIMIT 5<br>    &quot;&quot;&quot;<br>}<br><br><br>def lambda_handler(event, context):<br><br>    submitted_queries = []<br><br>    try:<br>        # Process the S3 event<br>        for record in event.get(&quot;Records&quot;, []):<br><br>            bucket = record[&quot;s3&quot;][&quot;bucket&quot;][&quot;name&quot;]<br>            key = urllib.parse.unquote_plus(<br>                record[&quot;s3&quot;][&quot;object&quot;][&quot;key&quot;]<br>            )<br><br>            # Only process CSV files in the raw-sales-data folder<br>            if not key.startswith(&quot;raw-sales-data/&quot;) or not key.endswith(&quot;.csv&quot;):<br>                print(f&quot;Skipping file: {key}&quot;)<br>                continue<br><br>            print(f&quot;Processing uploaded file: s3://{bucket}/{key}&quot;)<br><br>            # Submit all five queries to Athena<br>            for query_name, query_string in QUERIES.items():<br><br>                response = athena.start_query_execution(<br>                    QueryString=query_string,<br>                    QueryExecutionContext={<br>                        &quot;Database&quot;: DATABASE_NAME<br>                    },<br>                    WorkGroup=ATHENA_WORKGROUP,<br>                    ResultConfiguration={<br>                        &quot;OutputLocation&quot;: ATHENA_OUTPUT<br>                    }<br>                )<br><br>                query_id = response[&quot;QueryExecutionId&quot;]<br><br>                submitted_queries.append({<br>                    &quot;query_name&quot;: query_name,<br>                    &quot;query_execution_id&quot;: query_id<br>                })<br><br>                print(f&quot;Submitted {query_name}: {query_id}&quot;)<br><br>        print(&quot;All queries submitted successfully.&quot;)<br><br>        return {<br>            &quot;statusCode&quot;: 200,<br>            &quot;body&quot;: json.dumps({<br>                &quot;status&quot;: &quot;SUCCESS&quot;,<br>                &quot;submitted_query_count&quot;: len(submitted_queries),<br>                &quot;queries&quot;: submitted_queries<br>            })<br>        }<br><br>    except Exception as error:<br><br>        print(f&quot;Error processing sales data: {str(error)}&quot;)<br><br>        raise error</pre><p>I tested the function manually before connecting it to S3. The test completed successfully, and Lambda submitted all five queries to Athena. This confirmed that the function and its permissions were working as expected.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*UuAN1cBSfsl_rvMDK4R9vA.png" /><figcaption>Figure 11: Lambda function configured to submit the five sales analytics queries to Athena.</figcaption></figure><p><strong>6. Configuring the S3 Event Notification</strong></p><p>After setting up the Lambda function, I connected it to Amazon S3 so that the analysis could run automatically whenever a new sales file was uploaded.</p><p>In the S3 bucket, I created an event notification named <strong>SalesAnalyticsCSVUploadTrigger</strong>. I set it up to listen for object-created events, with the following filters:</p><ul><li><strong>Prefix:</strong> raw-sales-data/</li><li><strong>Suffix:</strong> .csv</li></ul><p>I then selected <strong>SalesAnalyticsAutomationFunction</strong> as the destination.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*q4-7GOtUU8Ivj9Di5UvO5A.png" /><figcaption>Figure 12: S3 event notification configured to trigger the Lambda function when a CSV file is uploaded.</figcaption></figure><p>With this configuration, uploading a new CSV file to the <strong>raw-sales-data/</strong> folder triggers the Lambda function. The function then submits SQL queries to Athena, which processes the sales data and saves the results in the <strong>athena-results/</strong> folder.</p><p>This completed the automation setup. The next step was to upload a new test file and confirm that the entire workflow worked as expected.</p><p><strong>7. Testing and Results</strong></p><p>I uploaded a small test CSV file named <strong>retail_sales_test.csv</strong> into the <strong>raw-sales-data/</strong> folder in my S3 bucket. The upload triggered the <strong>SalesAnalyticsAutomationFunction</strong> Lambda function automatically.</p><p>I then checked the function’s execution logs in Amazon CloudWatch. The logs showed that Lambda had detected the uploaded file and submitted all queries to Amazon Athena.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*6Jjhz2jv-vqa-056Isevew.png" /><figcaption>Figure 13: Athena query result files saved in the S3 results folder after the automated run.</figcaption></figure><p>Next, I opened Athena and checked the query execution history. All five queries had completed successfully. I also checked the <strong>athena-results/</strong> folder in S3 and confirmed that the query result files had been saved there.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*UuAN1cBSfsl_rvMDK4R9vA.png" /><figcaption>Figure 11: Lambda function configured to submit the five sales analytics queries to Athena.</figcaption></figure><h3>Limitations and Future Improvements</h3><p>Although the automation worked as expected, there are a few things I would improve in a future version.</p><p>The project used a sample dataset with a relatively small number of records. A production system would need to handle larger datasets and more frequent uploads.</p><p>A future improvement would be to build a dashboard using Amazon QuickSight or another visualisation tool. This would allow a business owner to view sales trends, compare and identify top-selling products without opening Athena or checking the saved query results in S3.</p><p>These improvements would make the solution more useful for a real retail business and better suited to a larger amount of sales data.</p><h3>Conclusion</h3><p>Retail businesses need a simple way to understand their sales performance without having to analyse CSV files manually or write SQL queries every time new data becomes available.</p><p>In this project, I built a serverless sales analytics pipeline that addresses this problem by bringing together Amazon S3, AWS Glue, Amazon Athena and AWS Lambda.</p><p>The result is a solution that reduces repetitive work and makes it easier to generate sales reports from newly uploaded data. Although this project used a sample dataset and does not yet include a visual dashboard, it provides a foundation that could be extended into a more complete retail analytics platform.</p><p>Happy building!</p><p>Follow me on <a href="https://www.linkedin.com/in/kevin-kiruri/">LinkedIn</a> for more demos and networking <a href="https://www.linkedin.com/in/kevin-kiruri/">Kevin Kiruri</a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=96efbb82929c" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Building Serverless Microservices on AWS with ECS Fargate, ECR, and Terraform]]></title>
            <link>https://kevinkiruri.medium.com/building-serverless-microservices-on-aws-with-ecs-fargate-ecr-and-terraform-4fe35bbc5327?source=rss-4f39f6231241------2</link>
            <guid isPermaLink="false">https://medium.com/p/4fe35bbc5327</guid>
            <category><![CDATA[terraform]]></category>
            <category><![CDATA[aws]]></category>
            <category><![CDATA[serverless]]></category>
            <category><![CDATA[microservices]]></category>
            <category><![CDATA[fargate]]></category>
            <dc:creator><![CDATA[Kevin Kiruri]]></dc:creator>
            <pubDate>Fri, 02 Jan 2026 17:36:52 GMT</pubDate>
            <atom:updated>2026-01-06T16:20:27.258Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/896/0*KW487vJVG9iAAuEy.jpg" /></figure><p>In the evolving world of cloud-native architectures, serverless doesn’t always mean Lambda. With Amazon ECS Fargate, you can run containers without managing servers, combining the scalability of containers with the simplicity of serverless operations.</p><p>In this article, we’ll build a serverless microservices architecture using AWS ECS Fargate, ECR, and Terraform.<br> Our setup includes two Django-based microservices:</p><p>Reader Service — handles read operations from a shared database.</p><p>Writer Service — handles write operations to the same database.</p><p>Both services connect to a shared Amazon RDS PostgreSQL instance. The entire infrastructure is defined and provisioned through Infrastructure as Code (IaC) using Terraform.</p><h3>Key Components</h3><p><strong>Amazon ECS Fargate</strong> — Runs containerized Django microservices without provisioning EC2 instances.</p><p><strong>Amazon ECR (Elastic Container Registry)</strong> — Hosts Docker images for both services.</p><p><strong>Amazon RDS (PostgreSQL)</strong> — Central database for the microservices.</p><p><strong>Application Load Balancer (ALB)</strong> — Distributes traffic between services.</p><p><strong>Terraform</strong> — Automates infrastructure provisioning.</p><h3>1. Containerizing the Microservices</h3><p>Each Django service (Reader and Writer) is containerized using Docker. Here is the docker compose file that builds both services and a local db container for local testing (test containers before pushing to cloud).</p><pre>  db:<br>    image: postgres:15<br>    container_name: postgres-db<br>    environment:<br>      - POSTGRES_DB=${DB_NAME:-library_db}<br>      - POSTGRES_USER=${DB_USER:-postgres}<br>      - POSTGRES_PASSWORD=${DB_PASSWORD:-password}<br>    ports:<br>      - &quot;5432:5432&quot;<br>    volumes:<br>      - postgres_data:/var/lib/postgresql/data<br>    healthcheck:<br>      test: [&quot;CMD-SHELL&quot;, &quot;pg_isready -U ${DB_USER:-postgres}&quot;]<br>      interval: 10s<br>      timeout: 5s<br>      retries: 5<br><br>  reader-service:<br>    container_name: reader-svc<br>    build: ./app/reader-service/reader<br>    ports:<br>      - &quot;8000:8000&quot;<br>    environment:<br>      - DB_USER=${DB_USER:-postgres}<br>      - DB_PASSWORD=${DB_PASSWORD:-password}<br>      - DB_NAME=${DB_NAME:-library_db}<br>      - DB_HOST=db<br>      - DB_PORT=5432<br>    depends_on:<br>      db:<br>        condition: service_healthy<br>    healthcheck:<br>      test: [&quot;CMD&quot;, &quot;curl&quot;, &quot;-f&quot;, &quot;http://localhost:8000/health/&quot;]<br>      interval: 30s<br>      timeout: 10s<br>      retries: 3<br>      start_period: 40s<br>    deploy:<br>      resources:<br>        limits:<br>          memory: 512M<br>        reservations:<br>          memory: 256M<br>    restart: unless-stopped<br><br>  writer-service:<br>    container_name: writer-svc<br>    build: ./app/writer-service/writer<br>    ports:<br>      - &quot;8001:8000&quot;<br>    environment:<br>      - DB_USER=${DB_USER:-postgres}<br>      - DB_PASSWORD=${DB_PASSWORD:-password}<br>      - DB_NAME=${DB_NAME:-library_db}<br>      - DB_HOST=db<br>      - DB_PORT=5432<br>    depends_on:<br>      db:<br>        condition: service_healthy<br>    healthcheck:<br>      test: [&quot;CMD&quot;, &quot;curl&quot;, &quot;-f&quot;, &quot;http://localhost:8000/health/&quot;]<br>      interval: 30s<br>      timeout: 10s<br>      retries: 3<br>      start_period: 40s<br>    deploy:<br>      resources:<br>        limits:<br>          memory: 512M<br>        reservations:<br>          memory: 256M<br>    restart: unless-stopped<br><br>volumes:<br>  postgres_data:</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/800/0*GnX9VVTHTuJeI40F.png" /></figure><h3>2. After testing locally, build and push each image to ECR.</h3><pre>aws ecr create-repository --repository-name reader-service<br>aws ecr create-repository --repository-name writer-service<br><br>docker build -t reader-service .<br>docker tag reader-service:latest &lt;account-id&gt;.dkr.ecr.&lt;region&gt;.amazonaws.com/reader-service:latest<br>docker push &lt;account-id&gt;.dkr.ecr.&lt;region&gt;.amazonaws.com/reader-service:latest</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/800/0*jG2NkgSLsZ8mxmzb.png" /></figure><h3>3. Defining Infrastructure with Terraform</h3><p>Your Terraform project will include these key modules:</p><p>VPC and Networking</p><p>RDS Instance</p><p>ECR Repositories</p><p>ECS Cluster and Task Definitions</p><p>Application Load Balancer</p><pre>resource &quot;aws_ecs_cluster&quot; &quot;main&quot; {<br>  name = &quot;serverless-cluster&quot;<br>}<br><br>resource &quot;aws_ecs_task_definition&quot; &quot;reader_task&quot; {<br>  family                   = &quot;reader-service&quot;<br>  network_mode             = &quot;awsvpc&quot;<br>  requires_compatibilities = [&quot;FARGATE&quot;]<br>  cpu                      = 256<br>  memory                   = 512<br>  execution_role_arn       = aws_iam_role.ecs_task_execution_role.arn<br>  task_role_arn            = aws_iam_role.ecs_task_execution_role.arn<br><br>  container_definitions = jsonencode([<br>    {<br>      name      = &quot;reader&quot;<br>      image     = &quot;${aws_ecr_repository.reader.repository_url}:latest&quot;<br>      essential = true<br>      portMappings = [<br>        {<br>          containerPort = 8000<br>          hostPort      = 8000<br>        }<br>      ]<br>      environment = [<br>        { name = &quot;DB_HOST&quot;, value = aws_db_instance.app_db.address },<br>        { name = &quot;DB_NAME&quot;, value = &quot;app_db&quot; },<br>        { name = &quot;DB_USER&quot;, value = &quot;admin&quot; },<br>        { name = &quot;DB_PASS&quot;, value = var.db_password }<br>      ]<br>    }<br>  ])<br>}<br><br>resource &quot;aws_ecs_service&quot; &quot;reader_service&quot; {<br>  name            = &quot;reader-service&quot;<br>  cluster         = aws_ecs_cluster.main.id<br>  task_definition = aws_ecs_task_definition.reader_task.arn<br>  desired_count   = 2<br>  launch_type     = &quot;FARGATE&quot;<br><br>  network_configuration {<br>    subnets          = aws_subnet.public[*].id<br>    assign_public_ip = true<br>    security_groups  = [aws_security_group.ecs_service.id]<br>  }<br><br>  load_balancer {<br>    target_group_arn = aws_lb_target_group.reader.arn<br>    container_name   = &quot;reader&quot;<br>    container_port   = 8000<br>  }<br><br>  depends_on = [aws_lb_listener.frontend]<br>}</pre><p>Repeat a similar configuration for the Writer Service.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/800/0*TmGgzv_ljofPdTq-.png" /></figure><h3>4. RDS Database and Networking</h3><p>Both services share the same RDS instance, accessed via private networking for security.</p><pre># create database subnet group<br>resource &quot;aws_db_subnet_group&quot; &quot;database_subnet_group&quot; {<br>  name        = &quot;${var.project_name}-${var.environment}-database-subnets&quot;<br>  subnet_ids  = [aws_subnet.private_data_subnet_az1.id, aws_subnet.private_data_subnet_az2.id]<br>  description = &quot;subnets for database instance&quot;<br><br>  tags = {<br>    Name = &quot;${var.project_name}-${var.environment}-database-subnets&quot;<br>  }<br>}<br><br># create the rds instance<br>resource &quot;aws_db_instance&quot; &quot;database_instance&quot; {<br>  engine                 = &quot;postgres&quot;<br>  engine_version         = &quot;14&quot;<br>  multi_az               = var.multi_az_deployment<br>  identifier             = var.database_instance_identifier<br>  username               = var.db_user<br>  password               = var.db_password<br>  db_name                = var.db_name<br>  instance_class         = var.database_instance_class<br>  allocated_storage      = 200<br>  db_subnet_group_name   = aws_db_subnet_group.database_subnet_group.name<br>  vpc_security_group_ids = [aws_security_group.database_security_group.id]<br>  availability_zone      = data.aws_availability_zones.available_zones.names[0]<br>  skip_final_snapshot    = true<br>  publicly_accessible    = var.publicly_accessible<br>}</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/800/0*PfpIQpzH5xbP6tUo.png" /></figure><p>Find the terraform template at: <a href="https://github.com/Kevin-byt/AWS-Projects/tree/64ad6a19deb4473689efbf78cee032e153c00170/ecs-django-microservices/terraform/env">ECS Django Microservices</a></p><p>then run:</p><pre>Terraform init<br><br>Terraform plan<br><br>Terraform apply</pre><h3>5. Why ECS Fargate is Serverless</h3><p>When most developers hear serverless, they immediately think of AWS Lambda. But serverless is more than just event-driven functions, it’s a paradigm. It’s about abstracting infrastructure management, automating scaling, and paying only for what you use.<br> That’s exactly what Amazon ECS Fargate delivers for containerized workloads.</p><h3>Serverless Principles and How Fargate Fits In</h3><p><strong>Let’s break it down:</strong> <strong>No Server Management</strong></p><p>With ECS Fargate, you don’t provision, scale, or patch EC2 instances.<br> You define your task requirements (vCPU, memory, networking), and AWS automatically runs your containers in a managed compute environment. This removes the operational overhead of managing ECS clusters on EC2, configuring capacity providers, or dealing with auto scaling groups.</p><p>Your focus shifts entirely to application logic, not infrastructure maintenance.</p><p><strong>Automatic Scaling and Orchestration</strong></p><p>Fargate integrates natively with ECS Service Auto Scaling and Application Auto Scaling.<br> When your traffic spikes, ECS launches more Fargate tasks; when traffic drops, it scales down automatically, no manual intervention needed.</p><p>This elasticity ensures cost-efficiency and performance stability without managing the underlying scaling policies or EC2 lifecycle.</p><p><strong>Pay-per-Use Model</strong></p><p>You only pay for the exact CPU and memory resources your containers use, billed per second while tasks are running.<br> There’s no idle server cost, no over-provisioned clusters, and no unused capacity. This aligns perfectly with serverless economics — true usage-based billing.</p><p><strong>Seamless AWS Integration</strong></p><p>Fargate runs as part of the broader AWS ecosystem. It integrates smoothly with:</p><p>CloudWatch Logs for centralized logging,</p><p>CloudWatch Alarms for health and performance metrics,</p><p>IAM Roles for Tasks for fine-grained permissions,</p><p>AWS X-Ray for distributed tracing, and</p><p>ECR for secure container image storage and retrieval.</p><p>This ecosystem synergy allows teams to build secure, observable, and automated serverless container platforms end-to-end.</p><p><strong>Security and Isolation</strong></p><p>Each Fargate task runs in its own isolated compute environment, with dedicated kernel and network interfaces.<br> Unlike EC2-based ECS, where containers share the host kernel, Fargate tasks achieve stronger isolation, closer to the Lambda-level security boundary, making it ideal for multi-tenant or microservice architectures.</p><p><strong>Infrastructure as Code Ready</strong></p><p>By pairing Fargate with Terraform, you extend serverless principles to your entire infrastructure.<br> Your ECS services, networking, IAM roles, and monitoring configurations all live in code that is reproducible, version-controlled, and automated. This enables serverless operations not just in runtime, but also in provisioning and deployment.</p><h3>6. Observability and Monitoring</h3><p>Add logging and metrics using:</p><p>AWS CloudWatch Logs — For container logs and AWS CloudWatch Alarms — For task health and CPU/memory usage using Terraform</p><pre># ECS CloudWatch Monitoring<br>resource &quot;aws_cloudwatch_log_group&quot; &quot;ecs_logs&quot; {<br>  name              = &quot;/ecs/${var.project_name}&quot;<br>  retention_in_days = 7<br>}<br><br># ECS Service Alarms<br>resource &quot;aws_cloudwatch_metric_alarm&quot; &quot;ecs_cpu_high&quot; {<br>  alarm_name          = &quot;${var.project_name}-ecs-cpu-high&quot;<br>  comparison_operator = &quot;GreaterThanThreshold&quot;<br>  evaluation_periods  = &quot;2&quot;<br>  metric_name         = &quot;CPUUtilization&quot;<br>  namespace           = &quot;AWS/ECS&quot;<br>  period              = &quot;300&quot;<br>  statistic           = &quot;Average&quot;<br>  threshold           = &quot;80&quot;<br>  alarm_description   = &quot;ECS CPU utilization is too high&quot;<br>  alarm_actions       = [aws_sns_topic.alerts.arn]<br><br>  dimensions = {<br>    ServiceName = aws_ecs_service.ecs_service1.name<br>    ClusterName = aws_ecs_cluster.ecs_cluster.name<br>  }<br>}<br><br>resource &quot;aws_cloudwatch_metric_alarm&quot; &quot;ecs_memory_high&quot; {<br>  alarm_name          = &quot;${var.project_name}-ecs-memory-high&quot;<br>  comparison_operator = &quot;GreaterThanThreshold&quot;<br>  evaluation_periods  = &quot;2&quot;<br>  metric_name         = &quot;MemoryUtilization&quot;<br>  namespace           = &quot;AWS/ECS&quot;<br>  period              = &quot;300&quot;<br>  statistic           = &quot;Average&quot;<br>  threshold           = &quot;80&quot;<br>  alarm_description   = &quot;ECS Memory utilization is too high&quot;<br>  alarm_actions       = [aws_sns_topic.alerts.arn]<br><br>  dimensions = {<br>    ServiceName = aws_ecs_service.ecs_service1.name<br>    ClusterName = aws_ecs_cluster.ecs_cluster.name<br>  }<br>}<br><br># ALB Target Health<br>resource &quot;aws_cloudwatch_metric_alarm&quot; &quot;alb_unhealthy_targets&quot; {<br>  alarm_name          = &quot;${var.project_name}-alb-unhealthy-targets&quot;<br>  comparison_operator = &quot;GreaterThanThreshold&quot;<br>  evaluation_periods  = &quot;2&quot;<br>  metric_name         = &quot;UnHealthyHostCount&quot;<br>  namespace           = &quot;AWS/ApplicationELB&quot;<br>  period              = &quot;300&quot;<br>  statistic           = &quot;Average&quot;<br>  threshold           = &quot;0&quot;<br>  alarm_description   = &quot;ALB has unhealthy targets&quot;<br>  alarm_actions       = [aws_sns_topic.alerts.arn]<br><br>  dimensions = {<br>    LoadBalancer = aws_lb.application_load_balancer.arn_suffix<br>  }<br>}<br><br># SNS Topic for Alerts<br>resource &quot;aws_sns_topic&quot; &quot;alerts&quot; {<br>  name = &quot;${var.project_name}-alerts&quot;<br>}<br><br>resource &quot;aws_sns_topic_subscription&quot; &quot;email&quot; {<br>  topic_arn = aws_sns_topic.alerts.arn<br>  protocol  = &quot;email&quot;<br>  endpoint  = var.alert_email<br>}</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/800/0*CU2yqLTt1z_z3_N3.png" /></figure><h3>Conclusion</h3><p>This setup demonstrates that serverless microservices don’t have to rely on AWS Lambda.<br>With ECS Fargate, ECR, and Terraform, you can build production-grade, scalable, and cost-efficient systems while maintaining full control over your architecture.</p><p>It’s the perfect middle ground between full container control and the simplicity of serverless.</p><p>Happy coding!</p><p>Follow me for more demos and networking. <a href="https://www.linkedin.com/in/kevin-kiruri/">Kevin Kiruri LinkedIn</a><br>Find the source code here: <a href="https://github.com/Kevin-byt/AWS-Projects/tree/64ad6a19deb4473689efbf78cee032e153c00170/ecs-django-microservices">ECS Django Microservices</a></p><p><em>Originally published at </em><a href="https://dev.to/aws-builders/building-serverless-microservices-on-aws-with-ecs-fargate-ecr-and-terraform-4ocm"><em>https://dev.to</em></a><em> on January 2, 2026.</em></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=4fe35bbc5327" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[AWS Aurora DSQL for Django Developers: A Step-by-Step Guide]]></title>
            <link>https://kevinkiruri.medium.com/aws-aurora-dsql-for-django-developers-a-step-by-step-guide-e89398657a36?source=rss-4f39f6231241------2</link>
            <guid isPermaLink="false">https://medium.com/p/e89398657a36</guid>
            <category><![CDATA[python]]></category>
            <category><![CDATA[django]]></category>
            <category><![CDATA[aurora]]></category>
            <category><![CDATA[aws]]></category>
            <category><![CDATA[serverless]]></category>
            <dc:creator><![CDATA[Kevin Kiruri]]></dc:creator>
            <pubDate>Fri, 07 Feb 2025 06:16:22 GMT</pubDate>
            <atom:updated>2025-02-07T06:40:56.567Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1000/0*LCanQCIZ9yRO_zzl.jpg" /></figure><p>Amazon Aurora is a cloud-native relational database service that provides high performance and scalability. With the introduction of Aurora DSQL (Distributed SQL), developers can now leverage distributed database capabilities to enhance reliability and performance. This guide will walk you through setting up Aurora DSQL for a Django project in four steps:</p><h3>Prerequisites</h3><p>Before proceeding, ensure you have the following:</p><h3>Step 1.Provision an Aurora DSQL Cluster</h3><ul><li>Navigate to the Aurora DSQL Console.</li><li>Create Cluster</li><li>Security Group for network access. Inbound — Best practice: Restrict inbound access to your VPC CIDR (if the application runs inside the same VPC) or your server’s specific IP and avoid using 0.0.0.0/0 unless for testing purposes Outbound — Is set to Allow all traffic from 0.0.0.0/0 by default</li></ul><h3>Step 2: Connect Django to Aurora DSQL</h3><p>Change Database settings in your django app’s settings.py</p><h3>Step 3: Migrate the Database</h3><p>Run the following commands to apply migrations:</p><p>Remember to clear any previous migrations, using:</p><h3>Step 4: Monitor and Scale</h3><p>Use Amazon CloudWatch and Performance Insights to monitor query performance and optimize configurations.</p><p>Aurora DSQL enhances Django applications by providing high availability, scalability, and better performance. By following this guide, you can integrate Aurora DSQL into your Django project efficiently. Explore additional optimizations and best practices to get the most out of your Aurora database setup.</p><p>Follow me for more demos and networking. <a href="https://www.linkedin.com/in/kevin-kiruri/">Kevin Kiruri LinkedIn</a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=e89398657a36" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Amazon Aurora DSQL: The New Era of Distributed SQL]]></title>
            <link>https://kevinkiruri.medium.com/amazon-aurora-dsql-the-new-era-of-distributed-sql-f10fd4654d29?source=rss-4f39f6231241------2</link>
            <guid isPermaLink="false">https://medium.com/p/f10fd4654d29</guid>
            <category><![CDATA[aws]]></category>
            <category><![CDATA[database]]></category>
            <category><![CDATA[sql]]></category>
            <category><![CDATA[aurora]]></category>
            <category><![CDATA[serverless]]></category>
            <dc:creator><![CDATA[Kevin Kiruri]]></dc:creator>
            <pubDate>Mon, 30 Dec 2024 18:36:08 GMT</pubDate>
            <atom:updated>2024-12-31T19:18:48.682Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1000/0*q7HkMpPzHr3MmKiL.jpg" /></figure><p>Amazon Aurora DSQL (dee-sequel) is a groundbreaking serverless, distributed SQL database that delivers active-active high availability. Announced at AWS re:Invent 2024, it is now available in preview across US East (N. Virginia), US East (Ohio), and US West (Oregon). With its distributed architecture, Aurora DSQL enables simultaneous read and write operations across multiple regions while maintaining strong consistency. Transactions are processed locally, with cross-region concurrency checks performed only at commit, ensuring performance and reliability.</p><h3>What sets Aurora DSQL apart?</h3><ul><li><strong>Effortless Scalability:</strong> Seamlessly scale reads, writes, and storage independently, providing virtually unlimited horizontal scaling for workloads of any size.</li><li><strong>Availability:</strong> Ensures 99.99% uptime in single-region deployments and an impressive 99.999% for multi-region clusters.</li><li><strong>Performance:</strong> Up to 4x faster read and write operations compared to popular distributed SQL databases, making it a top choice for performance-critical applications.</li><li><strong>Compatibility:</strong> PostgreSQL-compatible, allowing developers to use well-known relational database concepts, accelerating adoption.</li><li><strong>Management:</strong> Completely serverless, Aurora DSQL eliminates the complexities of patching, upgrades, and maintenance, freeing developers to focus on innovation.</li><li><strong>Event-Driven and Microservice Ready:</strong> Optimized for serverless and microservices architectures.</li></ul><h3>Core Architecture Components</h3><h3>Distributed Design:</h3><p>Amazon Aurora DSQL is built on a robust distributed architecture, at its core, the architecture is composed of four key components:</p><ul><li>Relay/Connectivity: Manages client connections, routing requests to appropriate resources while ensuring seamless communication across the system.</li><li>Compute/Databases: Handles query execution and processing, leveraging PostgreSQL compatibility for familiar operations.</li><li>Transaction Log and Concurrency Control: Provides atomicity and isolation for transactions while ensuring consistency across nodes and regions.</li><li>User Storage: Ensures data durability and redundancy by replicating user data across multiple Availability Zones.</li></ul><p>These components are orchestrated through a control plane that maintains coordination and redundancy across three Availability Zones. This design enables self-healing capabilities, automatic scaling, and high availability, ensuring that infrastructure failures do not impact database operations.</p><h3>Multi-Region Clusters:</h3><p>Aurora DSQL takes availability and resilience to the next level with multi-region linked clusters, enabling applications to achieve high performance and reliability on a global scale.</p><ul><li>Linked Clusters: Multi-region clusters operate in active-active mode, allowing read and write operations in multiple regions simultaneously. Data is synchronously replicated across regions, ensuring strong consistency and eliminating replication lag.</li><li>Resilience: Each linked cluster provides independent endpoints for concurrent operations. In the event of a failure in one region, the other region continues operating seamlessly, ensuring uninterrupted service.</li><li>Cross-Region Consistency: Transactions are processed locally in their originating region, with cross-region concurrency checks performed during the commit phase. This approach minimizes latency while maintaining strong data consistency.</li></ul><h3>Exploring Aurora DSQL Multi-Region linked clusters</h3><p>Curious about Amazon’s latest innovation in distributed SQL databases, I decided to put Multi-Region clusters (one of Aurora DSQL’s Core Architecture Components) to the test by creating clusters in different regions to demonstrate cross-region replication and consistent reads from both endpoints.</p><p><strong>Creating a cluster in Aurora DSQL:</strong></p><p>Navigate to <a href="https://console.aws.amazon.com/dsql">https://console.aws.amazon.com/dsql</a> and simply create a cluster, add linked regions and choose a region for your linked cluster region.</p><p><strong>Connecting to the cluster using an authentication token and running SQL commands in aurora DSQL:</strong></p><p>Choose the cluster you want to connect to, copy the end point, then use the command below to use psql to start a connection to your cluster. You should see a prompt to provide a password. generate an authentication token and use it as the password.</p><p>PGSSLMODE=require \<br> psql --dbname postgres \<br> --username admin \<br> --host uiabtxahshv6at5pcfidcxfnbq.dsql.us-east-1.on.aws \<br> --password <strong>Writing in one region and reading from the second region:</strong></p><p>You can perform write operations in one region and immediately read the data from another region.</p><p><strong>Some of the queries to try:</strong></p><pre>INSERT INTO example (id, name) VALUES (1, &#39;Aurora DSQL Test&#39;);</pre><pre>SELECT * FROM example WHERE id = 1;</pre><h3>Why this stands out:</h3><ul><li>Both regions(us-east-1 and us-east-2)operate as active endpoints, enabling concurrent read and write operations across regions.</li><li>Aurora DSQL guarantees that all reads and writes return the latest committed writes no matter the region.</li><li>If us-east-1 experiences downtime, all operations can seamlessly continue in us-east-2 ensuring high availability.</li><li>Applications in different regions access the same database, reducing complexity.</li><li>Ease of management.</li></ul><p>This setup is ideal for globally distributed applications such as e-commerce platforms, financial systems, or multi-region SaaS services, where uptime, consistency, and performance are critical.</p><p>Follow me for more demos and networking. <a href="https://www.linkedin.com/in/kevin-kiruri/">Kevin Kiruri LinkedIn</a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=f10fd4654d29" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Monitoring AWS Lambda Functions with AWS X-Ray and CloudWatch: Advanced Technique]]></title>
            <link>https://kevinkiruri.medium.com/monitoring-aws-lambda-functions-with-aws-x-ray-and-cloudwatch-advanced-technique-160ee7b758b7?source=rss-4f39f6231241------2</link>
            <guid isPermaLink="false">https://medium.com/p/160ee7b758b7</guid>
            <category><![CDATA[aws]]></category>
            <category><![CDATA[cloudwatch]]></category>
            <category><![CDATA[cloud-computing]]></category>
            <category><![CDATA[lambda]]></category>
            <category><![CDATA[serverless]]></category>
            <dc:creator><![CDATA[Kevin Kiruri]]></dc:creator>
            <pubDate>Mon, 16 Dec 2024 23:33:58 GMT</pubDate>
            <atom:updated>2024-12-19T07:55:13.566Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1000/0*Xxyxhup2If_nLpy4.jpg" /></figure><p>It is essential to understand the state of your Lambda-based application to ensure its reliability and health. Monitoring provides information that helps you detect and resolve performance problems, outages and errors in your workloads. Lambda-based applications often integrate with multiple services, which makes it just as important that you monitor each service endpoint. In this article, we will explore advanced techniques that allow you to leverage AWS X-Ray and CloudWatch to monitor and observe your applications.</p><h3>AWS X-Ray Advanced features with Lambda:</h3><p>Since most serverless applications consist of multiple service integrations, troubleshooting performance issues or errors usually involves tracking a request from the source caller through all involved services. In this case, AWS X-Ray is a faster and more convenient tool to trace distributed capabilities as it allows you to visualise and analyse the flow of requests across your application.</p><p>To use X-Ray in Lambda you can activate it in the Lambda console for a specific function or enable it in the AWS SAM template:</p><pre>Resources: GetLocations: Type: AWS::Serverless::Function Properties: Tracing: Active</pre><p>Make sure you also provide permission using the <em>AWSXRayDaemonWriteAccess</em> managed policy. You must also activate X-Ray for each service in your workload. Once enabled X-Ray starts collecting tracing data for events. You can instrument your code using AWS X-Ray SDK to <strong>annotate</strong> traces and add custom <strong>metadata</strong>. In python:</p><pre>from aws_xray_sdk.core import xray_recorder @xray_recorder.capture(&#39;process_event&#39;) def lambda_handler(event, context): # Your function logic here pass</pre><p>You can isolate which part of the system is causing the most latency using the X-Ray <strong>service map</strong>, which visually represents the communication between your lambda function and other services on the X-Ray console.</p><p>Finally, using <strong>Subsegments</strong>, leverage granular tracing within a single Lambda invocation. In python:</p><pre>with xray_recorder.in_segment(&#39;custom_subsegment&#39;): # Specific operations to trace pass</pre><h3>AWS CloudWatch</h3><p>AWS CloudWatch provides a platform for centralised monitoring and logging. All Lambda functions are automatically integrated with CloudWatch and log various standard metrics that get published to CloudWatch metrics. Key CloudWatch metrics for lambda include invocations, Errors, Duration, Concurrent executions and memory usage. You can create CloudWatch alarms based on these metrics to alert on threshold. Leverage the creation of custom Dashboards to display key metrics such as errors, execution duration, and invocations in one view. Analyze Logs using CloudWatch Logs Insights to query your application’s logs for debugging and optimization.</p><h3>Advanced Monitoring with X-Ray and CloudWatch</h3><ul><li>Correlate Traces and Logs: Use trace IDs to link X-Ray traces with CloudWatch logs for detailed debugging. This enables you to trace a request’s journey and investigate logs for specific trace segments, offering precise context during troubleshooting.</li><li>Real-time Anomaly Detection: Enable anomaly detection in CloudWatch to establish dynamic thresholds based on historical patterns. This helps automatically identify unusual behaviours in your metrics, such as unexpected spikes in errors or latency, reducing the manual effort in monitoring setups.</li><li>Automating Alerting and Remediation: Set up CloudWatch alarms based on Lambda performance or error trends and automate remediation steps using SNS or step functions to handle issues as soon as they arise</li><li>Optimise Performance: Regularly analyze X-Ray traces and CloudWatch metrics to identify inefficiencies, such as redundant function calls or underutilized resources. These insights allow you to fine-tune function configurations, optimize code, and reduce operational costs.</li></ul><p>Monitoring Lambda-based applications with AWS X-Ray and CloudWatch enables you to maintain high availability, optimize performance, and quickly resolve issues. By focusing on critical metrics like errors, execution time, and throttling, and leveraging the advanced capabilities of these tools, you can build robust and resilient serverless applications.</p><p>Happy Monitoring! <br><a href="https://www.linkedin.com/in/kevin-kiruri/">Let’s connect on LinkedIn</a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=160ee7b758b7" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Serverless Architecture on AWS]]></title>
            <link>https://kevinkiruri.medium.com/serverless-architecture-on-aws-be3d6bd13f9a?source=rss-4f39f6231241------2</link>
            <guid isPermaLink="false">https://medium.com/p/be3d6bd13f9a</guid>
            <category><![CDATA[sns]]></category>
            <category><![CDATA[serverless]]></category>
            <category><![CDATA[dynamodb]]></category>
            <category><![CDATA[lambda]]></category>
            <category><![CDATA[aws]]></category>
            <dc:creator><![CDATA[Kevin Kiruri]]></dc:creator>
            <pubDate>Wed, 14 Feb 2024 11:56:38 GMT</pubDate>
            <atom:updated>2024-02-14T11:56:38.718Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*DoTlyiBpG_YEx9UUxFxc8Q.png" /></figure><blockquote><strong>Introduction</strong></blockquote><p>Let’s assume you want to receive notifications regularly based on some event occurrences. This is the case in this demo walk through. Assume you have stores in various locations. At the end of the day, you would like to know the products and store locations where the inventory is below 5 every day. This serverless infrastructure can adequately send you notifications for your knowledge. Applications for this use case would be endless in business. To put the icing on the case, there is a CloudFormation yaml template that creates the infrastructure in the click of a button.</p><blockquote><strong>Prerequisites</strong></blockquote><ol><li>Have an AWS account. If you don’t have one, sign up <a href="https://aws.amazon.com/">here</a> and enjoy the benefits of the <a href="https://aws.amazon.com/free/?trk=99f831a2-d162-429a-9a77-a89f6b3bd6cd&amp;sc_channel=ps&amp;ef_id=Cj0KCQjw06-oBhC6ARIsAGuzdw03qBfKD7AA8ejOZPfdI7Pg4XshpRd57Q1p-a8--id9VCM5Cjg78xEaAmJtEALw_wcB%3AG%3As&amp;s_kwcid=AL%214422%213%21645125273267%21e%21%21g%21%21aws+free+tier%2119574556890%21145779847112&amp;all-free-tier.sort-by=item.additionalFields.SortRank&amp;all-free-tier.sort-order=asc&amp;awsf.Free+Tier+Types=*all&amp;awsf.Free+Tier+Categories=*all">Free-Tier Account</a></li><li>View project files in my <a href="https://github.com/Kevin-byt/AWS-Projects/tree/main/inventory">GitHub portfolio</a></li></ol><blockquote><strong>Creating an S3 Bucket</strong></blockquote><ol><li>On the AWS console, search for S3 and click on it to open the S3 Console Page</li><li>Under Buckets click on Create bucket</li></ol><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*aNznTyh3X-r2DWzJHPdSYw.png" /></figure><p>3. Enter your preferred AWS Region and Bucket name. Remember that a bucket name should be unique across the entire region. Scroll down and click on Create bucket . Note the ARN of the created bucket. We will be using it later</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*eNkWhlHEeOInjahnRHfGJA.png" /></figure><blockquote><strong>Create a Dynamo Db Table</strong></blockquote><ol><li>Search for Dynamo DB on the Services search box. Once opened, click on Create table</li></ol><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Z9GZq2ILCVtLrJ7SASfZVw.png" /></figure><p>2. Enter the Table name , Partition key and Sort Key . Scroll to the bottom and click on Create table</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*zO4XPTzO0aVm_qfPpLZz4w.png" /></figure><p>3. The table is created and appears on your list of tables. Note the DynamoDB ARN of the table. We will use it later.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*HoTbazqkRPRfxgIAvp7dew.png" /></figure><blockquote><strong>Create an SNS Topic</strong></blockquote><ol><li>In the Amazon SNS dashboard, under the Topics page, click on Create topic</li></ol><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*mCl6B7XKhoydPITl3mGRjQ.png" /></figure><p>2. Select the Standard type, then give your preferred name for the SNS Topic. Scroll down and click on Create topic</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*0OHYCOrU2XdbzJV6ZmAmuw.png" /></figure><p>3. Once created, on the Subscriptions tab, click on Create subscription</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*RGUR9e5SOJl8x1eGYMoGzA.png" /></figure><p>4. In the Create subscription page, set the Protocol as Email and enter your preferred email address as the Endpoint . Scroll down and click on Create subscription</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*k8nk3Dt_fU6FyYPQVStgMQ.png" /></figure><p>5. You will receive an email asking you to confirm the subscription. Click on Confirm subscription .</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*amx2w8IAzDWU1Lq_PBHUBA.png" /></figure><p>You should then get a Subscription confirmed notification</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/898/1*6U1A1t4EilHMWXTHXpjNHw.png" /></figure><blockquote><strong>Creating a Role for Lambda to read from S3 and update the DynamoDB</strong></blockquote><ol><li>In the IAM Console , under Policies click on Create policy</li></ol><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*h1zaS1kAUmz0br7RuasM3g.png" /></figure><p>2. Select the json tab of the policy and paste the json policy below</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*NNxKSTJWzkp5b_36QVsn8w.png" /></figure><pre>{<br> &quot;Version&quot;: &quot;2012-10-17&quot;,<br>    &quot;Statement&quot;: [<br>      {<br>        &quot;Effect&quot;: &quot;Allow&quot;,<br>        &quot;Action&quot;: [<br>          &quot;s3:GetObject&quot;,<br>          &quot;s3:ListBucket&quot;<br>        ],<br>        &quot;Resource&quot;: [<br>          &quot;{S3-Bucket ARN}/*&quot;,<br>          &quot;{S3-Bucket ARN}&quot;<br>        ]<br>      },<br>      {<br>        &quot;Effect&quot;: &quot;Allow&quot;,<br>        &quot;Action&quot;: [<br>          &quot;dynamodb:PutItem&quot;,<br>          &quot;dynamodb:BatchWriteItem&quot;<br>        ],<br>        &quot;Resource&quot;: &quot;{DynamoDB ARN}&quot;<br>      }<br>    ]<br>}</pre><p>Replace {S3-Bucket ARN} with the ARN of the S3 bucket you created and {DynamoDB ARN} with the ARN of your DynamoDB Table</p><p>3. Scroll to the bottom and click on Next</p><p>4. Give the policy a name then scroll down and click on Create policy</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*aDlC5BItTTh2UkyQT3kPAA.png" /></figure><p>5. On the IAM Console click on Roles then Create role</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*GnsfdJSCZUxAxr0FVjFUNA.png" /></figure><p>6. Under Trusted entity type , select AWS service , select Lambda as the Use case then click on Next</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*2MKPt_PsaLqZ8QekLA-JNg.png" /></figure><p>7. In the Add permissions page, search for the policy we just created, select the check box then click on Next</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*HzI0GsTKZECh4WEGmW-Z8w.png" /></figure><p>8. Give the Role a name, than scroll down and click on Create role</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*a8nW9-6tFQ1oMOGkMld1uQ.png" /></figure><blockquote><strong>Creating a Role for Lambda to read from DynamoDB and send an SNS Notification</strong></blockquote><ol><li>Follow the steps followed in creating the previous role but use the JSON policy below</li></ol><pre>{<br> &quot;Version&quot;: &quot;2012-10-17&quot;,<br> &quot;Statement&quot;: [<br>  {<br>   &quot;Effect&quot;: &quot;Allow&quot;,<br>   &quot;Action&quot;: [<br>    &quot;dynamodb:PutItem&quot;,<br>    &quot;dynamodb:UpdateItem&quot;,<br>    &quot;dynamodb:GetItem&quot;,<br>    &quot;dynamodb:Scan&quot;,<br>    &quot;dynamodb:Query&quot;<br>   ],<br>   &quot;Resource&quot;: &quot;{DynamoDB Table ARN}&quot;<br>  },<br>  {<br>            &quot;Effect&quot;: &quot;Allow&quot;,<br>            &quot;Action&quot;: [<br>                &quot;dynamodb:GetRecords&quot;,<br>                &quot;dynamodb:GetShardIterator&quot;,<br>                &quot;dynamodb:DescribeStream&quot;,<br>                &quot;dynamodb:ListStreams&quot;<br>            ],<br>            &quot;Resource&quot;: &quot;{DynamoDB Table ARN}/Stream/*&quot;<br>        },<br>  {<br>   &quot;Effect&quot;: &quot;Allow&quot;,<br>   &quot;Action&quot;: &quot;sns:Publish&quot;,<br>   &quot;Resource&quot;: &quot;{SNS Topic ARN}&quot;<br>  }<br> ]<br>}</pre><p>Insert your Dynamo DB ARN and the SNS Topic ARN in the respective positions in the policy above</p><blockquote><strong>Create a Lambda Function to Read from S3 and Upload to Dynamo DB</strong></blockquote><ol><li>Search for Lambda and open the Lambda Console. Click on Create function</li></ol><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*zx5GKWjLZWYAdDmQRf89MQ.png" /></figure><p>2. Under Create function, choose Author from scratch, enter your preferred Function name and Runtime . Scroll down and click on Create function</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*pBGNrkBcvs6S3lGna1EBmw.png" /></figure><p>3. In the Code source section, enter the code below and click on Deploy</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*2fNA8gBTUJdpu3cPdmUavQ.png" /></figure><pre>import json<br>import boto3<br>import csv<br>import os<br><br>def lambda_handler(event, context):<br>   <br>    # Get the S3 bucket and object information from the event<br>    s3_bucket = event[&#39;Records&#39;][0][&#39;s3&#39;][&#39;bucket&#39;][&#39;name&#39;]<br>    s3_key = event[&#39;Records&#39;][0][&#39;s3&#39;][&#39;object&#39;][&#39;key&#39;]<br><br>    # Test lambda function manually<br>    # s3_bucket = &#39;productinventory240215&#39;<br>    # s3_key = &#39;inventory.csv&#39;<br><br>    # Set the DynamoDB table name<br>    dynamo_table_name = &#39;AppleInventoryTable&#39;<br><br>    # Create a DynamoDB client<br>    dynamodb = boto3.resource(&#39;dynamodb&#39;)<br>    table = dynamodb.Table(dynamo_table_name)<br><br>    # Create an S3 client<br>    s3 = boto3.client(&#39;s3&#39;)<br><br>    # Download the CSV file from S3<br>    temp_file_path = &#39;/tmp/temp.csv&#39;<br>    s3.download_file(s3_bucket, s3_key, temp_file_path)<br><br>    # Read and process the CSV file, skipping the header row<br>    with open(temp_file_path, &#39;r&#39;) as csvfile:<br>        csvreader = csv.reader(csvfile)<br>        # Skip the header row<br>        next(csvreader, None)<br>        <br>        for row in csvreader:<br>            store = row[0]<br>            product = row[1]<br>            count = int(row[2])<br><br>            # Update DynamoDB table<br>            table.put_item(<br>                Item={<br>                    &#39;StoreRegion&#39;: store,<br>                    &#39;Product&#39;: product,<br>                    &#39;ProductCount&#39;: count<br>                }<br>            )<br>  <br>    # Clean up the temporary file<br>    os.remove(temp_file_path)<br><br>    return {<br>        &#39;statusCode&#39;: 200,<br>        &#39;body&#39;: &#39;CSV processing complete&#39;<br>    }</pre><p>4. Unser the Configuration tab, select Triggers then click on Add trigger</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*U74Hxf7UXFU2pnV2Skk7bw.png" /></figure><p>5. Select the Source as S3 . Select the Bucket you created. For Event types use All object create events. Add .csv as the Suffix. (We will be uploading csv files with the data). Select the Check box then click on Add</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/764/1*YpiT5jbtEmUB4TvUShYfag.png" /></figure><p>6. Under the Permissions select Edit</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*9SKkmfRdiwmSXXs0K9OMTQ.png" /></figure><p>7. Scroll down. Increase the Timeout to 5 minutes . This will help deal with large files. Under Execution role select Use an existing role then select the role we created to read data from S3 and upload to DynamoDB. Click on Save</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*iDVBsSkR8NOpPPyOQoO7Qw.png" /></figure><blockquote><strong>Create a Lambda Function to Read from DynamoDB and send an SNS Notification for Low Inventory</strong></blockquote><ol><li>Navigate to the Lambda Console page and click on Create function</li></ol><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*eyiiAYaLpHMr9Mupmvn2yA.png" /></figure><p>2. Select Author from scratch , give the function a name and select the Python runtime. Scroll down and click on Create function</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Ysr-1-aMi7ddjtJff4f5Uw.png" /></figure><p>3. Under the code tab, insert the Python code below then click on Deploy</p><pre>import boto3<br>import json<br>import os<br><br>def lambda_handler(event, context):<br>    # Define DynamoDB and SNS clients<br>    dynamo_table_name = &#39;AppleInventoryTable&#39;<br>    dynamodb = boto3.resource(&#39;dynamodb&#39;)<br>    table = dynamodb.Table(dynamo_table_name)<br>    sns = boto3.client(&#39;sns&#39;)<br>    region = os.environ[&#39;AWS_REGION&#39;]<br>    account_id = context.invoked_function_arn.split(&quot;:&quot;)[4]<br><br>    # Scan DynamoDB table to get items with count of 5 or less than 5<br>    response = table.scan(<br>        FilterExpression=boto3.dynamodb.conditions.Attr(&#39;ProductCount&#39;).lt(6)<br>    )<br>    <br>    # Check if there are items with count less than 5<br>    items = response.get(&#39;Items&#39;, [])<br>    <br>    if items:<br>        # Prepare notification message<br>        notification_message = &quot;Low inventory alert:\n&quot;<br>        for item in items:<br>            notification_message += f&quot;StoreRegion: {item[&#39;StoreRegion&#39;]}, Item: {item[&#39;Product&#39;]}, Count: {item[&#39;ProductCount&#39;]}\n&quot;<br><br>        # Publish notification to SNS<br>        sns.publish(<br>            # TopicArn=&#39;arn:aws:sns:us-east-1:471285348599:LowInventoryTopic&#39;,<br>            TopicArn = f&#39;arn:aws:sns:{region}:{account_id}:LowInventoryTopic&#39;,<br>            Message=notification_message,<br>            Subject=&#39;Low Inventory Alert&#39;<br>        )<br>        <br>        return {<br>            &#39;statusCode&#39;: 200,<br>            &#39;body&#39;: &#39;Notification sent successfully.&#39;<br>        }<br>    else:<br>        return {<br>            &#39;statusCode&#39;: 200,<br>            &#39;body&#39;: &#39;No items with count less than 5.&#39;<br>        }</pre><p>4. Under the Configuration tab, select Permissions then click on Edit</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*e8OcrDov_K2mdGFFkCvtpw.png" /></figure><p>5. On the Edit basic settings page, scroll down and set the timeout at 5 min . Under Execution role, select Use an existing role and select the role you created to read from DynamoDB and send a notification of low inventory. Scroll to the bottom and click on Save</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/808/1*dbXXr42NPLmqJffbTIgBvA.png" /></figure><p>6. Under the Configuration tab, select Triggers then click on Add trigger</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*h1zm66A1F3AMKBI_eUZs5g.png" /></figure><p>5. Select DynamoDB as the Source. Select the DynamoDB Table we created earlier . Check the box for Activate trigger. Enter 1,000 as the Batch size then scroll down and click on Add</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/869/1*rLypHwWr3UlhpuK139PkKw.png" /></figure><blockquote><strong>Testing</strong></blockquote><ol><li>Upload a csv file with 3 columns for Store, Product and Count. You can get a sample file <a href="https://github.com/Kevin-byt/AWS-Projects/blob/main/inventory/inventory.csv">here</a></li><li>Navigate to S3 and select the bucket we created</li><li>Click on Upload</li></ol><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Cx4WcOO-flCKhdx_nngRZQ.png" /></figure><p>4. Click on Add files then navigate to your files and select the required file. Click on Upload at the bottom</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/984/1*Dn8Ng5LouBO8Mi8wkUvdRg.png" /></figure><p>5. Once uploaded, the lambda function should upload the items to DynamoDB</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1003/1*sae5uv1E5qg2XUtJprsw3A.png" /></figure><p>6. An email should also be sent, showing the Products and Stores with less than 5 items in inventory</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/698/1*z00819CfkO8nM5uiHoi2jw.png" /></figure><p>7. This ensures that the system works end to end.</p><blockquote><strong>Clean Up</strong></blockquote><ol><li>Delete the uploaded file from the S3 bucket</li><li>Delete the S3 bucket</li><li>Delete the SNS Topic</li><li>Delete the Dynamo DB table</li><li>Delete the Lambda functions</li><li>Delete the IAM roles created</li><li>Delete the IAM policies created</li></ol><blockquote><strong>Infrastructure as Code</strong></blockquote><p>The beauty of setting up infrastructure on AWS is that we have access to tools used for Infrastructure as Code (IaC).</p><ol><li>Navigate to CloudFormation on the AWS console. Under Stacks click on Create stack</li></ol><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*GAW-15ThXefZ57wAWbd9Fg.png" /></figure><p>2. Select Template is ready and Upload a template file so that we can upload our <a href="https://github.com/Kevin-byt/AWS-Projects/blob/main/inventory/inventory.yaml">yaml template</a>. You can get the yaml template in the <a href="https://github.com/Kevin-byt/AWS-Projects/blob/main/inventory/inventory.yaml">GitHub repo</a>. Click Next</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*_yfBjUbnoNvMC0J5LoaJPQ.png" /></figure><p>3. Provide a Stack name , the notification email and your preferred s3 bucket name</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*6NtazoI-l64mQq2OctDQXQ.png" /></figure><p>4. click on Next on the next 2 pages then click on Submit at the bottom. Give time for the resources to create. You can monitor the creation of resources on the Resources tab. Once complete, the status will show CREATE_COMPLETE . In the event of a ROLL_BACK check to confirm what the error is and correct it</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*8lZVfwkcz-cp_jmNqeEAwQ.png" /></figure><p>5. Confirm subscription to the SNS topic from your email</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*at3zn_LI9V3bf70JilrE9Q.png" /></figure><p>6. Proceed to the Testing section as done in the manual setup</p><blockquote><strong>Clean Up</strong></blockquote><ol><li>Once done, click on Delete on the CloudFormation template page. Theis will delete all the resources created</li></ol><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Lp3QRvhGMl8v6uLL7nUWFA.png" /></figure><p>2. On the pop up that appears, click on Delete and the infrastructure will be pulled down</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/941/1*k0pAHivdr_pCXvmalMRqrg.png" /></figure><blockquote><strong>Conclusion</strong></blockquote><p>This demo steps give a walkthrough on how to create a serverless on AWS. You could customize the configurations to fit your use case.</p><p>Follow for more demos and networking. <br><a href="https://www.linkedin.com/in/kevin-kiruri/">Kevin Kiruri LinkedIn</a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=be3d6bd13f9a" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Full-stack Observability and Monitoring on AWS]]></title>
            <link>https://kevinkiruri.medium.com/observability-and-monitoring-on-aws-42628f13b569?source=rss-4f39f6231241------2</link>
            <guid isPermaLink="false">https://medium.com/p/42628f13b569</guid>
            <category><![CDATA[monitoring]]></category>
            <category><![CDATA[cloudwatch]]></category>
            <category><![CDATA[aws-x-ray]]></category>
            <category><![CDATA[observability]]></category>
            <category><![CDATA[aws]]></category>
            <dc:creator><![CDATA[Kevin Kiruri]]></dc:creator>
            <pubDate>Fri, 29 Dec 2023 21:05:47 GMT</pubDate>
            <atom:updated>2023-12-29T21:19:01.830Z</atom:updated>
            <content:encoded><![CDATA[<p>When running applications and offering online services, it is crucial to determine, at any given time, whether the applications and systems are running as expected. Downtimes can lead to great financial losses. Hence, it is paramount that every organization has a real time view of its operations. AWS offers observability and monitoring tools that help organizations with obtaining logs, metrics and traces that are used to quickly make decisions. I watched a presentation by Rich McDonough, a Senior World Wide Cloud Ops Specialist at AWS made at the AWS Summit SF 2022 on Full-stack observability and application monitoring that shed a lot of insight on the topic. You can find the whole presentation below.</p><iframe src="https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Fwww.youtube.com%2Fembed%2For7uFFyHIX0%3Ffeature%3Doembed&amp;display_name=YouTube&amp;url=https%3A%2F%2Fwww.youtube.com%2Fwatch%3Fv%3Dor7uFFyHIX0&amp;image=https%3A%2F%2Fi.ytimg.com%2Fvi%2For7uFFyHIX0%2Fhqdefault.jpg&amp;key=a19fcc184b9711e1b4764040d3dc5c07&amp;type=text%2Fhtml&amp;schema=youtube" width="854" height="480" frameborder="0" scrolling="no"><a href="https://medium.com/media/b0d85ddedde7659da2af6487544255bb/href">https://medium.com/media/b0d85ddedde7659da2af6487544255bb/href</a></iframe><p>The following are the takeaways from the presentation</p><blockquote><strong>Monitoring vs Observability</strong></blockquote><p>Monitoring are activities that make one get information from a system. It comprises of activities such as tracing and logging. Observability on the other hand describes how well we can understand a system. It determines the metrics, logs and traces that we wish to collect from a system.</p><blockquote><strong>Pillars of Observability</strong></blockquote><p>There are 3 pillars of observability and include:</p><ol><li><em>Logs</em>. Logging data should be sequence aware and logging data should be received in the correct order and should should not be able to inject data which helps us to recreate the series of events happening in our environment</li><li><em>Metrics</em>. Measures taken from systems eg CPU utilization, Disk usage etc</li><li><em>Traces.</em> Application traces tell what an app is doing, how long it takes and how well it is doing it.</li></ol><blockquote><strong>Components to build a full stack observability solution</strong></blockquote><p>Building an observability solution requires tools. We can make use of AWS Cloud native tools or Open source managed services. In this blog, we will be focussing on the AWS-native services.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*I_6egsRTByQxs8432K_cew.png" /></figure><p>Data is collected by collectors(agents) such as CloudWatch agent that collects logs and metrics from applications, servers, services etc. X-ray agent collects application trace data.</p><p>The data is then fed into CloudWatch in the form of metrics, traces and logs which are then analyzed to give meaningful information to a user.</p><blockquote><strong>Full stack observability strategies</strong></blockquote><p>When evaluating your observability solution, you can choose to watch things from the Outside-in or from the Inside-out. The choice is entirely up to you and depends on your Service Level Objectives (SLO) and what matters to your business.</p><p>The main question asked when analyzing observability solutions is: How does good look like?</p><p>When using the Outside-in strategy,we ask what does good like to end users? Typical Service Level Objectives(SLOs) in this strategy would be page load time, successful purchases, JS and HTML errors, conversion rates and new customer acquisition, new feature adoption rates and search engine traffic. The SLOs are dependent on end-user behavior and what they observe on the application.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/982/1*CaCF38x7ODqMvcuXMZqipw.png" /></figure><p>When using the Inside-out strategy, we ask what does good look like for your backend applications? Typical SLOs that are backend focused include: slow SQL queries, integration health or container restarts, High/low CPU utilization, disk usage (IOPS), API response tome, errors, faults and retries. Comprises of internal-facing signals.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*JFK7uVlTVTTZK0VrqofKLw.png" /></figure><p>You should observe what matters to your business. Your business objectives should shape your goals, objectives and approach to observability. You should determine the signals you wish to receive from your workloads and what to create alarms and notifications for. Build a full stack observability solution that reduces your mean time to resolution to the lowest possible.</p><p>The best application should combine both worlds. The solution should have both the Outside-in view and the Inside-out view. The solution should be inclusive to know what is happening in all environments and be in a position to assess all outcomes.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*85gLojfasgtFeyYbpETirQ.png" /></figure><blockquote><strong>AWS CloudWatch</strong></blockquote><p>The solution will be different for every organization. We need a service to help us analyze what is happening in all environments. AWS CloudWatch comes in handy in the cloud-native environment. It is:</p><ul><li>Highly resilient</li><li>Fault tolerant</li><li>Built for scale</li></ul><p>AWS CloudWatch can be easily integrated with all services on AWS which makes it a crucial element in observability</p><blockquote><strong>Hybrid, Distributed and On-premises workloads</strong></blockquote><p>AWS also offers solutions for hybrid and on-premises workloads. The hybrid, distributed and on-premises workloads can consume the AWS monitoring and observability tools through the internet through the Pay As You Go model. You can connect through a VPN, the public internet or Direct connect to access the services based on the required level of privacy. The tools would work on these environments just as they would in an AWS environment</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*DN2C06CFS-dkIvKNOqcJfw.png" /></figure><blockquote><strong>Conclusion</strong></blockquote><p>In conclusion, keeping a close eye on how our applications and systems perform is important for businesses, and AWS provides some handy tools for this job. Rich McDonough’s talk at the AWS Summit SF 2022 taught us that observability is about understanding what’s happening in our systems, not just getting data from them. The three key things to watch out for are logs, metrics, and traces, which help us know what’s going on and fix issues quickly. AWS CloudWatch is like a superhero in this world, resilient and scalable, making it easy for us to keep an eye on things, especially in the cloud. Whether your stuff is in the cloud, a mix of places, or even your own space, AWS has got tools to help you see what’s happening, and that’s crucial for making sure everything runs smoothly and businesses stay on track.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=42628f13b569" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[How to install and use Amazon CodeWhisperer on VS Code]]></title>
            <link>https://kevinkiruri.medium.com/how-to-install-and-use-amazon-codewhisperer-on-vs-code-404e6624ec95?source=rss-4f39f6231241------2</link>
            <guid isPermaLink="false">https://medium.com/p/404e6624ec95</guid>
            <category><![CDATA[codewhisperer]]></category>
            <category><![CDATA[generative-ai-tools]]></category>
            <category><![CDATA[aws]]></category>
            <category><![CDATA[code]]></category>
            <category><![CDATA[ai]]></category>
            <dc:creator><![CDATA[Kevin Kiruri]]></dc:creator>
            <pubDate>Fri, 29 Dec 2023 10:33:14 GMT</pubDate>
            <atom:updated>2023-12-29T10:52:17.422Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/300/1*JW7EQi2l7zF15H9nXKJ9Rw.jpeg" /></figure><p>AI is coming up greatly and our buddies at AWS have not been left behind. We now have <a href="https://aws.amazon.com/codewhisperer/">AWS CodeWhisperer</a>. Let’s look at how to install it on VS Code, how to use it and establish whether it’s a worthwhile tool</p><blockquote><strong><em>Prerequisites</em></strong></blockquote><ol><li>Install <a href="https://code.visualstudio.com/download">Visual Studio Code</a> (<a href="https://code.visualstudio.com/download">VS Code</a>). You can get it <a href="https://code.visualstudio.com/download">here</a></li></ol><blockquote><strong>Install the AWS Toolkit extension</strong></blockquote><ol><li>On VS Code, click on the extensions icon on the left hand panel</li></ol><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*PJ1GU3zXeR__ICrX_yDD6A.png" /></figure><p>2. Search for AWS Toolkit on the search bar. Select the toolkit created by Amazon Web Services and click on Install</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*bizcJSFg8GuS_tSaY8PqqQ.png" /></figure><p>3. Once installed, the AWS Toolkit icon should appear on the left hand panel</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*_XjtwOKS4avGyoHkELjCCQ.png" /></figure><blockquote><strong>Sign in with your AWS Builder ID or Sign up if you do not have one</strong></blockquote><ol><li>Click on the AWS Toolkit icon then click on Sign in to get started and select Use for free wih AWS Builder ID .</li></ol><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*egBHTCVAsX1H55S_2TVwyQ.png" /></figure><p>2. Open the link in your browser. You are asked to confirm the characters on your browser with the characters that appear on your VS Code screen. (Giving some Bluetooth connection vibes 😅). Click on Confirm and continue</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/800/1*5Yt2IJtcnyiO2POUCgdJRQ.png" /></figure><p>3. Create an AWS Builder ID if you don’t have one. If you already do, you can proceed to Sign in</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/777/1*IdthIiaTSIOLKgxsqrO3Ew.png" /></figure><p>4. Once connected, your screen on VS Code will show that you are Connected to AWS Builder ID</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*JhjtDNrQyUK1n4Qj2yXsNw.png" /></figure><blockquote><strong>Testing</strong></blockquote><p>Now that we have the power of AI, let’s test it out</p><p>We’ll let CodeWhisperer create a function that takes in a string and returns the number of vowels in the string. Let’s see how it does</p><ol><li>Open the file in the language you want to code in. We will be using a Python file (.py)</li><li>Enter the command starting with // . In our case, we will enter //create a function that takes in a string and returns the number of vowels in the string</li><li>Click on Alt+C (Option+C on Mac) to run CodeWhisperer. CodeWhisperer makes suggestions and you can accept them by hitting Tab</li><li>Once done, comment the CodeWhisperer commands and run your code to test it</li></ol><iframe src="https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Fwww.youtube.com%2Fembed%2Fgq_Pd06MLkg%3Ffeature%3Doembed&amp;display_name=YouTube&amp;url=https%3A%2F%2Fwww.youtube.com%2Fwatch%3Fv%3Dgq_Pd06MLkg&amp;image=https%3A%2F%2Fi.ytimg.com%2Fvi%2Fgq_Pd06MLkg%2Fhqdefault.jpg&amp;key=a19fcc184b9711e1b4764040d3dc5c07&amp;type=text%2Fhtml&amp;schema=youtube" width="854" height="480" frameborder="0" scrolling="no"><a href="https://medium.com/media/e6340ed0361a1f5e79253e10cbde1e2e/href">https://medium.com/media/e6340ed0361a1f5e79253e10cbde1e2e/href</a></iframe><blockquote><strong>Verdict</strong></blockquote><p>There is no doubt that AI is here to help us make our work easier. Using this as a stepping stone, let’s build even more products in a faster and more efficient way. AWS CodeWhisperer is definitely a tool to use. Can’t wait to see AWSome products from it.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=404e6624ec95" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Build a Real Time Data Streaming System with Amazon Kinesis Data Streams]]></title>
            <link>https://kevinkiruri.medium.com/build-a-real-time-data-streaming-system-with-amazon-kinesis-data-streams-08d62301d1cd?source=rss-4f39f6231241------2</link>
            <guid isPermaLink="false">https://medium.com/p/08d62301d1cd</guid>
            <category><![CDATA[lambda]]></category>
            <category><![CDATA[kinesis]]></category>
            <category><![CDATA[data-streaming]]></category>
            <category><![CDATA[data]]></category>
            <category><![CDATA[aws]]></category>
            <dc:creator><![CDATA[Kevin Kiruri]]></dc:creator>
            <pubDate>Thu, 28 Dec 2023 21:11:25 GMT</pubDate>
            <atom:updated>2023-12-28T21:11:25.245Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Arf0mLfyeb3WFPWSxCZxZw.png" /></figure><p>Let’s talk streaming data. You want to read data and stream it into another program or different uses for gain. Your data suddenly comes to life and can be used to do amazing things. You could even launch a rocket using a text file if you wanted. Here, your imagination is your limit. Let’s dive into a step-by-step journey on how to stream data on AWS. Let’s dig in.</p><blockquote><strong>Prerequisites</strong></blockquote><ol><li>Have an AWS account. If you don’t have one, sign up <a href="https://aws.amazon.com/">here</a> and enjoy the benefits of the <a href="https://aws.amazon.com/free/?trk=99f831a2-d162-429a-9a77-a89f6b3bd6cd&amp;sc_channel=ps&amp;ef_id=Cj0KCQjw06-oBhC6ARIsAGuzdw03qBfKD7AA8ejOZPfdI7Pg4XshpRd57Q1p-a8--id9VCM5Cjg78xEaAmJtEALw_wcB%3AG%3As&amp;s_kwcid=AL%214422%213%21645125273267%21e%21%21g%21%21aws+free+tier%2119574556890%21145779847112&amp;all-free-tier.sort-by=item.additionalFields.SortRank&amp;all-free-tier.sort-order=asc&amp;awsf.Free+Tier+Types=*all&amp;awsf.Free+Tier+Categories=*all">Free-Tier Account</a></li><li>View <a href="https://github.com/Kevin-byt/AWS-Projects/tree/main/12weeksworkshop/Data-streaming">project files</a></li></ol><blockquote><strong>Creating a Kinesis data stream</strong></blockquote><ol><li>Search for kinesis on the Services search bar and select the Kinesis service</li></ol><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*L76We33ioglX4Jp76ZGWVA.png" /></figure><p>2. Click on Create data stream</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*u2iOqsNwek_RAAzuJslRCg.png" /></figure><p>3. Set the following configurations:</p><ul><li>Data stream name: learn-data-stream</li><li>Capacity mode: Provisioned</li><li>Provisioned shards: 1</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/944/1*00ytGQsho-OSAr46xzlt3w.png" /></figure><p>4. Scroll down and click on Create data stream</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/968/1*E_tMJVymwE6RzK9xTNnrmA.png" /></figure><p>5. Once the data stream page, select the Configuration tab. Scroll down to Encryption and click on Edit</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*WKiZ9URI_3Cjet1bj67UbQ.png" /></figure><p>6. Check Enable server-side encryption and select Use AWS managed CMK. Click on Save changes</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1013/1*fbPpvGEgyVZHeRj-Hh9l8w.png" /></figure><blockquote><strong>Create an S3 Bucket</strong></blockquote><ol><li>Search for S3 on the Services search box</li><li>click on Create bucket</li></ol><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*eDwE77o8BQGKGO6ap3eXvw.png" /></figure><p>3. Give the bucket a unique name. (Remember the bucket name should be globally unique). I have named mine learning-datasource-12222023</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*iR7TXcFPKJcXJ6eN1pEvXQ.png" /></figure><p>4. scroll down and Enable Bucket versioning</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*x3tH1vkgEv2SeE8EXM8NqA.png" /></figure><p>5. Scroll down and click on Create bucket</p><blockquote><strong>Create Lambda role</strong></blockquote><ol><li>Navigate to the IAM service console</li></ol><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*k5X1W9JEWlndhD83-rS1Pw.png" /></figure><p>2. Select Policies on the left-hand menu then click on Create policy</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*wODnFTQ03n-hiMc9IWAm0w.png" /></figure><p>3. Under Specify permissions select JSON then enter the following JSON template</p><pre>{<br> &quot;Version&quot;: &quot;2012-10-17&quot;,<br> &quot;Statement&quot;: [<br>  {<br>   &quot;Sid&quot;: &quot;kinesisAccess&quot;,<br>   &quot;Effect&quot;: &quot;Allow&quot;,<br>   &quot;Action&quot;: [<br>    &quot;kinesis:*&quot;<br>   ],<br>   &quot;Resource&quot;: &quot;*&quot;,<br>   &quot;Condition&quot;: {<br>    &quot;StringEquals&quot;: {<br>     &quot;aws:RequestedRegion&quot;: &quot;us-east-1&quot;<br>    }<br>   }<br>  },<br>  {<br>   &quot;Effect&quot;: &quot;Allow&quot;,<br>   &quot;Action&quot;: [<br>    &quot;logs:*&quot;<br>   ],<br>   &quot;Resource&quot;: &quot;arn:aws:logs:*:*:*&quot;<br>  },<br>  {<br>   &quot;Effect&quot;: &quot;Allow&quot;,<br>   &quot;Action&quot;: [<br>    &quot;s3:*&quot;<br>   ],<br>   &quot;Resource&quot;: &quot;arn:aws:s3:::*&quot;<br>  }<br> ]<br>}</pre><p>4. Scroll to the bottom and click Next</p><p>5. Give the policy a name (I named mine lambda_datastream )</p><p>6. Scroll down and click on Create policy</p><p>7. Select Roles on the left-hand menu and click on Create role</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*S-rDIBTZOfxex3ltTtwF9A.png" /></figure><p>8. Set the following configurations then click on Next at the bottom.</p><ul><li>Trusted entity type: AWS Service</li><li>Use case: Lambda</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*aj5fGQJrpRKDJdX3uGd9rg.png" /></figure><p>9. Under Add permissions search for the policy you created and check it when it appears, then click Next</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Um_xkfgMRQDH62Pia2oqzQ.png" /></figure><p>10. Give the role a name. (I named mine lambda_datastream )</p><p>11. Scroll down and click on Create role</p><blockquote><strong>Creating a Producer Lambda Function</strong></blockquote><ol><li>Search for Lambda in the Services search box and select the Lambda service</li></ol><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*989vA_h5qFLsI2lcQfiy4Q.png" /></figure><p>2. Click on Create function</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*7Yie5oLg2zNL5gZ5nW_4mA.png" /></figure><p>3. Set the following configuration:</p><ul><li>Create function: Author from scratch</li><li>Name: Producer</li><li>Runtime: Node.js 14.x</li><li>Execution role: Use an existing role , then select lambda_datastream</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*mjI86r5v8U9Jkjoys7aKOA.png" /></figure><p>4. Scroll to the bottom and click on Create function</p><p>5. On the Code tab, replace the existing code in the index.js file with the following:</p><pre>const AWS = require(&#39;aws-sdk&#39;);         <br>AWS.config.update({             <br>    region: &#39;us-east-1&#39;                     <br>})                      <br>const s3 = new AWS.S3();                    <br>const kinesis = new AWS.Kinesis();                      <br>exports.handler = async (event) =&gt; {                <br>    console.log(JSON.stringify(event));                 <br>    const bucketName = event.Records[0].s3.bucket.name;             <br>    const keyName = event.Records[0].s3.object.key;             <br>    const params = {        <br>        Bucket: bucketName,             <br>        Key: keyName                <br>    }   <br>    await s3.getObject(params).promise().then(async (data) =&gt; {     <br>        const dataString = data.Body.toString();                <br>        const payload = {   <br>            data: dataString    <br>        }<br>        await sendToKinesis(payload, keyName);              <br>    }, error =&gt; {       <br>        console.error(error);                   <br>    })              <br>};                  <br>async function sendToKinesis(payload, partitionKey) {                   <br>    const params = {        <br>        Data: JSON.stringify(payload),              <br>        PartitionKey: partitionKey, <br>        StreamName: &#39;learn-data-stream&#39;      <br>    }           <br>    await kinesis.putRecord(params).promise().then(response =&gt; {        <br>        console.log(response);      <br>    }, error =&gt; {       <br>        console.error(error);               <br>    })                  <br>}</pre><p>6. Click on Deploy to save the function</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*-wISKKAxB1tBuWihQd8cPA.png" /></figure><blockquote><strong>Create an event notification</strong></blockquote><ol><li>Navigate to S3</li><li>Click on the bucket we created and select the Properties tab</li></ol><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*n0HIWUFKhg_uuHXILTKAdQ.png" /></figure><p>3. Scroll down to the Event notifications section and click on Create event notification .</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*u1D2I_iIFOcTOihmjxmM6A.png" /></figure><p>4. Under Create an event notification set the following:</p><ul><li>Event name: upload-event</li><li>Prefix: default (blank)</li><li>Suffix: .txt</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/977/1*ecNoTPxGLS9mccQOzVcpJw.png" /></figure><p>5. Under Event types select All object create events</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*ksardVYQLqxWfIsmxfQ7_w.png" /></figure><p>6. Under Destination select Lambda function and choose the producer function from your list of functions and click on Save changes</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*4HE_5RO0bHE9n-he9kvusQ.png" /></figure><p>7. This will trigger the producer function each time an object is uploaded to the bucket</p><blockquote><strong>Create the consumer lambda functions</strong></blockquote><ol><li>Set up a lambda environment with the Node.js 14.x runtime and name the function Consumer1 and the lambda_datastream role</li></ol><figure><img alt="" src="https://cdn-images-1.medium.com/max/974/1*InYgHXxztuHyfToTo9O0yQ.png" /></figure><p>2. Under the code section, replace the existing code with the following code and Deploy to save the changes</p><pre>exports.handler = async (event) =&gt; {<br>    console.log(JSON.stringify(event));<br>    for (const record of event.Records) {<br>    const data = JSON.parse(Buffer.from(record.kinesis.data, &#39;base64&#39;));<br>    console.log(&#39;consumer #1&#39;, data);<br>    }<br>    };</pre><p>3. Move to the Configurations tab and select Triggers and click on Add trigger</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Eg1vrKiJfzaOLypoe1P-bA.png" /></figure><p>4. Under Trigger configuration</p><ul><li>Source: Kinesis</li><li>Kinesis stream: Kinesis/learn-data-stream</li><li>Check Activate trigger</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*-cQZFgecYxfrk4aOej8f9g.png" /></figure><p>5. Scroll to the bottom and click on Add</p><p>6. Create a second consumer function. Name it Consumer2 and use the following code with it. Remember to use the lambda_datastream role we created.</p><pre>exports.handler = async (event) =&gt; {<br>    console.log(JSON.stringify(event));<br>    for (const record of event.Records) {<br>    const data = JSON.parse(Buffer.from(record.kinesis.data, &#39;base64&#39;));<br>    console.log(&#39;consumer #2&#39;, data);<br>    }<br>    };</pre><p>7. Move to the Configurations tab and select Triggers and click on Add trigger just like we did for Consumer1</p><p>8. Under Trigger configuration</p><ul><li>Source: Kinesis</li><li>Kinesis stream: Kinesis/learn-data-stream</li><li>Check Activate trigger</li></ul><blockquote><strong>Uploading a txt file to S3</strong></blockquote><ol><li>Using a text editor on your computer create a txt file. I created one with the following content:</li></ol><pre>Data Streaming exercise<br>We are learning how to stream data <br>This is a lot of fun!!!</pre><p>2. Navigate to S3 and select the bucket we created earlier</p><p>3. Click on Upload</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*dBPO6r_ftMJWOexcly8Ifw.png" /></figure><p>4. On the Upload page, click on Add files and select the txt file we created. Then click on Upload at the bottom of the page.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*DbVmnmxaO4zt0nObAMX47g.png" /></figure><p>5. Click on Close to close the upload status page</p><blockquote><strong>Testing the configuration</strong></blockquote><blockquote><strong>Producer Function</strong></blockquote><ol><li>The producer function should get triggered upon upload of the txt file</li><li>We test by checking the lambda functions logs</li><li>Navigate to CloudWatch on the Services menu</li><li>Under Logs , click on Log groups and search for the /aws/lambda/ . You should see logs from the 3 Lambda functions</li></ol><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*2-ntbj7ZAhEJ_BdmY9TVjg.png" /></figure><p>5. Click on the /aws/lambda/Producer log group</p><p>6. You should see the Log stream at the bottom of the page. Click on it</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*XfKAYvoTwY2ov5uzAJ138A.png" /></figure><p>7. On expanding the lo0gs, we can see that it was able to p[ick the object from the bucket</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*0_lDpwZ8u5bkqxjfExdazw.png" /></figure><p>8. Then the data is sent to Kinesis .</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/984/1*fa6TrqUCTyfytv97VHHCZA.png" /></figure><blockquote><strong>Consumer1 Function</strong></blockquote><ol><li>Under the /aws/lambda/Consumer1 log streams, we can see that it was able to get the decoded data from Kinesis</li></ol><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*lqWpZs0yN3B4t04PuzkYLg.png" /></figure><p>2. The function then takes the data and reads it out</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Dqrs7vBx79F8u4ANnegxjw.png" /></figure><blockquote><strong>Consumer2 Function</strong></blockquote><ol><li>Viewing the /aws/lambda/Consumer2 log stream, we can see that the function is also able to read and decode data from Kinesis</li></ol><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*i-zW_FLW_V_2_nptDmRTaQ.png" /></figure><blockquote><strong>Clean Up</strong></blockquote><ol><li>Delete the Kinesis Data stream</li><li>Delete the txt file upload to S3</li><li>Delete the created S3 bucket</li><li>Delete the Producer, Consumer1 and Consumer2 functions</li><li>Delete the lambda role created</li><li>Delete the lambda policy that we had created</li></ol><blockquote><strong>Conclusion</strong></blockquote><p>The blog showed us a simple guide on how to work wth data. May it serve as a guide to help you proceed in your learning journey. Thanks and see you in the next blog.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=08d62301d1cd" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[AWS Event-Driven Architecture]]></title>
            <link>https://kevinkiruri.medium.com/aws-event-driven-architecture-098db8bee1b0?source=rss-4f39f6231241------2</link>
            <guid isPermaLink="false">https://medium.com/p/098db8bee1b0</guid>
            <category><![CDATA[sns]]></category>
            <category><![CDATA[aws]]></category>
            <category><![CDATA[event-driven-architecture]]></category>
            <category><![CDATA[automation]]></category>
            <category><![CDATA[lambda]]></category>
            <dc:creator><![CDATA[Kevin Kiruri]]></dc:creator>
            <pubDate>Fri, 08 Dec 2023 15:30:02 GMT</pubDate>
            <atom:updated>2023-12-08T15:30:02.928Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*9-iXqe4YFXsE34DxpxV_AA.png" /></figure><p>Event-Driven Architecture (EDA) is a great way to make things happen automatically when certain events occur. AWS offers tools to make events trigger actions across different services. In this blog, we’ll walk through a hands-on example using three AWS tools: Amazon S3, AWS Lambda, and Simple Notification Service (SNS). Imagine setting up a special storage space (S3 bucket) where, whenever you put a file in it, it automatically kicks off a little program (Lambda function). This program then sends a quick message to an email address using SNS. It’s like creating a chain reaction of events without manual intervention. Join us as we break down the steps and explore how this event-driven magic works on AWS!</p><blockquote><strong><em>Prerequisites</em></strong></blockquote><ol><li>Have an AWS account. If you don’t have one, sign up <a href="https://aws.amazon.com/">here</a> and enjoy the benefits of the <a href="https://aws.amazon.com/free/?trk=99f831a2-d162-429a-9a77-a89f6b3bd6cd&amp;sc_channel=ps&amp;ef_id=Cj0KCQjw06-oBhC6ARIsAGuzdw03qBfKD7AA8ejOZPfdI7Pg4XshpRd57Q1p-a8--id9VCM5Cjg78xEaAmJtEALw_wcB%3AG%3As&amp;s_kwcid=AL%214422%213%21645125273267%21e%21%21g%21%21aws+free+tier%2119574556890%21145779847112&amp;all-free-tier.sort-by=item.additionalFields.SortRank&amp;all-free-tier.sort-order=asc&amp;awsf.Free+Tier+Types=*all&amp;awsf.Free+Tier+Categories=*all">Free-Tier Account</a></li><li>View <a href="https://github.com/Kevin-byt/AWS-Projects/tree/main/12weeksworkshop/Event-Driven">project files</a></li></ol><blockquote><strong>Creating an S3 Bucket</strong></blockquote><ol><li>On the S3 bucket console on AWS, click on Create bucket</li></ol><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*OQE5wwXCVxmVQdGrvg_9NQ.png" /><figcaption>v</figcaption></figure><p>2. Select the region you want to host the bucket and give the bucket a unique name for all buckets within that region</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Ijxz3T2_eAt_4P8SbrfJ0w.png" /></figure><p>3. Keep the other settings as default and scroll to the bottom and click on Create bucket</p><blockquote><strong>Creating an SNS Topic</strong></blockquote><ol><li>On the Amazon SNS Dashboard, select Topic on the left-hand menu and click on Create topic</li></ol><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*N9fPtcBPMSs5uIN4UqOfug.png" /></figure><p>2. Select the Standard type, and give it a name. `</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*vcwfA3tHIW79-jpyNgM-xQ.png" /></figure><p>3. Scroll to the bottom and click on Create topic</p><p>4. Note the ARN of the Topic, we will be using it later</p><blockquote><strong>Subscribing to a Topic</strong></blockquote><p>We will create an email subscription to the topic</p><ol><li>On the Topic dashboard, click on Create subscription</li></ol><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*cZmv53QbFcQXgIVMHlP72w.png" /></figure><p>2. Select Email as the Protocol and put the desired email as the Endpoint</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*qabKurvBfEJSc2zMXRqPQA.png" /></figure><p>3. Scroll to the bottom and click on Create subscription</p><p>4. In the email endpoint, you will receive an email from AWS Notifications for you to Confirm the subscription</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*wzKnRlreY4YG4uey8zjDmg.png" /></figure><p>5. Open the mail and click on the Confirm subscription link</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*fCDzNENGNJGbHemXcSbBxg.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/986/1*NXjw35QJz0otjHvY6_UxBg.png" /></figure><blockquote><strong>Creating a Lambda Function</strong></blockquote><ol><li>On the Lambda Console dashboard, on the left navigation plane, select Functions, then click on Create function</li></ol><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*gxRo959PkHBWsNT5rF4v-Q.png" /></figure><p>2. Select Author from scratch, then give the function a name and select the most updated Python runtime (Currently as at the time I am writing this blog, it’s Python 3.11)</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*CT68dIgd4ftHonKH30BT3g.png" /></figure><p>3. Scroll to the bottom and click on Create function</p><p>4. On the code tab, input the following code.</p><pre>import os<br>import json<br>from datetime import datetime<br>import boto3<br><br>def lambda_handler(event, context):<br>    # Get information about the uploaded object from the S3 event<br>    s3_event = event[&#39;Records&#39;][0][&#39;s3&#39;]<br>    bucket_name = s3_event[&#39;bucket&#39;][&#39;name&#39;]<br>    object_name = s3_event[&#39;object&#39;][&#39;key&#39;]<br>    object_size = s3_event[&#39;object&#39;][&#39;size&#39;]<br><br>    # Get the current timestamp<br>    upload_time = datetime.now().strftime(&quot;%Y-%m-%d %H:%M:%S&quot;)<br><br>    # Prepare message to be sent to SNS<br>    message = {<br>        &#39;bucket_name&#39;: bucket_name,<br>        &#39;object_name&#39;: object_name,<br>        &#39;object_size&#39;: object_size,<br>        &#39;upload_time&#39;: upload_time<br>    }<br><br>    # Convert the message to JSON<br>    message_json = json.dumps(message)<br><br>    # Set up the SNS client<br>    sns_client = boto3.client(&#39;sns&#39;)<br><br>    # Replace &#39;YOUR_SNS_TOPIC_ARN&#39; with your actual SNS topic ARN<br>    sns_topic_arn = &#39;YOUR_SNS_TOPIC_ARN&#39;<br><br>    # Publish the message to the SNS topic<br>    sns_client.publish(<br>        TopicArn=sns_topic_arn,<br>        Message=message_json,<br>        Subject=f&quot;Object Uploaded: {object_key}&quot;<br>    )<br><br>    print(f&quot;Message published to SNS: {message_json}&quot;)<br><br>    return {<br>        &#39;statusCode&#39;: 200,<br>        &#39;body&#39;: json.dumps(&#39;Message published to SNS successfully!&#39;)<br>    }</pre><p>3. Replace YOUR_SNS_TOPIC_ARN with the ARN of the Topic we created earlier</p><p>3. Click on Deploy</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*u4tGM41GFJjDCI1malLVtg.png" /></figure><blockquote><strong>Adding the Trigger to the Lambda Function</strong></blockquote><ol><li>On the Function overview page, click on Add trigger</li></ol><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*MKsIEbL1bMone76chsiIrA.png" /></figure><p>2. Select S3 as the source and select the bucket we created earlier. Select PUT as the Event type so that the function only runs when an object is uploaded. Scroll to the bottom and click on Add</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/785/1*39IreapuujwLYHINJjHKAQ.png" /></figure><blockquote><strong>Edit the Lambda Role</strong></blockquote><ol><li>Click on the Configuration tab and select the Role name link</li></ol><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*l4eh5QOfR2iRwS-0M4ywpg.png" /></figure><p>2. It opens on a new tab. Scroll down to Permission policies and select the Policy name link</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*biOWnNgHvPY4hcCmuaGphw.png" /></figure><p>3. Click on Edit</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*_etj-NbWiOcxGdqZcwRFCg.png" /></figure><p>4. Select the visual tab and click on Add more permissions</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*zXnQ5CsaIlVn1h9BgUshOw.png" /></figure><p>5. Select the Service as SNS, for the actions allowed, select Publish and you can tick the box for Any in this account . Click on Next</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*b4bfG2O8-57eGANjhMwMZA.png" /></figure><p>6. Click on Save changes</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*0e3kYiYGOWxqBnoPmjvqyQ.png" /></figure><blockquote><strong>Let’s test it</strong></blockquote><ol><li>Open the bucket we created on S3 and click on Upload</li></ol><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*kpx4jti80k2IkOFykyJG_A.png" /></figure><p>2. Click on Add files , select the file you want to upload, then click on Upload</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/879/1*Bwy_7EOzEmWbk5jdmyYoCA.png" /></figure><p>3. You should get an email with the object details:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*uQaXOAV5HmCIDZDOjab3eg.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*yZLqicJPPJDTIcUp5X5hxw.png" /></figure><blockquote><strong>CleanUp</strong></blockquote><ol><li>Delete the uploaded object</li><li>Delete the S3 bucket</li><li>Delete the SNS topic</li><li>Delete the Lambda function</li></ol><blockquote><strong>Conclusion</strong></blockquote><p>Using Event-Driven Architecture on AWS shows us a smart way to make things happen automatically when certain events occur. Our journey with Amazon S3, AWS Lambda, and Simple Notification Service (SNS) demonstrated how a small event, like uploading a file, can set off a chain of actions. This isn’t just about making things easier — it’s about making systems work faster and smarter. Understanding this approach helps us build efficient, flexible solutions in cloud. Event-Driven Architecture is like giving our systems a superpower, making them respond and adapt to changes in a way that’s both clever and simple.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=098db8bee1b0" width="1" height="1" alt="">]]></content:encoded>
        </item>
    </channel>
</rss>