<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by HarbeyLefty on Medium]]></title>
        <description><![CDATA[Stories by HarbeyLefty on Medium]]></description>
        <link>https://medium.com/@harbeylefty?source=rss-a9dda9de9043------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/1*dmbNkD5D-u45r44go_cf0g.png</url>
            <title>Stories by HarbeyLefty on Medium</title>
            <link>https://medium.com/@harbeylefty?source=rss-a9dda9de9043------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Fri, 20 Nov 2026 13:02:57 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@harbeylefty/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[Serverless Image Processing Pipeline on AWS: Step-by-Step Guide With AWS SAM]]></title>
            <link>https://medium.com/@harbeylefty/serverless-image-processing-pipeline-on-aws-step-by-step-guide-with-aws-sam-7c8ad6e785f1?source=rss-a9dda9de9043------2</link>
            <guid isPermaLink="false">https://medium.com/p/7c8ad6e785f1</guid>
            <dc:creator><![CDATA[HarbeyLefty]]></dc:creator>
            <pubDate>Wed, 11 Jun 2025 20:02:37 GMT</pubDate>
            <atom:updated>2025-06-11T20:02:37.986Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/750/1*I4UlWfUm8urrA4xoPU5LRg.jpeg" /></figure><p>Hello and welcome back! This is Part 3, the final instalment in our series on building an event-driven serverless image processing pipeline on AWS.</p><ul><li>In <strong>part 1:</strong> <a href="https://medium.com/@harbeylefty/serverless-image-processing-on-aws-step-by-step-guide-with-aws-lambda-ec75b16dfb1f"><strong>Serverless Image Processing on AWS: Step-by-Step Guide with AWS Lambda</strong></a>, we detailed the Python logic for each of our five Lambda functions.</li><li>In <strong>part 2: </strong><a href="https://medium.com/@harbeylefty/aws-step-functions-orchestrating-an-event-driven-image-processing-workflow-0f041da58b4d"><strong>AWS Step Functions: Orchestrating an Event-Driven Image Processing Workflow</strong></a>, we designed the Amazon States Language (ASL) workflow to manage these Lambdas, handle errors, and send notifications.</li></ul><p>Now, with our core logic and orchestration designed, it’s time to automate everything! In this article, we’ll focus on the <strong>AWS Serverless Application Model (SAM)</strong> to define our entire infrastructure. From S3 buckets and DynamoDB tables to IAM roles, all our Lambda functions, the Step Functions state machine, its S3 trigger, and our API Gateway endpoint. We’ll walk through key parts of the template.yaml file, cover the deployment process, and finally test our fully automated, SAM-deployed pipeline.</p><p>As always, the complete project, including the final template.yaml and all source code, is available on <a href="https://github.com/Harbeylefty/Image-Processing-Pipeline">GitHub</a></p><p><strong>Why AWS SAM for our serverless pipeline?</strong></p><p>Manually creating and configuring over a dozen interconnected AWS resources (S3 buckets, Lambdas, IAM roles, a Step Function, DynamoDB table, SNS topic, API Gateway, EventBridge rule) for our pipeline would be time-consuming, error-prone, and difficult to replicate or version control. This is where Infrastructure as Code (IaC) becomes essential.</p><p><strong>AWS SAM</strong> is an open-source framework built on AWS CloudFormation that provides a simplified syntax specifically for defining serverless applications. For our project, SAM offers:</p><ul><li><strong>Conciseness:</strong> A much shorter way to define serverless resources like Lambdas, APIs, and Step Functions compared to raw CloudFormation.</li><li><strong>Automation:</strong> Enables automated provisioning and updates of our entire application stack.</li><li><strong>Repeatability &amp; consistency:</strong> Ensures our pipeline can be deployed identically across different environments or accounts.</li></ul><p>1.<strong> Setting up the foundation: S3, DynamoDB, &amp; SNS in SAM</strong></p><p>First, we need our core storage and messaging services:</p><p><strong>S3 Buckets:</strong> We require two S3 buckets: one for incoming original images (UploadsBucket) and another for the generated thumbnails (ThumbNailsBucket). In SAM, these are defined as AWS::S3::Bucket resources. We ensure public access is blocked for security. The UploadsBucket is also configured to send event notifications to Amazon EventBridge, which is crucial for triggering our pipeline.</p><pre>UploadsBucket:<br>    Type: AWS::S3::Bucket  <br>    Properties:<br>      BucketName: YourUniqueBucketname<br>      PublicAccessBlockConfiguration:<br>        BlockPublicAcls: true<br>        BlockPublicPolicy: true<br>        IgnorePublicAcls: true<br>        RestrictPublicBuckets: true<br>      NotificationConfiguration:<br>        EventBridgeConfiguration:<br>          EventBridgeEnabled: true # This tells S3 to send events to EventBridge<br><br>  ThumbNailsBucket:<br>    Type: AWS::S3::Bucket<br>    Properties:<br>      BucketName: YourUniqueBucketName<br>      PublicAccessBlockConfiguration:<br>        BlockPublicAcls: true<br>        BlockPublicPolicy: true<br>        IgnorePublicAcls: true<br>        RestrictPublicBuckets: true</pre><p><strong>DynamoDB Table:</strong> A DynamoDB table stores all the metadata extracted from the images, along with references to the original image and its thumbnails. We define this using AWS::DynamoDB::Table, specifying ImageKey (the S3 object key) as our string partition key and using PAY_PER_REQUEST billing mode for cost efficiency.</p><pre>ImageProcessingDynamoDBTable:<br>    Type: AWS::DynamoDB::Table<br>    Properties:<br>      TableName: ImageProcessingDb<br>      AttributeDefinitions:<br>        - AttributeName: ImageKey<br>          AttributeType: S<br>      KeySchema:<br>        - AttributeName: ImageKey<br>          KeyType: HASH<br>      BillingMode: PAY_PER_REQUEST</pre><p><strong>SNS Topic &amp; subscription:</strong> For notifications, we define an AWS::SNS::Topic (e.g., ImageProcessingNotifications). An AWS::SNS::Subscription is then created to send these notifications to an email address , which requires confirmation via a link sent to the email after deployment.</p><pre>ImageProcessingSNSTopic:<br>    Type: AWS::SNS::Topic<br>    Properties:<br>      TopicName: ImageProcessingNotifications<br><br>  EmailNotificationSubscription:<br>    Type: AWS::SNS::Subscription<br>    Properties:<br>      TopicArn: !Ref ImageProcessingSNSTopic<br>      Protocol: email<br>      Endpoint: Your-Email@gmail.com</pre><p>2.<strong> Defining IAM roles and permissions</strong></p><p>We define two specific IAM roles ImageProcessingLambdaRole and ImageProcessingStepFunctionRole using AWS::IAM::Role in our SAM template to ensure our services operate under the principle of least privilege.</p><p><strong>ImageProcessingLambdaRole: </strong>This role is assumed by all our Lambda functions. It’s granted the AWSLambdaBasicExecutionRole managed policy for CloudWatch logging. Its custom inline policy provides precise permissions to:</p><ul><li>Read objects from the UploadsBucket (specifically the uploads/* path).</li><li>Write objects to the ThumbNailsBucket (to the thumbnails/* path).</li><li>Perform PutItem and GetItem actions on our DynamoDB Table.</li><li>Call rekognition:DetectLabels.</li></ul><p>These resource permissions use SAM/CloudFormation intrinsic functions like !Sub and !GetAtt to dynamically refer to the S3 buckets and DynamoDB table created within the same template.</p><p><strong>ImageProcessingStepFunctionRole: </strong>This role is assumed by our Step Functions state machine. Its policy allows it to:</p><ul><li>Invoke the specific Lambda functions that are part of the image processing workflow (using !GetAtt LambdaLogicalId.Arn).</li><li>Publish messages to our SNS Topic (using !RefImageProcessingSNSTopic).</li><li>Write its own execution logs to CloudWatch</li><li>Securely pass the ImageProcessingLambdaRole to the Lambda service when invoking functions, thanks to an iam:PassRole permission with a condition restricting it to the Lambda service.</li><li>Send trace data to AWS X-Ray.</li></ul><pre>ImageProcessingLambdaRole:<br>    Type: AWS::IAM::Role<br>    Properties:<br>      RoleName: ImageProcessingLambdaRole<br>      AssumeRolePolicyDocument:<br>        Version: &#39;2012-10-17&#39;<br>        Statement:<br>          - Effect: Allow<br>            Principal:<br>              Service: lambda.amazonaws.com<br>            Action: sts:AssumeRole<br>      ManagedPolicyArns:<br>        - arn:aws:iam::aws:policy/service-role/AWSLambdaBasicExecutionRole #For lambda logging to cloudwatch<br>      Policies:<br>        - PolicyName: LambdaImageProcessingPolicy<br>          PolicyDocument:<br>            Version: &#39;2012-10-17&#39;<br>            Statement:<br>              # Statement 1: Allow reading from the uploads Bucket <br>              - Sid: S3ReadFromUploadsFolder<br>                Effect: Allow<br>                Action: s3:GetObject<br>                Resource: !Sub arn:aws:s3:::${UploadsBucket}/uploads/*<br><br>              #Statement 2: Allow write to the thumbnails s3 bucket <br>              - Sid: S3WriteToThumbnailsFolder<br>                Effect: Allow<br>                Action: s3:PutObject<br>                Resource: !Sub arn:aws:s3:::${ThumbNailsBucket}/thumbnails/*<br><br>              #Statement 3: Allow Read and write to DynamoDB Table <br>              - Sid: DynamoDBTableAccess<br>                Effect: Allow<br>                Action:<br>                  - dynamodb:PutItem<br>                  - dynamodb:GetItem<br>                Resource: !GetAtt ImageProcessingDynamoDBTable.Arn<br><br>              # Rekognition Access <br>              - Sid: RekognitionAccess<br>                Effect: Allow<br>                Action: rekognition:DetectLabels<br>                Resource: &#39;*&#39; # For detect labels resource is usually *</pre><pre>ImageProcessingStepFunctionRole:<br>    Type: AWS::IAM::Role<br>    Properties:<br>      RoleName: ImageProcessingStepFunctionsRole<br>      AssumeRolePolicyDocument:<br>        Version: &#39;2012-10-17&#39;<br>        Statement:<br>          - Effect: Allow<br>            Principal:<br>              Service: !Sub states.${AWS::Region}.amazonaws.com<br>            Action: sts:AssumeRole<br>      Policies:<br>        - PolicyName: StepFunctionImageProcessingPolicy<br>          PolicyDocument:<br>            Version: &#39;2012-10-17&#39;<br>            Statement:<br>              # Allow for invoking of lambda functions <br>              - Sid: InvokeLambdaFunctions<br>                Effect: Allow<br>                Action: lambda:InvokeFunction<br>                Resource:<br>                  - !GetAtt ImageValidationLambda.Arn<br>                  - !GetAtt GenerateThumbnailsLambda.Arn<br>                  - !GetAtt ExtractMetadataLambda.Arn<br>                  - !GetAtt StoreResultsLambda.Arn<br><br>              - Sid: PublishToSnsTopic<br>                Effect: Allow<br>                Action: sns:Publish<br>                Resource: !Ref ImageProcessingSNSTopic<br><br>              - Sid: AllowStepFunctionToWriteItsexecutionLogs<br>                Effect: Allow<br>                Action:<br>                  - logs:CreateLogDelivery<br>                  - logs:GetLogDelivery<br>                  - logs:UpdateLogDelivery<br>                  - logs:DeleteLogDelivery<br>                  - logs:ListLogDeliveries<br>                  - logs:PutResourcePolicy<br>                  - logs:DescribeResourcePolicies<br>                  - logs:DescribeLogGroups<br>                Resource: &#39;*&#39;<br><br>              - Sid: AllowStepFunctionsToPassLambdaExecutionRole<br>                Effect: Allow<br>                Action: iam:PassRole<br>                Resource: !GetAtt ImageProcessingLambdaRole.Arn<br>                Condition:<br>                  StringEquals:<br>                    iam:PassedToService: lambda.amazonaws.com<br>              - Sid: XrayAccessForTracing<br>                Effect: Allow<br>                Action:<br>                  - xray:PutTraceSegments<br>                  - xray:PutTelemetryRecords<br>                  - xray:GetSamplingRules<br>                  - xray:GetSamplingTargets<br>                Resource: &#39;*&#39;</pre><p>If you look at the SAM template.yaml code, you&#39;ll see special tags like !Ref, !GetAtt, and !Sub. These are powerful AWS CloudFormation Functions that SAM uses to dynamically link your resources together without hardcoding values. Here’s a quick guide:</p><ul><li><strong>!Ref</strong> <strong>(Reference)</strong>: Gets the main physical identifier (like a name or an ARN, depending on the resource type) of another resource defined in your template. For example, in the project, to link our email subscription to the correct SNS topic, we use TopicArn: !Ref ImageProcessingSNSTopic. This tells CloudFormation to use the ARN of the ImageProcessingSNSTopic resource.</li><li><strong>!GetAtt (Get Attribute):</strong> Fetches a <em>specific attribute</em> of a resource (like its full ARN if !Ref only provides the name, or other specific properties like a Lambda&#39;s ARN).</li><li><strong>!Sub: </strong>Builds a new string by inserting values from other resources, parameters, or AWS pseudo parameters (like ${AWS::Region}) into a template string. Inside a !Sub string, ${LogicalId} acts like !Ref LogicalId, and ${LogicalId.AttributeName} acts like !GetAtt LogicalId.AttributeName. <br>For example in our project, To allow our Lambda functions to read only from the uploads/ folder in our UploadsBucket, we construct the specific S3 resource ARN like this: Resource: !Sub arn:aws:s3:::${UploadsBucket}/uploads/*. Here, ${UploadsBucket} gets replaced with the actual name of our UploadsBucket.</li></ul><p>3.<strong> Define Lambda Functions</strong></p><p>It’s time to bring our processing logic to life by defining the Lambda functions. These functions, written in Python will perform the actual image processing tasks.<br>Let’s take a look at the GenerateThumbnailsLambda definition as an illustrative example.</p><pre>GenerateThumbnailsLambda:<br>    Type: AWS::Serverless::Function<br>    Properties:<br>      FunctionName: GenerateThumbnailsFunction<br>      Description: Generates thumbnails for images uploaded<br>      Runtime: python3.12<br>      Architectures:<br>        - x86_64<br>      Handler: lambda_function.lambda_handler<br>      Role: !GetAtt ImageProcessingLambdaRole.Arn<br>      Timeout: 20<br>      MemorySize: 256<br>      CodeUri: ../../src/generate-thumbnails-lambda/<br>      Environment:<br>        Variables:<br>          THUMBNAILS_S3_BUCKET: !Ref ThumbNailsBucket<br>          THUMBNAIL_SIZES: 100x100,640x480<br>      Layers:<br>        - arn:aws:lambda:us-east-1:770693421928:layer:Klayers-p312-Pillow:6</pre><p>In this definition:</p><ul><li>The CodeUri points to the specific source code for this Lambda.</li><li>Under Environment.Variables, THUMBNAILS_S3_BUCKET is dynamically set using !Ref ThumbNailsBucket, linking it directly to the S3 bucket resource we defined earlier in the same SAM template. THUMBNAIL_SIZES is passed as a configurable string that our Python code parses.</li><li>The Layers property includes the ARN for a pre-compiled Pillow library. This is crucial for using Pillow in Lambda, as it has C extensions that need to be compatible with the Lambda (Amazon Linux) runtime environment. Using a layer keeps our deployment package small (just our Python script) and handles the Pillow dependency effectively.</li></ul><p>4. <strong>Orchestrating the magic: AWS Step Functions</strong></p><p>Our image processing pipeline relies on AWS Step Functions to orchestrate the sequence of Lambda invocations, manage data flow, and handle errors. The detailed logic for this workflow, written in Amazon States Language (ASL), resides in our workflow.asl.json file. <em>(We covered the ins and outs of this ASL definition, including its states, error handling, and notification mechanisms, in Part 2 of this series)</em></p><p><strong>Defining the state machine in SAM</strong></p><pre>ImageProcessingStateMachine:<br>    Type: AWS::Serverless::StateMachine<br>    Properties:<br>      Name: ImageProcessingWorkflow<br>      DefinitionSubstitutions:<br>        ImageValidationLambdaArn: !GetAtt ImageValidationLambda.Arn<br>        GenerateThumbnailsLambdaArn: !GetAtt GenerateThumbnailsLambda.Arn<br>        ExtractMetadataLambdaArn: !GetAtt ExtractMetadataLambda.Arn<br>        StoreResultsLambdaArn: !GetAtt StoreResultsLambda.Arn<br>        ImageProcessingSNSTopicArn: !Ref ImageProcessingSNSTopic<br>      Role: !GetAtt ImageProcessingStepFunctionRole.Arn<br>      Events:<br>        S3UploadTrigger:<br>          Type: EventBridgeRule<br>          Properties:<br>            EventBusName: default<br>            InputPath: $.detail # This will pass the S3 event details to your Step Function<br>            Pattern:<br>              source:<br>                - aws.s3<br>              detail-type:<br>                - Object Created <br>              detail:<br>                bucket:<br>                  name:<br>                    - !Ref UploadsBucket<br>                object:<br>                  key:<br>                    - prefix: uploads/<br>      Logging:<br>        Level: ALL<br>        IncludeExecutionData: true<br>        Destinations:<br>          - CloudWatchLogsLogGroup:<br>              LogGroupArn: !GetAtt StateMachineLogGroup.Arn<br>      Tracing:<br>        # EnableAWS X-ray tracing<br>        Enabled: true<br>      DefinitionUri: ../stepfunctions/workflow.asl.json<br><br>  # Create Log group where logs from statemachine should go to <br>  StateMachineLogGroup:<br>    Type: AWS::Logs::LogGroup<br>    Properties:<br>      RetentionInDays: 7</pre><p>This definition links our ASL file, substitutes the dynamic ARNs, assigns the correct IAM role, configures detailed logging to a specific CloudWatch Log Group we also define, enables X-Ray tracing, and sets up the S3 event trigger via an EventBridge rule. The InputPath: $.detail ensures our first Lambda gets the direct S3 event data.</p><p>5.<strong> Exposing the status check: API Gateway via SAM</strong></p><p>To query the status of processed images, our StatusCheckLambda is fronted by an Amazon API Gateway HTTP API. This is concisely defined in SAM by adding an HttpApi event source directly to the StatusCheckLambda resource definition</p><pre># Inside StatusCheckLambda definition in template.yaml<br>Events:<br>        StatusCheckApi:<br>          Type: HttpApi <br>          Properties:<br>            Path: /images/uploads/{filename}/status<br>            Method: GET</pre><p>SAM automatically creates the API Gateway, the /images/uploads/{filename}/status route for GET requests, and the Lambda integration.</p><p>6.<strong> Deployment with AWS SAM</strong></p><p>Clone the repository and navigate to the SAM application directory</p><pre>git clone https://github.com/Harbeylefty/Image-Processing-Pipeline.git<br>cd image-processing-pipeline/infrastructure/sam/</pre><p>Build the SAM application</p><pre>sam build --use-container</pre><p>This command builds Lambda function dependencies inside a Docker container that mimics the AWS Lambda environment.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*FgKQ6L_vpVnEQsmiQFBSGg.png" /></figure><p><strong>Deploy the SAM application:</strong> The first time you deploy, use the --guided flag to specify deployment parameters.</p><pre>sam deploy --guided</pre><p>You will be prompted for:</p><ul><li>Stack Name: Choose a name for your CloudFormation stack. This needs to be unique in your AWS account and region for new stacks.</li><li>AWS Region: Your desired AWS region.</li><li>Confirm changes before deploy: Enter Y to review changes.</li><li>Allow SAM CLI IAM role creation: Enter Y.</li><li>Disable rollback: Enter N (allows CloudFormation to roll back to the previous stable state if the deployment fails).</li><li>Save arguments to configuration file samconfig.toml?: Enter Y.</li><li>Accept defaults for other prompts</li><li>You will be asked to confirm if you want to deploy the resources, Enter Y to deploy</li><li>You should see the resources being created and a message “Successfully created/updated stack” shown on screen after successful deployments of all resources.</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*lsE4aa35QCzWyaPQiLSTTA.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*vEWye7cdhomFZbANXLHzMw.png" /></figure><p>After the stack is successfully deployed, an email will be sent from AWS notifications to the email address specified in template.yaml for the SNS subscription. You must click the confirmation link in this email to activate the subscription and start receiving notifications from the topic.<br>Also, check the AWS console to confirm the creation of all resources.</p><p>For subsequent deployments after making changes to your template or Lambda code, you can often just run sam deploy.</p><p><strong>7. Testing the pipeline</strong></p><p>Triggering the pipeline</p><ul><li>Navigate to the s3 console</li><li>Go to the s3 created by SAM for uploads</li><li>Upload a valid image file into the uploads/prefix folder. You can do this by using the “Create folder” button in the S3 console to create an uploads folder, then navigating into it and clicking “Upload”.</li></ul><p><strong>Expected outputs and verification</strong></p><ul><li>Step Functions: Monitor the execution in the AWS Step Functions console for the state machine . Look for a successful (all green) execution.</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*8W2mZggwPUbeO9fJAqJf0Q.png" /></figure><ul><li>Thumbnails: Check the S3 bucket for thumbnails inside its thumbnails/ prefix.</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*9FUma_xLO0L4aQIUQcPzQg.png" /></figure><ul><li>DynamoDB: Verify a new item with the image’s metadata and thumbnail details is created in the ImageProcessingDb table. The ImageKey will be the full S3 key (e.g., uploads/your-image-name.jpg)</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*7LVlj9A8RnY2RNbSo1xQIw.png" /></figure><ul><li>SNS Notification: Check your configured email for a success or failure notification from the SNS topic.</li></ul><p><strong>Using the status check API</strong></p><p>i. After sam deploy completes successfully with the API defined, navigate to API Gateway in the console, find your HTTP API , go to Stages, select the $default stage, and copy the Invoke URL.</p><p>ii. The endpoint structure is: GET&lt;InvokeURL&gt;/images/uploads/{filename}/status. Replace &lt;InvokeURL&gt; with your actual API invoke URL and {filename} with the name of an image file (e.g., filename.png) that has been processed. <br>Example using curl:</p><pre>curl https://&lt;your-api-id&gt;.execute-api.&lt;your-region&gt;.amazonaws.com/images/uploads/Kubernetes2.png/status</pre><p>Alternatively you can also use postman for testing</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*PafxpyVGGJY_XjbCHt_dxw.png" /></figure><p>iii. Expected Responses:</p><ul><li>200 OK: With a JSON body containing the image’s processing details from DynamoDB.</li><li>404 Not Found: If the image ID (derived from uploads/filename) is not found in DynamoDB, with a body like {&quot;message&quot;: &quot;Image details not found for the given image identifier.&quot;}.</li></ul><p><strong>Cleaning up/deleting the stack</strong></p><p>To remove all AWS resources created by this SAM application</p><p>i. Empty S3 Buckets: Manually delete all objects from the UploadsBucket and ThumbNailsBucket that were created by this stack. CloudFormation cannot delete buckets that contain objects.</p><p>ii. Delete the SAM/CloudFormation Stack: Navigate to your infrastructure/sam/ directory in the terminal and run</p><pre>sam delete --stack-name &lt;YourStackName&gt;</pre><p>Confirm the deletion when prompted. Alternatively, you can delete the stack directly from the AWS CloudFormation console.</p><p><strong>8. Challenges faced and lesson learned</strong></p><p>No project comes without its learning moments, and for this pipeline, a notable one involved the Pillow library. When I first deployed my Lambda functions for thumbnailing and metadata extraction, I hit a common error; the cannot import name &#39;_imaging&#39; from &#39;PIL&#39; runtime error.</p><p>This error arises because Pillow includes C extensions, which, when installed locally on a Mac or Windows, are compiled in an environment that differs from Amazon Linux environment that AWS Lambda uses for its python runtimes. As a result, those binaries aren’t compatible with the Lambda’s execution environment.</p><p>The clear solution was to ensure Pillow was provided to my Lambda functions in a pre-compiled, Lambda-compatible format. After standardizing my Lambdas on Python 3.12, I achieved this by utilizing a public AWS Lambda layer for Pillow (specifically, a Klayers version for Python 3.12). This Layer ARN was simply added to the Layers property of the relevant functions in my AWS SAM template.yaml. This approach not only resolved the compatibility errors instantly but also kept my Lambda deployment packages lean (containing only my function code) and made dependency management cleaner. This experience also highlighted the value of sam build --use-container for other projects, it builds your dependencies inside a Docker container that replicates the Lambda environment, preventing such issues.</p><p><strong>Key takeaways:</strong></p><ul><li><strong>AWS Step Functions:</strong> Is invaluable for reliably managing complex, multi-step serverless workflows, including reliable error handling and state transitions.</li><li><strong>AWS SAM (Serverless Application Model): </strong>SAM simplifies defining, deploying, and managing resources like Lambda, API Gateway, Step Functions, and DynamoDB using infrastructure as code.</li><li><strong>AWS Lambda dependency management</strong>: For Python packages with compiled dependencies (e.g., Pillow), using Lambda Layers or containerized builds (sam build --use-container) ensures compatibility with the Lambda runtime and smooth deployment.</li></ul><p><strong>Conclusion</strong></p><p>In this guide, we’ve walked through the creation of a complete, event-driven serverless image processing pipeline on AWS, managed end-to-end with AWS SAM. This project is a testament to the power of combining services like <strong>Step Functions, Lambda, S3, DynamoDB, Rekognition, API</strong> <strong>Gateway </strong>and <strong>SAM </strong>to build scalable and automated solutions.</p><p>Thank you for reading, and I hope this walkthrough provides practical insights and inspiration for your own serverless adventures on AWS.</p><p>The possibilities with serverless are immense, and I encourage you to explore them further. Happy building!</p><p><strong>Resources</strong></p><p><a href="https://docs.aws.amazon.com/step-functions/latest/dg/welcome.html">Step Functions Developer Guide</a></p><p><a href="https://docs.aws.amazon.com/serverless-application-model/latest/developerguide/what-is-sam.html">AWS SAM Documentation</a></p><p><a href="https://docs.aws.amazon.com/lambda/latest/dg/welcome.html">AWS Lambda Documentation</a></p><p><a href="https://docs.aws.amazon.com/eventbridge/latest/userguide/eb-what-is.html">Amazon EventBridge Documentation</a></p><p>Connect with me</p><p><a href="https://www.linkedin.com/in/habeebyusuf/"><strong>LinkedIn</strong></a><br><a href="https://x.com/harbeylefty"><strong>X(Twitter)</strong></a><a href="https://x.com/harbeylefty"><br></a><a href="http://Harbeysegun@gmail.com/"><strong>Email</strong></a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=7c8ad6e785f1" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[AWS Step Functions: Orchestrating an Event-Driven Image Processing Workflow]]></title>
            <link>https://medium.com/@harbeylefty/aws-step-functions-orchestrating-an-event-driven-image-processing-workflow-0f041da58b4d?source=rss-a9dda9de9043------2</link>
            <guid isPermaLink="false">https://medium.com/p/0f041da58b4d</guid>
            <dc:creator><![CDATA[HarbeyLefty]]></dc:creator>
            <pubDate>Wed, 11 Jun 2025 18:02:02 GMT</pubDate>
            <atom:updated>2025-06-11T18:02:02.062Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*8W2mZggwPUbeO9fJAqJf0Q.png" /></figure><p>In <strong>Part 1: </strong><a href="https://medium.com/@harbeylefty/serverless-image-processing-on-aws-step-by-step-guide-with-aws-lambda-ec75b16dfb1f"><strong>Serverless Image Processing on AWS: Step-by-Step Guide with AWS Lambda</strong></a>, we took a close look at the individual AWS Lambda functions that serve as the “workers” in our system. We explored the specific logic each function uses for tasks like image validation, thumbnail generation, metadata extraction, storing results in DynamoDB, and handling API status requests.</p><p>Now that we have these Lambda functions ready, how do we connect them into a seamless, automated, and resilient end-to-end workflow? How do we ensure data flows correctly from one step to the next, and what happens if one of those steps encounters an error?</p><p>This is where <strong>AWS Step Functions</strong> steps in as our master orchestrator! <br>In this article, we’ll dive into how I designed and implemented the Step Functions state machine using Amazon States Language (ASL) to manage our image processing pipeline. We’ll explore how it calls our Lambda functions in sequence, handles data, manages errors gracefully, and even sends out notifications.</p><p>All the code, including the Step Functions ASL definition (workflow.asl.json) we&#39;ll be discussing, can be found in the project&#39;s <a href="https://github.com/Harbeylefty/Image-Processing-Pipeline">GitHub</a> repository</p><p><strong>Why AWS Step Functions for our image pipeline?</strong></p><p>Before we jump into the code, let’s briefly touch upon why AWS Step Functions is such a great fit for orchestrating a multi-step serverless process like ours.</p><p>At its heart, <strong>AWS Step Functions </strong>is a serverless workflow service that lets you coordinate multiple AWS services (including Lambda functions, SNS, DynamoDB, and more) into a visual workflow. Instead of writing complex state management and retry logic within your Lambda functions or having Lambdas call other Lambdas directly, Step Functions allows you to define your entire workflow as a state machine.</p><p>For our image pipeline, this means we get:</p><ul><li><strong>Clear visual workflow:</strong> We can see exactly how our image processing progresses from one stage to the next.</li><li><strong>Built-in state management:</strong> Step Functions keeps track of the state of our workflow and the data being passed between steps.</li><li><strong>Error handling:</strong> We can define Catch blocks to handle errors from any Lambda function.</li><li><strong>Service integrations:</strong> It can directly integrate with services like SNS for notifications, reducing the amount of custom Lambda code needed.</li><li><strong>Scalability and durability:</strong> Being a managed service, it handles the underlying infrastructure for running and scaling our workflows.</li><li><strong>Separation of concerns:</strong> Our Lambda functions can focus on their specific business logic, while Step Functions handles the overall orchestration, error handling, and flow control.</li></ul><p>Essentially, Step Functions acts as the “Project Manager” for our image processing tasks, ensuring each Lambda “worker” is called at the right time, with the right information, and that any problems are handled gracefully.</p><p><strong>Designing the state machine: A walkthrough of our ASL workflow</strong></p><p>The definition of a Step Functions state machine is written in <strong>Amazon States Language (ASL)</strong>, which is a JSON-based, structured language. Our workflow is defined in the workflow.asl.json file within the infrastructure/stepfunctions/ directory of our project.</p><p>This ASL file specifies each state in our pipeline, the transitions between them, and how data is handled. Let’s break down some key parts.</p><p>(A quick but important note: Inside the ASL, you’ll see Resource ARNs for Lambda functions written as placeholders like ${ImageValidationLambdaArn}. Similarly, the SNS Topic ARN will be ${ImageProcessingSNSTopicArn}. These placeholders are resolved by <strong>AWS SAM </strong>during deployment (which we&#39;ll cover in Part 3 of this series), when SAM injects the actual ARNs of the Lambda functions and SNS topic it creates based on our template.yaml file. This keeps our ASL definition clean and dynamically linked to our SAM-managed resources.)</p><p>Let’s look at the overall structure and then break down the key states:</p><p><strong>The starting point: </strong><strong>ImageValidationState</strong></p><p>Every ASL definition has a basic structure:</p><pre><br>{<br>  &quot;Comment&quot;: &quot;Image processing workflow orchestrating lambda functions for image processing and notifications&quot;,<br>  &quot;StartAt&quot;: &quot;ImageValidationState&quot;,<br>  &quot;States&quot;: {<br>   // #All our state definitions will live inside this &quot;States&quot; object<br>  }<br>}</pre><ul><li><strong>Comment</strong>: A human-readable description.</li><li><strong>StartAt</strong>: Specifies the name of the first state to execute when the workflow begins. For us, it&#39;s ImageValidationState.</li><li><strong>States</strong>: This object contains the definitions for all the individual states in our pipeline.</li></ul><p>Now, let’s walk through each state.</p><ol><li><strong>ImageValidationState: </strong>This is our initial processing step, invoking the ImageValidationLambda (whose Python logic we explored in Part 1).</li></ol><pre>&quot;ImageValidationState&quot;: {<br>      &quot;Type&quot;: &quot;Task&quot;,<br>      &quot;Resource&quot;: &quot;${ImageValidationLambdaArn}&quot;,<br>      &quot;Comment&quot;: &quot;Invoke the Image Validation Lambda function&quot;,<br>      &quot;Next&quot;: &quot;GenerateThumbnailsState&quot;,<br>      &quot;Catch&quot;: [<br>        {<br>          &quot;ErrorEquals&quot;: [&quot;States.ALL&quot;],<br>          &quot;Next&quot;: &quot;NotifyFailureState&quot;,<br>          &quot;Comment&quot;: &quot;HandleImageValidationFailure&quot;<br>        }<br>      ]<br>    },</pre><ul><li><strong>Type: &quot;Task&quot;</strong>: This tells Step Functions this state will perform work, in this case, by invoking a Lambda function.</li><li><strong>Resource: &quot;${ImageValidationLambdaArn}&quot;</strong>: Specifies the Lambda function to execute. The placeholder will be replaced by SAM with the actual Lambda ARN.</li><li><strong>Next: &quot;GenerateThumbnailsState&quot;</strong>: If ImageValidationLambda completes successfully (i.e., doesn&#39;t raise an error), the workflow transitions to the state named GenerateThumbnailsState. The output of the current Lambda becomes the input to the next state by default.</li><li><strong>Catch: [...]</strong>: This is our crucial error handling mechanism.</li><li><strong>ErrorEquals: [&quot;States.ALL&quot;]</strong>: This is a powerful filter that catches <em>any</em> error raised by the ImageValidationLambda (e.g., if it throws a ValueError for an unsupported image type, or any other unhandled Python exception).</li><li><strong>Next: &quot;NotifyFailureState&quot;</strong>: If an error is caught, the workflow is redirected to our NotifyFailureState.</li></ul><p><strong>2. GenerateThumbnailsState: </strong>If image validation succeeds, we move on to creating thumbnails. This state invokes the GenerateThumbnailsLambda</p><pre>&quot;GenerateThumbnailsState&quot;: {<br>      &quot;Type&quot;: &quot;Task&quot;,<br>      &quot;Resource&quot;: &quot;${GenerateThumbnailsLambdaArn}&quot;,<br>      &quot;Comment&quot;: &quot;Invoke the Generate thumbnails lambda function&quot;,<br>      &quot;Next&quot;: &quot;ExtractMetadataState&quot;,<br>      &quot;Catch&quot;: [<br>        {<br>          &quot;ErrorEquals&quot;: [&quot;States.ALL&quot;],<br>          &quot;Next&quot;: &quot;NotifyFailureState&quot;,<br>          &quot;Comment&quot;: &quot;Handle GenerateThumbnailsFailure&quot;<br>        }<br>      ]<br>    },</pre><p>This state follows the exact same pattern as ImageValidationState: it&#39;s a Task that calls its respective Lambda. On success, it proceeds to ExtractMetadataState. If GenerateThumbnailsLambda fails, the Catch block sends the workflow to NotifyFailureState with the error information.</p><p><strong>3. ExtractMetadataState - </strong>Next, we extract basic and AI-driven metadata using our ExtractMetadataLambda.</p><pre>&quot;ExtractMetadataState&quot;: {<br>  &quot;Type&quot;: &quot;Task&quot;,<br>  &quot;Resource&quot;: &quot;${ExtractMetadataLambdaArn}&quot;,<br>  &quot;Comment&quot;: &quot;Invoke the ExtractMetadata lambda function&quot;,<br>  &quot;Next&quot;: &quot;StoreResultinDynamoDBState&quot;,<br>  &quot;Catch&quot;: [<br>    {<br>      &quot;ErrorEquals&quot;: [&quot;States.ALL&quot;],<br>      &quot;Next&quot;: &quot;NotifyFailureState&quot;,<br>      &quot;Comment&quot;: &quot;Handle Extractmetdata Failure&quot;<br>    }<br>  ]<br>},</pre><p>Again, a consistent structure. If metadata extraction is successful, it moves to StoreResultinDynamoDBState. Failures are caught and routed.</p><p><strong>4. StoreResultinDynamoDBState - </strong>This is the final core processing step, invoking StoreResultsLambda</p><pre>&quot;StoreResultinDynamoDBState&quot;: {<br>  &quot;Type&quot;: &quot;Task&quot;,<br>  &quot;Resource&quot;: &quot;${StoreResultsLambdaArn}&quot;,<br>  &quot;Comment&quot;: &quot;Invoke the StoreResults lambda function&quot;,<br>  &quot;Next&quot;: &quot;NotifySuccessState&quot;,<br>  &quot;Catch&quot;: [<br>    {<br>      &quot;ErrorEquals&quot;: [&quot;States.ALL&quot;],<br>      &quot;Next&quot;: &quot;NotifyFailureState&quot;,<br>      &quot;Comment&quot;: &quot;Handles StoreResultsinDynamoDB Failure&quot;<br>    }<br>  ]<br>},</pre><p>if this Lambda successfully stores all the data in DynamoDB, the workflow transitions to NotifySuccessState</p><p><strong>Handling success: The </strong><strong>NotifySuccessState</strong> If all the previous processing steps complete without error, this state is responsible for sending a success notification via Amazon SNS.</p><pre>&quot;NotifySuccessState&quot;: {<br>      &quot;Type&quot;: &quot;Task&quot;,<br>      &quot;Resource&quot;: &quot;arn:aws:states:::sns:publish&quot;, <br>      &quot;Parameters&quot;: {<br>        &quot;TopicArn&quot;: &quot;${ImageProcessingSNSTopicArn}&quot;, <br>        &quot;Subject&quot;: &quot;SUCCESS: Image Processing Complete&quot;,<br>        &quot;MessageStructure&quot;: &quot;json&quot;, <br>        &quot;Message&quot;: {<br>          // Creates a human-readable message using data from the workflow<br>          &quot;default.$&quot;: &quot;States.Format(&#39;Image processing for S3 object [{}] completed successfully. Overall status: {}.&#39;, $.s3_key, $.overall_processing_status)&quot;<br>        },<br>        &quot;MessageAttributes&quot;: { <br>          &quot;processing_status&quot;: {<br>            &quot;DataType&quot;: &quot;String&quot;,<br>            &quot;StringValue&quot;: &quot;SUCCESS&quot;<br>          },<br>          &quot;imageId&quot;: {<br>            &quot;DataType&quot;: &quot;String&quot;,<br>            &quot;StringValue.$&quot;: &quot;$.s3_key&quot; <br>          }<br>        }<br>      },<br>      &quot;Comment&quot;: &quot;Notify successful completion via SNS.&quot;,<br>      &quot;End&quot;: true <br>    }</pre><ul><li><strong>Resource: &quot;arn:aws:states:::sns:publish&quot;</strong>: This special ARN tells Step Functions to directly call the SNS Publish API without needing an intermediary Lambda.</li><li><strong>Parameters</strong>: This object defines what to send to SNS:</li><li><strong>TopicArn:</strong> The ARN of our ImageProcessingNotifications topic (substituted by SAM).</li><li><strong>MessageStructure: “json”</strong>: We’re sending a JSON-structured message.</li><li><strong>Message.default.$</strong>: This defines the default message body (used by email subscribers). States.Format() is a Step Functions intrinsic function that allows us to build a string dynamically using values from the current state&#39;s data payload (e.g.,$.s3_key, $.overall_processing_status, which would have been passed along from the StoreResultsLambda&#39;s output).<br>MessageAttributes: We add custom attributes like processing_status which can be used by SNS subscribers for filtering.</li><li><strong>End: true</strong>: This indicates that if this state completes, the entire state machine execution is considered successful and terminates.</li></ul><p><strong>Managing failures: </strong>If any of our core Lambda tasks fail and are caught, the workflow transitions to NotifyFailureState.</p><pre>&quot;NotifyFailureState&quot;: {<br>  &quot;Type&quot;: &quot;Task&quot;,<br>  &quot;Resource&quot;:  &quot;arn:aws:states:::sns:publish&quot;,<br>  &quot;Parameters&quot;: {<br>    &quot;TopicArn&quot;: &quot;${ImageProcessingSNSTopicArn}&quot;,<br>    &quot;Subject&quot;: &quot;FAILURE: Image Processing Error&quot;, <br>    &quot;MessageStructure&quot;: &quot;json&quot;, <br>    &quot;Message&quot;: {<br>      &quot;default.$&quot;: &quot;States.Format(&#39;Image Processing FAILED. Error Type: {}. Cause: {}&#39;, $.Error, $.Cause)&quot;<br>    },<br>    &quot;MessageAttributes&quot;: {<br>      &quot;processing_status&quot;: {<br>        &quot;DataType&quot;: &quot;String&quot;,<br>        &quot;StringValue&quot;: &quot;FAILURE&quot;<br>      }         <br>    }<br>  },<br>  &quot;Comment&quot;: &quot;Notify Failure Via SNS&quot;,<br>  &quot;Next&quot;: &quot;WorkFlowFailedState&quot;</pre><p>This state is very similar to NotifySuccessState but sends a failure-specific subject and message. The Message.default.$ uses States.Format() to include the Error and Cause details that were captured by the Catch block&#39;s ResultPath: &quot;$.error_info&quot; from the failed Lambda state. After sending the notification, it transitions to WorkFlowFailedState.</p><p>Finally, our explicit failure state:</p><pre>&quot;WorkFlowFailedState&quot;: {<br>      &quot;Type&quot;: &quot;Fail&quot;,<br>      &quot;Comment&quot;: &quot;Explicitly mark the workflow as failed after sending failure notification&quot;,<br>      &quot;Error&quot;: &quot;ImageProcessingPipelineError&quot;,<br>      &quot;Cause&quot;: &quot;The image processing workflow encountered an error. See execution logs and SNS failure notification for details.&quot;<br>    }<br>  }<br>}</pre><ul><li><strong>Type: &quot;Fail&quot;</strong>: This is a terminal state that definitively marks the entire state machine execution as &quot;Failed&quot; in the Step Functions console. It also allows you to specify a custom Error name and Cause for the overall workflow failure, which can be useful for monitoring and high-level error reporting.</li></ul><p>This ASL structure defines our complete, resilient image processing workflow, ready to be deployed by AWS SAM!</p><p><strong>Conclusion</strong></p><p>We’ve now designed a complete AWS Step Functions workflow using Amazon States Language (ASL), orchestrating our Lambda functions into a reliable and event-driven image processing pipeline. With the Lambda logic in place from Part 1 and the orchestration designed here in Part 2, the next step is to package and deploy this system as Infrastructure as Code.</p><p>In <strong>Part 3: </strong><a href="https://medium.com/@harbeylefty/serverless-image-processing-pipeline-on-aws-step-by-step-guide-with-aws-sam-7c8ad6e785f1"><strong>Serverless Image Processing Pipeline on AWS: Step-by-Step guide with AWS SAM</strong></a>, we’ll use the AWS Serverless Application Model<strong> </strong>(SAM)<strong> </strong>to define all our resources, set up the S3 trigger, deploy the pipeline, and add an API for status checks.</p><p>Connect with me</p><p><a href="https://www.linkedin.com/in/habeebyusuf/"><strong>LinkedIn</strong></a><br><a href="https://x.com/harbeylefty"><strong>X(Twitter)</strong></a><br><a href="http://Harbeysegun@gmail.com/"><strong>Email</strong></a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=0f041da58b4d" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Serverless Image Processing on AWS: Step-by-Step Guide With AWS Lambda]]></title>
            <link>https://medium.com/@harbeylefty/serverless-image-processing-on-aws-step-by-step-guide-with-aws-lambda-ec75b16dfb1f?source=rss-a9dda9de9043------2</link>
            <guid isPermaLink="false">https://medium.com/p/ec75b16dfb1f</guid>
            <dc:creator><![CDATA[HarbeyLefty]]></dc:creator>
            <pubDate>Wed, 11 Jun 2025 11:28:03 GMT</pubDate>
            <atom:updated>2025-06-11T11:28:03.598Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Kx0MunAnmkGYN-4AdXF2HQ.png" /></figure><p>In today’s cloud-native landscape, serverless architectures are the preferred approach for building scalable, event-driven applications without the burden of managing infrastructure. In this project, I designed a fully serverless image processing pipeline on AWS that automatically responds to new image uploads, performs background processing, tracks processing status, and sends notifications upon completion or failure. All built to run reliably without manual intervention.</p><p>To manage the flow of operations such as triggering processing, updating status, and sending notifications, I used <strong>AWS Step Functions</strong> to orchestrate multiple <strong>Lambda</strong> functions in a reliable and maintainable way. This approach allowed for better visibility, error handling, and separation of concerns across each stage of the pipeline.</p><p>This article is Part 1 of a 3-part series walking through how I designed, built, and deployed this solution.</p><ul><li><strong>Part 1 : </strong><a href="https://medium.com/@harbeylefty/serverless-image-processing-on-aws-step-by-step-guide-with-aws-lambda-ec75b16dfb1f"><strong>Serverless Image Processing on AWS: Step-by-Step Guide with AWS Lambda </strong></a>— We’ll explore the overall design of our serverless pipeline, understand the role of each AWS service, and then dive into the Python code for the AWS Lambda functions that perform the core image processing tasks.</li><li><strong>Part 2: </strong><a href="https://medium.com/@harbeylefty/aws-step-functions-orchestrating-an-event-driven-image-processing-workflow-0f041da58b4d"><strong>AWS Step Functions: Orchestrating an Event-Driven Image Processing Workflow</strong></a>— In the next instalment, we’ll focus on designing the state machine using Amazon States Language (ASL) to manage our Lambdas, handle errors, and integrate notifications.</li><li><strong>Part 3:</strong><a href="https://medium.com/@harbeylefty/serverless-image-processing-pipeline-on-aws-step-by-step-guide-with-aws-sam-7c8ad6e785f1"><strong> Serverless Image Processing Pipeline on AWS: Step-by-Step guide with AWS SAM</strong></a>— The final part will bring everything together, showing how to define and deploy the entire infrastructure as code using AWS SAM, including the S3 trigger and an API Gateway endpoint for status checks.</li></ul><p>You can find all the Python code for these Lambda functions and the complete project on <a href="https://github.com/Harbeylefty/Image-Processing-Pipeline">GitHub</a></p><h4>Architecture overview</h4><p>This serverless image processing pipeline is built with the following AWS services:<br><strong>Amazon S3</strong>: Two buckets are created, one for uploads and the other for thumbnails.</p><p><strong>Amazon EventBridge: </strong>Captures events from S3 and triggers the processing workflow</p><p><strong>AWS Step Functions</strong>: Orchestrates the multi-step processing pipeline</p><p><strong>AWS Lambda</strong>: Handles all the custom processing logic in our pipeline. Each function focuses on a specific task, making the system efficient and easy to maintain.</p><p><strong>Rekognition</strong>: Performs AI-powered label detection on uploaded image.</p><p><strong>DynamoDB</strong>: Stores processing results and metadata.</p><p><strong>SNS</strong>: Sends success or failure notifications to subscribed email endpoints</p><p><strong>Amazon API Gateway (HTTP API)</strong>: Exposes a GET endpoint to check image processing status</p><p><strong>IAM</strong>: Enforces least privilege access through service-specific roles</p><p><strong>AWS SAM</strong>: Manages Infrastructure as Code and deployment</p><p><strong>CloudWatch</strong>: Monitoring and Logging</p><h4>How it works: The image processing pipeline</h4><ol><li>A user uploads an image to the S3 bucket’s uploads/ prefix.</li><li>S3 triggers an ObjectCreated event, which is routed through Amazon EventBridge.</li><li>EventBridge triggers the AWS Step Functions state machine.</li><li>The state machine orchestrates the following Lambda functions:</li></ol><ul><li><strong>ImageValidationLambda</strong> checks that the uploaded file is a valid image</li><li><strong>GenerateThumbnailsLambda</strong> creates thumbnails and saves them to the thumbnails bucket.</li><li><strong>ExtractMetadataLambda</strong> : Gathers image metadata and labels using Rekognition</li><li><strong>StoreResultsInDynamoDBLambda</strong> Saves results in DynamoDB.</li></ul><p>5. Upon completion, Step Functions publishes a success or failure message to an SNS topic. Subscribers receive a notification via email.</p><p>6. Separately, clients can retrieve processing status via the GET /images/uploads/{filename}/status API Gateway endpoint, which triggers the StatusCheckLambda to query DynamoDB.</p><h4>Prerequisites</h4><p>To deploy and run this project, you will need:</p><ul><li>An AWS Account. Click <a href="https://aws.amazon.com/console/">here</a> to create one</li><li>AWS CLI installed and configured with necessary permissions. You can install <a href="https://docs.aws.amazon.com/cli/latest/userguide/getting-started-install.html">here</a></li><li>AWS SAM CLI installed. You can install <a href="https://docs.aws.amazon.com/serverless-application-model/latest/developerguide/install-sam-cli.html">here</a></li><li>Docker installed and running (required for sam build --use-container).</li></ul><p>Now that we’ve gone over how the pipeline works, let’s take a closer look at the core of it all: the five AWS Lambda functions. Each one handles a specific task to keep the whole system running smoothly and efficiently.</p><p><strong>Crafting the core: Our image processing Lambda Functions</strong></p><p>Our serverless image processing pipeline runs on several AWS Lambda functions each written in Python and tailored to perform a specific task. These functions form the core engine of the workflow. In this section, we’ll dive into each function’s responsibilities, logic, and how they interconnect to drive the entire pipeline.</p><p><em>(Note: The following Python snippets illustrate the core logic of each Lambda. For the complete, runnable code with all necessary imports, error handling, please refer to the project’s Github repository</em>)</p><ol><li>ImageValidationLambda:</li></ol><ul><li><strong>What it does:</strong> This Lambda is the first checkpoint. When a new file arrives, its main job is to check if it’s an image type we actually support (like a .jpg, .jpeg, or .png). If it&#39;s something else (like a text file or PDF), this Lambda flags it as unsupported, and the image won&#39;t proceed further in our pipeline, saving us from trying to process unsuitable files. It also makes sure to correctly read filenames that might have spaces or special characters.</li></ul><pre>import os<br>import urllib.parse<br><br>SUPPORTED_IMAGE_TYPES = [&#39;.jpg&#39;, &#39;.jpeg&#39;, &#39;.png&#39;]<br><br>def lambda_handler(event, context):<br>    bucket_name = event[&#39;bucket&#39;][&#39;name&#39;]<br>    object_key = event[&#39;object&#39;][&#39;key&#39;]<br>    <br>    object_key = urllib.parse.unquote_plus(object_key) # Decode S3 key<br>    _, file_extension = os.path.splitext(object_key.lower())<br><br>    if file_extension in SUPPORTED_IMAGE_TYPES:<br>        output = {<br>            &#39;s3_bucket&#39;: bucket_name,<br>            &#39;s3_key&#39;: object_key,<br>            &#39;image_type&#39;: file_extension,<br>            &#39;validation_status&#39;: &#39;SUCCESS&#39;<br>        }<br>        return output<br>    else:<br>        raise ValueError(f&quot;Unsupported image type: {file_extension}&quot;)</pre><p>2. GenerateThumbnailsLambda:<strong> </strong>The Image Resizer</p><ul><li><strong>What it does:</strong> Once an image is validated, this function steps in to create smaller versions of it: our thumbnails. It can create multiple sizes (like “100x100”, “640x480”, configurable via an environment variable THUMBNAIL_SIZES). It uses the Pillow library to perform the resizing and then uploads these new thumbnails to a dedicated S3 thumbnails bucket (name provided by the THUMBNAILS_S3_BUCKET environment variable) under a thumbnails/ prefix.</li><li><strong>Key considerations:</strong> This Lambda relies on the Pillow library for image manipulation. For Pillow to work correctly in Lambda’s Amazon Linux environment (due to its C extensions), we utilize a pre-compiled Lambda Layer (this will be specified in our SAM template in Part 3 of this guide). It also needs to know the target S3 bucket and desired thumbnail sizes from its configuration (environment variables).</li></ul><pre>from PIL import Image<br>import io<br>import os<br>import boto3 # For S3 interaction<br><br>s3_client = boto3.client(&#39;s3&#39;)<br><br>def lambda_handler(event, context):<br>    # Assume original_bucket, original_key_unquoted, base_filename are from event<br>    # Assume THUMBNAIL_SIZES (e.g., [(100,100), (640,480)]) is parsed from env var<br>    # Assume THUMBNAILS_S3_BUCKET is from env var<br>    # Assume image_data (bytes) and original_format (&#39;JPEG&#39;/&#39;PNG&#39;) are loaded from S3<br><br>    img = Image.open(io.BytesIO(image_data))<br>    thumbnail_locations = {} # To store S3 URIs of generated thumbnails<br><br>    for width, height in THUMBNAIL_SIZES:<br>        img_copy = img.copy()<br>        img_copy.thumbnail((width, height)) # Resizes while maintaining aspect ratio<br>        <br>        thumb_buffer = io.BytesIO()<br>        # Determine save format, defaulting to JPEG if original is not JPEG/PNG<br>        save_format = original_format if original_format.upper() in [&#39;JPEG&#39;, &#39;PNG&#39;] else &#39;JPEG&#39;<br>        if save_format.upper() == &#39;JPEG&#39; and img_copy.mode == &#39;RGBA&#39;: # Handle transparency for JPEG<br>            img_copy = img_copy.convert(&#39;RGB&#39;)<br>        <br>        img_copy.save(thumb_buffer, format=save_format)<br>        thumb_buffer.seek(0)<br><br>        thumbnail_key = f&quot;thumbnails/{base_filename}_{width}x{height}.{save_format.lower()}&quot;<br>        <br>        s3_client.put_object(<br>            Bucket=THUMBNAILS_S3_BUCKET,<br>            Key=thumbnail_key,<br>            Body=thumb_buffer,<br>            ContentType=Image.MIME[save_format] <br>        )<br>        thumbnail_locations[f&quot;{width}x{height}&quot;] = f&quot;s3://{THUMBNAILS_S3_BUCKET}/{thumbnail_key}&quot;<br>    <br>    # event.update({&#39;thumbnails&#39;: thumbnail_locations, &#39;thumbnail_generation_status&#39;: &#39;SUCCESS&#39;})<br>    # return event</pre><p>3. ExtractMetadataLambda:</p><ul><li><strong>What it does:</strong> This Lambda gathers rich information about the original image. It uses the Pillow library (from the same Lambda Layer) to extract basic properties like dimensions, format, and filesize. If enabled by its USE_REKOGNITION environment variable (set to &quot;true&quot;), it also calls Amazon Rekognition&#39;s detect_labels API to get AI-powered content labels.</li><li><strong>Key considerations:</strong> Needs the Pillow Layer. The decision to call Rekognition is configurable, allowing for flexibility and cost management.</li></ul><pre>import os<br>import boto3<br>from PIL import Image # For basic metadata<br>import io<br><br># s3_client = boto3.client(&#39;s3&#39;)<br># rekognition_client = boto3.client(&#39;rekognition&#39;)<br># USE_REKOGNITION = os.environ.get(&#39;USE_REKOGNITION&#39;, &#39;false&#39;).lower() == &#39;true&#39;<br><br>def lambda_handler(event, context):<br>    # Assume original_bucket, original_key_unquoted are from event.<br>    # Assume image_data (bytes) is downloaded from S3 for Pillow.<br>    <br>    img = Image.open(io.BytesIO(image_data))<br>    metadata = {<br>        &#39;filename&#39;: os.path.basename(original_key_unquoted),<br>        &#39;filesize_bytes&#39;: len(image_data),<br>        &#39;format&#39;: img.format,<br>        &#39;width_pixels&#39;: img.width,<br>        &#39;height_pixels&#39;: img.height,<br>        &#39;mode&#39;: img.mode<br>    }<br><br>    if USE_REKOGNITION: # From environment variable<br>        rek_response = rekognition_client.detect_labels(<br>            Image={&#39;S3Object&#39;: {&#39;Bucket&#39;: original_bucket, &#39;Name&#39;: original_key_unquoted}},<br>            MaxLabels=10,<br>            MinConfidence=75.0<br>        )<br>        rek_labels = [{&#39;Name&#39;: label[&#39;Name&#39;], &#39;Confidence&#39;: label[&#39;Confidence&#39;]}<br>                      for label in rek_response.get(&#39;Labels&#39;, [])]<br>        metadata[&#39;rekognition_labels&#39;] = rek_labels<br>        <br>    # event.update({&#39;extracted_metadata&#39;: metadata, &#39;metadata_extraction_status&#39;: &#39;SUCCESS&#39;})<br>    # return event</pre><p>4. StoreResultsLambda: The Record Keeper</p><ul><li><strong>What it does:</strong> This function takes all the data accumulated through the pipeline, original image details, validation status, thumbnail URLs, and all extracted metadata and stores it as a single item in our Amazon DynamoDB table (named ImageProcessingDb, which it gets from the DYNAMODB_TABLE_NAME environment variable). It uses the full S3 object key (e.g., uploads/filename.jpeg) as the primary key (ImageKey) in DynamoDB.</li><li><strong>Key considerations:</strong> A crucial step here is ensuring numeric data (like Rekognition confidence scores, which are floats) is converted to Python’s Decimal type before sending to DynamoDB, as DynamoDB&#39;s Number type requires this.</li></ul><pre>import boto3<br>import decimal<br>import json<br>import time<br>import os<br># import urllib.parse<br><br># Assume DYNAMODB_TABLE_NAME is from environment variable.<br># Assume DecimalEncoder class is defined in the full code.<br># dynamodb_table = boto3.resource(&#39;dynamodb&#39;).Table(DYNAMODB_TABLE_NAME)<br><br>def lambda_handler(event, context):<br>    # image_id = urllib.parse.unquote_plus(event[&#39;s3_key&#39;]) # Use unquoted key<br>    image_id = event[&#39;s3_key&#39;] # Assumes s3_key from event is already what you want for ImageKey<br><br>    item_to_store = {<br>        &#39;ImageKey&#39;: image_id, # Primary Key for DynamoDB<br>        &#39;s3_bucket_original&#39;: event.get(&#39;s3_bucket&#39;),<br>        &#39;s3_key_original&#39;: event.get(&#39;s3_key&#39;),<br>        &#39;image_type&#39;: event.get(&#39;image_type&#39;),<br>        &#39;validation_status&#39;: event.get(&#39;validation_status&#39;),<br>        &#39;thumbnails&#39;: event.get(&#39;thumbnails&#39;, {}),<br>        &#39;thumbnail_generation_status&#39;: event.get(&#39;thumbnail_generation_status&#39;),<br>        &#39;extracted_metadata&#39;: event.get(&#39;extracted_metadata&#39;, {}),<br>        &#39;metadata_extraction_status&#39;: event.get(&#39;metadata_extraction_status&#39;),<br>        &#39;overall_processing_status&#39;: &#39;COMPLETED&#39;,<br>        &#39;created_at&#39;: int(time.time()),<br>        &#39;updated_at&#39;: int(time.time())<br>    }<br>    <br>    # Convert Python dict to use Decimals for numbers before sending to DynamoDB<br>    item_json_string_with_decimals = json.dumps(item_to_store, cls=DecimalEncoder)<br>    item_cleaned_for_dynamodb = json.loads(item_json_string_with_decimals,<br>                                         parse_float=decimal.Decimal,<br>                                         parse_int=decimal.Decimal)<br>    <br>    dynamodb_table.put_item(Item=item_cleaned_for_dynamodb)<br>    <br>    # event.update({&#39;dynamodb_storage_status&#39;: &#39;SUCCESS&#39;, &#39;overall_processing_status&#39;: &#39;COMPLETED&#39;})<br>    # return event</pre><p>5. StatusCheckLambda:</p><ul><li><strong>What it does:</strong> This Lambda isn’t part of the main S3-triggered processing pipeline. Instead, it serves our API Gateway endpoint (e.g., GET /images/uploads/{filename}/status). When a client calls this API with a filename, this Lambda receives the request, queries our DynamoDB, and returns the image&#39;s full record as a JSON response (or a 404 &quot;Not Found&quot; message).</li><li><strong>Key considerations:</strong> It needs the DYNAMODB_TABLE_NAME from an environment variable. It also uses a DecimalEncoder (similar to StoreResultsLambda) to convert DynamoDB&#39;s Decimal numbers back into standard JSON numbers for the API response.</li></ul><pre>import json<br>import boto3<br>import decimal<br>import os<br><br># Assume DecimalEncoder class is defined.<br># Assume table is initialized:<br># DYNAMODB_TABLE_NAME = os.environ.get(&#39;DYNAMODB_TABLE_NAME&#39;)<br># table = boto3.resource(&#39;dynamodb&#39;).Table(DYNAMODB_TABLE_NAME)<br><br>def lambda_handler(event, context):<br>    # Example of extracting filename from API Gateway event<br>    filename_from_path = event.get(&#39;pathParameters&#39;, {}).get(&#39;filename&#39;)<br>    if not filename_from_path:<br>        # Return 400 Bad Request if filename is missing<br>        return {&#39;statusCode&#39;: 400, &#39;body&#39;: json.dumps({&#39;error&#39;: &#39;filename path parameter is missing.&#39;})}<br><br>    image_id_to_query = f&quot;uploads/{filename_from_path}&quot; # Construct the full ImageKey<br><br>    response_ddb = table.get_item(Key={&#39;ImageKey&#39;: image_id_to_query})<br><br>    if &#39;Item&#39; in response_ddb:<br>        item_body = json.dumps(response_ddb[&#39;Item&#39;], cls=DecimalEncoder) # Use DecimalEncoder for API response<br>        return {<br>            &#39;statusCode&#39;: 200,<br>            &#39;headers&#39;: {&#39;Content-Type&#39;: &#39;application/json&#39;, &#39;Access-Control-Allow-Origin&#39;: &#39;*&#39;},<br>            &#39;body&#39;: item_body<br>        }<br>    else: # Item not found<br>        return {<br>            &#39;statusCode&#39;: 404,<br>            &#39;headers&#39;: {&#39;Content-Type&#39;: &#39;application/json&#39;, &#39;Access-Control-Allow-Origin&#39;: &#39;*&#39;},<br>            &#39;body&#39;: json.dumps({&quot;message&quot;: &quot;Image details not found for the given image identifier.&quot;})<br>        }</pre><p><strong>Conclusion</strong></p><p>In this article, we looked at how our image processing pipeline works and walked through the Python code behind the five AWS Lambda functions. We now have a clear understanding of these individual processing units, what specific task each performs, their core logic.</p><p>But having a team of skilled functions is only half the battle. How do we make these independent Lambda functions collaborate effectively as a cohesive unit? How do we manage the flow of data from one function to the next, ensure that errors are handled gracefully, and that the entire process runs reliably from start to finish?</p><p>That’s precisely where our master conductor, <strong>AWS Step Functions </strong>steps onto the stage.</p><p><strong>Join me in Part 2: </strong><a href="https://medium.com/@harbeylefty/aws-step-functions-orchestrating-an-event-driven-image-processing-workflow-0f041da58b4d"><strong>AWS Step Functions: Orchestrating an Event-Driven Image Processing Workflow</strong></a><strong>,</strong> where we’ll explore how to design and implement the state machine using Amazon States Language (ASL). We’ll connect our Lambda functions into a resilient workflow, manage their execution sequence, implement reliable error handling, and integrate notifications. <br>Get ready to see how these individual functions are brought together to perform an intricate serverless symphony!</p><p>Connect with me</p><p><a href="https://www.linkedin.com/in/habeebyusuf/"><strong>LinkedIn</strong></a><br><a href="https://x.com/harbeylefty"><strong>X(Twitter)</strong></a><strong><br></strong><a href="http://Harbeysegun@gmail.com/"><strong>Email</strong></a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=ec75b16dfb1f" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Serverless Contact Form with AWS: Lambda, DynamoDB, API Gateway, and SNS]]></title>
            <link>https://medium.com/@harbeylefty/serverless-contact-form-with-aws-lambda-dynamodb-api-gateway-and-sns-221c23843935?source=rss-a9dda9de9043------2</link>
            <guid isPermaLink="false">https://medium.com/p/221c23843935</guid>
            <dc:creator><![CDATA[HarbeyLefty]]></dc:creator>
            <pubDate>Sat, 03 May 2025 11:46:21 GMT</pubDate>
            <atom:updated>2025-05-27T00:27:11.409Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*IUjbrjcsrpQik3AMF1xRtQ.png" /></figure><p><strong>Project Overview</strong></p><p>In this guide, we’ll explore how to create a serverless contact form system that handles and processes user submissions efficiently, with no need for managing servers. It leverages <strong>AWS Lambda</strong>, <strong>API Gateway</strong>, <strong>SNS</strong>, and <strong>DynamoDB</strong> to process data automatically, store it securely, and send real-time notifications.</p><p>User input is captured, saved to DynamoDB, and a notification is sent to the admin via SNS. The entire system is scalable, low-maintenance, and designed with security in mind, using IAM roles to give each service only the permissions it needs.</p><h4>Table of Contents</h4><ol><li>Project Overview</li><li>Highlights</li><li>Prerequisites</li><li>Create DynamoDB Table</li><li>Create SNS topic</li><li>Setting up IAM role and permissions for Lambda</li><li>Deploy Lambda Function</li><li>Set Up API Gateway to trigger lambda function</li><li>Conclusion</li></ol><h4>Highlights</h4><ul><li>Captures form submissions from users</li><li>API Gateway triggers the Lambda function when the form is submitted.</li><li>Processes and stores submission data in DynamoDB using AWS Lambda</li><li>Sends real-time notifications when a new form is submitted using Amazon SNS</li><li>Follows the principle of least privilege using IAM roles for secure access</li></ul><h4>Prerequisites</h4><ul><li>AWS Account</li><li>Basic knowledge of AWS services especially Lambda, API Gateway, and IAM</li><li>Familiarity with Python; used for writing the Lambda function logic</li><li>IAM permissions: Ability to create roles, policies, and resources in AWS</li><li>Understanding of CORS and HTTP basics, helpful for configuring API Gateway</li></ul><p><strong>Project implementation<br></strong>For this project, we have a static frontend hosted on S3 for the contact form. You can copy the code from the index.html file in the repository <a href="https://github.com/Harbeylefty/serverless-contact-form">here</a></p><p><strong>1. Create DynamoDB table</strong><br> We’ll create a DynamoDB table to store the data submitted by users.</p><ul><li>Go to the DynamoDB service in the AWS Management Console.</li><li>Click Create table.</li><li>Enter Table name ( eg ContactFormSubmissions )</li><li>Set the Primary Key - Partition Key: submissionId (string)</li><li>Leave the rest of the settings as default.</li><li>Click Create table</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*19bbLULcRpHuaUnp1Qrp6A.png" /></figure><p><strong>2. Create SNS Topic</strong></p><ul><li>In the AWS Management Console, search for and select SNS</li><li>Click on Create topic under the Topics section and select standard as topic type</li><li>Name the topic.</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*gx9u9ZALnCjjSn_4fZ8xow.png" /></figure><ul><li>Click on create topic to finish</li></ul><p><strong>2.1 Add subscription to the SNS topic</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*LQJSof2Myi-LieHwQ3qs3w.png" /></figure><ul><li>Click on create subscription</li><li>Select your topic ARN</li><li>Choose the protocol and select Email from the menu. For SMS, select SMS</li><li>Enter the Endpoint<strong>, </strong>For Email, enter the email address where notifications should be sent. For SMS, enter the phone number</li><li>Click create subscription to create the subscription</li><li>Confirm the subscription<strong>. </strong>If you selected Email as the protocol, you will receive a confirmation email. Click on the Confirm subscription link in your email to complete the setup</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*uF2hcKbffVYIVTeIuiueCA.png" /></figure><p><strong>3. Setting up IAM role and permissions for Lambda</strong> <br>To allow the lambda function to send data and send notifications, we need to create an IAM role with the right permissions. This role will be assumed by the function during execution.</p><p>First let’s create the Policy for DynamoDB and SNS access.</p><ul><li>Go to the IAM service in the AWS management console</li><li>Click on policies, Create policy, Click on JSON and paste the json policy below into the policy editor</li></ul><pre>{<br>    &quot;Version&quot;: &quot;2012-10-17&quot;,<br>    &quot;Statement&quot;: [<br>        {<br>            &quot;Sid&quot;: &quot;AllowDynamoDBAndSNSAccess&quot;,<br>            &quot;Effect&quot;: &quot;Allow&quot;,<br>            &quot;Action&quot;: [<br>                &quot;sns:Publish&quot;,<br>                &quot;dynamodb:PutItem&quot;<br>            ],<br>            &quot;Resource&quot;: [<br>                &quot;arn:aws:dynamodb:REGION:ACCOUNT_ID:table/ContactFormSubmissions&quot;,<br>                &quot;arn:aws:sns:REGION:ACCOUNT_ID:ContactFormTopic&quot;<br>            ]<br>        }<br>    ]<br>}</pre><ul><li>Click next and give the policy a name (e.g ContactFormPermissions), then click on create Policy to create the policy.</li></ul><p>This policy ensures the function has just the access it needs, nothing more — following the principle of least privilege. It allows the function to upload submitted form data to a specific DynamoDB table and publish notifications to an SNS topic.</p><p><strong>3.1 Create IAM role</strong> <br>- Go to the IAM service in the AWS management console<br>- Click roles, create role. <br>- Select AWS service and choose lambda as use case <br>- Click next and attach the following policies to the role: <br> <strong>AWSLambdaBasicExecutionRole<br> ContactFormPermissions</strong> (The policy we created earlier)<strong><br>-</strong> Click next to review, Give the role a name. <br>- Click create role to create the role.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*pCD5XNbKswoTqeIJ9F61Dw.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*RO3gg8BW_50rYxCpbDrYlw.png" /></figure><p><strong>4. Deploy Lambda Function</strong></p><p>With our permissions in place, the next step is to deploy the Lambda function that handles the form submissions. This function will:<br>- Receive the submitted data,<br>- Store it in DynamoDB,<br>- And send a notification via SNS</p><p>First we create the function</p><ul><li>Navigate to the lambda service in the AWS management console</li><li>Create a function</li><li>Select Author from scratch and give the function a name (eg <strong>contact</strong>-<strong>form-function</strong>) and choose python 3.13 as runtime</li><li>Under the change execution role, click on use existing role and attach the role we created earlier to the function</li><li>Click create function to create the function.</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*z8PrDeBuoeGoRNbuFDqsag.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*NnDpcatPgz6IxgRLSN-b9Q.png" /></figure><p><strong>4.1 Lambda Function Code<br></strong>To handle the form submissions, below is the code for the Lambda function that processes the contact form data</p><pre>import json<br>import boto3<br>import logging<br>import time<br><br>logging.getLogger().setLevel(logging.INFO)<br>sns_client = boto3.client(&#39;sns&#39;)<br>dynamodb = boto3.resource(&#39;dynamodb&#39;)<br><br>def lambda_handler(event, context):<br>    headers = {<br>        &#39;Content-Type&#39;: &#39;application/json&#39;,<br>        &#39;Access-Control-Allow-Origin&#39;: &#39;*&#39;,<br>        &#39;Access-Control-Allow-Headers&#39;: &#39;Content-Type,X-Amz-Date,Authorization,X-Api-Key&#39;,<br>        &#39;Access-Control-Allow-Methods&#39;: &#39;POST,OPTIONS&#39;<br>    }<br>    <br>    try:<br>        body = json.loads(event.get(&#39;body&#39;, &#39;{}&#39;))<br>        name = body.get(&#39;name&#39;)<br>        email = body.get(&#39;email&#39;)<br>        message = body.get(&#39;message&#39;)<br><br>        if not all([name, email, message]):<br>            return {<br>                &#39;statusCode&#39;: 400,<br>                &#39;headers&#39;: headers,<br>                &#39;body&#39;: json.dumps({&#39;message&#39;: &#39;Missing required fields&#39;})<br>            }<br><br>        # Save to DynamoDB<br>        table = dynamodb.Table(&#39;ContactFormSubmissions&#39;)<br>        timestamp = time.strftime(&#39;%Y%m%d-%H%M%S&#39;)<br>        submission_id = f&quot;{timestamp}-{email}&quot;<br>        table.put_item(<br>            Item={<br>                &#39;submissionId&#39;: submission_id,<br>                &#39;name&#39;: name,<br>                &#39;email&#39;: email,<br>                &#39;message&#39;: message,<br>                &#39;timestamp&#39;: timestamp<br>            }<br>        )<br>        logging.info(f&quot;Saved submission to DynamoDB: {submission_id}&quot;)<br><br>        # Publish to SNS<br>        topic_arn = &#39;arn:aws:sns:REGION:ACCOUNT_ID:ContactFormTopic&#39;<br>        message_body = f&quot;&quot;&quot;<br>        New contact form submission:<br><br>        Name: {name}<br>        Email: {email}<br>        Message: {message}<br>        &quot;&quot;&quot;<br>        response = sns_client.publish(<br>            TopicArn=topic_arn,<br>            Message=message_body,<br>            Subject=&#39;New Contact Form Submission&#39;<br>        )<br>        logging.info(f&quot;SNS Publish Response: {response}&quot;)<br><br>        return {<br>            &#39;statusCode&#39;: 200,<br>            &#39;headers&#39;: headers,<br>            &#39;body&#39;: json.dumps({&#39;message&#39;: &#39;Form submitted successfully!&#39;})<br>        }<br><br>    except json.JSONDecodeError:<br>        logging.error(&quot;Invalid JSON in request body&quot;)<br>        return {<br>            &#39;statusCode&#39;: 400,<br>            &#39;headers&#39;: headers,<br>            &#39;body&#39;: json.dumps({&#39;message&#39;: &#39;Invalid JSON format&#39;})<br>        }<br>    except Exception as e:<br>        logging.error(f&quot;Error: {str(e)}&quot;)<br>        return {<br>            &#39;statusCode&#39;: 500,<br>            &#39;headers&#39;: headers,<br>            &#39;body&#39;: json.dumps({&#39;message&#39;: &#39;Error processing form&#39;})<br>        }</pre><p>This Lambda function processes the contact form submissions by:</p><ol><li>Validating the input (ensuring all required fields are present).</li><li>Storing the submission data securely in a DynamoDB table</li><li>Sending an email notification via SNS when a new form is submitted</li><li>Returning a success or error response to the frontend ensuring smooth communications with the user interface.</li></ol><ul><li>Copy and Paste the code into the code editor of the function and click on Deploy to save. Ensure you replace the DynamoDB table name and SNS topic arn with yours.</li><li>Click on the test tab to test the function, give the test a name. In the test editor paste this into the editor and save.</li></ul><pre>{<br>  &quot;body&quot;: &quot;{\&quot;name\&quot;:\&quot;John Wick\&quot;,\&quot;email\&quot;:\&quot;john.wick@example.com\&quot;,\&quot;message\&quot;:\&quot;This is a sample message!\&quot;}&quot;<br>}</pre><ul><li>Click on test to see the outcome. You should see a status code 200 with the message “Form submitted successfully” and from the logs you should see that submissions has been saved to DynamoDB and notifications has been sent.</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*XtFWR-josSdMCDXElxp7YQ.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*YkG_uA9ty3JxWgDWDYrp-A.png" /></figure><ul><li>Check your Mail to verify that notifications is being sent. You should see a mail with the user’s details and message</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Sp7X9eM2ersQ8LMl14xsWw.png" /></figure><ul><li>Click on Explore items to check the DynamoDB table we created for the data submission and confirm that users data are being stored.</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*5N8QVaIsA3-LcNbHcCj67g.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*oSWKZ5XpQAvu2icTmVNfYA.png" /></figure><p><strong>5. Set Up API Gateway to trigger lambda function <br></strong>To expose the Lambda function to the internet and allow the frontend to trigger it, we need to set up API Gateway.</p><ul><li>Go to the API Gateway service in the AWS management console</li><li>Click on create API, choose REST API and click on build</li><li>Select New API, Give the API a name</li><li>Select regional for API endpoint type and click on create API</li><li>In the next page, click on create resource</li><li>Give the resource a name and tick the CORS option (This will be the URL path eg api-url/contactform.</li><li>Create resource</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*LQ6nVAaZvUY4mpvK-_JvAw.png" /></figure><ul><li>Select the new resource and click Create Method.</li><li>Select POST for Method type (since the form submits data via POST), and Lambda function for integration type</li><li>In the Lambda function field, select the Lambda function you deployed earlier and check the box for Lambda proxy integration.</li><li>Click Create method to create POST method</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*MIdD4E2R0w5d_9hbYvZxXg.png" /></figure><ul><li>Click on Deploy API and select new stage, then give the stage a name (e.g test) and click on Deploy</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*EcB8_QLo-VqRiubNr208UQ.png" /></figure><ul><li>Once the API is deployed, you’ll see an Invoke URL in the stages section</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*0VbqevDPYQathfGnT7z90w.png" /></figure><ul><li>You can test the form by manually submitting data to this URL using a tool like <strong>Postman</strong> or <strong>curl</strong>, or update your frontend to submit the form to this API Gateway URL.</li></ul><p>Let’s test the URL using Postman before connecting the frontend to the API Gateway.</p><ul><li>Open Postman and click on New to create a new request.</li><li>Select HTTP, and set the HTTP method to POST</li><li>In the URL field, enter the Invoke URL from API Gateway, and append /contactform to it. (since that is the name of the resource we created) It should look like this ( <a href="https://2vk6zqwyv1.execute-api.us-east-1.amazonaws.com/test">https://2vkhghr.executee-api.us-east-1.amazonaws.com/test</a>/contactform)</li><li>Click on the Body tab in Postman, Select raw and then choose JSON from the menu on the right.</li><li>In the body section, input the JSON data you want to send with the form</li></ul><pre>{<br>  &quot;name&quot;: &quot;Joshua Jackson &quot;,<br>  &quot;email&quot;: &quot;joshuajackson@example.com&quot;,<br>  &quot;message&quot;: &quot;This is a test message!&quot;<br>}</pre><ul><li>Go to the headers tab to ensure the Content-Type is set to application/json. Postman should automatically handle this, but it’s good to verify.</li><li>Click the Send button.</li><li>Postman will send the request to your API Gateway, which will trigger your Lambda function and send the email via SNS.</li><li>You should see a “message: form submitted successfully”</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*UFzXDKWmfZv3se2OL4HdKw.png" /></figure><p>So everything is working as expected, Next step is to connect the frontend to the API Gateway. Go to the html file in the repository and replace the API URL with your own API URL</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*kJbiL1YBaB2ZCuxAl5ulNg.png" /></figure><ul><li>Then re-upload this file to the s3 bucket hosting your static website and when you open the website, fill in the necessary details and you should see a “message sent successfully”</li><li>Ensure to test all functionalities, leave a space empty and you should get a “Please fill out this field” message shown on the screen.</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*a5IcVEhFR9ZMgPm6n8OrkA.png" /></figure><ul><li>Check your Email and you should get notifications from SNS about the form submission.</li></ul><p><strong>Conclusion <br></strong>We’ve built a Serverless contact form that’s efficient, secure, and easy to maintain. Using API Gateway, Lambda, SNS, and DynamoDB, the form collects user submissions, stores the data safely, and sends out notifications in real time.<br>Everything runs without the need to manage any servers, keeping operations simple and costs low. By setting up IAM role, we also made sure each part of the system only has the permissions it needs.<br>This project shows how you can use AWS’s Serverless tools to build flexible and dependable systems without the need to manage servers.</p><p><strong>Next steps to Consider</strong></p><ul><li><strong>Expand Form Features</strong>: Add more fields to the contact form (e.g., phone number, file attachments).</li><li><strong>Integrate AWS S3</strong>: Allow users to upload files with their submission and store the files securely in an S3 bucket.</li><li><strong>Use AWS SES</strong>: Send confirmation emails to users after successful submission using Amazon Simple Email Service (SES).</li></ul><p>Thank you for following along. I hope this article proves valuable for your Serverless projects!</p><p>Connect with me on</p><p><a href="http://www.linkedin.com/in/habeebyusuf">LinkedIn</a><br><a href="https://x.com/harbeylefty">X(Twitter)<br></a><a href="http://Harbeysegun@gmail.com/">Email</a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=221c23843935" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[HNG Stage Zero: Setting Up a simple web server through NGINX on an AWS EC2 Instance]]></title>
            <link>https://medium.com/@harbeylefty/hng-stage-zero-setting-up-a-simple-web-server-through-nginx-on-an-aws-ec2-instance-7ce7f454e728?source=rss-a9dda9de9043------2</link>
            <guid isPermaLink="false">https://medium.com/p/7ce7f454e728</guid>
            <dc:creator><![CDATA[HarbeyLefty]]></dc:creator>
            <pubDate>Fri, 31 Jan 2025 09:02:41 GMT</pubDate>
            <atom:updated>2025-01-31T15:07:52.110Z</atom:updated>
            <content:encoded><![CDATA[<p><strong>Introduction</strong></p><p>As part of the HNG Stage 0 task, I was tasked with setting up and configuring NGINX on a fresh Ubuntu server on AWS. This task is a great introduction to working with web server configurations, as it involves launching an EC2 instance, installing and configuring NGINX, and serving a custom HTML page. Here, I’ll walk through the steps I took to complete this task.</p><p><strong>Pre-requisites:</strong><br>1. AWS Account. <br>2. Basic knowledge of using AWS Services.</p><p><strong>Step 1: Create an Ec2 Instance</strong></p><p>Login to the AWS Management console, Navigate to the search bar in the AWS console, and search for EC2, this will take you to the EC2 dashboard.<br>Click on instances, and you would see a “Launch instances” icon, click on this to start the creation of your EC2 instance.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*zCTlLniNVYMju429QUTb5A.png" /></figure><p>On the next page, you would be asked to fill in details for your EC2 instance, fill in the name of your server and select Ubuntu as the preferred Server to use for this task.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*WkNDlnSRdzWqfFiAKwT2rw.png" /></figure><p>On the next page, you would be asked to select instance type, and also select or create a new key pair if you do not have one existing. For me i already have one existing so i will just go with that, you can create a new keypair also. The keypair would be downloaded to a folder in your local Machine and this is the folder where we would navigate to ssh into our ec2 instance. For instance type, select t2.micro.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*TFiRaYJh6-zb57CSf8NL5g.png" /></figure><p>For network settings, Leave the default vpc as it is, click on create a new security group, tick the HTTP and SSH options, this allows your instance to be accessible remotely through SSH and HTTP(Internet)</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*ZwusvCc6jUn5fhRm3URVsw.png" /></figure><p>Leave the other settings as they are, and click on Launch instance by the bottom right corner.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*RY6xvExmA6OYYYqIljsGbA.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*0vSXj3Ac7lSrzg4ABNZWXQ.png" /></figure><p>You can see from the above image that our instance has been created and is in a running state.</p><p><strong>Step 2: SSH Into the EC2 Instance</strong></p><p>Click on your newly created Instance, you would see a connect icon at the top, click the icon and this will take to the page where you would see instructions on how to connect to your instance through SSH.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*MS9V6QHaVuTGQU3I33KYuQ.png" /></figure><p>Copy the comand shown at the bottom, open your terminal and navigate to the folder where you have your keypair downloaded and paste the command you copied. it will ask you for a prompt and you type “yes”</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*h01TbTHNvdrb9kJpeP7GpA.png" /></figure><p>As seen above, i have successfully ssh into my Ec2 Instance and now we can go ahead to configure and install Nginx on our server.</p><p><strong>Step 3: Configure and Install Nginx.</strong></p><p>Now that we are inside our instance, run the following commands to setup and install Nginx.</p><pre>sudo apt update<br>sudo apt install nginx -y<br>sudo systemctl start nginx<br>sudo systemctl status nginx</pre><p>You should see a running status</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*6fPiyCTMqnB-IpWkPBGG1Q.png" /></figure><p><strong>Step 4: Configure Nginx to Serve a custom HTML page</strong></p><p>Navigate to the default html directory and create an index.html file,</p><pre>cd /var/www/html<br>sudo vim index.html</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*GVJxOvyy5H8UFA9ByZ7X1w.png" /></figure><p>A Blank Vim editor page opens, copy and paste the code below. Replace [Your Name] and [SlackName] with your actual name and Slack username.</p><pre>&lt;html&gt;<br>&lt;head&gt;&lt;title&gt;Welcome&lt;/title&gt;&lt;/head&gt;<br>&lt;body&gt;<br>    &lt;h1&gt;Welcome to DevOps Stage 0 - [Your Name]/[SlackName]&lt;/h1&gt;<br>&lt;/body&gt;<br>&lt;/html&gt;</pre><p>Exit and Save this file. <br>Now to confirm that NGINX is accessible, copy the public ip of your instance and paste in your browser, The result should be something like this:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*EKGwB4tuf4Jd-FSUcF_b-g.png" /></figure><p><strong>Conclusion</strong></p><p>We have walked through the process of configuring a simple webpage using NGINX on an AWS Ubuntu server. This project helped to refresh my memory and reinforce my understanding of basic web server configuration, And i faced no real challenge coming up with this task. Looking forward to the next stage and more tasks.</p><p>Looking to Hire DevOps Engineers or Platform Engineers? HNG has the best <a href="https://hng.tech/hire/devops-engineers">DevOps Engineers,</a> and <a href="https://hng.tech/hire/platform-engineers">Platform Engineers</a> available for hire and freelance jobs anywhere.</p><p>Connect with me on <br><a href="https://www.linkedin.com/in/habeeb-yusuf-265254252/">LinkedIn</a></p><p><a href="https://x.com/harbeylefty">Twitter</a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=7ce7f454e728" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Deploying and Monitoring a Full-Stack Application with Docker, Prometheus, Grafana, and Nginx Proxy…]]></title>
            <link>https://medium.com/@harbeylefty/deploying-and-monitoring-a-full-stack-application-with-docker-prometheus-grafana-and-nginx-proxy-3e945d9b437a?source=rss-a9dda9de9043------2</link>
            <guid isPermaLink="false">https://medium.com/p/3e945d9b437a</guid>
            <dc:creator><![CDATA[HarbeyLefty]]></dc:creator>
            <pubDate>Tue, 03 Dec 2024 00:45:28 GMT</pubDate>
            <atom:updated>2025-04-11T15:55:41.840Z</atom:updated>
            <content:encoded><![CDATA[<h3><strong>Deploying and Monitoring a Full-Stack Application with Docker, Prometheus, Grafana, and Nginx Proxy Manager on a Custom Domain.</strong></h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*i5LpMjWbMthizSy9QVSKzQ.png" /></figure><h4>Introduction</h4><p>When developers build full-stack applications and deploy them to production, they often face challenges related to scaling, performance, and observability. As traffic grows, the need for efficient infrastructure becomes crucial to maintain the reliability and responsiveness of the application. To address these challenges, containerization with Docker has become a go-to solution for managing and scaling applications across environments.</p><p>In this article, we will walk through the process of containerizing and deploying a full-stack application Using Docker Compose and also setting up Monitoring and Logging tools to monitor the application’s perfomance in real time.</p><h4><strong>Requirements</strong></h4><p>Before we start this project, you need to have the following:</p><ul><li>Knowledge of how to set up EC2 Instance with the Necessary Security groups</li><li>Basic Knowledge of Docker</li><li>Basic Knowledge of Prometheus and Grafana</li><li>Visual Studio Code (VsCode) and knowledge of how to connect Vscode to your EC2 instance</li></ul><h4><strong>Project Overview:</strong></h4><p>This Project involves containerizing and deploying a full-stack application consisting of a FastAPI backend and a ReactJS frontend. We will deploy the application on AWS EC2, using Docker for containerization and Nginx Reverse Proxy Manager to efficiently manage traffic between the frontend and backend.</p><p>Furthermore, we will explore how to configure a custom domain, enhance security with SSL certificates from Let’s Encrypt, and integrate observability tools like Prometheus and Grafana to monitor the application’s performance in real time. In Summary:</p><ol><li>Containerize and deploy a full-stack FastAPI and ReactJS application.</li><li>Set up traffic management with Nginx Reverse Proxy Manager.</li><li>Implement a custom domain and SSL certificates for secure access.</li><li>Monitor the application using Prometheus, Grafana, and Loki.</li></ol><h4>Project Implementation</h4><p>Before Starting the Project, spin up an ec2 instance and configure with the neccesary security groups, and ssh into your instance.</p><p>Next, clone the project repository, and navigate to the directory. Also for easy access you can connect your VsCode to your Ec2 instance.</p><pre>git clone https://github.com/Harbeylefty/Deploying-and-Monitoring-Fullstack-Application.git<br>cd Deploying-and-Monitoring-Fullstack-Application</pre><p>Next let’s start to dockerize our application, navigate to the backend directory and create a Dockerfile for the fastAPI, and paste this configuration into the Dockerfile:</p><pre>FROM python:3.11-slim<br><br># Set the working directory to /app <br>WORKDIR /app <br><br># Update package list and install neccesary build dependencies <br>RUN apt-get update -y &amp;&amp; apt-get install -y --no-install-recommends \<br>    curl \<br>    bash \<br>    &amp;&amp; rm -rf /var/lib/apt/lists/*<br><br># Install Poetry<br>RUN curl -sSL https://install.python-poetry.org | python3 -<br><br># Add poetry to PATH <br>ENV PATH=&quot;/root/.local/bin:$PATH&quot;<br><br># Copy poetry config files to take advantage of Docker Cache<br>COPY pyproject.toml poetry.lock ./<br>    <br># Copy the rest of the source code into the container <br>COPY . . <br><br># Install dependencies <br>RUN poetry install <br><br># Make the prestart.sh script executable <br>RUN chmod +x ./prestart.sh<br><br># Expose the port the application runs on <br>EXPOSE 8000<br><br># Command to run the executable and start the server<br>CMD [&quot;sh&quot;, &quot;-c&quot;, &quot;poetry run bash ./prestart.sh &amp;&amp; poetry run uvicorn app.main:app --host 0.0.0.0 --port 8000 --reload&quot;]</pre><p>Navigate to the frontend directory, create a Dockerfile and paste this into the Dockerfile:</p><pre>FROM node:22-alpine3.18<br><br>WORKDIR /app<br><br>COPY package*.json ./<br><br>RUN npm install <br><br>COPY . .<br><br>EXPOSE 5173<br><br>CMD [&quot;npm&quot;, &quot;run&quot;, &quot;dev&quot;, &quot;--&quot;, &quot;--host&quot;]</pre><p>Next we’ll use Docker Compose to Containerize our application and run all the services simultaneously. <br>create a compose.yml file in the root directory and add this:</p><pre>include:<br>  - compose.monitoring.yml<br><br>services: <br>  backend:<br>    build:<br>      context: ./backend <br>      dockerfile: Dockerfile <br>    container_name: backend_fast_api<br>    ports: <br>      - 8000:8000<br>    env_file:<br>      - backend/.env <br>    networks:<br>      - backend-network <br>      - frontend-network <br><br>    depends_on:<br>      - db <br>  <br>  frontend: <br>    build: <br>      context: ./frontend <br>      dockerfile: Dockerfile <br>    container_name: react-frontend <br>    ports: <br>      - 5173:5173 <br>    env_file: <br>      - frontend/.env <br>    depends_on: <br>      - backend <br>    networks:<br>      - frontend-network<br>  <br>  db: <br>    image: postgres:13<br>    environment: <br>      POSTGRES_USER: app<br>      POSTGRES_PASSWORD: changethis123<br>      POSTGRES_DB: app <br>    ports:<br>      - 5432:5432 <br>    volumes: <br>      - postgres_data:/var/lib/postgresql/postgres_data <br>    networks: <br>      - backend-network <br>  <br>  adminer: <br>    image: adminer<br>    restart: always <br>    environment:<br>      ADMINER_DEFAULT_SERVER: db <br>    ports:<br>      - 8080:8080<br>    networks:<br>      - backend-network  <br>  <br>  proxy_manager:<br>    image: jc21/nginx-proxy-manager:2.10.4<br>    restart: always<br>    ports: <br>      - 80:80<br>      - 443:443<br>      - 81:81<br>    environment:<br>      DB_SQLITE_FILE: &quot;/data/database.sqlite&quot;<br>    volumes:<br>      - data:/data<br>      - letsencrypt:/etc/letsencrypt<br>       #- ./Nginx/nginx.conf:/data/nginx/custom/http_top.conf<br>    depends_on:<br>      - frontend<br>      - backend <br>      - db <br>      - adminer <br>    networks: <br>      - frontend-network<br>      - backend-network <br><br>networks:<br>  frontend-network:<br>  backend-network:<br>    <br>volumes:<br>  postgres_data: <br>  data:<br>  letsencrypt: </pre><ul><li>The setup consists of the <strong>frontend (React)</strong>, <strong>backend (FastAPI)</strong>, <strong>PostgreSQL database</strong>, <strong>Adminer</strong> for managing the database, and <strong>Nginx Proxy Manager</strong> for routing traffic to the containers.</li><li>The frontend communicates with the backend, which in turn connects to the database.</li><li>The Nginx proxy manager handles incoming web traffic and routes it to the appropriate service (frontend, backend, etc.).</li><li>Adminer is provided as a web interface for easy management of the PostgreSQL database.</li><li>Also the commented “nginx.conf” will be used later to configure routing of traffic to the containers when we set up our domain.</li></ul><h3>Deploying the Monitoring Stack</h3><p>In this section step we will set up the monitoring stack using Prometheus, Grafana, and Loki. We will then define these services in a second Docker Compose file while running both compose files with a single <strong>docker-compose up command</strong>. These tools will help us collect metrics, visualize them, and aggregate logs for easy troubleshooting.</p><p>First let us set up the configurations for this monitoring tools: <br>Create a Folder named Monitoring, Inside it create a file named prometheus.yml and add this configuration:</p><pre>global:<br>  scrape_interval: 15s <br>  evaluation_interval: 15s <br>  <br>scrape_configs:<br>  - job_name: &quot;prometheus&quot;<br>    metrics_path: &quot;/prometheus/metrics&quot;<br>    static_configs:<br>      - targets: [&quot;prometheus:9090&quot;]<br><br>  - job_name: &quot;cadvisor&quot;<br>    static_configs:<br>      - targets: [&quot;cadvisor:8080&quot;]</pre><p>This configuration file defines how Prometheus scrapes metrics at regular intervals, specifying target endpoints for collecting metrics from both Prometheus and cAdvisor for monitoring purposes.</p><p>Next, create a promtail-config.yml file and add this:</p><pre>server:<br>  http_listen_port: 9080<br>  grpc_listen_port: 0<br><br>positions:<br>  filename: /tmp/positions.yaml<br><br>clients:<br>  - url: http://loki:3100/loki/api/v1/push<br><br>scrape_configs:<br>  - job_name: system<br>    static_configs:<br>      - targets:<br>          - &quot;172.31.18.250&quot;<br>        labels:<br>          job: varlogs<br>          __path__: /var/log/*log<br>        labels:<br>          job: docker_logs<br>          __path__: /var/lib/docker/containers/*/*-json.log</pre><p>This file configures the server settings for log scraping, defining how logs are collected from system and Docker sources and sent to Loki for storage and analysis.</p><p>Next create a Loki-config.yml file:</p><pre>auth_enabled: false<br><br>server:<br>  http_listen_port: 3100<br>  grpc_listen_port: 9096<br><br>common:<br>  instance_addr: 127.0.0.1<br>  path_prefix: /tmp/loki<br>  storage:<br>    filesystem:<br>      chunks_directory: /tmp/loki/chunks<br>      rules_directory: /tmp/loki/rules<br>  replication_factor: 1<br>  ring:<br>    kvstore:<br>      store: inmemory<br><br>query_range:<br>  results_cache:<br>    cache:<br>      embedded_cache:<br>        enabled: true<br>        max_size_mb: 100<br><br>schema_config:<br>  configs:<br>    - from: 2020-10-24<br>      store: tsdb<br>      object_store: filesystem<br>      schema: v13<br>      index:<br>        prefix: index_<br>        period: 24h<br></pre><p>This file sets up Loki’s log collection, storage, and caching processes, ensuring logs are effectively managed and easy to access for analysis.</p><p>Now let’s create our compose.monitoring.yml file for containerization of the monitoring services:</p><pre>services: <br>  prometheus:<br>    image: prom/prometheus:latest<br>    volumes: <br>      - ./monitoring/prometheus.yml:/etc/prometheus/prometheus.yml <br>    ports:<br>      - 9090:9090<br>    command: <br>      - &quot;--config.file=/etc/prometheus/prometheus.yml&quot;<br>      - &quot;--web.external-url=/prometheus&quot;<br>    networks:<br>      - frontend-network <br>    depends_on: <br>      - cadvisor <br>  <br>  loki:<br>    image: grafana/loki:latest <br>    ports: <br>      - 3100:3100 <br>    command: <br>      - --config.file=/etc/loki/loki-config.yml<br>    volumes: <br>      - ./monitoring/loki-config.yml:/etc/loki/loki-config.yml <br>    networks:<br>      - frontend-network<br>  <br>  promtail:<br>    image: grafana/promtail:latest <br>    networks:<br>      - frontend-network<br>      - backend-network <br>    volumes: <br>      - ./monitoring/promtail-config.yml:/etc/promtail/promtail-config.yml<br>      - /var/log:/var/log <br>    depends_on: <br>      - loki <br>  <br>  grafana:<br>    image: grafana/grafana:latest <br>    ports: <br>      - 3000:3000<br>    volumes:<br>      - grafana_storage:/var/lib/grafana<br>    networks:<br>      - frontend-network <br>    environment:<br>      - GF_SERVER_ROOT_URL=http://54.159.253.249:3000/grafana<br>      - GF_SERVER_SERVE_FROM_SUB_PATH=true<br>    depends_on:<br>      - loki <br>  <br>  cadvisor:<br>    image:  gcr.io/cadvisor/cadvisor:v0.47.0<br>    ports:<br>      - 8081:8080<br>    volumes:<br>      - /:/rootfs:ro<br>      - /var/run:/var/run:ro<br>      - /sys:/sys:ro<br>      - /var/lib/docker/:/var/lib/docker:ro<br>    networks:<br>      - frontend-network<br>      - backend-network <br><br>volumes:<br>  grafana_storage: </pre><p>This Docker Compose configuration sets up a monitoring and logging stack with Prometheus, Grafana, Loki, and other components, each serving distinct functions in the observability stack.<br>Prometheus,Loki, and promtail will use their respective configurations from the monitoring folder.</p><p>Next, let’s set up Docker to send container logs directly to Loki by installing the Loki log driver:</p><pre>docker plugin install grafana/loki-docker-driver:2.9.1 --alias loki --grant-all-permissions</pre><p>Now, let’s edit the <strong>/etc/docker/daemon.json</strong> file with the configuration provided below. This will enable Docker to use the Loki log driver for capturing and sending container logs:</p><pre>{<br>     &quot;log-driver&quot;: &quot;loki&quot;,<br>     &quot;log-opts&quot;: {<br>       &quot;loki-url&quot;: &quot;http://localhost:3100/loki/api/v1/push&quot;,<br>       &quot;loki-batch-size&quot;: &quot;400&quot;<br>     }<br>   }</pre><p><strong>Configure Env Variables</strong></p><p>Before testing our deployment, let us configure the environment variables so that our application can run smoothly as intended. <br>In the backend directory, open the .env file and replace localhost with your instance ip address, it shouldlook like this:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*SjsbhmdlxI0ZxGffSXvs2Q.png" /></figure><p>Navigate to your frontend directory and also change the localhost to your ip address in your .<strong>env</strong> file. This will ensure the frontend connects successfully to the backend.</p><p>Also Ensure that the database credentials in the <strong>backend .env</strong> file match the PostgreSQL credentials to ensure proper connection. You can access the database through adminer with these credentials.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*-12-dKFsvHA8it4gyvxy8Q.png" /></figure><p><strong>Testing Deployment</strong></p><p>Let us test our deployment to see if everything goes well, and also verify that the containers are all running:</p><pre>docker compose up -d<br>docker ps -a </pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*XegHlW3wt7wHDXTUCg_UMQ.png" /></figure><p>As we can see from the image above, all our containers are running as they should. If any one doesn’t run as it should, you can check what the issue is with the command:</p><pre>docker logs container_id</pre><p>Test if all the applications are accessible from your web browser and also check if you can login successfully.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*JrKKXfyzBbAG-_-fC67FHw.png" /><figcaption>Frontend is running succesfully</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*9GV3CpgMHyc0WldBgDjwFw.png" /><figcaption>Postgresql database.</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*dBZresnXjp-jk6pgkT6J9Q.png" /><figcaption>Grafana</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*fmIN_OdJyUiAvhknG7TmFA.png" /><figcaption>Prometheus</figcaption></figure><p>As we can see from the images above, we were able to access our react frontend with the login credentials, also we were able to access the postgreSQL database through adminer, and also prometheus and Grafana were accessed through our browser as well.</p><h3>Deploying with a custom domain</h3><p>In this section, we will set up a domain which we will use to deploy our Application, we will configure Nginx Proxy Manager and generate SSL certificates to securely route traffic from our domain to the containers, we will also configure a redirection from www to non-www, and http to https redirection. <br>You can get free subdomains from <a href="https://freedns.afraid.org/">AfraidDNS</a>. Sign up and click on subdomain to create subdomains, also make sure the domain is pointing to your ip address.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*odIu2aTCj7if5RNohq3NBw.png" /></figure><h4>Generating SSL Certificates</h4><p>Now Let us generate ssl certificates for our custom domains to secure our custom domain and establish a secure connection. You can generate SSL certificates by accessing the Nginx Proxy Manager UI with your “<strong>ip_address:81”. admin@example.com </strong>as default username and<strong> changeme </strong>as password<strong>.</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*V4GbNGR-5E2FrDsdIjp4ow.png" /><figcaption>SSL Certificates</figcaption></figure><p>Generate SSL certificates for each of your domains by entering your domain, add a valid email address and save.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*QHIA3rMmIOEcXBnnQgf1QQ.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*TdO29ut2Cg7ni8SptrnRXQ.png" /></figure><p>After generating the SSL ceretificates for each of your domain, check the logs for your Nginx Proxy Manager container and take note of the paths to the SSL certificates for each of the domain. We will use this paths in the nginx.conf file.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*WDstrKSDA6977opGJk18ZQ.png" /></figure><h4>Configure Nginx Proxy Manager</h4><p>Now that we have generated SSL certicates for our domain, next step is to configure Nginx Proxy Manager to route traffic from our domains to the containers. This will ensure we are able to access the services through our domains. <br>Now let’s bring our containers down:</p><pre>docker compose down</pre><p>Create a folder named Nginx in the root directory, and create a file nginx.conf inside the folder. Copy and paste this in the file:</p><pre><br># Configuration for harbeyy.mooo.com<br>server {<br>    listen 80;<br>    server_name harbeyy.mooo.com www.harbeyy.mooo.com;<br><br>    # Redirect HTTP to HTTPS<br>    location / {<br>        if ($host = &#39;www.harbeyy.mooo.com&#39;) {<br>            return 301 https://harbeyy.mooo.com$request_uri;<br>        }<br><br>        return 301 https://$host$request_uri;<br>    }<br>}<br><br>server {<br>    listen 443 ssl;<br>    server_name harbeyy.mooo.com www.harbeyy.mooo.com;<br><br>    # Redirect www to non-www<br>    if ($host = &#39;www.harbeyy.mooo.com&#39;) {<br>        return 301 https://harbeyy.mooo.com$request_uri;<br>    }<br><br>    # SSL certificate for harbeyy.mooo.com<br>    ssl_certificate /etc/letsencrypt/live/npm-1/fullchain.pem;<br>    ssl_certificate_key  /etc/letsencrypt/live/npm-1/privkey.pem;<br><br>    # Proxy pass settings for services<br>    location /api {<br>        proxy_pass http://backend:8000/api;  <br>        proxy_set_header Host $host;<br>        proxy_set_header X-Real-IP $remote_addr;<br>        proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;<br>        proxy_set_header X-Forwarded-Proto $scheme;<br>    }<br><br>    location /docs {<br>        proxy_pass http://backend:8000/docs; <br>        proxy_set_header Host $host;<br>        proxy_set_header X-Real-IP $remote_addr;<br>        proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;<br>        proxy_set_header X-Forwarded-Proto $scheme;<br>    }<br><br>    location /redoc {<br>        proxy_pass http://backend:8000/redoc; <br>        proxy_set_header Host $host;<br>        proxy_set_header X-Real-IP $remote_addr;<br>        proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;<br>        proxy_set_header X-Forwarded-Proto $scheme;<br>    }<br><br>    location /prometheus {<br>        proxy_pass http://prometheus:9090; <br>        proxy_set_header Host $host;<br>        proxy_set_header X-Real-IP $remote_addr;<br>        proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;<br>        proxy_set_header X-Forwarded-Proto $scheme;<br>    }<br><br>    location /grafana {<br>    proxy_pass http://grafana:3000;<br>    proxy_set_header Host $http_host;<br>    proxy_set_header X-Real-IP $remote_addr;<br>    proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;<br>    proxy_set_header X-Forwarded-Proto $scheme;<br>}<br><br><br>    location / {<br>        proxy_pass http://frontend:5173; <br>        proxy_set_header Host $host;<br>        proxy_set_header X-Real-IP $remote_addr;<br>        proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;<br>        proxy_set_header X-Forwarded-Proto $scheme;<br>    }<br><br><br>}<br><br># Configuration for db.harbeyy.mooo.com<br>server {<br>    listen 80;<br>    server_name db.harbeyy.mooo.com www.db.harbeyy.mooo.com;<br><br>    # Redirect HTTP to HTTPS<br>    location / {<br>        if ($host = &#39;www.db.harbeyy.mooo.com&#39;) {<br>            return 301 https://db.harbeyy.mooo.com$request_uri;<br>        }<br><br>        return 301 https://$host$request_uri;<br>    }<br>}<br><br>server {<br>    listen 443 ssl;<br>    server_name db.harbeyy.mooo.com www.db.harbeyy.mooo.com;<br><br>    # Redirect www to non-www<br>    if ($host = &#39;www.db.harbeyy.mooo.com&#39;) {<br>        return 301 https://db.harbeyy.mooo.com$request_uri;<br>    }<br><br>    # SSL certificate for db.harbeyy.mooo.com<br>    ssl_certificate /etc/letsencrypt/live/npm-2/fullchain.pem;<br>    ssl_certificate_key /etc/letsencrypt/live/npm-2/privkey.pem;<br><br>    location / {<br>        proxy_pass http://adminer:8080;  <br>        proxy_set_header Host $host;<br>        proxy_set_header X-Real-IP $remote_addr;<br>        proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;<br>        proxy_set_header X-Forwarded-Proto $scheme;<br>    }<br>}<br><br># Configuration for proxy.harbeyy.mooo.com<br>server {<br>    listen 80;<br>    server_name proxy.harbeyy.mooo.com www.proxy.harbeyy.mooo.com;<br><br>    # Redirect HTTP to HTTPS<br>    location / {<br>        if ($host = &#39;www.proxy.harbeyy.mooo.com&#39;) {<br>            return 301 https://proxy.harbeyy.mooo.com$request_uri;<br>        }<br><br>        return 301 https://$host$request_uri;<br>    }<br>}<br><br>server {<br>    listen 443 ssl;<br>    server_name proxy.harbeyy.mooo.com www.proxy.harbeyy.mooo.com;<br><br>    # Redirect www to non-www<br>    if ($host = &#39;www.proxy.harbeyy.mooo.com&#39;) {<br>        return 301 https://proxy.harbeyy.mooo.com$request_uri;<br>    }<br><br>    # SSL certificate for proxy.harbeyy.mooo.com<br>    ssl_certificate /etc/letsencrypt/live/npm-3/fullchain.pem;<br>    ssl_certificate_key   /etc/letsencrypt/live/npm-3/privkey.pem;<br><br>    location / {<br>        proxy_pass http://proxy_manager:81;  <br>        proxy_set_header Host $host;<br>        proxy_set_header X-Real-IP $remote_addr;<br>        proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;<br>        proxy_set_header X-Forwarded-Proto $scheme;<br>    }<br>}</pre><p>The above Nginx configuration handles the routing and redirection for our subdomains (harbeyy.mooo.com, db.harbeyy.mooo.com, and proxyy.harbey.mooo.com), implementing HTTP to HTTPS redirection, non-www to www redirection, and proxying requests to each of our services. <br>Also ensure that each of the SSL paths you used matches the one displayed when you checked the logs.</p><p>Now update your compose.yml file:</p><pre>include:<br>  - compose.monitoring.yml<br><br>services: <br>  backend:<br>    build:<br>      context: ./backend <br>      dockerfile: Dockerfile <br>    container_name: backend_fast_api<br>    ports: <br>      - 8000:8000<br>    env_file:<br>      - backend/.env <br>    networks:<br>      - backend-network <br>      - frontend-network <br><br>    depends_on:<br>      - db <br>  <br>  frontend: <br>    build: <br>      context: ./frontend <br>      dockerfile: Dockerfile <br>    container_name: react-frontend <br>    ports: <br>      - 5173:5173 <br>    env_file: <br>      - frontend/.env <br>    depends_on: <br>      - backend <br>    networks:<br>      - frontend-network<br>  <br>  db: <br>    image: postgres:13<br>    environment: <br>      POSTGRES_USER: app<br>      POSTGRES_PASSWORD: changethis123<br>      POSTGRES_DB: app <br>    ports:<br>      - 5432:5432 <br>    volumes: <br>      - postgres_data:/var/lib/postgresql/postgres_data <br>    networks: <br>      - backend-network <br>  <br>  adminer: <br>    image: adminer<br>    restart: always <br>    environment:<br>      ADMINER_DEFAULT_SERVER: db <br>    ports:<br>      - 8080:8080<br>    networks:<br>      - backend-network  <br>  <br>  proxy_manager:<br>    image: jc21/nginx-proxy-manager:2.10.4<br>    restart: always<br>    ports: <br>      - 80:80<br>      - 443:443<br>      - 81:81<br>    environment:<br>      DB_SQLITE_FILE: &quot;/data/database.sqlite&quot;<br>    volumes:<br>      - data:/data<br>      - letsencrypt:/etc/letsencrypt<br>      - ./Nginx/nginx.conf:/data/nginx/custom/http_top.conf<br>    depends_on:<br>      - frontend<br>      - backend <br>      - db <br>      - adminer <br>    networks: <br>      - frontend-network<br>      - backend-network <br><br>networks:<br>  frontend-network:<br>  backend-network:<br>    <br>volumes:<br>  postgres_data: <br>  data:<br>  letsencrypt: <br> </pre><p>Before starting our Application, Navigate to the backend.env file and change the <strong>Environment=production</strong>, and also <strong>domain=yourdomain</strong>.<br>Also modify the frontend.env too and change to your domain.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*ENSzsrqXYvXNg_iy7QLFzA.png" /></figure><p>Now start the Application with the command:</p><pre>docker compose up -d</pre><h4>Testing Deployments</h4><p>Verify that each of the subdomains are working fine and routing traffic accordingly to each of the services.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*j8vg_yMxathSQ00yyyw0Yg.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*PPX8TZmFdNqpjpo0dHpE7Q.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*9JzxH10jlnRw6YYlArHngg.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*PUonyaBLZ2W30FD39Q8ImA.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*atRDy1RK-spMCpQSX7LtPA.png" /><figcaption>Frontend</figcaption></figure><h4>Grafana Dashboard Set Up</h4><p>Grafana is an Incredible Visualization tool. It allows you to create dashboards where you can view and analyze the data collected by prometheus(or other data sources). It turns that data into graphs, so you can easily see how your systems are perfoming in real time.</p><p>First we will set up our data sources: <strong>Prometheus and Loki.</strong></p><p>Log in to Grafana, on the left side bar click under connections: click on data sources, and select prometheus as a data source, Enter the Prometheus service URL http://prometheus:9090/prometheus , leave other settings as default, and click &quot;save and test”.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*KangkPeN36ZN015IP75O9A.png" /></figure><p>Do the same for Loki, and add the Loki Service URL <a href="http://loki:3100/">http://loki:3100</a></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*rPTiBc6FDVjEZmm8td8zUA.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*JDErys0mH1hrMGBzmN_DPw.png" /></figure><h4>Container Dashboard</h4><p>To set up dashboard for our containers, click on Dashboards, Click on the + icon in the upper right and import dashboard. Use the Dashboard ID <strong>19792</strong> and Load. Give your Dashboard a Name, Select Prometheus as data source and Import.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*ytqk2dq2FehKvx3ottNjvg.png" /></figure><p>This will display a very beautiful dashboard with visualizations for each of your container metrics.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*I7-ssGEni6jU6nKAaRWKvQ.png" /></figure><h4>Logs Dashboard</h4><p>Next we’ll setup our Logs dashboard, This will show visualizations for system logs.</p><p>Go to Dashboards, Click on the + icon, New dashboard. <br>Click Add Visualization and select Loki as data source. <br>Click on select labels, and choose job label, then in select value, select the varlogs, and then click on run query.<br>Choose Switch to table, to see the system logs in table format.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*EfliLe2pqkBYYcGH8Ge53w.png" /></figure><h3>Conclusion</h3><p>In this article, we’ve explored the setup and configuration of a robust monitoring, logging, and reverse proxy system using Docker and Nginx Proxy Manager. By combining tools like Prometheus, Grafana, Loki, and Nginx, we’ve created a highly scalable and secure environment that efficiently handles containerized applications and metrics collection.</p><p><strong>Key takeaways include</strong>:</p><ul><li><strong>Prometheus</strong> provides a powerful monitoring solution with customizable configurations for scraping and visualizing metrics.</li><li><strong>Grafana</strong> serves as an intuitive dashboard for displaying data from Prometheus and Loki, enabling easy monitoring and analysis.</li><li><strong>Loki and Promtail</strong> work together to collect, aggregate, and visualize logs in Grafana, ensuring full-stack observability.</li><li><strong>Nginx Proxy Manager</strong> simplifies the management of reverse proxy rules, secure SSL certificates, and routing across multiple services.</li></ul><p>Thank you for reading and i hope you find this article helpful.</p><p>connect with me on</p><p><a href="https://x.com/harbeylefty">Twitter</a><br><a href="https://www.linkedin.com/in/habeeb-yusuf-265254252/">LinkedIn</a><br><a href="http://harbeysegun@gmail.com/">Email</a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=3e945d9b437a" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Deploying an Application on Kubernetes: A Step by Step Guide.]]></title>
            <link>https://medium.com/@harbeylefty/deploying-an-application-on-kubernetes-a-step-by-step-guide-ec872433cd76?source=rss-a9dda9de9043------2</link>
            <guid isPermaLink="false">https://medium.com/p/ec872433cd76</guid>
            <dc:creator><![CDATA[HarbeyLefty]]></dc:creator>
            <pubDate>Fri, 08 Nov 2024 14:39:36 GMT</pubDate>
            <atom:updated>2024-11-08T15:06:55.718Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*wBN1Yump0J7SJAEnYwPIXQ.png" /></figure><p><strong>Introduction</strong></p><p><strong>Kubernetes</strong> is an open-source platform designed to automate the deployment, scaling, and management of containerized applications. It is a powerful tool that helps you manage applications that are packaged in containers, like Docker. It automates many tasks, such as starting your app, keeping it running, and scaling it up or down depending on the need. Instead of manually managing where and how your app runs, Kubernetes handles it for you. This makes it easier to run and maintain even complex applications without worrying about downtime, crashes, or handling a lot of traffic.</p><p><strong>Importance Of Kubernetes</strong></p><p>Kubernetes has become an essential tool for managing modern applications because it simplifies how we deploy, manage, and scale them. Kubernetes stands out as a solution for managing containerized applications due to its unique set of features and benefits, Here are few reasons why Kubernetes is important:</p><ol><li><strong>Automated Operations: </strong>Kubernetes has built-in commands to handle a lot of the heavy lifting that goes into application management, allowing you to automate day to day operations. Tasks like deploying updates, scaling applications when traffic increases, or restarting containers if they crash are handled automatically. This saves a lot of time and reduces the chances of human errors.</li><li><strong>Resilience and High Availability</strong>: Kubernetes is built to ensure that your application remains available, even when parts of the system fail. It automatically detects and replaces failed containers, distributing workloads across healthy servers. This minimises downtime and keeps your services running smoothly.</li><li><strong>Service Health Monitoring:</strong> Kubernetes continuously runs health checks against your services, restarting containers that fails or have stalled, and only making available services to users when it has confirmed they are running.</li><li><strong>Scalability</strong>: Whether your application has to handle 10 users or 10 million, Kubernetes scales up or down automatically to meet the demand, ensuring that your application performs well under any load.</li></ol><p>Now that we have an understanding of Kubernetes and the importance of Kubernetes, let us take a walk through the process of deploying an application to a Kubernetes cluster.</p><p><strong>Requirements</strong></p><p>Before we dive deep into deploying our application to Kubernetes, you need to have the following requirements:</p><p>Basic Knowledge of Docker.</p><p>Docker installed on your local machine.</p><p>A Kubernetes cluster. This can be setup locally using Minikube. Click <a href="https://minikube.sigs.k8s.io/docs/start/?arch=%2Fmacos%2Fx86-64%2Fstable%2Fbinary+download">here</a> to install Minikube.</p><p>Click <a href="https://github.com/Harbeylefty/Deploy-app-k8s">here</a> to Fork and clone the application repo</p><p><strong>Step 1: Containerize The Application using Docker Compose</strong></p><p>For this project, we have a multi-service application that consists of three components: A backend, frontend, and a MongoDB database.</p><p><strong>Docker Compose</strong> is a tool specifically designed to manage multi-container Docker applications. It allows us to define how each service (container) should be built, configured, and linked to other services — all in a single file. By using Docker Compose, we simplify the development and deployment process, enabling:</p><ul><li><strong>Unified service management</strong>: All services (frontend, backend, and MongoDB) can be managed with one command.</li><li><strong>Networking between containers</strong>: Services can easily communicate with each other without complex network configurations.</li><li><strong>Dependency management</strong>: Compose ensures that services are started in the correct order, so the backend doesn’t start before MongoDB is ready.</li></ul><p>In this section, we’ll use Docker Compose to Containerize our application and run all the services simultaneously.</p><p>Below is the Dockerfile for the backend and frontend, and also the compose.yml file. You can see this in the Application Repository</p><pre>#Backend<br><br>FROM node:22-alpine<br><br>WORKDIR /app<br><br>COPY package*.json ./<br><br>RUN npm install<br><br>COPY . .<br><br>EXPOSE 5000<br><br>CMD npm start </pre><pre>#Frontend<br><br># Stage 1: Build the React app<br>FROM node:22 As build<br><br>WORKDIR /app<br><br>COPY package*.json ./<br><br>RUN npm install<br><br>COPY . .<br><br>RUN npm run build<br><br># Stage 2: Serve the app using Nginx<br>FROM nginx:alpine<br><br>#Copy the build output to the appropriate Nginx directory<br>COPY --from=build /app/build /usr/share/nginx/html<br><br># Copy the custom Nginx configuration<br>COPY nginx.conf /etc/nginx/conf.d/default.conf<br><br>EXPOSE 80<br><br>CMD [&quot;nginx&quot;, &quot;-g&quot;, &quot;daemon off;&quot;]</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*p98Z3ecIW-nR-t-PTIvETw.png" /></figure><p>Use the following commands to build and push the images to Dockerhub. It doesn’t necessarily have to be Dockerhub, you can use any container registry of your choice like AWS Elastic Container Registry(ECR), or Azure Container Registry.</p><pre>docker login<br>docker compose up<br>docker tag image_id your-dockerhub-username/your-image-name<br>docker push your-dockerhub-username/your-image-name</pre><p>docker login ensures you’re logged in to Dockerhub from the terminal.</p><p>docker compose up builds the image and run the containers</p><p>docker tag is used to tag your image. In this case we have the frontend and backend image, so you tag both the frontend and backend images.</p><p>docker push is used to push the images to your Dockerhub</p><p><strong>Step 2: Setting up the Kubernetes Cluster.</strong></p><p>For this project we will be using Minikube to setup our cluster locally.</p><p>Run the command below to Start Minikube:</p><pre>minikube start</pre><p><strong>Step 3: Deploying Application to Kubernetes</strong></p><p>Now that we’ve containerized our application and pushed the Docker images to Docker Hub, the next step is to deploy the application to the Kubernetes cluster. For this project, we’ll use Kubernetes manifests that are already prepared to define the desired state of our application within the cluster. This manifest files can be found in the Application repo.</p><p>Kubernetes manifests are YAML files that specify how different parts of our application (frontend, backend, and database) should be deployed and managed in the cluster. Let’s walk through the deployment process.</p><p><strong>3.1 MongoDB Deployment and Service.</strong></p><p><strong>Secrets and ConfigMap</strong></p><p>Before deploying MongoDB, we need to ensure that sensitive informations like the MongoDB username and password are handled securely. To achieve this, we’ll use a <strong>Secret</strong> to store these sensitive values, these values are stored in a <strong>base64-encoded</strong> format to ensure they are not exposed as plain text in the manifest, and a <strong>ConfigMap</strong> to store non-sensitive information like the MongoDB connection string.</p><p>Let’s define both of these components:</p><p><strong>Secret: (mongo-secret.yml)</strong></p><pre>apiVersion: v1<br>kind: Secret <br>metadata:<br>  name: mongo-secret <br>type: Opaque <br>data:<br>  mongo-user: bW9xxxxxxxxxxx #Replace with base64 encoded value<br>  mongo-password: bW9xxxxxxxx #Replace with your password</pre><p>To convert the mongo-user and mongo-password to base64 encoded format, run the following commands</p><pre>#For Linux/MacOS<br>echo -n &#39;mongo-user&#39; | base64<br>echo -n &#39;mongo-password&#39; | base64<br><br>#For Windows <br>[System.Convert]::ToBase64String([System.Text.Encoding]::UTF8.GetBytes(&quot;mongo-user&quot;))<br><br>[System.Convert]::ToBase64String([System.Text.Encoding]::UTF8.GetBytes(&quot;mongo-password&quot;))</pre><p><strong>ConfigMap: (mongo-config.yml).</strong></p><pre>apiVersion: v1<br>kind: ConfigMap <br>metadata:<br>  name: mongo-config<br>data:<br>  MONGO_URI: mongodb://mongo-user:mongo-password@mongodb:27017/mydb</pre><p>The <strong>ConfigMap</strong> stores the MongoDB connection string, which points to the MongoDB instance and uses the username and password from the <strong>mongo-secret</strong>. This allows the MongoDB deployment to easily access and connect to the database.</p><p>— — — — — — — — — — — — — — — — — — — — — — — — — — — — — — — — — — — — —</p><p>Now that we have securely stored the MongoDB credentials using a <strong>Secret</strong> and set up the connection string with a <strong>ConfigMap</strong>, we can move on to deploying MongoDB.</p><p>The <strong>mongo-deployment</strong> will reference the <strong>mongo-secret</strong> for the username and password, and the <strong>mongo-config</strong> for the database connection string.</p><p>Here’s the Deployment and Service configuration for MongoDB:</p><pre>apiVersion: apps/v1 <br>kind: Deployment <br>metadata: <br>  name: mongo-deployment<br>  labels:<br>    app: mongo <br>spec:<br>  replicas: 1<br>  selector:<br>    matchLabels:<br>      app: mongo<br>  template:<br>    metadata:<br>      labels:<br>        app: mongo<br>    spec:<br>      containers:<br>      - name: mongodb <br>        image: mongo:latest <br>        ports:<br>        - containerPort: 27017<br>        env:<br>        - name: MONGO_INITDB_ROOT_USERNAME<br>          valueFrom:<br>            secretKeyRef:<br>              name: mongo-secret <br>              key: mongo-user <br>        - name: MONGO_INITDB_ROOT_PASSWORD<br>          valueFrom:<br>            secretKeyRef:<br>              name: mongo-secret<br>              key: mongo-password <br>        resources:<br>          requests:<br>            memory: &quot;128Mi&quot;<br>            cpu: &quot;250m&quot;<br>          limits:<br>            memory: &quot;256Mi&quot;<br>            cpu: &quot;500m&quot;<br><br>---<br>apiVersion: v1 <br>kind: Service <br>metadata:<br>  name: mongo-service<br>spec:<br>  selector: <br>    app: mongo <br>  ports: <br>  - protocol: TCP<br>    port: 27017<br>    targetPort: 27017</pre><p>In this Manifest file, we define the <strong>Deployment</strong> and <strong>Service</strong> for MongoDB in Kubernetes. The deployment creates a MongoDB instance and uses <strong>Kubernetes Secrets</strong> to manage sensitive data like credentials.</p><p>Instead of hardcoding the database credentials, the deployment references a <strong>Secret</strong> called mongo-secret to securely provide the MongoDB root username and password. This prevents sensitive data from being exposed in the manifest file</p><p><strong>3.2 Backend Deployment and Service (backend-deployment.yml)</strong></p><pre>apiVersion: apps/v1 <br>kind: Deployment <br>metadata:<br>  name: backend-deployment<br>  labels:<br>    app: backend <br>spec:<br>  replicas: 1 <br>  selector: <br>    matchLabels:<br>      app: backend <br>  template:<br>    metadata: <br>      labels:<br>        app: backend<br>    spec: <br>      containers: <br>      - name: backend <br>        image: harbeylefty17/k8s-demo-backend<br>        ports:<br>        - containerPort: 5000<br>        env: <br>        - name: USER_NAME<br>          valueFrom: <br>            secretKeyRef:<br>              name: mongo-secret <br>              key: mongo-user<br>        - name: USER_PWD <br>          valueFrom: <br>            secretKeyRef:<br>              name: mongo-secret <br>              key: mongo-password <br>        - name: MONGO_URI <br>          valueFrom:<br>            configMapKeyRef:<br>              name: mongo-config <br>              key: MONGO_URI <br>        resources:<br>          requests:<br>            memory: &quot;128Mi&quot;<br>            cpu: &quot;250m&quot;<br>          limits: <br>            memory: &quot;256Mi&quot;<br>            cpu: &quot;500m&quot;<br><br>---<br>apiVersion: v1 <br>kind: Service <br>metadata: <br>  name: backend-service <br>spec: <br>  type: NodePort <br>  selector:<br>    app: backend <br>  ports: <br>  - protocol: TCP <br>    port: 5000<br>    targetPort: 5000<br>    nodePort: 30001</pre><p>In this Manifest file we define the <strong>Deployment</strong> and <strong>Service</strong> for the backend of our application.</p><ul><li>The backend container uses the image <strong>harbeylefty17/k8s-demo-backend</strong> and listens on port <strong>5000</strong>.</li><li><strong>Environment variables</strong> are used to connect to MongoDB</li><li>The <strong>USER_NAME</strong> and <strong>USER_PWD</strong>are securely fetched from a Kubernetes <strong>Secret</strong> (mongo-secret).</li><li>The <strong>MONGO_URI</strong> is sourced from a <strong>ConfigMap</strong> (mongo-config), allowing flexibility in configuring the MongoDB connection string without modifying the container.</li><li>The <strong>backend-service</strong> is of type <strong>NodePort</strong>, which makes the backend accessible externally through a specific node port (in this case, <strong>30001</strong>).</li><li>It listens on port <strong>5000</strong>, matching the backend container’s port, and routes traffic accordingly.</li></ul><p><strong>3.3 Defining NGINX Configuration for the Frontend</strong></p><p>For the frontend, we need to define an <strong>NGINX configuration</strong> that will control how the application is served. This configuration is stored in a <strong>ConfigMap</strong> called nginx-config, which contains the NGINX server settings.</p><pre>apiVersion: v1<br>kind: ConfigMap<br>metadata:<br>  name: nginx-config<br>data:<br>  default.conf: |<br>    server {<br>      listen 80;<br>      server_name localhost;<br><br>      location / {<br>        root   /usr/share/nginx/html;<br>        index  index.html index.htm;<br>        try_files $uri $uri/ /index.html;<br>      }<br><br>      error_page   500 502 503 504  /50x.html;<br>      location = /50x.html {<br>        root   /usr/share/nginx/html;<br>      }<br>    }</pre><p>This <strong>ConfigMap</strong> defines the NGINX server behaviour, including how requests are handled and served to the frontend application. The configuration will be mounted as a volume in the frontend deployment, ensuring that NGINX uses these custom settings.</p><p><strong>3.4 Frontend Deployment and Service (frontend-deployment.yml)</strong></p><pre>apiVersion: apps/v1 <br>kind: Deployment <br>metadata:<br>  name: frontend-deployment<br>  labels:<br>    app: frontend  <br>spec:<br>  replicas: 1 <br>  selector: <br>    matchLabels:<br>      app: frontend <br>  template:<br>    metadata: <br>      labels:<br>        app: frontend <br>    spec: <br>      containers: <br>      - name: frontend  <br>        image: harbeylefty17/k8s-demo-frontend <br>        ports:<br>        - containerPort: 80<br>        volumeMounts:<br>        - name: nginx-config-volume<br>          mountPath:  /etc/nginx/conf.d/default.conf<br>          subPath: default.conf<br>        resources:<br>          requests:<br>            memory: &quot;512Mi&quot;<br>            cpu: &quot;250m&quot;<br>          limits: <br>            memory: &quot;1Gi&quot;<br>            cpu: &quot;500m&quot;<br>      volumes:<br>      - name: nginx-config-volume<br>        configMap: <br>          name: nginx-config <br><br>---<br>apiVersion: v1 <br>kind: Service <br>metadata: <br>  name: frontend-service <br>spec: <br>  type: NodePort <br>  selector:<br>    app: frontend <br>  ports: <br>  - protocol: TCP <br>    port: 80<br>    targetPort: 80<br>    nodePort: 30100</pre><p>In this manifest, we define the <strong>Deployment</strong> and <strong>Service</strong> for the frontend in Kubernetes. The deployment runs a frontend instance and uses <strong>ConfigMap</strong> to manage the NGINX configuration.</p><p>Instead of hardcoding the NGINX settings, the deployment references a <strong>ConfigMap</strong> called nginx-config to securely provide the configuration file, ensuring that the settings are applied without exposing them in the manifest. The frontend service is exposed using a <strong>NodePort</strong>, making it accessible externally.</p><p><strong>Step 4: Apply the Manifest files to Deploy Application to Kubernetes</strong></p><p>With the manifest files ready, the next step is to apply them to your Kubernetes cluster. Make sure that your cluster is up and running. If your Cluster is not up run the minikube start command again to start the cluster.</p><p>To apply the configurations use the commands:</p><pre>kubectl apply -f mongo-secret.yml<br>kubectl apply -f mongo-config.yml<br>kubectl apply -f nginx-configmap.yml<br>kubectl apply -f mongo-deployment.yml<br>kubectl apply -f backend-deployment.yml<br>kubectl apply -f frontend-deployment.yml</pre><p>This will:</p><p>Create Deployments for the Frontend, backend, and mongoDB.</p><p>Expose each service with the appropriate service type (ClusterIP or NodePort)</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*5tYpgZQvhRhK5cCwo8Sswg.png" /></figure><p><strong>Verify that the Pods are running properly</strong></p><pre>kubectl get pods</pre><p>You should see the pods and their current status:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*NbJrQWAXjD79LYEPN1nB5A.png" /></figure><p><strong>Access the Application.</strong></p><p>To access our Application, first we need to get the frontend service by running the command:</p><pre>kubectl get svc</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*lDKKgGMeed1BtjNNBeZZ3w.png" /></figure><p>Now that we have the services, let’s try to Access the Application. we will use our frontend service to access the application. Run the command:</p><pre>minikube service frontend-service --url</pre><p>This command will generate a URL that is required to access your Application. If you input this url in your browser, you should see a simple login page with Email and Password. This indicates that our application has been successfully deployed to Kubernetes!!!</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*yJkRlQfKLYhmvV-Eulp-_g.png" /></figure><p><strong>Conclusion</strong></p><p>We have gone through the process of deploying a multi-service Application to Kubernetes. We started by Containerizing our Frontend, Backend, and MongoDB using Docker, and then Deploying our Application using Kubernetes manifests, and we also saw how to reference Secrets and configMap in our deployments. We can see now that Kubernetes simplifies the orchestration of multiple services and also ensuring that your Applications runs smoothly and efficiently.</p><p><strong>I hope you find this article helpful and thanks for reading.</strong></p><p><strong>Connect with me on:</strong></p><p><a href="https://x.com/harbeylefty?t=_3AvKFiw2E4jlrZWr2wOxw&amp;s=08">Twitter</a></p><p><a href="http://Harbeysegun@gmail.com/">Email</a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=ec872433cd76" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Hosting a Static website on AWS S3 using Terraform and AWS]]></title>
            <link>https://medium.com/@harbeylefty/hosting-a-static-website-on-aws-s3-using-terraform-and-aws-86b076a6b4f0?source=rss-a9dda9de9043------2</link>
            <guid isPermaLink="false">https://medium.com/p/86b076a6b4f0</guid>
            <dc:creator><![CDATA[HarbeyLefty]]></dc:creator>
            <pubDate>Sat, 05 Oct 2024 11:08:34 GMT</pubDate>
            <atom:updated>2024-10-05T21:44:01.244Z</atom:updated>
            <content:encoded><![CDATA[<h3>Hosting a Static website on AWS S3 using Terraform and AWS.</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/720/1*Xt1ispxUFnS8CZ1dw4rMKg.png" /></figure><p><strong>Introduction</strong></p><ul><li><strong>Terraform:</strong> Terraform is an open-source <strong>infrastructure-as-code</strong> tool that allows you to automate the setup, management, and deployment of infrastructure using simple configuration files.</li><li><strong>AWS S3</strong>: Amazon S3 is an object storage service that stores data as objects within buckets.</li></ul><p><strong>Prerequisites</strong> :</p><p>AWS account.</p><p>Terraform installed on your Local machine.</p><p>AWS CLI configured on your local machine</p><p>click <a href="https://github.com/Harbeylefty/Terraform-s3-static-website-cicd/tree/master/terraform-s3">here</a> to clone the repository that contains the source code used for this project</p><p><strong>STEP 1: Define Provider</strong></p><p>Create a file named provider.tf with the following code to define the AWS provider and region.</p><pre>terraform {<br>  required_providers {<br>    aws = {<br>      source = &quot;hashicorp/aws&quot;<br>      version = &quot;5.66.0&quot;<br>    }<br>  }<br>}<br><br>provider &quot;aws&quot; {<br>  region = &quot;us-east-1&quot;<br>}</pre><p><strong>STEP 2</strong>: After defining the provider the next step is to prepare our working directory for use by terraform by running the command:</p><pre>terraform init</pre><p>This allows us to be able to start working with terraform. You should get a response like this that says terraform has been successfully initialized.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*CJrKKAB2evu617TupvbYpg.png" /></figure><p><strong>STEP 3</strong>: Now the next step is to create s3 bucket using terraform. To do that create a file with the name main.tf. This is the file where we will define all our configurations.</p><p>Now that we have created a main.tf file, we can create our s3 bucket with the following code</p><pre>resource &quot;aws_s3_bucket&quot; &quot;mybucket&quot; {<br>    bucket = &quot;terraformstaticbucket2024&quot;<br>}</pre><p>Now that we have our code for creating our s3 bucket, we apply the changes so that terraform creates our s3 bucket in AWS. To apply the changes, execute the following commands:</p><pre>terraform validate<br>terraform plan<br>terraform apply -auto-approve</pre><ul><li>The terraform validate command is used to validate our terraform configuration files to check for any errors before we apply our changes.</li><li>terraform plan is used to see the actions that will be taken and the resources that will be created without applying the changes</li><li>terraform apply -auto-approve creates our infrastructure.</li></ul><p>After executing the terraform apply command, terraform will create the s3 bucket in your AWS account.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1008/1*BghrbJ3ZmgWVs_9NwbPJkw.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*prOXSpol9TiUzqcUQnBOrA.png" /></figure><p><strong>STEP 4:</strong> Now that we have created our s3 bucket, we need to add some additional configurations. The first is to define ownership control of the bucket.</p><pre>resource &quot;aws_s3_bucket_ownership_controls&quot; &quot;ownership&quot; {<br>  bucket = aws_s3_bucket.mybucket.id<br><br>  rule {<br>    object_ownership = &quot;BucketOwnerPreferred&quot;<br>  }<br>}</pre><p><strong>STEP 5</strong>: Next we need to make the bucket public by defining the public access block</p><pre>resource &quot;aws_s3_bucket_public_access_block&quot; &quot;public_access&quot; {<br>  bucket = aws_s3_bucket.mybucket.id<br><br>  block_public_acls       = false<br>  block_public_policy     = false<br>  ignore_public_acls      = false<br>  restrict_public_buckets = false<br>}</pre><p><strong>STEP 6</strong>: Next step is to add acl (access control lists) to make sure that the bucket is ready for the website.</p><p>After applying these changes we will see that the bucket is now ready to set up our static website on it.</p><pre>resource &quot;aws_s3_bucket_acl&quot; &quot;bucket_acl&quot; {<br>  depends_on = [<br>    aws_s3_bucket_ownership_controls.ownership,<br>    aws_s3_bucket_public_access_block.public_access,<br>  ]<br><br>  bucket = aws_s3_bucket.mybucket.id<br>  acl    = &quot;public-read&quot;<br>}</pre><p>Before applying the changes if we check our bucket permissions we will see that it blocks all public access making it impossible for the public to be able to access our website.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*CbT_UvjtqqxEDUF5ahLKbg.png" /></figure><p>Run terraform apply now to update our changes and let’s see the outcome.</p><p>If you reload the AWS console page then you should see the outcome has changed to this</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*_IoGb4nts_VQoekHoqzoVQ.png" /></figure><p><strong>STEP 7</strong>: Now that our bucket is public the next step is to upload our index.html file and error.html file to the bucket since the bucket is empty.</p><pre>resource &quot;aws_s3_object&quot; &quot;index&quot; {<br>    bucket = aws_s3_bucket.mybucket.id<br>    key = &quot;index.html&quot;<br>    source = &quot;index.html&quot;<br>    acl = &quot;public-read&quot;<br>    content_type = &quot;text/html&quot;<br>}<br><br>resource &quot;aws_s3_object&quot; &quot;error&quot; {<br>    bucket = aws_s3_bucket.mybucket.id<br>    key = &quot;error.html&quot;<br>    source = &quot;error.html&quot;<br>    acl = &quot;public-read&quot;<br>    content_type = &quot;text/html</pre><p>Run terraform apply to update and upload our files to the bucket</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*uSdxWk8UxyxgBxRdn_9l9A.png" /></figure><p>Now we can see our index.html and error.html file inside our bucket.</p><p><strong>STEP 8</strong>: After uploading our files to the bucket then we setup the website for static hosting. This is where we get the link to access our website, as you can see below static website hosting is disabled</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*kJvTODlCUk6_d1DwjNHHug.png" /></figure><p>Copy and paste the code below to setup website for static hosting</p><pre># Setup websi <br>resource &quot;aws_s3_bucket_website_configuration&quot; &quot;website&quot; {<br>  bucket = aws_s3_bucket.mybucket.id<br><br>  index_document {<br>    suffix = &quot;index.html&quot;<br>  }<br><br>  error_document {<br>    key = &quot;error.html&quot;<br>  }<br>  depends_on = [aws_s3_bucket_acl.bucket_acl]<br>}</pre><p>Run terraform apply -auto-approve to apply our changes</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*2g_5jU4cK7oyFgHysCW20g.png" /></figure><p>Now we can see that the static website hosting has been enabled and on the screen we have the link to visit our website.</p><p>Lastly, in order to get the website link without needing to go to the AWS console, we can output the website endpoint. Create an output.tf file and paste the following code</p><pre> <br>output &quot;website_endpoint&quot; {<br>    value = aws_s3_bucket.mybucket.website_endpoint <br>}</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1018/1*Y06vkRSuKzde3LxErh5-Yg.png" /></figure><p>Copy the website link printed on the screen and paste into your browser and this should be the outcome:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*9oC8SEpyLZ3carpM-oeDpA.png" /></figure><p><strong>Conclusion:</strong></p><p>We have gone through the process of hosting a static website on AWS s3 using terraform, and we can see that by using terraform we were able to easily automate the setup of our infrastructure.</p><p><strong>I hope you find this article helpful and thanks for reading.</strong></p><p><strong>connect with me on:</strong></p><p><a href="https://x.com/harbeylefty">Twitter</a>:</p><p><a href="http://harbeysegun@gmail.com/">Email</a>:</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=86b076a6b4f0" width="1" height="1" alt="">]]></content:encoded>
        </item>
    </channel>
</rss>