<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Yesa on Medium]]></title>
        <description><![CDATA[Stories by Yesa on Medium]]></description>
        <link>https://medium.com/@yesa1?source=rss-98e2b0c14f89------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/1*LAZzWR-DFJSzRCuwxD_xNw.png</url>
            <title>Stories by Yesa on Medium</title>
            <link>https://medium.com/@yesa1?source=rss-98e2b0c14f89------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Thu, 19 Nov 2026 13:29:07 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@yesa1/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[Building a Scalable Image Service with Amazon S3 + CloudFront,  DynamoDB and Lambda]]></title>
            <link>https://medium.com/@yesa1/building-a-scalable-image-service-with-amazon-s3-cloudfront-dynamodb-and-lambda-cbe3a648ef66?source=rss-98e2b0c14f89------2</link>
            <guid isPermaLink="false">https://medium.com/p/cbe3a648ef66</guid>
            <category><![CDATA[cloudfront]]></category>
            <category><![CDATA[cloudfront-and-s3]]></category>
            <category><![CDATA[amazon-s3]]></category>
            <dc:creator><![CDATA[Yesa]]></dc:creator>
            <pubDate>Thu, 05 Sep 2024 17:55:24 GMT</pubDate>
            <atom:updated>2024-09-11T02:36:07.653Z</atom:updated>
            <content:encoded><![CDATA[<h3>Building a Scalable Image Service with Amazon S3 + CloudFront, DynamoDB and Lambda</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*vToK0txGgDdh9sFZ" /><figcaption>Photo by <a href="https://unsplash.com/@laurenmancke?utm_source=medium&amp;utm_medium=referral">Lauren Mancke</a> on <a href="https://unsplash.com/?utm_source=medium&amp;utm_medium=referral">Unsplash</a></figcaption></figure><p>In this guide, I’ll walk you through the steps to create a scalable image service using Amazon Web Services (AWS) components: S3, CloudFront, DynamoDB, and Lambda.</p><h3>Overview</h3><h4>Why AWS?</h4><p>Amazon Web Services (AWS) provides a suite of cloud computing services that offer scalability, reliability, and cost-effectiveness. By leveraging AWS, you can build highly available and resilient systems that can handle a wide range of workloads, including serving images at scale.</p><h4>Architecture Overview</h4><p>The image service architecture comprises the following components:</p><ul><li><strong>Amazon S3:</strong> Serves as the primary storage for original and processed images, ensuring high durability and availability.</li><li><strong>CloudFront:</strong> Acts as a global content delivery network, caching images at edge locations for faster delivery to users.</li><li><strong>DynamoDB:</strong> Stores image metadata, such as filenames, dimensions, formats, and associated tags, enabling efficient search and retrieval.</li><li><strong>Lambda:</strong> Handles image processing tasks, such as resizing, cropping, and format conversion, triggered by events from S3 or API calls.</li></ul><h3>Design</h3><h4><strong>Image Upload</strong></h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*lzYWJGSC8t_aQ7_qm1rXLg.png" /><figcaption>Image upload flow</figcaption></figure><ul><li>Users upload images via a web or mobile application.</li><li>The application sends the image data to an API Gateway endpoint.</li><li>A Lambda function receives the image, generates unique identifiers, and stores the original image in private S3 buckets. The upload bucket serves as an event source that will trigger image transformation.</li><li>The Lambda function extracts metadata from the image and stores it in DynamoDB.</li></ul><p><em>Be aware that Lambda has a 6MB limit for the payload size. Alternatively, you could use an S3 pre-signed URL for the upload.</em></p><h4><strong>Image Processing</strong></h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*veIZ13imSA3x0-YFhYyJKw.png" /><figcaption>Image processing flow</figcaption></figure><ul><li>When an image is uploaded, S3 publishes event to an SNS topic that triggers a Lambda function.</li><li>The Lambda function retrieves the original image from upload bucket, performs the image transformation, and stores the transformed images in the S3 asset bucket, which is integrated with CloudFront for efficient content delivery. To manage storage costs effectively, the original image is deleted from the upload bucket after the transformation is completed.</li></ul><h4><strong>Image Delivery</strong></h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*LJINB_9jPYpg28RDNzKeYA.png" /><figcaption>Image delivery flow</figcaption></figure><ul><li>Users request images through a web or mobile application.</li><li>The application constructs URLs pointing to CloudFront distributions associated with the S3 buckets.</li><li>CloudFront serves the cached images from the nearest edge location, reducing latency and improving performance.</li><li>If the requested image or transformation is not cached, CloudFront fetches it from S3. If the fetch from S3 results in a 404 error (image not found), Lambda@Edge will be triggered to serve a default image. Alternatively you can set up CloudFront with <a href="https://docs.aws.amazon.com/AmazonCloudFront/latest/DeveloperGuide/high_availability_origin_failover.html">origin failover</a> (<em>fallback to another S3 bucket or a web server)</em> for scenarios that require high availability.</li></ul><h4><strong>Managing Image Metadata with DynamoDB</strong></h4><p>While S3 efficiently stores and serves our images, DynamoDB plays a crucial role in managing associated metadata. This metadata includes essential information about each image, such as:</p><ul><li><strong>Unique Identifier:</strong> A unique ID generated during the upload process to distinguish each image.</li><li><strong>Filename:</strong> The original filename of the uploaded image.</li><li><strong>Dimensions:</strong> Width and height of the original and processed images.</li><li><strong>Formats:</strong> The image formats (e.g., JPEG, PNG).</li><li><strong>Tags:</strong> User-defined tags or categories for easier image organization and search.</li><li><strong>Upload Timestamp:</strong> The date and time the image was uploaded.</li></ul><p><strong>Use Cases for DynamoDB in Image Management</strong></p><ul><li><strong>Image Search:</strong> Users can search for images based on specific criteria, such as tags, dimensions, or upload date.</li><li><strong>Image Filtering:</strong> Filter images based on metadata attributes to display relevant subsets.</li></ul><h3>Cost Estimation</h3><p>Let’s assume we have a social media app with 40K daily active users. 30% of them upload one new image per day, with an average image size of 1.5 MB. Our system transforms the image into several variants and reduces the image size by ~85% of the original size. The other 70% are primarily readers, viewing an average of 50 images per day, with average image size of 75 KB. This translates to:</p><ul><li><strong>Write Traffic:</strong> 12K users * 1 request * 30 days = 360K requests / month</li><li><strong>Read Traffic: </strong>28K users * 15 requests * 30 days = 12.6M requests / month</li><li><strong>Write Throughput: </strong>360K requests * 1.5 MB = 540 GB/ month</li><li><strong>Read Throughput:</strong> 12.6M requests * 75 KB = 901 GB / month</li><li><strong>Transformed Images Storage: </strong>120KB compressed original image + 75KB medium image + 35KB small image = 230 KB / image processed</li></ul><p>And the components we are using are:</p><ul><li>1 AWS Lambda for upload, with memory size 128MB and average execution time of 300ms / invocation</li><li>1 AWS Lambda for transform, with memory size 320MB and average execution time of 2s / invocation</li><li>1 AWS Lambda@Edge to handle 404 response from origin server, with memory size 128MB and average execution time of 150ms</li><li>1 Amazon API Gateway with HTTP API</li><li>1 Amazon DynamoDB table with on-demand capacity mode</li><li>1 Amazon CloudFront CDN Distribution</li><li>1 Amazon S3 standard bucket for upload, and 1 for asset storage</li></ul><p>Assuming we are hosting in ap-southeast-1 region to serve customers in APAC. Below is the detailed estimation, which totals $23.81 per month.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/d9890bb75cd27587a25f1ac79ac83119/href">https://medium.com/media/d9890bb75cd27587a25f1ac79ac83119/href</a></iframe><h3><strong>Benefits</strong></h3><ul><li><strong>Scalability:</strong> The architecture leverages the inherent scalability of AWS services, handling increasing image volumes and user requests seamlessly.</li><li><strong>Performance:</strong> CloudFront’s global network ensures fast image delivery to users worldwide.</li><li><strong>Cost-efficiency:</strong> S3’s tiered storage options and Lambda’s pay-per-execution model optimize costs.</li><li><strong>Flexibility:</strong> The service can be easily extended to support additional image processing features or integrations with other AWS services.</li></ul><h3><strong>Conclusion</strong></h3><p>Building a scalable image service with Amazon S3, CloudFront, DynamoDB, and Lambda provides a powerful solution for managing and delivering images efficiently. This architecture enables you to focus on your core application while AWS handles the complexities of image storage, processing, and delivery. By leveraging these managed services, you can create a high-performing, cost-effective, and scalable image infrastructure that meets the demands of modern web applications.</p><p><strong><em>Note:</em></strong><em> This article provides a conceptual overview. The actual implementation may involve additional considerations and configurations based on specific requirements.</em></p><p>Bonus: I’ve provided the CloudFormation template to deploy the necessary AWS resources. Go check it out at my github repository <a href="https://github.com/yezarela/image-service-cf/">https://github.com/yezarela/image-service-cf/</a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=cbe3a648ef66" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Architecting AWS Lambda Functions in Golang: A Clean Approach]]></title>
            <link>https://medium.com/@yesa1/architecting-aws-lambda-functions-in-golang-a-clean-approach-f0f6d95dbfcd?source=rss-98e2b0c14f89------2</link>
            <guid isPermaLink="false">https://medium.com/p/f0f6d95dbfcd</guid>
            <category><![CDATA[aws-lambda]]></category>
            <category><![CDATA[clean-architecture]]></category>
            <category><![CDATA[go]]></category>
            <dc:creator><![CDATA[Yesa]]></dc:creator>
            <pubDate>Mon, 09 Oct 2023 11:46:48 GMT</pubDate>
            <atom:updated>2024-01-09T08:26:32.983Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*JPIgnx-4XbTyZAWn" /><figcaption>Photo by <a href="https://unsplash.com/@lanceanderson?utm_source=medium&amp;utm_medium=referral">Lance Anderson</a> on <a href="https://unsplash.com/?utm_source=medium&amp;utm_medium=referral">Unsplash</a></figcaption></figure><p>Keeping code structure clean when using AWS Lambda can be a bit trickier, since there will be a bunch of entry points in our code. In fact, I usually spend some time finding how to structure my code before starting to code, to achieve “maintainable” and “easy to read” code. In this post, we’ll try out how to use Clean Architecture with AWS Lambda and Go.</p><p><em>*In </em><a href="https://medium.com/@yesaritonga/unveiling-clean-architecture-guidelines-benefits-and-trade-offs-d030831a3e53"><em>previous post</em></a><em>, I’ve summarized some concepts of what Clean Architecture is and the pros cons, please check it out if you are keen!</em></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*pzzvLDAAirzGOf4scdbI8w.png" /></figure><p>The first purpose of Clean Architecture is “<strong>Independent of Frameworks”</strong> so we will not depend on any framework rule and limitations. In this project, we will use 4 layers to structure our code with Clean Architecture:</p><h4>Domains</h4><p>This layer holds all models / interfaces that will be used across layers. The reason of putting them all together is to avoid potential circular dependency issue, since most of the time a model will be referenced by another model.</p><p>For example, this code is placed in domain/user.go:</p><pre>package domain<br><br>...<br><br>type User struct {<br>  ID        uint      `json:&quot;id&quot;`<br>  Name      string    `json:&quot;name&quot;`<br>  ...<br>}<br><br>type UserRepository interface {<br>  GetUser(ctx context.Context, id uint) (*User, error)<br>  ...<br>}<br><br>type UserUsecase interface {<br>  GetByID(ctx context.Context, id uint) (*User, error)<br>  ...<br>}</pre><h4>Repositories</h4><p>This layer is responsible for communicating with data sources, whether it is Database, another services, or external APIs. There should be no business logic here other than converting data source models to our application models defined in domain folder.</p><p>This layer should expose general purpose methods regardless what the data source is, for example using GetUserByEmail instead of QueryUserByGSIEmail. With this in mind, we can easily swap the data source with another data source in the future without nudging the repository interface which is already consumed by another layers.</p><p>All repositories will be placed inside each module that represents it. For example, this code is placed in user/repository/mysql_repository.go :</p><pre>package repository<br><br>...<br><br>type repository struct {<br>   db *sql.DB<br>}<br><br>func NewMysqlRepository(db *sql.DB) domain.UserRepository {<br>   return &amp;repository{db}<br>}<br><br>func (m *repository) GetUser(ctx context.Context, id uint) (*domain.User, error) {  <br>  rows, err := m.db.QueryContext(ctx, &quot;SELECT * FROM user WHERE id = ?&quot;, id)  <br>  ...<br>}</pre><h4>Use cases</h4><p>This layer holds the business logic of our application in each related module, it combines the use of some repositories and some conditions to achieve the desired results. This layer should not care about who will consume it, whether it is a REST API, CLI, etc. Thus, no handler specific implementation imports allowed to be here, such as to directly return 400 status code, etc.</p><p>This layer should care more about data validation before calling repository layer, and most likely will expose some error codes that can be understood by outer layer if validation fails or something went wrong. This layer really depends on repository layer, so please be aware when making changes to repository layer.</p><p>All use cases will be placed inside related module together with their repository dependencies. For example, this code is placed in user/usecase/user_usecase.go :</p><pre>package usecase<br><br>...<br><br>type usecase struct {<br>  userRepo domain.UserRepository<br>}<br><br>func NewUsecase(r domain.UserRepository) domain.UserUsecase {<br>  return &amp;usecase{<br>    userRepo: r,<br>  }<br>}<br><br>func (m *usecase) GetByID(ctx context.Context, id uint) (*domain.User, error) {<br>  ...<br>  if res == nil {<br>    return nil, domain.ErrNotFound<br>  }<br>  ...<br>}</pre><h4>Handlers</h4><p>This is where our application entry point resides, every Lambda handlers for the related module will be defined in this layer. This layer handles request parameters, validates them, pass them to use case layer, and return the result to Lambda function which will then processed by API Gateway.</p><p>This layer is responsible for being the interface between user and our system, accept the inputs and return the outputs. In some cases, this layer will need to convert the models to some custom formats.</p><p>All handlers will also be placed under the related module directory as repository and use cases does. For example, this code is placed in user/handler/user-get/main.go :</p><pre>package main<br><br>...<br><br>var (<br> db          *sql.DB<br> userRepo    domain.UserRepository<br> userUsecase domain.UserUsecase<br>)<br><br>func handler(req events.APIGatewayProxyRequest) (events.APIGatewayProxyResponse, error) {<br>  ...<br>  id, _ := strconv.Atoi(req.PathParameters[&quot;id&quot;])<br><br>  res, err := userUsecase.GetByID(ctx, id)<br>  if err != nil {<br>    return api.APIServerError(err)<br>  }<br><br>  if res == nil {<br>    return api.ErrResponse(404, &quot;Data not found&quot;)<br>  }<br><br>  return api.APIResponse(200, users)<br>}<br><br>func main() {<br> db = database.NewMySQLConnection(os.Getenv(&quot;DBDataSourceName&quot;))<br> userRepo = _userRepo.NewMysqlRepository(db)<br> userUsecase = _userUsecase.NewUsecase(userRepo)<br><br> lambda.Start(handler)<br>}</pre><h4>Conclusion</h4><p>Finally, we have seen how each layer communicates and one thing we should never forget that <em>“Source code dependencies can only point inwards”.</em></p><p>To conclude, this is how the directory looks like:</p><pre>.<br>├── domain                           &lt;-- All domains/entities belong here<br>├── user                             &lt;-- Module directory<br>│   └── handler                     <br>│       └── user-get                 &lt;-- Source code for a lambda function<br>│           ├── main.go              &lt;-- Lambda function code<br>│           └── main_test.go         &lt;-- Unit tests<br>│   └── repository <br>│       ├── mysql_repository.go      &lt;-- Repository for our db<br>│       └── mysql_repository_test.go &lt;-- Unit tests<br>│   └── usecase <br>│       ├── user_usecase.go          &lt;-- Business rules<br>│       └── user_usecase_test.go     &lt;-- Unit tests<br>│<br>├── infra                            &lt;-- All external framework/drivers <br>├── utils                            &lt;-- Shared utilities<br>├── Makefile                         &lt;-- Make to automate build<br>└── template.yaml                    &lt;-- Cloudformation template</pre><p>As you can see, we can now write unit tests for each layer independently as one of the Clean Architecture purpose is <strong>Testable</strong>! And also if in the future you want to replace Lambda with <a href="https://echo.labstack.com/">Echo</a> or CLI, it can be done in handler layer level without touching repository/use case layers.</p><p>If you are interested, the source code for this post can be found here:</p><p><a href="https://github.com/yezarela/go-lambda/">GitHub - yezarela/go-lambda: Go architecture for AWS Lambda</a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=f0f6d95dbfcd" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Unveiling Clean Architecture: Guidelines, Benefits, and Trade-offs]]></title>
            <link>https://medium.com/@yesa1/unveiling-clean-architecture-guidelines-benefits-and-trade-offs-d030831a3e53?source=rss-98e2b0c14f89------2</link>
            <guid isPermaLink="false">https://medium.com/p/d030831a3e53</guid>
            <category><![CDATA[clean-architecture]]></category>
            <category><![CDATA[clean-code]]></category>
            <dc:creator><![CDATA[Yesa]]></dc:creator>
            <pubDate>Tue, 26 Sep 2023 15:38:24 GMT</pubDate>
            <atom:updated>2023-09-26T15:38:24.040Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*ipNNgSD_SKiPf-ta" /><figcaption>Photo by <a href="https://unsplash.com/@srosinger3997?utm_source=medium&amp;utm_medium=referral">Samantha Gades</a> on <a href="https://unsplash.com/?utm_source=medium&amp;utm_medium=referral">Unsplash</a></figcaption></figure><p>You may have heard about Clean Architecture before and still figuring out what that is and what it is trying to solve, yes I was there too. In this post, I’ll try to explain and summarize Clean Architecture from some sources I read in a simple and hopefully quite straightforward way for engineers to understand.</p><h3>What is Clean Architecture?</h3><p>Clean architecture is a software design philosophy that <strong>separates</strong> the elements of a software design into ring levels. An important goal of clean architecture is to provide developers with a way to organize code in such a way that it encapsulates the business logic but keeps it separate from the delivery mechanism. As you can see in the image below, there are some common layers used in Clean Architecture to fit this goal:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/772/0*KXkQmsgDoJwd5Zhy.jpeg" /><figcaption>Source: blog.cleancoder.com</figcaption></figure><p>We will talk about what each layer does later, but first let’s see what is actually the purpose of this design, what is Clean Architecture trying to solve?</p><ul><li><strong>Independent of Frameworks</strong>. The architecture does not depend on the existence of some library of feature laden software. This allows you to use such frameworks as tools, rather than having to rely on their limited constraints.</li><li><strong>Testable</strong>. The purer a function, class or module is, the easier it is to predict the result. The business rules can be tested without the UI, Database, Web Server, or any other external element.</li><li><strong>Independent of UI</strong>. The UI can change easily, without changing the rest of the system. A Web UI could be replaced with a console UI, for example, without changing the business rules.</li><li><strong>Independent of Database</strong>. You can swap out MySQL or PostgreSQL, for MongoDB, BigTable, CouchDB, or something else. Your business rules are not bound to the database.</li><li><strong>Independent of any external element</strong>. If at some point your system need a library, another system or any element to connect, it should be easily assembled and modularized. In fact, for the system, this outer layer should be abstracted.</li></ul><h3>Rule of Clean Architecture</h3><p><em>Source code dependencies can only point inwards, nothing in an inner circle can know anything at all about something in an outer circle</em>.</p><p>The above statement is the core principle when designing with Clean Architecture. Each layer should not know anything at all about how will it consumed by the outer layer.</p><ol><li><strong>Entities / Models / Domains<br></strong>This layer encapsulate <em>Enterprise wide</em> business rules, least likely to change when something external changes.</li><li><strong>Use Cases<br></strong>Set of application business rules. We do not expect changes in this layer to affect the entities. We also do not expect this layer to be affected by changes to externalities such as the database, the UI, or any of the common frameworks.</li><li><strong>Interface Adapters<br></strong>Set of adapters that convert data from the format most convenient for the use cases and entities, to the format most convenient for some external agency such as Web, UI, Devices.</li><li><strong>Frameworks and Drivers<br></strong>The outermost layer is generally composed of frameworks and tools such as the Database, the Web Framework, etc. This layer is where all the details go. The Web is a detail. The database is a detail. We keep these things on the outside where they can do little harm.</li></ol><p>There’s no rule that says you must always have just these four. However, <em>The Dependency Rule</em> always applies. As you move inwards the level of abstraction increases.</p><h3>What are the Pros and Cons of using Clean Architecture?</h3><p>As we often hear the tired old cliché that says <em>“there is no silver bullet”</em>, yes this also applies in software architecture design. Let’s see what are the pros first:</p><ul><li>Consistent business logic through the app</li><li>Swappable data persistence, external systems integration, and presentation</li><li>Enables carving out a vertical feature slice into a separate service (microservice, nanoservice, serverless) without much difficulty</li><li>Promotes more testable design patterns (Core business logic tends to always be testable)</li></ul><p>And what are the cons?</p><ul><li>Requires more intentional design (you can’t reference data persistence APIs directly in your business logic)</li><li>Due to the business logic being agnostic of the outer layers you can loose optimizations in being closer to library features and implementations with the benefit of looser coupling</li><li>Can be overkill when only a CRUD app is needed</li></ul><h3>Conclusion</h3><p>Clean Architecture helps us to write maintainable code, by the fact that its layers are independent. Thus, changes to certain part tend not to interfere with the rest of the application. This architecture is ideal when you have a long-term project.</p><p>On the other hand, this requires more time to work on and takes longer time for just a simple MVP application to see the first results.</p><p>It is good to know Clean Architecture principles but it is better to use them wisely otherwise your solution can become overcomplicated with too many unnecessary abstractions and interfaces.</p><p>This post is summarized from some articles I read when trying to understand Clean Architecture, please read the links below for more:</p><ul><li><a href="https://blog.cleancoder.com/uncle-bob/2012/08/13/the-clean-architecture.html">https://blog.cleancoder.com/uncle-bob/2012/08/13/the-clean-architecture.html</a></li><li><a href="https://dev.to/hurricaneinteractive/clean-architecture-your-approach-5ae6">https://dev.to/hurricaneinteractive/clean-architecture-your-approach-5ae6</a></li><li><a href="https://dev.to/rubemfsv/clean-architecture-the-concept-behind-the-code-52do">https://dev.to/rubemfsv/clean-architecture-the-concept-behind-the-code-52do</a></li></ul><p>Thanks for reading this post, and feel free if you have any comments or feedbacks!</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=d030831a3e53" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Optimizing Costs of Google Kubernetes Engine]]></title>
            <link>https://medium.com/gravel-engineering/optimizing-costs-of-google-kubernetes-engine-7f47c0723b0?source=rss-98e2b0c14f89------2</link>
            <guid isPermaLink="false">https://medium.com/p/7f47c0723b0</guid>
            <category><![CDATA[cost-optimization]]></category>
            <category><![CDATA[infrastructure]]></category>
            <category><![CDATA[google-cloud-platform]]></category>
            <dc:creator><![CDATA[Yesa]]></dc:creator>
            <pubDate>Fri, 15 Sep 2023 06:26:11 GMT</pubDate>
            <atom:updated>2023-09-15T06:41:07.727Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*tsMt_DHyJvrlPi79" /><figcaption>Photo by <a href="https://unsplash.com/@cgbriggs19?utm_source=medium&amp;utm_medium=referral">Chris Briggs</a> on <a href="https://unsplash.com/?utm_source=medium&amp;utm_medium=referral">Unsplash</a></figcaption></figure><p>With increasing need for flexibility, scalability and cost-effectiveness, many organizations decided to migrate and leverage the benefits of cloud infrastructure. But ironically, cloud spending has become one of the biggest cost center for many businesses. While the cloud offers great advantages, it can also lead to higher expenses if not managed effectively.</p><p>This led us to take a closer look at how we were using cloud resources, especially in our Google Kubernetes Engine (GKE) clusters. Let’s dig into how we could optimize our GKE cost by cutting ~75% of spending.</p><h3><strong>Identify the Usage</strong></h3><p>Identifying resource consumption is the first step of optimization, we need to get information about how much resources requested by our services vs the actual usage. Fortunately, GKE provides us enough data to see, from optimization recommendations to utilization metrics.</p><ul><li>GKE cluster cost optimization recommendation: This is the fastest way to get the total utilizations in a cluster and recommended node size based on utilizations.</li><li>GKE workload cost optimization recommendation: This gives us quick actual usage vs requested resource of each workloads in certain time-range.</li><li>CPU &amp; Memory utilization metrics: This gives us a more detailed utilizations data of every workloads and a more flexible way to visualize it. Thankfully, GKE already comes with its monitoring dashboards, the most useful one in this case is <a href="https://console.cloud.google.com/monitoring/dashboards/integration/gke.gke-compute-resources-workload-view">GKE Compute Resources — Workload View</a>.</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*o3mVi2CBq112-fLYPxsK8A.png" /><figcaption>GKE Compute Resources — Workload View Dashboard</figcaption></figure><p>At the end of this step, we found out that our total utilizations never went above 10%. Additionally, almost all of the workloads didn’t even reach 10% of resources they requested, which conclude that we have been over-allocating resources.</p><h3>Optimization Plan</h3><p>After having the big picture of our usage, next step is optimization planning. For the context, we were using GKE cluster with 1 node pool (node autoscaler enabled) with average 5 nodes running with 8 vCPU and 32 GB memory in every nodes. We have 20 micro services running in this cluster, each with their fixed number of replicas. To give an idea, GCP charges about $263 per month for every node.</p><p>Since this is a production cluster, we should not aggressively change everything, so I started with some plan I have in mind:</p><p><strong>Enabling HPA (Horizontal Pod Autoscaler)</strong></p><p>For an unknown reason, pod autoscaling has not been activated in any of the workloads. Instead, a fixed number of replicas. After seeing the utilizations, it becomes evident that enabling autoscaling is essential for optimizing efficiency. I’m planning to use CPU utilization as the autoscaling metric with target 60% of utilizations, which means that our workloads has a 40% CPU buffer for handling requests while new replicas are spinning up.</p><pre>Target CPU Utilization: 60%</pre><p><strong>Workload Rightsizing</strong></p><p>This is an obvious way we need to do considering the low utilizations, but the sizes may vary on each workloads depend on their utilizations. I’m planning to create a new sizes which is based on average mean and max usage in the last 7 days, while keeping it at 60% of the new size.</p><pre>New CPU request formula:<br>- Average (mean, max usage) * 1000m / 60 * 100<br>- Rounded up to nearest 50<br>- Minimum value is 100m<br><br>New Memory request formula:<br>- Average (mean, max usage) * 1000M / 60 * 100<br>- Rounded up to nearest 50<br><br>New Limits formula:<br>- Use the same amount of memory for requests and limits <br>- The CPU limit will be set at twice the value of the CPU request</pre><p>Some of the decisions here are based on best practices for running cost-optimized GKE by Google which can be seen <a href="https://cloud.google.com/architecture/best-practices-for-running-cost-effective-kubernetes-applications-on-gke">here</a>.</p><p><strong>Node Rightsizing</strong></p><p>After enabling HPA and rightsizing workloads, there could be opportunity for us to rightsize the node after reviewing the actual usage.</p><h3>Time to Optimize!</h3><p>We began with <strong>enabling HPA</strong> on some low-traffic to high-traffic services, and completed it in 3 days. I used kubectl and <a href="https://github.com/robscott/kube-capacity">kube-capacity</a> to monitor the HPA target and resource utilizations. The outcome? Nearly all services now require only a single replica. After completing this, our cluster’s node count dropped drastically from 5 to 3 nodes! This change saved us around $526 per month!</p><pre>kubectl get hpa<br><br>kube-capacity --pods</pre><p>The next step is <strong>workload rightsizing</strong>. The strategy is still the same, we began with the low-traffic services to high-traffic ones and completed it in 3 days. I still use the same tools to monitor the utilizations and the services are still running well. This results in 1 more node reduction and $263 saving per month! Our node count has now dropped from 3 to 2!</p><p>After reviewing the total utilizations in a few days after workload rightsizing, we found out that we have an opportunity to <strong>rightsize the node</strong> as expected. The plan is to downsize node from 8 vCPU and 32 GB memory to 6 vCPU and 12 GB memory.</p><p>Rightsizing the node can be accomplished by creating a new node with new size and migrating all the workloads to the new node. Thanks to the kubectl cordon feature, this process can be executed without any downtime. I started with creating the new node pool with the new size.</p><p><strong>Isolate the old node pool</strong>:<strong> </strong>This operation marks the nodes in the existing node pool (old-pool) as <em>unschedulable</em>. Kubernetes stops scheduling new Pods to these nodes once we mark them as <em>unschedulable</em><strong>.</strong></p><pre>for node in $(kubectl get nodes -l cloud.google.com/gke-nodepool=old-pool -o=name); do<br>    kubectl cordon &quot;$node&quot;;<br>done</pre><p><strong>Drain the existing node pool: </strong>This operation relocates the workloads running on the nodes of the existing node pool (old-pool) gracefully.</p><pre>for node in $(kubectl get nodes -l cloud.google.com/gke-nodepool=old-pool -o=name); do<br>    kubectl drain --force --ignore-daemonsets --delete-emptydir-data --grace-period=10 &quot;$node&quot;;<br>done</pre><p>And finally we can delete the old node pool. This step saves us about $100 per month for each node!</p><h3>Conclusion</h3><p>At the end, we could save about 75% of our cost spending for GKE with no downtime.</p><p>In the fast-paced world of cloud infrastructure, striking a balance between cost and cloud efficiency is a never-ending process. It’s important to regularly review your usage and costs to identify new opportunities for savings. The cloud can unlock countless benefits, but it can also quietly drain your budget.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=7f47c0723b0" width="1" height="1" alt=""><hr><p><a href="https://medium.com/gravel-engineering/optimizing-costs-of-google-kubernetes-engine-7f47c0723b0">Optimizing Costs of Google Kubernetes Engine</a> was originally published in <a href="https://medium.com/gravel-engineering">Gravel Product &amp; Tech</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
    </channel>
</rss>