<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Khoa Le on Medium]]></title>
        <description><![CDATA[Stories by Khoa Le on Medium]]></description>
        <link>https://medium.com/@tkhoa2711?source=rss-ef729da18394------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/0*yCtwofJgmYPVsgYU.jpeg</url>
            <title>Stories by Khoa Le on Medium</title>
            <link>https://medium.com/@tkhoa2711?source=rss-ef729da18394------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Wed, 18 Nov 2026 10:28:49 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@tkhoa2711/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[Our way of doing code review and pull request]]></title>
            <link>https://medium.com/homes-engineering/our-way-of-doing-code-review-and-pull-request-cb99f5754207?source=rss-ef729da18394------2</link>
            <guid isPermaLink="false">https://medium.com/p/cb99f5754207</guid>
            <category><![CDATA[team-culture]]></category>
            <category><![CDATA[code-review]]></category>
            <category><![CDATA[pull-request]]></category>
            <dc:creator><![CDATA[Khoa Le]]></dc:creator>
            <pubDate>Tue, 11 Jan 2022 02:51:16 GMT</pubDate>
            <atom:updated>2022-01-11T02:51:53.498Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*tss5R1eROKMuNqrJ" /><figcaption>Photo by <a href="https://unsplash.com/@alvarordesign?utm_source=medium&amp;utm_medium=referral">Alvaro Reyes</a> on <a href="https://unsplash.com/?utm_source=medium&amp;utm_medium=referral">Unsplash</a></figcaption></figure><p>At <a href="http://homes.com.au/">homes.com.au</a>, our engineering team has developed our own approach to conduct code reviews and pull requests (PRs) that enables us to move fast and not break things (as much as we could, that is).</p><h3>TL;DR</h3><p>We embrace a hybrid approach of trunk-based development and PR model. There is no hard and fast rules. We will use whatever makes the most sense in a given circumstance. We acknowledge that it is okay to <strong>not</strong> use PRs when appropriate.</p><p>However, we encourage people to use a PR, and review code, if:</p><ul><li>There is something in the PR that you want to <strong>share</strong> with other engineers</li><li>You want to <strong>learn</strong> from each other’s code</li><li>You want to <strong>discuss</strong> the solution and ideas with the team, perhaps even coming up with a better alternative</li></ul><h3>Background</h3><p>To give it a bit more context, prior to settling on an agreed approach, we enforced PRs for all changes to our codebase, via the repo settings on BitBucket. It worked fine — but not great — as common problems started to emerge; such as PRs were getting blocked from merging due to slow review, pushing hot fixes for production issues took longer as they needed to be approved and merged.</p><p>When we migrated to GitHub (one of the best decision we ever made!), we took the opportunity to rethink our process — effectively how we could make it more enjoyable and efficient. There were benefits to the PR model that we didn’t want to lose. For instance, the ability to have another pair of eyes to review the code to ensure code quality, catch unnoticed bugs or missed requirements.</p><p>At the same time, alternative approaches such as <a href="https://trunkbaseddevelopment.com/">trunk-based development</a> was lucrative as it would enable us to move fast — given we were an early stage startup. As reviewing PR can be time consuming, it can be frustrating for the PR author to wait for approval as well as resolving discussions — due to the asynchronous nature of it. Trunk-based development did sound attractive, but we felt that we were not ready for it yet. It comes with its own problems and requires a certain level of discipline from the team to adopt it well.</p><p>In the end, we agreed on a hybrid approach and trusted our people to pick the most appropriate choice. It also aligned with a principle in our <a href="https://homes.com.au/engineering/manifesto">engineering manifesto</a>:</p><blockquote><em>“We give our people autonomy to make the right decisions.”</em></blockquote><blockquote><a href="https://homes.com.au/engineering/manifesto">— Engineering Manifesto @ Homes.com.au</a></blockquote><h3>Guidelines to write a good pull request</h3><p>No two pull requests are equal. There are ways to improve them, such as adding more context, that makes reviewing them a more satisfactory experience. Here are some guidelines we look out for to make a better PR:</p><ul><li>Meaningful title and description</li><li>The smaller the change the better. Would be even better if it is <a href="https://en.wikipedia.org/wiki/Atomic_commit">atomic</a></li><li>Well-maintained commit history</li><li>Add visuals such as images, videos or diagrams whenever possible</li></ul><p>Most importantly, at the minimum engineers should explain <strong>why</strong> they are making this change.</p><h3>Suggested format</h3><p>Following the broad guidelines mentioned above, we came up with a suggested structure to the PR with the aim that it would help articulate the changes. Having said that, these are still suggestions and not every point must be included every time.</p><p><strong>The why</strong></p><ul><li>Why is this change needed in the first place?</li><li>What problem does it solve?</li><li>Any background or context the reviewers need to know?</li></ul><p><strong>The what</strong></p><ul><li>What was changed?</li></ul><p><strong>The how</strong></p><ul><li>Reasons for choosing this approach</li><li>Have we considered any other approach?</li></ul><p><strong>The bonus</strong></p><ul><li>What else do reviewers need to know?</li><li>Any gotchas?</li><li>Is something still WIP?</li><li>Links to a reference materials — such as articles or (internal) documentation?</li></ul><h3>Examples</h3><p>An example speaks a thousand words so we have 2 examples of 2 PR descriptions that we will try to apply the principles mentioned previously to gradually improve them.</p><p><strong>Example #1</strong>: a simple UI change</p><p>👎</p><blockquote><em>Update margin of the map component</em></blockquote><p>👍</p><blockquote><em>Update margin of the map component so it is in the centre of the page</em></blockquote><p>👍👍</p><blockquote><em>Update margin of the map component so it is in the centre of the page based on peer review. Confirmed the design with the Design team.</em></blockquote><p><strong>Example #2</strong>: a more complicated change</p><p>👎👎</p><blockquote><em>Set lambda timeout to 10 seconds</em></blockquote><p>👎</p><blockquote><em>Increase lambda timeout from 5 to 10 seconds</em></blockquote><p>👍</p><blockquote><em>Increase lambda timeout from 5 to 10 seconds.</em></blockquote><blockquote><em>The lambda was timing out intermittently when processing large files.</em></blockquote><p>👍👍</p><blockquote><em>Increase lambda timeout from 5 to 10 seconds.</em></blockquote><blockquote><em>The lambda was timing out intermittently when processing large files.</em></blockquote><blockquote><em>I’ve tried to increase the memory for the lambda but it didn’t help since we’re already processing the files by chunks. The file server is not controlled by us so there’s not much we can do on the other side of processing. Increasing timeout is the only option.</em></blockquote><blockquote><em>We will probably need to implement a retry mechanism when errors happen. But it seems to be quite complicated; I would need to learn how to implement it first. Will do that in a separate PR.</em></blockquote><h3>Guidelines to code review</h3><p><strong>Respect other people’s time</strong> — from both sides of a PR:</p><ul><li>Reviewing a PR can be time consuming</li><li>Waiting for a PR to be reviewed can block other tasks</li></ul><p>Always provide <strong>constructive</strong> <strong>feedback:</strong></p><p>👎</p><blockquote><em>This is just wrong. Why don’t you use X to implement this?</em></blockquote><p>👍</p><blockquote><em>I suggest we use X to implement this since it will be more concise and actually is an idiomatic pattern in Y language. See the docs here [provide link].</em></blockquote><p><strong>Keep the ego out</strong> of code reviews:</p><ul><li>Seniority doesn’t always mean being right</li><li>Provide reasons, not feelings</li></ul><p><strong>Feel empowered to defend your solution</strong>, with clear reasonings and probably links to articles or docs to support your point.</p><p><strong>When in doubt, ask</strong>. There’s no stupid question.</p><p>Written arguments can be easily misunderstood, <strong>resort to a quick call or in-person chat</strong> if arguments get heated, comments/replies drag on, or there are too many “I don’t get it” type comments. Post a follow-up comment summarising the discussion.</p><p>If suggesting an alternative approach, <strong>be specific</strong> what that should be. Even better if concrete examples are given.</p><p>If there are shortcuts or temporary solutions made, add <strong>explicit comments and/or TODOs</strong> so other people know about them and there are records of them in the code. No record means it will never be addressed — like a tech debt.</p><p>Review the PR keeping the <strong>bigger picture</strong> in mind. In a complicated system, applications (or services) are often coupled. For example:</p><ul><li>Changing a field type in an object model can affect other applications using that model. Make sure you’ve checked all other repositories that may use that model.</li><li>If it’s a breaking change, consider how the system will behave during deployment. Will there be any downtime. If there’s a failure, how do we rollback. We must consider all possible scenarios. Reliability and availability don’t come for free easily.</li></ul><p>You don’t need to be a senior engineer to review a senior engineer’s code or a frontend dev to review frontend code. Everyone is free to review other people’s code, even if the purpose is learning.</p><h3>Conclusion</h3><p>It has been 3 months since we started applying these practices and the feedback from the team members have been positive. Straightforward and non-code-related changes can be committed directly to the main branch. It results in less wait time for such trivial changes to be merged and enforces another principle in our <a href="https://homes.com.au/engineering/manifesto">engineering manifesto</a>:</p><blockquote><em>“We take full ownership of the things we build.”</em></blockquote><p>That means more happy engineers 😃. Otherwise, the team still actively use PRs and reap the benefits of high-quality code reviews.</p><p>In conclusion, we are satisfied with the choices we’ve made — adopting best practices from the industry and adjusting them to fit our team dynamics. We understand this is not the end and that, as we grow, we may need to revise the process further.</p><p><em>Thanks to </em><a href="https://medium.com/u/659c0089c254"><em>Benny Thomas</em></a><em> for reviewing this article.</em></p><p><em>We are also actively looking for awesome engineers to join our engineering team. If you share the same view on these values and are interested in solving exciting challenges with us, please get in touch </em><a href="https://www.linkedin.com/feed/update/urn:li:activity:6861441278274629632/"><em>https://www.linkedin.com/feed/update/urn:li:activity:6861441278274629632/</em></a><em>.</em></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=cb99f5754207" width="1" height="1" alt=""><hr><p><a href="https://medium.com/homes-engineering/our-way-of-doing-code-review-and-pull-request-cb99f5754207">Our way of doing code review and pull request</a> was originally published in <a href="https://medium.com/homes-engineering">Engineering @ Homes.com.au</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Building a real-time image processing solution at edge]]></title>
            <link>https://medium.com/homes-engineering/building-a-real-time-image-processing-solution-at-edge-c9918c743076?source=rss-ef729da18394------2</link>
            <guid isPermaLink="false">https://medium.com/p/c9918c743076</guid>
            <category><![CDATA[realtime-image-processing]]></category>
            <category><![CDATA[edge-computing]]></category>
            <category><![CDATA[aws]]></category>
            <category><![CDATA[aws-lambda]]></category>
            <category><![CDATA[image-processing]]></category>
            <dc:creator><![CDATA[Khoa Le]]></dc:creator>
            <pubDate>Wed, 17 Nov 2021 05:26:24 GMT</pubDate>
            <atom:updated>2021-11-17T05:26:24.234Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*obkFuuk6ZuRDilD3" /><figcaption>Photo by <a href="https://unsplash.com/@tomrumble?utm_source=medium&amp;utm_medium=referral">Tom Rumble</a> on <a href="https://unsplash.com/?utm_source=medium&amp;utm_medium=referral">Unsplash</a></figcaption></figure><h3>Background</h3><p><a href="http://homes.com.au/">Homes.com.au</a> is a proptech startup based in Australia. Our goal is to make it easy for Australians to buy or rent their next home.</p><p>We currently process over 3 million unique images each month, most of them are of properties that are listed on our site. These images are sourced from different channels — mainly uploaded via our customers — and they differ in quality and size. We needed to ensure that there was conformity when presenting these images to users of the website — taking the device on which they were viewing the site into consideration.</p><p>Our initial attempt was to do this using an existing provider but found ourselves unsatisfied with the user experience that we delivered; so set ourselves an engineering challenge to “do better”. The following is an overview of our journey in exploring and delivering on that challenge.</p><h3>What’s the problem we’re trying to solve?</h3><p>We had been using a popular 3rd-party image service provider, Cloudinary, to transform and serve images to the end user. When images were requested, Cloudinary fetched them from our self-maintained S3 bucket and processed them before responding back to users. One of the benefits of using Cloudinary was the ability to cache “already fetched” images but what we soon found was that first-time fetching of images were surprisingly slow resulting in users spending a non-trivial amount of time waiting for images to load. Pre-caching these images as they came in to our platform was an option but that came at a financial cost.</p><p>We found Cloudinary to be rather expensive; its linear pricing model did not fit well with our <em>forecasted</em> growth in website traffic. It would have cost us ~$400 USD per 1 million images on just initial transformations. Any additional transformations and fetches would have only increased that cost.</p><p>So… we set out ourselves some goals:</p><ul><li>Achieve fast image load time with p99 &lt; 100ms</li><li>Cut current costs significantly</li><li>Improve data governance; we control how, where and when our data is stored</li><li>Connected infrastructure, we would like to pre-cache our images using existing AWS services</li></ul><h3>Proposed solution</h3><p>It basically came down to two options:</p><ul><li>Consider a different 3rd party service, or</li><li>Implement an in-house solution</li></ul><p>We knew that we would not need to do anything fancy with the images and hence, we felt, processing them ourselves should be relatively easy to achieve. Picking another third-party solution would not reduce the cost that much. Therefore we decided to build our own image service provider.</p><p>After a little bit of research we concluded that the most straightforward solution would be to store the images, process them and host them on a CDN. In the AWS world, the solutions for those three can be mapped to S3, Lambda@Edge and CloudFront.</p><p>Knowing that we weren’t really “breaking any new ground” with our chosen approach; some further research quickly led us to a few articles and blog posts about how other people have also solved similar problems:</p><ul><li><a href="https://awesomeopensource.com/projects/resize-images">https://awesomeopensource.com/projects/resize-images</a></li><li><a href="https://dev.to/horaceshmorace/an-open-source-and-completely-serverless-image-resizing-service-in-aws-589b">https://dev.to/horaceshmorace/an-open-source-and-completely-serverless-image-resizing-service-in-aws-589b</a></li><li><a href="https://aws.amazon.com/blogs/networking-and-content-delivery/resizing-images-with-amazon-cloudfront-lambdaedge-aws-cdn-blog/">https://aws.amazon.com/blogs/networking-and-content-delivery/resizing-images-with-amazon-cloudfront-lambdaedge-aws-cdn-blog/</a></li></ul><p>Although the principles and ideas can be applied to other platforms, we chose to focus on AWS for this implementation. CloudFront + S3 was a no-brainer when it came to choosing a cost-effective and reliable CDN solution. Real-time image transformation is listed by AWS themselves as one of the <a href="https://aws.amazon.com/lambda/edge/#Real-time_Image_Transformation">use cases of Lambda@Edge</a>.</p><p>We are a fan of infrastructure as code and therefore used Terraform to manage and automate deployments of these infrastructure components (e.g. CloudFront, S3, Lambda).</p><h3>Architecture overview</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*1jNWYUh_unULX0_cT9Ejyw.jpeg" /><figcaption>Overview of the architecture</figcaption></figure><p>This is roughly what the architecture looks like. CloudFront as a CDN provides 4 trigger points during the life cycle of a request such that developers can hook their own Lambda@Edge functions to them and manipulate the request/response. We have two Lambda@Edge functions and one normal Lambda:</p><ul><li>One that intercepts requests for images to validate the request and determine appropriate image formats</li><li>One to process images on the fly when the image with desired size and format is not found</li><li>One to process images asynchronously when a new image is available</li></ul><p>The third lambda was needed due to the way those images are made available to our system. We received them from an external system that we didn’t control.</p><h3>Pros and cons</h3><p>Pros:</p><ul><li>We fully own the solution and will be able to customise it as we’d like</li><li>Eliminate features that we don’t need and their associated cost</li><li>Better performance and lower cost</li><li>Finally, as always, expanding our technical knowledge in this area</li></ul><p>Cons:</p><ul><li>Development and maintenance cost. However, we will see these more like <em>pros</em> as their benefit outweighs the cost tremendously.</li></ul><p>There are also limitations and restrictions imposed on Lambda@Edge (the full list is <a href="https://docs.aws.amazon.com/AmazonCloudFront/latest/DeveloperGuide/edge-functions-restrictions.html">here</a>) as of this writing:</p><ul><li>Only Python and Node.js runtime are supported</li><li>No support for dead letter queue</li><li>No support for environment variables</li><li>It can’t be configured to access resources inside a VPC</li><li>Limit on the request body size</li><li>Limit to 10,000 request/second (can be increased)</li><li>The function must be deployed to us-east-1 region, as well as some related resources</li><li>Often a numbered version of the lambda must be used, not the the convenient $LATEST alias. This may not sound like a big deal but it has been a deal breaker in the past for a fully-automated IaC solution.</li></ul><p>Fortunately, none of these limitations were a major issue for us. Having said that, we would need to monitor the solution closely, especially the request traffic when our user base grows. Any throttling indicates we are hitting that limit of requests per second.</p><h3>Cost analysis</h3><p>In order to compare the cost between solutions, we made some assumptions about our usage for a month:</p><ul><li>Average image size: 50KB</li><li>Traffic — 10 million requests for images, 10% involves the original images to be processed while 90% of the traffic should be served via the CDN cache</li><li>Network — 500GB bandwidth used</li><li>Storage: 50GB * 7 different sizes = 350GB</li></ul><p>Also for the new solution, we assume:</p><ul><li>The AWS pricing was obtained for Sydney region</li><li>Average duration for viewer request lambda is 5ms</li><li>Average duration to process an image is 100ms</li><li>We preprocess 7 different sizes of an image and store them all in S3. This would result in 7 million images being stored in S3</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*kSyuimcGkSzqvvxJEBnnsQ.png" /><figcaption>Cost comparison</figcaption></figure><p>The AWS solution is a clear winner here at just ~20% of the cost.</p><p>Please note that there are other cost items that we intentionally not calculating due to its complication and which we know shouldn’t affect our comparison that much. Some of those are logging, CloudFront data transfer out to origin.</p><h3>What we need to consider?</h3><p>As always, there were several consideration we need to make for our solution.</p><ul><li>How will we process images on the fly? This is required for existing images as we transition to the new approach.</li><li>How will we process and store images we receive going forward? Should we process them all in advance so that we wouldn’t need to do it again in real time? A trade-off would have to be made between storage cost vs real-time performance.</li><li>Which image formats are we going to support? We also need to consider various browsers on both desktop and mobile.</li></ul><h3>Prototype</h3><h3>Choosing the language</h3><p>At the time of building this, Lambda@Edge only supported Node.js and Python <a href="https://docs.aws.amazon.com/AmazonCloudFront/latest/DeveloperGuide/edge-functions-restrictions.html">runtimes</a>. While it’s possible to use one language for the Lambda@Edge and another, such as Go, for the normal Lambda, we decided to be consistent and only use one language for both of our Lambda functions for now. Any premature optimisation is undesirable until being benchmarked properly.</p><p>Node.js was selected since it is a popular language of choice for edge functions such as <a href="https://workers.cloudflare.com/">Cloudflare Workers</a> and <a href="https://developer.akamai.com/akamai-edgeworkers-overview">Akamai EdgeWorkers</a>. Many of our engineers were also more familiar with Node.js more than Python, so it was a natural choice <em>for us</em>.</p><h3>Choosing the image processing library</h3><p>While looking at existing Node.js libraries for image processing, <a href="https://sharp.pixelplumbing.com/">sharp</a> came up as a strong candidate. It uses <a href="https://github.com/libvips/libvips">libvips</a> under the hood. It is praised for its high performance, support for several image formats as well as transformation operations on them. There is an existing benchmark of its performance against other popular libraries <a href="https://sharp.pixelplumbing.com/performance">here</a>.</p><h3>Building the prototype</h3><p>We started building out the prototype by creating the two lambdas following this article for a few reasons:<br><a href="https://aws.amazon.com/blogs/networking-and-content-delivery/resizing-images-with-amazon-cloudfront-lambdaedge-aws-cdn-blog/">https://aws.amazon.com/blogs/networking-and-content-delivery/resizing-images-with-amazon-cloudfront-lambdaedge-aws-cdn-blog/</a></p><ul><li>It was authoritative i.e. written by AWS, even though it was three years old</li><li>The implementation was written in Node.js</li><li>They also used sharp to process images</li></ul><p>It’s almost as if we were copying them and modifying it to fit our scenario!</p><p>Instead of CloudFormation, we used Terraform for Infrastructure-as-Code. There used to be an issue with Terraform (and also Serverless frameworks) not handling the full deployment lifecycle of Lambda@Edge completely. For instance, when a new version of the Lambda@Edge is published, developers would have to attach the newer version to the CloudFront distribution manually. That’d be a real deal-breaker for our continuous deployment setup. Luckily, that issue had been solved for Terraform at the time, enabling us to fully automate the deployment pipeline.</p><p>Here’s the code snippet for resizing the images:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/a9feada2d33270c27fbee11acf6d7142/href">https://medium.com/media/a9feada2d33270c27fbee11acf6d7142/href</a></iframe><h3>Testing image quality</h3><p>One thing we need to verify is the image quality of the new solution. Obviously we wanted to keep the image quality as much the same as possible. Any degradation would affect the user experience. We compared the image quality using Structural Similarity Index (SSIM) method.</p><ul><li><strong>Size</strong> — The new solution’s JPEG output is comparable to Cloudinary’s WebP, the size is also smaller for bigger images (+/-20%), Cloudinary’s WebP has smaller size for smaller images (+/-20%)</li><li><strong>Blockiness</strong> — Both solution performed relatively well</li><li><strong>Ringing</strong> — The old solution has considerable more ringing artefacts</li><li><strong>Aliasing</strong> — None for both (no upsizing)</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/831/1*L3zPMT6aCqDK6sYXesAqbg.jpeg" /><figcaption>Image quality comparison</figcaption></figure><h3>Implementation</h3><h3>Feature flag</h3><p>In order to test the new solution in staging and production, we used a feature flag to control the behaviour on the client side. We use <a href="https://launchdarkly.com/">Launch Darkly</a> as our feature flag provider. It allows us to enable the flag per environment, and at specific user level. This comes in handy when we tested the feature in production by turning it on for all of our internal users, including engineering and product team. Of course, we would need to login with our account so the app is able to determine our user type given the account email.</p><p>We also tested for backward compatibility by switching back and forth between the old and new approaches.</p><h3>Choosing the S3 bucket directory and URL structure</h3><p>We tried to structure our S3 bucket in the most pragmatic way to our current situation, but also considering any future extension, such as hosting images for a different product or other types of static assets using the same solution. Therefore, we chose to place all images for our current site in the img-homes-com-au folder.</p><pre>.<br>└── img-homes-com-au<br>    ├── 32<br>    ├── 64<br>    ├── 128<br>    ├── 384<br>    ├── 640<br>    ├── 1080<br>    └── 1920</pre><p>At the same time, each images will exist in various fixed sizes. We decided to store each version of a particular size in a particular sub-folder. One small benefit of doing so is the ability to keep the exact original image name in each sub-folder. And we can easily check for discrepancy in the number of images in each of the those sub-folders without having to do fancy filtering with S3 operations.</p><h3>Choosing the image formats</h3><p>At the beginning, we considered using <a href="https://developers.google.com/speed/webp">WebP</a> as it is a modern image format that provides superior lossless and lossy compression for images on the web. However, we had to take into account its support from different browsers and devices. On iOS, WebP is only supported on Safari from iOS 14.0 onwards, which is could be big problem for us — which means users on older iOS would have issue using our site effectively.</p><p>This article <a href="https://siipo.la/blog/is-webp-really-better-than-jpeg">https://siipo.la/blog/is-webp-really-better-than-jpeg</a> thoroughly analysed the difference the two formats produce under various circumstances. Long story short, it depends on the use case.</p><p>We decided to stick with JPEG for the time being. Device compatibility was more vital than performance to us at this stage. By using the mozjpeg engine for processing, we were able to preserve high image quality with comparable performance to WebP from our initial testing.</p><h3>Processing images in the background</h3><p>When the images are made available to us, we decided to preprocess them so that it’d be faster to serve them to the users when being requested.</p><p>For existing images, they are not preprocessed and hence need to be converted on the fly first. The real-time image processor then triggers the background processor lambda to process for the remaining sizes — in the background — so they will be ready for future requests.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1012/1*IP35A6Gv1yaz0GXRNfIEAQ.jpeg" /><figcaption>Real-time image processing flow</figcaption></figure><h3>Hardening the infrastructure</h3><p>We believe in good security and do our best to learn and follow best practices in the industry. The following are what we did to enhance our infrastructure:</p><ul><li>Enabled public access blocking for the S3 bucket</li><li>Only allowed access to S3 bucket via the designated CloudFront distribution</li><li>Enabled server-side encrypt for S3 data by default, including the images and CloudFront access logs</li><li>Disabled root object in CloudFront distribution to prevent listing objects in an S3 directory</li><li>Redirected all traffic to CloudFront to HTTPS</li><li>Restricted HTTP methods to GET, HEAD and OPTIONS only</li></ul><h3>Final architecture</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*SywxA0DXghQvJlT0kkjanA.jpeg" /><figcaption>Final architecture diagram</figcaption></figure><h3>Performance benchmarking</h3><h3>Measuring latency</h3><p>We used <a href="https://k6.io/">k6</a> to test the performance of the new solution against the old one. Essentially we wrote a script to fire requests continuously at a given URL and measure its latency.</p><p>The full result provided a good details of all the metrics:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*yVE-QXcC3shSpR6Pi_kUOA.png" /><figcaption>k6 test result</figcaption></figure><p>The final result was:</p><ul><li>New solution: P95 = 67ms</li><li>Old solution: P95 = 750ms</li></ul><p>Looking back at the result, we realised that the test configuration wasn’t at a desirable size. The script simulated 10 (virtual) users and sending 10 requests each — which is relatively small for a proper benchmarking exercise. This is a lesson learned to improve our benchmarking approach in the future.</p><h3>Real-life performance</h3><p>We also did a “realistic test” by simply browsing the website and inspecting the network requests <em>by eyes 👀</em>. In addition, by observing the image load time from the user perspective we could tell a significant improvement made.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/0bc464c2d79df5a4ef046a080e46dfdf/href">https://medium.com/media/0bc464c2d79df5a4ef046a080e46dfdf/href</a></iframe><h3>Further optimisation</h3><p>From the user interaction point of view, we knew users will likely to perform certain actions. For example, when users open the image carousel of a property, they will certainly scroll through the images. And we can optimise our implementation to cater for such behaviour.</p><ul><li>Increased react-virtualized total slide counts to preload more images</li><li>Prioritised loading images when users enter gallery mode</li><li>Prioritised loading images of those properties that are visible in the carousel of the current page</li></ul><p>We also looked at optimising the Lambda functions:</p><ul><li>Increased max memory for the real-time image processor to 1769 MB to fully utilise the power of 1 vCPU (see <a href="https://docs.aws.amazon.com/lambda/latest/dg/configuration-function-common.html#configuration-memory-console">doc</a> for reference). While lambdas for viewer requests and viewer responses are limited to 128 MB memory, lambdas attached to origin requests and responses don’t have such limitation. The timeout is also higher at 30s.Unfortunately, provisioned concurrency is not supported for Lambda@Edge</li><li>We did however set the reserved concurrency for background image processor, such that during a sudden influx of new images it wouldn’t suck up all the quota for concurrent execution per region</li></ul><h3>Monitoring</h3><p>We strongly believe no project ever goes complete without proper monitoring and alert setup. Observability of a system is just as important as building it. Without observability, we wouldn’t be able to understand how our system is performing or knowing things going wrong when <a href="https://en.wikipedia.org/wiki/Murphy%27s_law">Murphy’s Law</a> kicks in. We used Sumo Logic for centralised logging, metrics dashboards, as well as CloudWatch for alerts. This setup is worth another article to discuss on its own.</p><h3>Others</h3><h3>Real cost after going live</h3><p>This is the daily cost breakdown by service for the last month, obtained with AWS Cost Explorer.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*MYxKn4h8xJq1pG5OxY-2jA.png" /><figcaption>Cost breakdown for Oct 2021 from AWS Cost Explorer</figcaption></figure><p>It costs us ~$135 for the month, with about 3 million images processed, stored and served to the users. It could have been $2000+ if we were still using Cloudinary, due to its billing structure. As we could see, S3 and Lambda each takes about half of the total cost, while CloudFront costs virtually nothing.</p><h3>Extra-large image causing out-of-memory errors</h3><p>Processing images can take a lot of memory, especially when the image dimensions are large. For example, we once had an image with a size of 6717 x 14133 pixels (what a size!), which exceeds the pixel limit. Moreover, it would result in an out-of-memory error when we attempt to resize the image into 7 different sizes in parallel.</p><p>We could observe this error from the logs:</p><pre>width: 6717 [Error: Input image exceeds pixel limit]</pre><p>If the lambda execution is terminated due to out-of-memory, the log would look like this:</p><pre>Error: Runtime exited with error: signal: killed<br>Runtime.ExitError</pre><p>After this incident, we switched the non-realtime image processor lambda to resize the images sequentially, also increased lambda memory and timeout (to 60s) to accommodate those edge cases. This reduced most of our errors but we may need to refactor the lambda to be more performant if it turns out to be not enough.</p><h3>Conclusion</h3><p>In the end, we were very happy with the outcome. We set out to address not only a technical challenge, but also a problem affecting the experience of our users. We were able to build out a solution from scratch that is cheaper, faster and more customisable.</p><p>We will also open-source our solution at some point in the future. Image hosting and transformation are common tasks that someone will most likely come across; and we hope to help the developer community with our small experience in this matter.</p><h3>References</h3><p>Lambda@Edge design best practices <a href="https://aws.amazon.com/blogs/networking-and-content-delivery/lambdaedge-design-best-practices/">https://aws.amazon.com/blogs/networking-and-content-delivery/lambdaedge-design-best-practices/</a></p><p>Optimizing S3 performance <a href="https://docs.aws.amazon.com/AmazonS3/latest/userguide/optimizing-performance.html">https://docs.aws.amazon.com/AmazonS3/latest/userguide/optimizing-performance.html</a></p><p><em>Special thanks to </em><a href="https://medium.com/u/f50562c7af62"><em>Chris Chan</em></a><em> for working together with me on this project and contributing to this article, </em><a href="https://medium.com/u/659c0089c254"><em>Benny Thomas</em></a><em> for reviewing and editing this article.</em></p><p><em>We are also actively looking for awesome engineers to join our engineering team. If you are interested in solving similar challenges, please get in touch </em><a href="https://www.linkedin.com/feed/update/urn:li:activity:6861441278274629632/"><em>https://www.linkedin.com/feed/update/urn:li:activity:6861441278274629632/</em></a><em>.</em></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=c9918c743076" width="1" height="1" alt=""><hr><p><a href="https://medium.com/homes-engineering/building-a-real-time-image-processing-solution-at-edge-c9918c743076">Building a real-time image processing solution at edge</a> was originally published in <a href="https://medium.com/homes-engineering">Engineering @ Homes.com.au</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
    </channel>
</rss>