<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Saad Zaman on Medium]]></title>
        <description><![CDATA[Stories by Saad Zaman on Medium]]></description>
        <link>https://medium.com/@saadzaman?source=rss-4fc0318710ac------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/0*O2IozbNH4c0RS7c1.</url>
            <title>Stories by Saad Zaman on Medium</title>
            <link>https://medium.com/@saadzaman?source=rss-4fc0318710ac------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Wed, 18 Nov 2026 21:57:53 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@saadzaman/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[Frugally running monoliths on AWS through out the Start-up Journey]]></title>
            <link>https://medium.com/@saadzaman/frugally-running-monoliths-on-aws-through-out-the-start-up-journey-9399174fc87e?source=rss-4fc0318710ac------2</link>
            <guid isPermaLink="false">https://medium.com/p/9399174fc87e</guid>
            <category><![CDATA[containers]]></category>
            <category><![CDATA[cost-optimization]]></category>
            <category><![CDATA[monolithic-architecture]]></category>
            <category><![CDATA[aws]]></category>
            <category><![CDATA[startup]]></category>
            <dc:creator><![CDATA[Saad Zaman]]></dc:creator>
            <pubDate>Thu, 23 Mar 2023 04:39:39 GMT</pubDate>
            <atom:updated>2023-03-28T08:16:46.009Z</atom:updated>
            <content:encoded><![CDATA[<p>This blog sums up some steps and suggestions to take from Day 1 to design monoliths considering scalability and cost effectiveness in mind. This is a must read in cases where you have to deploy open source popular monoliths which are widely being used in the market. Like Magneto, Wordpress, Joomla, Drupal, Moodle , Self Hosted Bitbucket, Actitime, Jira, Redmine etc</p><h3>Why to consider this ?</h3><p>There are various myths and misconceptions clouding around developers who are working on monoliths. Most <strong>application developers</strong> due to their lack or fear of cloud, don’t explore the cloud services today, and usually start with a more unified design deploying everything on a single EC2 instance.</p><blockquote>Usually start up developers scale up their single EC2 instance. It takes some downtime however it works!!</blockquote><p>However it soon starts becoming a mess, which apart from the other things, starts adding substantial cost.</p><p>The servers are usually upsized according to the peak traffic load. For example if the peak load is 100CPUs for 30% of the time, the static server will be set to 100CPUs, however 70% of the time the resources will be being wasted.</p><p>So at some point in future they then contact a cloud architect or a system design expert. Who would then suggest some changes which would require a lot of effort to implement and test in that phase of the product.</p><h3><strong>Cost Optimisation Pillar</strong></h3><p>Leveraging cloud and following the AWS Pillars, the monolith should scale in and scale out whenever required from the first day it hits production.</p><p>For example for a LMS system operating in an school, the main usage time will be from 7am to 3pm. So effectively, the database, compute and storage systems should scale in after 3pm.</p><p>Similarly for an Ecommerce application operating in USA, selling fashion items. Most of the traffic on the website will be during the daytime.</p><p>Some systems might not be chunked on specific times of day. Then they should be designed such that they scale quickly as soon as the load hits the system.</p><p>I will break down some steps that the developers can take, which will help in scaling the infrastructure easily whenever required in their product journey.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*rcrke06Efkp2gOXp" /><figcaption><a href="https://medium.com/tradecraft-traction/5-phases-of-the-startup-lifecycle-morgan-brown-on-what-it-takes-to-grow-a-startup-50b4350f9d96">5 phases of startup lifecycle</a></figcaption></figure><h4>1) Identify major components in infrastructure</h4><p>So in this step, what we want to do is to identify every platform / service / component involved in the lifecycle of the request or processing of the workload.</p><p>For custom build application where the developers are making the applications from scratch, they know the complete flow of the code. So its easy in that case.</p><p>In case of most open source systems, their documentation gives a good insight of it. One simple way is to digest the config parameters the application exposes. Most of the applications default to configuration considering a single server in mind, however they have exposed it as a configurable parameters for it to be adjusted whenever required. <br> <br>As an example :</p><ul><li>The user will initiated a request from their browser</li><li>Request hits the code passing through the web server</li><li>Some data would be fetched from cache</li><li>Some data would be fetched from &amp; writted to database</li><li>A user session might be created</li><li>Some assets could be uploaded</li><li>An email service or some notification service might be used to trigger some notifications</li></ul><p>So in this example, we know that there are mutiple components being involved like</p><ol><li>Web Server which receives the request</li><li>Database</li><li>Session storage</li><li>Cache systems</li><li>File Storage</li><li>Email Service</li></ol><p>If there are static assets like javscript files, assets like photos, documents etc. We would also consider some type of <strong>7. Content Delivery System</strong></p><p>Usually a monolith application has the above components, in complex applications there are sometimes multiple layers of cache. Or diffferent types of cache for different purposes. It is also common for applications to <strong>use queues, message streaiming etc.</strong></p><p>So, inshort we want to identify what components does our application depends on.</p><h4>2) Strategise on how to deploy these components for development environment.</h4><p>As a personal rule of thumb, for developers while they are working on their machine, i would always strongly advise to run all components as containers.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*I25rzfQGeT_ZTrsjYBazlg.png" /><figcaption>Image taken from : <a href="https://blog.atulr.com/docker-local-environment/">https://blog.atulr.com/docker-local-environment/</a></figcaption></figure><p>To containerise everything on development environment, is something thats easy to do in 2023. As these days we get all popular services docker ready, the settings and configurations are tweaked for production. So you would never go wrong with them.</p><p>For example this one line code can run postgres in a container for you.</p><blockquote>docker run --name some-postgres -e POSTGRES_PASSWORD=mysecretpassword -d postgres</blockquote><p>going into the details on <a href="https://hub.docker.com/_/postgres">official docker hub page</a> you will see the customization parameters exposed for postgres.</p><blockquote>docker run -d \<br> --name some-postgres \<br> -e POSTGRES_PASSWORD=mysecretpassword \<br> -e PGDATA=/var/lib/postgresql/data/pgdata \<br> -v /custom/mount:/var/lib/postgresql/data \<br> postgres</blockquote><p>Similarly you can fine other components and deploy this on your local machine. <br><em>If you team has more experience, they can also look into Helm and Kubernetes to spawn the complete development environment quickly.</em></p><p>A major advantage using containers from the first day for development is that,</p><ol><li>You don’t face “ It works on my machine” ever from anyone in your team. As everyone in the team are using the same set of services with the exact same version.</li></ol><figure><img alt="" src="https://cdn-images-1.medium.com/max/1000/0*mKlZ4QjEQKapOigW.jpg" /></figure><p>2. Most importantly, this helps you write code and use the program how you intend to write it when the application has scaled to 100,000 users. This might seem trivial at first, however this distinction from Day 1 helps all developers understand what they need to look for where they need to optimise. And what are the limitations of the service, they need to take care of.</p><p>Often many developers use databases or file systems for cache and sessions. This works fine when everything is on a single node. Also sometimes the developers might have written logic not abstracting the type medium being used behind this. And when this ships to production and the DevOps tried to scale it horizontally, the application breaks.</p><p>At worst case scenario. A developer might be caching in a local filesystem, which is very quick and he might be just adding lines to file or rewriting the complete cache against a user or program in one file. When this gets reviewed, the developer would have to completely write this caching logic again for Redis.</p><p>If not this, the developer might have setup redis on his local machine, the version and the configuration settings might be different on his machine then on production. Which might show surprises when the application hits production.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/372/1*vnjrDt__dQB1YLg5JiY54A.png" /><figcaption>Containerised Deployments for <strong>Development &amp; BetaPhase</strong></figcaption></figure><h4>3) Moving to production, Find a sweet spot between managed and un-managed services.</h4><p>We can look for then critical and resource intensive components, which need to be scaled relatively more quickly to keep the application responsive.</p><p>From the listed components above, Database and Web Server are the two components that are always critical. So for this I would suggest to use AWS Managed Services for these two. <br> <br><strong>Database<br></strong>RDS or Aurora, even better Aurora Serverless in case of MySQL &amp; Postgres</p><p><strong>Compute</strong> <br>ECS with Fargate is the best option to take, for most of general use cases. Moreover, ECS is simple to adopt and works well for monolith applications.</p><p>So you are good to hit production with ECS as compute and RDS. For the rest of components, you can keep it on a single EC2 instance as separate containers.</p><p>So here with ECS &amp; Aurora Serverless, we would be paying for what we are using. And for other services, we can keep it even on a t2 small or t2 large instance.</p><blockquote>This keeps the bill to your lowest on initial days. This setup will probably work for you while you are in the MVP phase and are experimenting to get the right product fit.</blockquote><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*ehUZzxKJpeyxMIon6v99Bw.png" /><figcaption>Production in MVP / First 100 Customers</figcaption></figure><p>If you don’t expect much traffic on your production for first few months, but you want it live for your investors, or to showcase in pitches. Its necessary that you keep the system up and running efficiently at all times. However you are also conscious of the costs, at this point in the product journey. Its better to be<strong> frugal and be agile</strong> in making changes to the product at this point.</p><p>We should try to keep the infrastructure cost parallel to the actual users using the system. So if there are no users on the system, we want to keep the costs to minimum.</p><p>Other components, like Sessions and Cache, Redis is a popular option for it.<strong> Amazon Elasticache</strong> is the service for Redis, however Elasticache is a reltively expensive service. To initially save costs, we can deploy it ourselves on cheaper EC2 instances or as small Fargate task.</p><p>For Storage we can either use<strong> Amazon S3 </strong>if the application does not require POSIX compliant File Systems.</p><p>Otherwise, we can use <strong>Elastic File System</strong>, previously for a production application we had to provision the EFS with a certain bandwidth to get good performance. <a href="https://aws.amazon.com/blogs/aws/new-announcing-amazon-efs-elastic-throughput/">However as of 2023 we now have to option of Elastic Mode in EFS</a>, this enables EFS to automatically increase the bandwidth on demand. So you get a very good performance at all times and only pay for storage bandwidth being uses.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*jEallCYG3eb1ZY4zqrm9MA.png" /></figure><p>This has been something i did not see for a long time, we can seen storage system mostly static and this flexibility on EFS is unmatched</p><h4>4) Substitute with Managed Services</h4><p>Gradually as your application grows, you can gradually start shifting to managed services as required. We started using database as Managed, so we don’t have to do any major migration, and either it auto scales if its server less it runs on demand.</p><p>Other components usually do not save persistent data and can be easily swapped with minimal downtime.</p><p>So we have to decide weather to deploy the service as unmanaged or using managed AWS Services.</p><p>This decision would have to take in consideration some aspects :</p><ol><li>The budget you have</li><li>How much risk can you take with your end users</li><li>How critical are these components for you</li><li>How proficient is the team in tweaking these components for production and debugging if something goes wrong.</li><li>How complex is the scaling process of this component etc.</li><li>How complex will the step be if we decide to move this later to managed</li></ol><p><em>For example:</em> If you are running elastic search on AWS with AWS recommended production configurations. A decent setup would cost at least 600$ per month. However if its not a critical component for your application or you can tolerate some performance degradation on it, you can also run it as a container service on a M Large EC2 instance.</p><p>The diagram below can be another step on evolving the system, we are assuming in this complete article</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*l-iGqwd0F2fNys3q9xZjiQ.png" /><figcaption>Moving Cache, Elastic Search to Managed Services</figcaption></figure><h4>5) Monitor and Keep Evolving</h4><p>Either the services / components being used auto scales, if not then you should have defined standards and defined procedures on steps to scale a specific component.</p><p>Those procedures should contain the steps to scale,</p><ul><li>Whether or not it will require downtime ?</li><li>If yes how much downtime is required ?</li><li>Does the service store data? Is it persistent or non persistent ?</li><li>How will it be rewarmed ?</li><li>Will the endpoint change ?</li><li>Will the scaling require new code deployment</li></ul><p><strong>For ECS</strong> , Scalability is simple in most case, you just set a target CPU % and it spawns or kills the tasks to maintain that percentage.</p><p><strong>For Databases</strong>, if your workload is fine with how the serverless v2 operates thats a very easy to use Services that autoscales on demand. Other wise you can setup a DB Cluster, with writers and readers. And set readers to auto scale.</p><p>Lastly no matter what infrastructure combination you opt for, you need to make sure that you have defined thresholds against metrics of each components. Set the alarms on each component so that you are aware about whats happening in your application</p><blockquote>The last thing we want is getting notified about system degradation from the endusers.</blockquote><p>I will be writing in detail on what metrics to look in what type of workload. In short in context to monoliths</p><p><strong>ECS</strong>, you can set to track average<strong> </strong>CPU usage of containers.<br><strong>Database</strong>, we can set to track and put alarms on CPU utilisation, Deadlocks, Connections etc<br><strong>Queues</strong>, we can keep track number of visible messages in queue.</p><p>As the product grows and the user base grows, we might want to then continue moving parts in our infrastructure to a more stable and scalable approach.</p><p>Ideally ultimately we want to move everything to AWS Managed Services. So that we don’t have to worry about keeping them operational.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*tAUXHfo5Rh8izooOR4vzDg.png" /><figcaption>Here in this we have change ec2 with ecs so that it autoscales, added readers with scalability</figcaption></figure><blockquote>For compute to scale automatically, the components being shared need to be setup for expected performance.</blockquote><p>The article is based on my experience of working 10+ years with Startups in Silicon valley. Please feel free to disagree or correct me in comments :)</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*dPqsnn9ofoEJUQld.jpg" /></figure><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=9399174fc87e" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Deploying Magento 2 on ECS Cluster]]></title>
            <link>https://medium.com/@saadzaman/how-we-deploy-magento-2-as-cluster-on-aws-52ebbac193f6?source=rss-4fc0318710ac------2</link>
            <guid isPermaLink="false">https://medium.com/p/52ebbac193f6</guid>
            <category><![CDATA[magento]]></category>
            <category><![CDATA[cluster]]></category>
            <category><![CDATA[ecommerce]]></category>
            <category><![CDATA[ec]]></category>
            <category><![CDATA[aws]]></category>
            <dc:creator><![CDATA[Saad Zaman]]></dc:creator>
            <pubDate>Sat, 09 Jul 2022 00:03:06 GMT</pubDate>
            <atom:updated>2022-07-09T01:24:25.659Z</atom:updated>
            <content:encoded><![CDATA[<p>I will cover the steps we took to deploy Magento on AWS in a cluster. This post will cover the components and high level AWS services being used on each.</p><h3><strong>Compute</strong></h3><p>We had already containerised the Magento application, we will discuss this in more detail on the pipeline steps.</p><p>For computer we had options to chose between EKS and ECS, and then Fargate vs EC2.</p><p><strong>EKS vs ECS <br></strong>We avoided EKS because of the complexity and the list of tasks and stack that Kubernetes brings along. However most importantly our developer team was not familiar with kubernetes, and the client did not have a dedicated devops team.</p><p>We chose ECS mainly because its simple to use by the developers, they see whats going on and can debug basic stuff without any reliance on the DevOps team.</p><p><strong>Fargate vs EC2<br></strong>Although performance on EC2 is generally better than Fargate. We still chose fargate for simplicity. As we dont need to worry about old or stale ec2 instances running with out dated linux packages or linux kernel.</p><p>Another consideration was this that, we noticed fargate scaling is slightly more responsive then when we are using EC2 instances. ( Though on EC2 instances we can use warm instances for more responsive scaling )</p><p><strong>Multiple Services for better reliability <br></strong>We analysed our use case and noticed that different services / utilities are being heavily used on the Magento site. We can break this down in <br>1) <em>Frontend</em>: Which is the shop interface<br>2) <em>Backend</em> / Admin : This is where the shop admin logs in <br>3) <em>Cron</em> : The cron service which runs different tasks of Magento every minute<br>4) <em>API</em>: For our case these were mainly hits coming from odoo and ebay connectors<br>5) <em>SSH</em> : For several use cases, Magento developers would want access to ssh to debug or to fix issues on production. These would mainly be <em>/bin/magento</em> commands</p><p>So we decided to put all the extra work which is not directly related to consumer on a separate ECS services with its own <em>target group </em>and <em>auto scaling group</em>.</p><p>This isolated the experience which is the top most priority for the users. For the rest of the use cases, we created another service and using rules on Application Load Balancer we were able to direct the API requests and Admin requests to the separate fleet being the other ECS Service.</p><p><strong>TLDR;</strong></p><p>AWS ECS using AWS Fargate as compute<br>- 1 ECS Service for miscellaneous other purposes. (cron, ssh, admin)<br>- 1 ECS Service for serving site</p><h3>Session &amp; Caches</h3><p>Magento has different combinations of caches and session configurations that can be done. When we are on a single instance usually <strong>sessions</strong> are saved on filesystem. This is the default setting on Magento.</p><p>However, when we are doing a cluster deployment. The file storage which is used is shared between multiple nodes on the fleet. Hence its slower than local storage on the single machine. So as an obvious choice we have configured Redis to be used as session store.</p><p><strong>Cache</strong> configurations can be set similarly. However some thing to notice here is that, these cache settings do not contains static generated content or viewpreprocessed folders etc. These cache are data objects which are referenced at various places.</p><p><strong>TLDR;</strong></p><ol><li>AWS ElastiCache Redis service for frontend cache.</li><li>AWS ElastiCache Redis service for backend cache.</li><li>AWS ElastiCache Redis service for sessions.</li></ol><p><em>We did not implement Varnish as of yet. Its in the top of TODO list</em></p><h3><strong>Storage</strong></h3><p>For storing temporary files and media images, we had some options.</p><p><strong>S3</strong> — Using this would require us to use the fuse package, and add some custom logic as well on Magento end. <br>We did not see any value in using the block storage to save magento media files which were not to be publicly exposed or were to be made available as independent resources.</p><p><strong>Network File System </strong>This would require us to expose EBS Volume along with an EC2 instance over the network. This was a valid option, however it would require us to manage this EC2 instance, which we wanted to avoid.</p><p><strong>EFS </strong>Elastic File System was what we opted for media files and shared files among the nodes. It has all the perks of NFS along with managed automatic backups. It also has variable speed which can be instantly modified without any downtime. So scaling this is very easy as compared to NFS which would require downtime and modification of the instance,</p><p><strong>TLDR;</strong></p><p>AWS Elastic File System for shared media and /var directory</p><h3><strong>Elastic Search</strong></h3><p>This was a straight forward decision. We did not want to bring in complexity of deploying Elastic Search ourselves with all its complications. The previous setup of elastic search was deployed on a single instance and was just one failure away.</p><p>We had our mind set to choose managed services where ever applicable, so it was either Elastic Cloud which is Elastic’s own managed solution or AWS offering for Elastic Search. And we of course went with AWS Open Search. <br>Although we used the elastic search variant of it. You might want to check what versions of Elastic Search you want to use and wether its supported on AWS or not.</p><p>However, because Magento’s use of Elastic Search is fairly simple — So version of Elastic Search is not really an issue. We just went with the latest version AWS provided of Elastic Search.</p><p>A good thing was AWS launches Elastic Search in production ready state, with 2 master nodes and 3 worker nodes, this give a lot of reliability to it.</p><p><strong>TLDR;</strong></p><p>AWS OpenSearch ( Elastic Search )</p><h3><strong>Database</strong></h3><p>The database we wanted to setup, we wanted to manually scale it without downtime. And we wanted to have options to have readers, to <strong>manually scale </strong>the write node of the database with the least or no downtime. To have some level of reliability which means should at least be enough to stand <strong>outage of at least one availability zone</strong>.</p><p>In order to achieve both these things, we chose Aurora Multi AZ Setup, along with some basic cloud watch alarms at different CPU utilisation levels. When you are setting Multi AZ Cluster, you can promote a read replica in another availability zone as the master node, and this failover is done without or &lt; 2 seconds downtime usually.</p><p><strong>TLDR;</strong><br>AWS Multi AZ Aurora</p><h3>Diagram</h3><p>This is a simple diagram which gives an overview of main AWS services being used. Disclaimer : Please do not consider it a blueprint, as there are many minor details like security groups, iam roles, ecr, ecs etc missing.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/941/1*SzJX-2L1-Axed1VqeiTc6A.png" /><figcaption>Magento on AWS Overview</figcaption></figure><h3><strong>Deployment Process</strong></h3><p>This calls for its own post, and i will write this in detail soon. <br>We have dealt with all types of Magneto /bin/magento commands that run during the deployment process.</p><p><strong>TLDR;<br></strong>We used</p><ol><li>AWS Elastic Container Repository ( ECR )</li><li>AWS Code Pipeline</li></ol><h3>Magento specific issues</h3><p>We kept /pub/static and generated and view preprocessed in a different repository, which was pushed by developers. Generating this on demand was taking too much time, and was</p><p>Then we made sure the site goes to maintenance mode if there are any database changes being done by developers. In other words, does setup:upgrade need to be run.</p><p>Other than that we made sure scaling doesn&#39;t take longer, i.e new containers should not run di:compile again.</p><p>Overall we were able to deploy without downtime if there were no database changes.</p><p>Most of this was managed by a combination of entrypoint.sh file running while deploying the admin/cron service and while deploying the main app service.</p><p>Do leave your comments or reach me at szaman@folio3.com for any specific queries. Would be glad to hop on call to help you.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=52ebbac193f6" width="1" height="1" alt="">]]></content:encoded>
        </item>
    </channel>
</rss>