<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Hussain Ali Akbar on Medium]]></title>
        <description><![CDATA[Stories by Hussain Ali Akbar on Medium]]></description>
        <link>https://medium.com/@HussainAliAkbar?source=rss-3805c8ea5f9------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/1*0rLUQn9wBiQqQ2MEVfXHqw.jpeg</url>
            <title>Stories by Hussain Ali Akbar on Medium</title>
            <link>https://medium.com/@HussainAliAkbar?source=rss-3805c8ea5f9------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Wed, 18 Nov 2026 00:51:21 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@HussainAliAkbar/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[Effort vs Impact: What changed because you were here?]]></title>
            <link>https://medium.com/@HussainAliAkbar/effort-vs-impact-what-changed-because-you-were-here-5e5bd9da9a34?source=rss-3805c8ea5f9------2</link>
            <guid isPermaLink="false">https://medium.com/p/5e5bd9da9a34</guid>
            <category><![CDATA[management]]></category>
            <category><![CDATA[career-development]]></category>
            <category><![CDATA[productivity]]></category>
            <category><![CDATA[software-engineering]]></category>
            <category><![CDATA[leadership]]></category>
            <dc:creator><![CDATA[Hussain Ali Akbar]]></dc:creator>
            <pubDate>Thu, 20 Aug 2026 16:01:03 GMT</pubDate>
            <atom:updated>2026-08-20T16:01:03.088Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*AiI8hxuXq2pYt-3RT8HzAQ.jpeg" /><figcaption>Effort is what we experience. Impact is what changes because of it.</figcaption></figure><p>As we approach performance reviews at <strong>Metal</strong>, a question has continued to stay with me as I evaluate mine and my team’s impact –</p><blockquote>If I or this person was not at Metal, what would have been materially different?</blockquote><p>This has led me to think of the world that lies across effort and impact and how to think of it.</p><p>Our proximity to our own effort is more than anyone else. We remember the hard work involved, but our manager, our customer and the broader company experiences something different — the eventual result. This creates a natural gap in self-awareness. The individual is thinking, “<em>You don’t understand how difficult this was.</em>” The manager may fully understand that it was difficult, but is still asking:</p><blockquote><em>“</em>What was the impact?<em>”</em></blockquote><p>Both things can be true: Effort and impact are related, but they are not interchangeable. Something can require tremendous effort and still create limited impact. Something else can be completed with relatively little effort and create an outsized amount of impact.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*7nlnD03_gj1Ng6Nk" /><figcaption>Hard work matters. What it changes matters more.</figcaption></figure><h3>Not All Effort Leads To Impact</h3><p>It is helpful to distinguish between four things that are often grouped together:</p><ol><li><strong>Effort</strong> is the time, energy and difficulty involved in doing the work.</li><li><strong>Output</strong> is the thing that was produced.</li><li><strong>Outcome</strong> is the immediate change that happened because of the output.</li><li><strong>Impact</strong> is why that change mattered to the customer, the business, the team or the system.</li></ol><p>For example, an engineer spends two months building a new feature. The two months represent the <strong>effort</strong>. The feature itself is the <strong>output</strong>. In the first week, two customers start using it; that is the <strong>outcome</strong>. Those customers then engage with Metal more frequently and more deeply; that is the <strong>impact</strong>.</p><p>Performance discussions often stop at the first 2 levels: “<strong>I spent 2 months on this!</strong>” “<strong>I shipped 10 prototypes!</strong>” These statements describe work, They do not describe impact.</p><p>This distinction becomes especially important when we talk about exceeding expectations. If someone spends 50 hours in a week completing work that was reasonably expected within a standard work week, the additional ten hours do not by themselves automatically turn meeting expectations into exceeding expectations. The work may have been difficult, the person may have demonstrated commitment, there may even be important context around why it took longer than anticipated. <strong>But the additional effort does not, by itself, make the result exceptional.</strong></p><p>Another engineer may create greater impact in 30 hours because they identified a simpler solution, challenged an unnecessary requirement, automated a repeated process or realised that the original task was not worth doing in the first place.</p><p>The core point is that working longer does not automatically represent stronger performance. We need to move beyond how demanding the work was and look at whether it translated into something meaningful for the customer, the business, the team or the system.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1000/0*YLLd9QcsHzGAy2Ia" /><figcaption>The release is visible. The impact comes after.</figcaption></figure><h3>Shipping Is Not the Finish Line</h3><p>A recent example from a podcast on <a href="https://theskip.substack.com/p/inside-pm-at-stripe-what-comes-after">The Skip</a> captures this well.</p><p>A PM proudly reported that they had created ten prototypes in a week. But how many users had seen those prototypes? Zero. How many decisions had they influenced? Zero. Has the team developed greater conviction in a direction? No.</p><blockquote><em>Then what progress did we actually make?</em></blockquote><p>The issue is not that prototypes are useless. Exploration is a critical part of building products. The problem starts when the volume of output becomes a substitute for progress. Ten prototypes, Ten shipped features, One hundred closed tickets all sound impressive, but activity does not automatically mean progress.</p><p>A prototype that invalidates an expensive direction, or a failed experiment that prevents a team from spending the next three months building the wrong thing can be worth far more than a successful feature with limited adoption.</p><p>The right mindset therefore, is not to stop at shipping features. It is to stay with the work after it ships, to understand whether customers are using it, whether it solved the problem it was intended to solve, whether it changed an important metric or decision, and what needs to change if it did not.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*b4IzC3cMBaKyeuL7" /><figcaption>Impact isn’t always visible, immediate, or financial.</figcaption></figure><h3>What Impact Actually Looks Like</h3><p>Impact doesn’t always have to be tied to revenue. A salesperson may create impact by closing a customer. A product manager may create impact by increasing adoption or killing the wrong product direction early. An engineer may create impact by improving systems reliability, reducing infrastructure costs, prevent production outages, or creating a platform that allows ten other engineers to move faster.</p><p>Not every impactful work will immediately move the needle. Some of the most impactful work may be preventive, for example designing a stringent tech doc review process may avoid us rebuilding features every 6 months or dealing with tech debt.</p><p>In any case, we should be able to answer What became better? Who benefited? What evidence do we have? What risk was reduced? What future work was unlocked?</p><p>Similarly, impact must not be short-sighted. A feature that increases short-term adoption but creates severe technical instability does not represent strong net impact. Shipping quickly while leaving behind recurring incidents, burned-out teammates and an unmaintainable codebase is not high performance.</p><p>The impact that we value must be meaningful and sustainable.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/991/0*VKBuFP5KbyRzkyBt" /><figcaption>The strongest work keeps creating value after you step away.</figcaption></figure><h3>What Changed Because You Were Here?</h3><p>Everyone should look at the five pieces of work they are proudest of from the previous quarter and complete a simple sentence for each one: I did X, which changed Y, and I know that because of Z.</p><p>This single exercise forces clarity of thought and a distinction between activity and impact. What was done? What became different because of it? Who benefited? Did the work produce a temporary result, or lasting leverage?</p><p>If one cannot answer these questions, it does not automatically mean the work had no value. The impact may be too early to observe. It may be difficult to measure. The work may have prevented a problem that never became visible. But it should still prompt deeper reflection.</p><p>At the end of the quarter, the most important question is not how much work someone completed, It is:</p><blockquote>What changed because you were here?</blockquote><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=5e5bd9da9a34" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[“If only people stopped interrupting and started listening :)”]]></title>
            <link>https://medium.com/@HussainAliAkbar/if-only-people-stopped-interrupting-and-started-listening-2709fd1d2e87?source=rss-3805c8ea5f9------2</link>
            <guid isPermaLink="false">https://medium.com/p/2709fd1d2e87</guid>
            <category><![CDATA[habits]]></category>
            <category><![CDATA[empathy]]></category>
            <category><![CDATA[interruptions]]></category>
            <category><![CDATA[active-listening]]></category>
            <category><![CDATA[listening]]></category>
            <dc:creator><![CDATA[Hussain Ali Akbar]]></dc:creator>
            <pubDate>Wed, 16 Nov 2022 09:01:48 GMT</pubDate>
            <atom:updated>2022-11-16T09:01:48.783Z</atom:updated>
            <content:encoded><![CDATA[<h3>“If only people stopped interrupting and started listening :)”</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*QWrm8L7HgZMwXTzUQeqKCQ.jpeg" /><figcaption>Photo by <a href="https://unsplash.com/@jontyson?utm_source=unsplash&amp;utm_medium=referral&amp;utm_content=creditCopyText">Jon Tyson</a> on <a href="https://unsplash.com/s/photos/listening?utm_source=unsplash&amp;utm_medium=referral&amp;utm_content=creditCopyText">Unsplash</a></figcaption></figure><p>I recently attended a retrospective meeting that turned into a free-for-all. <strong>It was chaos</strong>.</p><p>Nobody was listening to anyone, just trying to get their viewpoint across. In the end, I had to call a time-out. It was so bad that I had to conduct a retrospective of the retrospective with the individual members of the team. Genuinely curious to know what happened and how we came to this, I realized that everyone (including me) was trying to say the same thing! Just in a different manner! To me, this seemed like a case of simple miscommunication, but I decided to dig deeper into the <strong>WHY </strong>of it.</p><p>And so I had a second round of discussions with the team. This time around, one of the teammates shared:</p><blockquote><strong>“If only people stopped interrupting and started listening”</strong></blockquote><p>And that’s when it dawned on me — nobody waited for the other person to complete their viewpoint but instead made an assumption on the half-delivered point and interrupted them. Had people been given a chance to complete what they wanted to say, everyone would’ve agreed easily.</p><p>And the moment we clarified this within the team and set the right expectations, we had another retrospective that turned out to be completely normal.</p><h3>But why do people Interrupt?</h3><p>As children, we’re taught this basic etiquette. As Managers, we’re taught this as part of our management training and yet, time and time again, we fail to realize and practice this simple rule. In fact, most people don&#39;t even realize that they’re interrupting the other person which is hurting the overall conversation.</p><blockquote>Most people don’t even realize that they’re interrupting</blockquote><p><strong>It also happens</strong> <strong>that</strong> if a speaker takes too long to get to the point, that may push the other colleagues towards interrupting to get to the point. We live in a world run by calendars, where schedules are very important and everyone always has to move on to the next task.</p><p><strong>And lastly,</strong> human beings, no matter how hard they try, can only keep so much context for conversations in their memories. If the speaker is sharing multiple arguments or points in one go, the listeners might forget some of the points that the speaker initially shared. And since everyone’s aware of their attention spans, they try to speak their mind about a given point or argument before they forget it.</p><blockquote><strong>Speaker 1</strong>: We should do A because of U, V, W, X, Y, Z,……<br><strong>Listener 1</strong>: (interrupts) Hold on, I have a concern about V, we should do B instead.</blockquote><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*oU0z1sfuO99RJZ0OXd_wlg.jpeg" /><figcaption>Photo by <a href="https://unsplash.com/@stefanbc?utm_source=unsplash&amp;utm_medium=referral&amp;utm_content=creditCopyText">Stefan Cosma</a> on <a href="https://unsplash.com/?utm_source=unsplash&amp;utm_medium=referral&amp;utm_content=creditCopyText">Unsplash</a></figcaption></figure><h3>What can we do to fix this?</h3><h4>Start with setting the right expectations.</h4><p>Don&#39;t assume that people are interrupting others on purpose. Have information and training sessions where the team level expectations are set and people are expected to listen to others completely. The team needs to realize that to become good speakers, people need to first become good listeners.</p><p>Behavioral changes take time to reflect on and become permanent habits. So don&#39;t expect an overnight impact, in fact, constantly enforce the right expectations now and then within team meetings and 1:1s around why this is important.</p><h4>Coach teammates to speak quickly and concisely.</h4><p>Make the teammates realize the importance of time and the adverse impacts of rambling. This too will require constant reinforcement and feedback to nail.</p><h4>Don&#39;t assume anything until the speaker has finished speaking.</h4><p>We’re quick to assume (<em>the worst</em>) without waiting for the speaker to complete. This is a big <strong>no-no</strong>. Not only will this cause us to lose focus and concentration on what&#39;s being said but it can also push us to interrupt thinking that the speaker has gone down the wrong path and feel the need to correct the speaker.</p><p>So, Wait. Wait for the speaker to complete and then think through what the speaker has said and respond. This may cause a pause in between the conversations but that would be much better than interrupting someone because of an incorrect assumption.</p><h4>Note down the questions so that you don&#39;t forget.</h4><p>If the speaker is sharing multiple points that you have a counter-argument to and are afraid of forgetting, just note them down!</p><p>One cool thing that we do at the current workplace now after the whole retro-fiasco is during any team meetings, everyone is taking personal notes while others are speaking that they can refer back to especially if they have any concerns about some things that the speaker is sharing. This ensures that nobody forgets anything but also doesn&#39;t have to interrupt anyone else.</p><h4>Use Technology! (duh).</h4><p>Tools like <strong>Google Meet</strong> and <strong>Zoom</strong> have a “<strong>Raise Hand</strong>” feature that notifies the speaker that you want to add something but also simultaneously does not interrupt them in their tracks. Use it!</p><p>Also, Almost all the virtual meeting tools have a chat feature that can be used to comment on the speaker’s thoughts, note down questions, and anything else. This helps you to say what you want to say when you want to say it while being respectful of the speaker as well.</p><p>It&#39;s interesting how small tweaks can cause big changes, and asking people to not interrupt is something very easy to implement but can yield big benefits. So let&#39;s be better co-workers, better engineers, and better <strong>HUMANS</strong>.</p><blockquote>Let&#39;s stop interrupting and start listening :)</blockquote><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=2709fd1d2e87" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Code can improve, customers lost will never be regained]]></title>
            <link>https://medium.com/@HussainAliAkbar/code-can-improve-customers-lost-will-never-be-regained-7a16fef265b7?source=rss-3805c8ea5f9------2</link>
            <guid isPermaLink="false">https://medium.com/p/7a16fef265b7</guid>
            <category><![CDATA[reflections]]></category>
            <category><![CDATA[learning]]></category>
            <category><![CDATA[software]]></category>
            <category><![CDATA[engineering]]></category>
            <dc:creator><![CDATA[Hussain Ali Akbar]]></dc:creator>
            <pubDate>Fri, 04 Feb 2022 14:50:04 GMT</pubDate>
            <atom:updated>2022-02-04T14:50:04.282Z</atom:updated>
            <content:encoded><![CDATA[<h3>Code can improve, Customers lost will never be regained</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Y5tEi_xpKQL0hE7vkhoQ1A.jpeg" /><figcaption>Photo by <a href="https://unsplash.com/@brett_jordan?utm_source=unsplash&amp;utm_medium=referral&amp;utm_content=creditCopyText">Brett Jordan</a> on <a href="https://unsplash.com/?utm_source=unsplash&amp;utm_medium=referral&amp;utm_content=creditCopyText">Unsplash</a></figcaption></figure><p>As I reflect on completing 2 years at a workplace that has continuously challenged me to push my limits, I ask myself: <strong><em>If I had to summarise my learnings in a few statements, what would they be?</em></strong></p><p>After some thought, the answer came to me, as clear as day:</p><blockquote><em>“Unlearning the notion that quality of a software is everything. Always put consumers’ interest first and ahead of any and every consideration. A consumers’ trust is everything and once lost, is very hard to regain.”</em></blockquote><p>When we are students, the concept that quality trumps all-else is instilled into us. As professional engineers we carry that lesson forth, doing our best to deliver optimised and clean code — one that has both 100% test coverage and an A grade from Sonarqube. But we often fail to realise that by following this mindset blindly, we’re putting the very people we’re working for at risk i.e. our consumers. What makes this even more challenging is when you as an individual have a tendency to be overly-obsessed with achieving perfection.</p><p><strong>Unlearning this lesson has been one of the hardest challenges that I have had to overcome.</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*PQKcs44Vx3-_ggLhharPvQ.jpeg" /><figcaption>Photo by <a href="https://unsplash.com/@tateisimikito?utm_source=unsplash&amp;utm_medium=referral&amp;utm_content=creditCopyText">Jukan Tateisi</a> on <a href="https://unsplash.com/?utm_source=unsplash&amp;utm_medium=referral&amp;utm_content=creditCopyText">Unsplash</a></figcaption></figure><p>My company was initially building a product for mass-transit. In March 2020, with the onset of the COVID-19 pandemic, we had to pivot into quick-commerce because of immediate lockdowns and mass transit service-suspension due to government regulations and health concerns. We did not have the capital to wait the pandemic out — we had to launch a new product, and launch it fast. This was a surmount challenge for the Engineering team that I was part of — we had to come forth with a complete end-to-end product as soon as possible to re-start operations.</p><p>In this process, we had to cut a lot of corners, basically skim over anything that wasn’t “<strong>essential</strong>” at the time which meant unit tests, documentation, scalable architecture and even proper QA at times. We were pushing code day in, day out. At times, I was pretty fed up with the way we were working — hastily, scrappily and messily.</p><blockquote>Looking back now, I realise that we made the right decision.</blockquote><p>Because not only did we manage to fix the code over time, but we also gained a first-mover advantage, by moving quickly and nimbly, pioneering quick commerce in Pakistan. Had we taken, let’s say 6 months to build a perfectly refined product with each and every corner case covered and a slick UI / UX, we probably would not have been able to make as big of a mark because the needs of the customers would’ve been addressed by someone else!</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*yg6t0ybpwIUPdt0rgVS_uQ.jpeg" /><figcaption>Photo by <a href="https://unsplash.com/@brett_jordan?utm_source=unsplash&amp;utm_medium=referral&amp;utm_content=creditCopyText">Brett Jordan</a> on <a href="https://unsplash.com/?utm_source=unsplash&amp;utm_medium=referral&amp;utm_content=creditCopyText">Unsplash</a></figcaption></figure><h4><strong>Life was simpler when my tech lead made the decisions and I simply followed them.</strong></h4><blockquote>“<em>Okay, I don’t completely agree with sacrificing the code’s quality but I trust my lead to make the correct decisions for the sake of the company.</em>”</blockquote><p>Things started to change when I became a lead myself and the decision of “delivering fast vs delivering quality” landed on me. After constantly struggling with this, I found a particularly nifty principle that I continue to abide by till date:<em> “move fast on type 1 decisions, slow on type 2 decisions”.</em> A type 1 decision can be easily reversed and hence, can be taken very quickly. A Type 2 decision is harder to reverse so must be made after careful consideration.</p><p>Relating the above to an example — let’s say you get a feature request that solves an enormous problem for your company and there are 2 ways to get about it: You can either deliver the feature quickly in a week but it can only sustain about 50 concurrent users beyond which, the feature will break. To make the feature truly scalable, you need at least a month. But since the feature is to be used by internal teams only, you don’t expect it to be used by more than 20–30 people for at least a couple of months.</p><p>The answer’s pretty simple right? One can argue that you will end up spending more engineering bandwidth in the long run by creating something inefficient and then reworking it completely. But the bigger picture that can allow us to answer this question is that while we engineers work on the v2 of the feature, can the v1 hold its own and allow the operations team to scale business, reach more customers to earn more revenue? If yes, we have our answer.</p><blockquote><strong>Code can improve. Customers lost, will never be regained.</strong></blockquote><p>Obviously, decisions in real life situations are neither as clear as above nor as easy to make. This, like everything else, is a skill that you get better at by failing at first.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*55qRevgpZ1D6TuCPaKqjyQ.jpeg" /><figcaption>Photo by <a href="https://unsplash.com/@timmossholder?utm_source=unsplash&amp;utm_medium=referral&amp;utm_content=creditCopyText">Tim Mossholder</a> on <a href="https://unsplash.com/s/photos/learn?utm_source=unsplash&amp;utm_medium=referral&amp;utm_content=creditCopyText">Unsplash</a></figcaption></figure><p>During the absolute early times, some users had found a very nasty loophole in our application and were exploiting it. A proper solution would have taken days to implement but until then, we would’ve lost a lot of money. A “<strong>quick fix</strong>” that would’ve been considered a “<strong>bad practice</strong>” by each and every software standard there is on the planet, but it did the trick, saved us a lot of money and held the fort until we worked on the proper solution.</p><h3>But wait, What about tech debt?</h3><p>Some might say that with the above approach, we might never get the time to take care of Tech Debt? And that is wrong again. Through a colleague, I <a href="https://www.linkedin.com/pulse/non-functional-requirements-technical-debt-business-problems-goulart/"><strong>learned</strong></a> that Technical Debt / Non Functional requirements are just as important as functional features. From a business’ point of view, we want to serve our customers right? What if our code reaches a point where it cannot continue to scale according to the demands of the customers and our systems are at a risk of a crash? This is not a Technical Debt anymore but a critical business requirement that needs to be taken up on priority. Again, it comes down to how you not only pitch technical debt to the product team / leadership but also how its up to you as Engineers to make sure that the Tech Debt continues to stay at bay and not become so large that it starts to cause problems for your customers.</p><blockquote>Moderation is key.</blockquote><p>On one end of the spectrum, we have Perfectionists who want to write the perfect code and don’t care for anything else and on the other hand, we have People who are filled with adrenaline to push out software as fast as possible.</p><p>I think it’s time that we start taking the middle path where we try to balance the two. A customer will neither use an application that’s still in development because the developers are perfecting the codebase, nor an application that’s so laggy and slow that it takes ages to move from one screen to another.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=7a16fef265b7" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Empathy — The secret to maintaining workplace sanity]]></title>
            <link>https://medium.com/@HussainAliAkbar/empathy-the-secret-to-maintaining-workplace-sanity-b9f8a4c711f6?source=rss-3805c8ea5f9------2</link>
            <guid isPermaLink="false">https://medium.com/p/b9f8a4c711f6</guid>
            <category><![CDATA[startup-culture]]></category>
            <category><![CDATA[workplace-culture]]></category>
            <category><![CDATA[empathy]]></category>
            <category><![CDATA[start-up-companies]]></category>
            <dc:creator><![CDATA[Hussain Ali Akbar]]></dc:creator>
            <pubDate>Sun, 05 Sep 2021 05:02:01 GMT</pubDate>
            <atom:updated>2021-09-05T05:02:01.207Z</atom:updated>
            <content:encoded><![CDATA[<h3><strong>Empathy — The secret to maintaining workplace sanity</strong></h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*ItXhaHthTd1oKDtNJil-Vw.jpeg" /><figcaption>Photo Credits: <a href="https://unsplash.com/photos/u4bZQUeo2Q8">Unsplash</a></figcaption></figure><p>When you are working at a rapidly-scaling, high-velocity start-up that ambitiously sets a lot of big, hairy and audacious goals, I can bet that you are bound to miss out on many, if not some of the targets that you initially set out to achieve. Maybe you incorrectly estimated a task or a peer was unable to complete their part of the feature on time or (unluckily) the entire requirement was wrong from the get-go. Whatever it is, <strong>things quite often do not go quite as you planned them.</strong></p><p>Such times are incredibly pivotal because it is at moments like these where frustration starts to build up and you keep on fussing over what went wrong as opposed to how it can be salvaged or improved — this often leads to the situation deteriorating even further.</p><blockquote>When people do not know the complete side of a picture, they tend to paint a one-sided one in their minds</blockquote><p>… which is (most of the time) very different from reality. The (not-so) secret here is to learn and lead with empathy which I firmly believe is the only way to maintain sanity at the workplace and ensure that you do not burn out or lose out on the long term vision.</p><p>There are 2 kinds of empathy that one should practice at the workplace. The former is easier of the two and something which most people get right — which is to empathize with yourself. If you make a mistake and that results in the team not meeting a target or missing out on a deadline or a goal, you have to empathize with yourself given the fact that you went all in and made your best effort and that is what matters in the end. You know you gave it your best shot. In the end, you need to sit down and think as to what you could have done better to avoid the problems that occurred and ensure that those mistakes are not repeated in the future.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*2emNFVHbgnjtWeBJec7MUQ.jpeg" /><figcaption>Photo credits: <a href="https://unsplash.com/photos/geM5lzDj4Iw">Unsplash</a></figcaption></figure><h4>And now comes the hard part</h4><p>Lets move to the latter that is harder to understand, even harder to practice — which is to empathize with others, being curious around why others did not perform as per the expectations, driving discussions and brainstorming around failures by assuming good intentions of other people.</p><blockquote>What is imperative is to realize that people are not intentionally or inherently “bad”.</blockquote><p>Nobody enjoys performing poorly, or purposelessly. For me, it is vital to never forget that human beings, by nature, always strive to be the best version of themselves and when they are unable to do that, there’s almost always a reason behind that. As coworkers, we need to be empathetic about the fact that people might have problems of all kinds (personal, financial, societal) that are simply not allowing them to perform up to the level they’re supposed to. Or maybe the other person is working at 100% but the expectation is unrealistic and way too high to be met or it can be any other problem entirely!</p><p>The tricky and the most difficult part to learn here would be to stop getting to conclusions as soon as things start to break and start being curious — talk to your coworkers, understand the situation they are in and what they are going through. Within the broader context of a raging pandemic — could it be that someone close to your coworker is ill? Especially due to the impersonality of remote working and work-from-home policies that often disconnect us, we often forget that our teammates and coworkers are, at the end of the day, not faces behind screens but real human beings, just like ourselves, with their own lives, needs and priorities.</p><p>Ever since I have started being intentional in reminding myself of this very fact, not only have I been able to foster a closer relationship with my teammates but this has also enabled me to maintain a (relatively frustration-free) sane mind during work.</p><p><strong>Practicing empathy is hard</strong> — it takes time and effort to adopt. Fortunately, it’s a learn-able skill — one that we’ll need more and more of as we move into an era where remote-working is becoming the norm and we rarely get to interact with our co-workers physically. So, the next time something goes south, speak to the person, get their perspective on the situation — I am sure, you’ll be much relieved!</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=b9f8a4c711f6" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Implementing pub/sub model with Apache Kafka and Node.js]]></title>
            <link>https://medium.com/weekly-webtips/implementing-pub-sub-model-with-apache-kafka-and-node-js-d913ce722557?source=rss-3805c8ea5f9------2</link>
            <guid isPermaLink="false">https://medium.com/p/d913ce722557</guid>
            <category><![CDATA[nodejs]]></category>
            <category><![CDATA[javascript]]></category>
            <category><![CDATA[apache-kafka]]></category>
            <category><![CDATA[message-queue]]></category>
            <category><![CDATA[web-development]]></category>
            <dc:creator><![CDATA[Hussain Ali Akbar]]></dc:creator>
            <pubDate>Mon, 17 Aug 2020 18:24:55 GMT</pubDate>
            <atom:updated>2020-09-03T06:45:03.514Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/975/1*8GbrXbHdH5uPGMb5epWhrg.png" /></figure><p>Apache Kafka is a distributed streaming platform with a lot of <a href="https://kafka.apache.org/uses">use cases</a> including acting as a message broker between your applications and processes. Today, We’ll look at how we can implement a publisher/subscriber model with Apache Kafka using Node.js!</p><h3>Basic Concepts of Apache Kafka</h3><p>Before we proceed with the code it is important to understand the basic concepts of Apache Kafka. Fortunately, there’s already tons of excellent resources available online that explain the basic concepts way better than I ever can!</p><ol><li><a href="https://kafka.apache.org/intro">Apache Kafka’s own website</a></li><li><a href="https://www.cloudkarafka.com/blog/2016-11-30-part1-kafka-for-beginners-what-is-apache-kafka.html">This excellent piece on Cloudkarafka</a></li><li><a href="https://medium.com/@jhansireddy007/how-to-parallelise-kafka-consumers-59c8b0bbc37a">This read up on Consumer Groups since that plays a major role in our pub/sub model</a></li></ol><p>Do go through these before proceeding further in case you’re not aware of the underlying concepts of Kafka!</p><h3>The Use Case</h3><p>Let&#39;s say we have an eCommerce website where every time a customer checks out an order, an event <strong>“orderCreated” </strong>will be generated<strong>. </strong>This event will be listened to by the following services:</p><ol><li><strong>Orders Service</strong>: This service is responsible for adding the order details to the Database</li><li><strong>Payments Service</strong>: This service is responsible for deducting credits from the Customer’s wallet</li><li><strong>Notifications Service</strong>: This service is responsible for notifying the customer that their order has been placed successfully.</li></ol><p>We can do these operations asynchronously in the background — The benefit of doing so is that we can quickly send a response back to the customer with the actual work being done in the background.</p><p>Now, there are 2 key challenges here:</p><ol><li><strong>We want our messages to be processed at least once.</strong> The listeners need to acknowledge that they’ve processed the events. In case a consumer service fails to process an event successfully, the event needs to be retried. For example, we cannot afford to lose events of credits that need to be deducted from customers wallets. <em>(Customers getting their orders for free!)</em></li><li><strong>We also want our messages to be processed at most once.</strong> In today’s era, We usually have a lot of instances of our application running behind a load balancer. This means that in our scenario, There will be multiple instances of the payment service (as well as others) running. Therefore, we don’t want the event to be processed by each instance. <em>(Customers getting charged double or triple times the order amount!)</em></li></ol><p>Let&#39;s see how we can achieve this with Kafka!</p><h3><strong>Setting up Kafka with Docker</strong></h3><p>I will once again recommend an excellent piece of article here that I used myself for setting up Kafka locally with Docker.</p><p><a href="https://itnext.io/how-to-install-kafka-using-docker-a2b7c746cbdc"><em>How to install Kafka using Docker</em></a></p><p>Here’s an updated version of the docker-compose.yml file:</p><pre>version: &#39;3.7&#39;<br><br>networks:<br>  kafka-net:<br>    driver: bridge<br><br>services:<br>  zookeeper-server:<br>    image: &#39;bitnami/zookeeper:latest&#39;<br>    networks:<br>      - kafka-net<br>    ports:<br>      - &#39;2181:2181&#39;<br>    environment:<br>      - ALLOW_ANONYMOUS_LOGIN=yes<br>  kafka-server1:<br>    image: &#39;bitnami/kafka:latest&#39;<br>    networks:<br>      - kafka-net<br>    ports:<br>      - &#39;9092:9092&#39;<br>    environment:<br>      - KAFKA_CFG_ZOOKEEPER_CONNECT=zookeeper-server:2181<br>      - KAFKA_CFG_ADVERTISED_LISTENERS=PLAINTEXT://localhost:9092<br>      - ALLOW_PLAINTEXT_LISTENER=yes<br>    depends_on:<br>      - zookeeper-server</pre><p>Just run docker-compose up in the terminal and the kafka server should start running!</p><p>You can set up <a href="https://www.conduktor.io/">Conduktor</a> as well in case you want to set up a desktop client.</p><h3>Time to Code!</h3><p>We’ll be using a package called <a href="https://kafka.js.org/">kafkajs</a> in order to interact with Kafka from Node.js.</p><ol><li>As a first step, We’ll need to create a topic orderCreated which will be used by publishers and subscribers.</li></ol><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/8640ef346778460e722f88927c414d63/href">https://medium.com/media/8640ef346778460e722f88927c414d63/href</a></iframe><p>The main point to note here is the number of partitions — Which we’ve currently set to 2. The reason behind this will become clear as we proceed.</p><p>Copy and run this code as is and if everything runs perfectly, the console should output done indicating that the topic has been created. If you have conduktor set up, you can verify this as well:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*nLvr3JVPpQDiiEJ9UzKOaA.png" /><figcaption>our topic “orderCreated” has been created with 2 partitions</figcaption></figure><p>2. The next step is to write a consumer that will subscribe to this topic and listen for events.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/dd4dce42c8c48c5fb40731d269d51e95/href">https://medium.com/media/dd4dce42c8c48c5fb40731d269d51e95/href</a></iframe><p>Before executing this code, let&#39;s understand a bit what’s happening here.</p><p>At the start of the process function, we’re creating 3 consumers to simulate our 3 services namely ordersConsumer, paymentsConsumer and notificationsConsumer. The groupId specifies the consumer group that these consumers will be a part of.</p><p>Adding the 3 consumers to 3 different consumer groups will ensure that all the 3 consumers will get the messages (which is what we want). If we add all the consumers to the same group then only one of these will get the message.</p><p>Next we subscribe the consumers to our topic “orderCreated” .</p><p>Finally, we start the consumers with.run which enables them to listen to any messages that are published on their subscribed topics. Each consumer does a simple job of logging the message that they receive.</p><p>The consumer file also expects a command-line argument which tells us the consumer number. Since these are our first consumers, let&#39;s execute this script asnode mediumConsumer.js 1 .</p><p>If everything works fine, we should see a lot of logs from Kafka saying that consumers have joined the group. This can be verified from Conduktor as well:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*YcYi-0m1pkr1s1vZiu1izA.png" /><figcaption>3 Consumer Groups are visible with 1 member each and both partitions assigned to them</figcaption></figure><p>One log that is important to note here is this one:</p><pre>{&quot;level&quot;:&quot;INFO&quot;,&quot;timestamp&quot;:&quot;2020-08-17T14:54:40.237Z&quot;,&quot;logger&quot;:&quot;kafkajs&quot;,&quot;message&quot;:&quot;[Runner] Consumer has joined the group&quot;,&quot;groupId&quot;:&quot;notifications&quot;,&quot;memberId&quot;:&quot;my-app-2d62f453-edd1-4b3f-a2ac-b926abcd9e12&quot;,&quot;leaderId&quot;:&quot;my-app-2d62f453-edd1-4b3f-a2ac-b926abcd9e12&quot;,&quot;isLeader&quot;:true,&quot;memberAssignment&quot;:{&quot;orderCreated&quot;:[0,1]},&quot;groupProtocol&quot;:&quot;RoundRobinAssigner&quot;,&quot;duration&quot;:20}</pre><p>Under memberAssignment, We see that both of our partitions (0, 1) are assigned to this single consumer i.e. any message that lands on either of these partitions will be received by this consumer. This log is for our “notifications” consumer. We should see the same log for the remaining consumers as well.</p><p>3. The final step is to write the publisher that will publish our messages:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/2a09edb1a56e6ff0e2d4f7347a5a5859/href">https://medium.com/media/2a09edb1a56e6ff0e2d4f7347a5a5859/href</a></iframe><p>The publisher’s code is very straightforward. We initialize a producer, connect it, and then send 3 messages on our topic in a loop.</p><p>With the consumer open in another terminal, execute the producer by running node mediumProducer.js which should publish all 3 messages then exit after outputting “done” . If everything works fine, the output on the consumer should be similar to this:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/789/1*h8sMKjcizMH6432-5XCB8A.png" /></figure><p>Since we have not specified which partition we want to send the message on, Kafka uses a Round Robin technique to send messages that’s why 2 messages were sent on partition 0 and 1 was sent on Partition 1.</p><p>So far so good. Let&#39;s run 2 consumers!</p><h3>Achieving “at most once” message processing with 2 consumers</h3><p>As mentioned initially under the use case section, we want to run multiple consumers in order to manage load however, we also want that our messages are only processed once.</p><p>To do this, we let our previous consumer keep running in a terminal and in another terminal, we start another consumer with the command node mediumConsumer.js 2 . Here 2 is indicating that this is our second consumer.</p><p>As soon as we execute the command, We should start to see some logs in the Consumer#1’s terminal indicating that the group is rebalancing:</p><pre>{&quot;level&quot;:&quot;ERROR&quot;,&quot;timestamp&quot;:&quot;2020-08-17T15:01:04.683Z&quot;,&quot;logger&quot;:&quot;kafkajs&quot;,&quot;message&quot;:&quot;[Runner] The group is rebalancing, re-joining&quot;,&quot;groupId&quot;:&quot;orders&quot;,&quot;memberId&quot;:&quot;my-app-e866f23c-53a4-4405-97b6-c49cd2b5b99b&quot;,&quot;error&quot;:&quot;The group is rebalancing, so a rejoin is needed&quot;,&quot;retryCount&quot;:0,&quot;retryTime&quot;:319}</pre><p>Once the rebalancing is complete, We should see this log:</p><pre>{&quot;level&quot;:&quot;INFO&quot;,&quot;timestamp&quot;:&quot;2020-08-17T15:01:14.697Z&quot;,&quot;logger&quot;:&quot;kafkajs&quot;,&quot;message&quot;:&quot;[Runner] Consumer has joined the group&quot;,&quot;groupId&quot;:&quot;notifications&quot;,&quot;memberId&quot;:&quot;my-app-2d62f453-edd1-4b3f-a2ac-b926abcd9e12&quot;,&quot;leaderId&quot;:&quot;my-app-2d62f453-edd1-4b3f-a2ac-b926abcd9e12&quot;,&quot;isLeader&quot;:true,&quot;memberAssignment&quot;:{&quot;orderCreated&quot;:[0]},&quot;groupProtocol&quot;:&quot;RoundRobinAssigner&quot;,&quot;duration&quot;:7}</pre><p>This time around, If we look at the memberAssignment property, we see that only 1 partition is assigned to Consumer#1. This is the log for our “notifications” consumer. We should see the same log for “orders” and “payments” as well. If we head over to the terminal of Consumer#2, we should see that Partition 1 has been assigned to Consumer#2.</p><p>What happened here is that during rebalancing, Kafka tried to divide all the partitions among the consumers equally. Since we had 2 partitions and 2 consumers, Kafka divided them equally among the 2 consumers. <strong>This also brings us to another important point:</strong></p><blockquote>If we have more consumers and less partitions for e.g. 3 consumers and 2 partitions then the extra number of consumers will be idle and no partition will be assigned to them.</blockquote><p>Now that both our Consumers are up and running, lets publish another message by executing the command node mediumProducer.js . After done is outputted on the console, head over to the consumers to examine the ouput:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/868/1*IBmxRb7lcclTX5T8JypRRA.png" /><figcaption>Consumer #1</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/817/1*OOekI8R5KuXwkSp-ge58Ww.png" /><figcaption>Consumer #2</figcaption></figure><p>The output on your terminals can be different from the above depending on the assignment of partitions by Kafka. The important thing to note here is that our load has been divided between 2 consumers and each consumer processes only the messages that are sent on the partitions assigned to them. And this is how we achieve “at most once” processing with 2 or more consumers!</p><h3>Achieving “at least once” message processing with 2 consumers</h3><p>Our 2nd objective as mentioned under the use case is that we want our messages to be processed at least once i.e. if any consumer fails to process a message due to any reason, the message should be retried until it is successful.</p><p>To simulate a failing consumer, let&#39;s modify our consumer’s code:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/de2aaeb35cd2453ccd8712ce9750c54e/href">https://medium.com/media/de2aaeb35cd2453ccd8712ce9750c54e/href</a></iframe><p>Most of the code is the same except for lines 42–48 where before logging our message, we throw an error. This way, the consumer won&#39;t be able to complete the message successfully and it will be retried again by Kafka.</p><p>To try this out, stop the Consumer#2 in the 2nd terminal, modify the consumer’s code as mentioned above and restart it with node mediumConsumer.js 2 . Again, we will see a lot of logs from Kafka indicating that it is rebalancing the topic and equally assigning the partitions among the 2 consumers.</p><p>Publish another message by executing the command node mediumProducer.js . After done is outputted on the console, check the Consumer#2’s output:</p><pre>{&quot;level&quot;:&quot;ERROR&quot;,&quot;timestamp&quot;:&quot;2020-08-17T16:29:21.569Z&quot;,&quot;logger&quot;:&quot;kafkajs&quot;,&quot;message&quot;:&quot;[Runner] Error when calling eachMessage&quot;,&quot;topic&quot;:&quot;orderCreated&quot;,&quot;partition&quot;:1,&quot;offset&quot;:&quot;3&quot;,&quot;stack&quot;:&quot;Error: some error got in the way which didnt let the message be consumed successfully\n ....</pre><p>We should see this message 6 times — One message for the initial attempt and the remaining for 5 retries. After this:</p><pre>{&quot;level&quot;:&quot;ERROR&quot;,&quot;timestamp&quot;:&quot;2020-08-17T16:29:31.309Z&quot;,&quot;logger&quot;:&quot;kafkajs&quot;,&quot;message&quot;:&quot;[Consumer] Crash: KafkaJSNumberOfRetriesExceeded: some error got in the way which didnt let the message be consumed successfully&quot;,&quot;groupId&quot;:&quot;notifications&quot;,&quot;retryCount&quot;:5,&quot;stack&quot;:&quot;KafkaJSNonRetriableError\n  Caused by: Error: some error got in the way which didnt let the message be consumed successfully\n ...</pre><pre>{&quot;level&quot;:&quot;INFO&quot;,&quot;timestamp&quot;:&quot;2020-08-17T16:29:31.318Z&quot;,&quot;logger&quot;:&quot;kafkajs&quot;,&quot;message&quot;:&quot;[Consumer] Stopped&quot;,&quot;groupId&quot;:&quot;notifications&quot;}</pre><p>This indicates that our consumer has crashed and stopped. If during the 5 retries, if the consumer successfully processes the message, then it won&#39;t crash (which in our case is not possible)`. By default, any crashing consumer will try to rejoin the consumer group, and surely enough, we see the log of consumer rejoining the group:</p><pre>{&quot;level&quot;:&quot;INFO&quot;,&quot;timestamp&quot;:&quot;2020-08-17T16:29:41.602Z&quot;,&quot;logger&quot;:&quot;kafkajs&quot;,&quot;message&quot;:&quot;[Runner] Consumer has joined the group&quot;,&quot;groupId&quot;:&quot;notifications&quot;,&quot;memberId&quot;:&quot;my-app-bde3d21a-4c83-4c03-9b04-0a26ee227c74&quot;,&quot;leaderId&quot;:&quot;my-app-2d62f453-edd1-4b3f-a2ac-b926abcd9e12&quot;,&quot;isLeader&quot;:false,&quot;memberAssignment&quot;:{&quot;orderCreated&quot;:[1]},&quot;groupProtocol&quot;:&quot;RoundRobinAssigner&quot;,&quot;duration&quot;:1933}</pre><p>But where did the message go if it wasn’t processed by Consumer#2? For that, let&#39;s look at Consumer#1’s logs:</p><pre>{&quot;level&quot;:&quot;ERROR&quot;,&quot;timestamp&quot;:&quot;2020-08-17T16:29:31.573Z&quot;,&quot;logger&quot;:&quot;kafkajs&quot;,&quot;message&quot;:&quot;[Connection] Response Heartbeat(key: 12, version: 1)&quot;,&quot;broker&quot;:&quot;localhost:9092&quot;,&quot;clientId&quot;:&quot;my-app&quot;,&quot;error&quot;:&quot;The group is rebalancing, so a rejoin is needed&quot;,&quot;correlationId&quot;:2325,&quot;size&quot;:10}</pre><pre>{&quot;level&quot;:&quot;ERROR&quot;,&quot;timestamp&quot;:&quot;2020-08-17T16:29:31.574Z&quot;,&quot;logger&quot;:&quot;kafkajs&quot;,&quot;message&quot;:&quot;[Runner] The group is rebalancing, re-joining&quot;,&quot;groupId&quot;:&quot;notifications&quot;,&quot;memberId&quot;:&quot;my-app-2d62f453-edd1-4b3f-a2ac-b926abcd9e12&quot;,&quot;error&quot;:&quot;The group is rebalancing, so a rejoin is needed&quot;,&quot;retryCount&quot;:0,&quot;retryTime&quot;:258}</pre><pre>{&quot;level&quot;:&quot;INFO&quot;,&quot;timestamp&quot;:&quot;2020-08-17T16:29:31.579Z&quot;,&quot;logger&quot;:&quot;kafkajs&quot;,&quot;message&quot;:&quot;[Runner] Consumer has joined the group&quot;,&quot;groupId&quot;:&quot;notifications&quot;,&quot;memberId&quot;:&quot;my-app-2d62f453-edd1-4b3f-a2ac-b926abcd9e12&quot;,&quot;leaderId&quot;:&quot;my-app-2d62f453-edd1-4b3f-a2ac-b926abcd9e12&quot;,&quot;isLeader&quot;:true,&quot;memberAssignment&quot;:{&quot;orderCreated&quot;:[0,1]},&quot;groupProtocol&quot;:&quot;RoundRobinAssigner&quot;,&quot;duration&quot;:5}</pre><p>Once the Notifications Consumer in our Consumer#2 crashed, Kafka initiated a rebalancing of the group and reassigned the partition that was left by Consumer#2 to Consumer#1. And sure enough, we can see the message processed by Consumer#1:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/737/1*PvaRDuktEd2qv-0UHprIVg.png" /><figcaption>Remaining Message on Consumer#1</figcaption></figure><p>This way we can ensure “at least once” message processing for all our messages.</p><h4>However, there is one important caveat!</h4><p>Our 2 consumers ran 2 different codes which is why one consumer was able to process the message while the other could not. This is a highly unlikely scenario. In the real world, this will not be the case and the codebase would be consistent across all the consumers.</p><p>This means that if there is a bug in the code that does not allow the message to be consumed successfully, the message will keep on circling between the consumers and never be processed successfully!</p><p>To handle this, we will need to implement a custom retry strategy that ignores a message /adds to a dead letter queue after a certain number of retries. But that is a topic for another day.</p><p>For now, what’s important is that Kafka ensures that all our messages are processed at least once and will keep on being retrying until they’re successful.</p><h3>Conclusion</h3><p>And that’s all there is to it! I have tried to keep the implementation simple but these basic concepts can be tweaked easily depending on the use cases! Please provide feedback if there’s anything in the article that’s incorrect or that can be improved. Let me know if you run into any issues while following this and I’ll gladly help!</p><p>As mentioned above, We still need to implement a retry mechanism that handles messages which cannot be consumed by the consumers — We will look into that in the next part!</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=d913ce722557" width="1" height="1" alt=""><hr><p><a href="https://medium.com/weekly-webtips/implementing-pub-sub-model-with-apache-kafka-and-node-js-d913ce722557">Implementing pub/sub model with Apache Kafka and Node.js</a> was originally published in <a href="https://medium.com/weekly-webtips">Webtips</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Introducing Coding Standards for Higher Efficiency]]></title>
            <link>https://medium.com/airlift-technologies/introducing-coding-standards-for-higher-efficiency-fbfe0dc1d96e?source=rss-3805c8ea5f9------2</link>
            <guid isPermaLink="false">https://medium.com/p/fbfe0dc1d96e</guid>
            <category><![CDATA[coding-standards]]></category>
            <dc:creator><![CDATA[Hussain Ali Akbar]]></dc:creator>
            <pubDate>Tue, 14 Jul 2020 09:57:36 GMT</pubDate>
            <atom:updated>2020-07-16T05:12:14.827Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*aJ4FD27mh5LghsQf1NsMVw.png" /></figure><h3><strong>Preface</strong></h3><p>I was busy coding on a sunny morning at the Karachi office, when I heard my colleague sigh and hold his head in frustration. He had been working on a feature in isolation for a few days and finally synced his code with the develop branch only to face a ton of merge conflicts. This was not because there was any major change in functionality of any of the files, but they were simply committed with different indentation settings.</p><blockquote><strong><em>We realized we needed to implement a strict set of coding standards to improve the efficiency of the development team.</em></strong></blockquote><h3><strong>The Status Quo</strong></h3><p>Although our backend codebase had <a href="https://palantir.github.io/tslint/"><strong>TSLint</strong></a> already set up:</p><ol><li>The TSLint configuration file was changed frequently to accommodate for the different coding styles of developers. This defeated the purpose of TSLint altogether since everyone was coding in their own style, and there was no standardization.</li><li>TSLint rules were not enforced and a developer could simply choose to ignore the warnings.</li></ol><h3><strong>Choosing the Right Tools</strong></h3><p>After some research and consultation with the Architects Group, we decided to go with the following tools:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/239/1*05ecjPv_7hBnqmP7YK1g4g.png" /><figcaption><strong>Eslint: </strong><a href="https://eslint.org/"><strong>https://eslint.org/</strong></a></figcaption></figure><p>We adopted ESLint because TSLint has been deprecated with the <a href="https://github.com/palantir/tslint/issues/4534">recommendation</a> that people migrate over to ESLint. For rules, we decided to go with <a href="https://www.npmjs.com/package/@typescript-eslint/eslint-plugin">Typescript-ESLint Recommended</a> rules with a few minor changes.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/224/1*hPdlm7uqlIPk38lCZAB9hw.png" /><figcaption><strong>Prettier: </strong><a href="https://prettier.io/"><strong>https://prettier.io/</strong></a></figcaption></figure><p>We also added a tool called <a href="https://prettier.io/">Prettier</a> and an accompanying utility<a href="https://github.com/azz/pretty-quick"> pretty-quick</a>, as these are specialized packages to automate formatting. Although Prettier has a limited set of rule options, it fulfills our requirement for maintaining a consistent code style.</p><p>This prevents us from running into conflicts due to different indentations and coding styles. I would recommend reading <a href="https://prettier.io/docs/en/why-prettier.html">why to use Prettier</a> and <a href="https://prettier.io/docs/en/comparison.html">how it’s different from Linters</a> for a better understanding of the advantages of using this tool. Popular editors such as VS Code can be configured to format code on every change.</p><h3><strong>Enforcement</strong></h3><p>Choosing tools was the simpler task, as the challenge lay in ensuring the usage of these tools. Our plan to enforce the new process was broken into two parts:</p><ol><li>Format all existing codebase according the the new rules of ESLint and Prettier</li><li>Add pre-commit hooks so that any new code also follows the same rules</li></ol><p>For the first part, we had to plan it in a way that no other developer on the team was affected by it. We communicated this plan over to the entire team in advance and asked them to check their pending code. On the planned day, we formatted the entire codebase according to the new format.</p><p>Secondly, we added the pre-commit hooks straight away so that moving forward, anyone who writes any new code adheres to the new coding standards.</p><blockquote>This ensures that any new code is auto-formatted before commit. If any code cannot be auto-formatted, the commit will fail and the developer will be required to fix the issues manually.</blockquote><h3><strong>The “Why” Behind This</strong></h3><p>It’s important to understand what impact these changes have on the efficiency of the overall team. Since the entire codebase is formatted and written in a consistent manner, it is easier for everyone to:</p><ol><li><strong><em>Read and understand it</em></strong></li><li><strong><em>Maintain it</em></strong></li><li><strong><em>Reuse and extend it</em></strong></li></ol><p>These benefits will continue to increase as team size increases.</p><h3><strong>Conclusion</strong></h3><p>It does not matter what standards you follow, but it is important to follow one specific standard or format to improve the efficiency of the entire team. This may be a small change but the benefits of it will be felt by your team members in the long run.</p><p><em>Hussain works on the Technology Team as a Full-Stack Software Engineer. He loves to read, explore new technologies and has recently taken up writing as a hobby. He can be reached at hussain.ali@rideairlift.com. </em><strong><em>To join our team</em></strong><em>, please click </em><a href="https://rideairlift.bamboohr.com/jobs/?source=other"><em>here</em></a><em> to visit our job board to browse open positions.</em></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=fbfe0dc1d96e" width="1" height="1" alt=""><hr><p><a href="https://medium.com/airlift-technologies/introducing-coding-standards-for-higher-efficiency-fbfe0dc1d96e">Introducing Coding Standards for Higher Efficiency</a> was originally published in <a href="https://medium.com/airlift-technologies">Airlift Technologies</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Introducing Airlift’s Engineering and Leadership Blog]]></title>
            <link>https://medium.com/airlift-technologies/introducing-airlifts-engineering-and-leadership-blog-8211f43eab76?source=rss-3805c8ea5f9------2</link>
            <guid isPermaLink="false">https://medium.com/p/8211f43eab76</guid>
            <category><![CDATA[startup]]></category>
            <category><![CDATA[leadership]]></category>
            <category><![CDATA[engineering]]></category>
            <category><![CDATA[airlift]]></category>
            <dc:creator><![CDATA[Hussain Ali Akbar]]></dc:creator>
            <pubDate>Mon, 08 Jun 2020 05:40:51 GMT</pubDate>
            <atom:updated>2020-06-10T06:06:23.861Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*_Nt5vZludYtKVLm7aVlZ9A.png" /></figure><p>It gives me immense pleasure to announce <strong>Airlift’s Engineering and Leadership Blog</strong> — a shared space where individuals and teams from all over the Company will be sharing their stories. The people behind Airlift — who are responsible for powering Pakistan’s first app-based mass transit system — never cease to amaze. The <strong>relentlessness</strong> and the <strong>hustle</strong> that I have seen from this specific group has been next to crazy. And it is because of this attitude, and this attitude only, that Airlift got to where it is today.</p><p>What does it mean to be <strong>relentless</strong> at Airlift? It means that we will be resourceful in finding solutions to problems, that we will break through walls to achieve end-outcomes. Being <strong>relentless</strong> in our pursuits lies at the core of our operating values.</p><h3>Year 1 at Airlift was super crazy in so many ways.</h3><p>The extreme level of <strong>development</strong>, <strong>hyper growth</strong>, <strong>raising the first ever Series A funding</strong> for a Pakistan-based startup, and the <strong>speed</strong> at which Airlift achieved growth has been phenomenal and unprecedented. This speed, however, came at a cost of technical debt on the engineering side and loose processes on the business side. We knew what these issues were. We knew what impact these issues would have in the long run. We just didn’t have the time to solve them. And although our customers helped us build a community that stuck by us despite the shortcomings, we <strong>needed</strong><em> </em>to solve these issues to better our product.</p><p>And that is why, when the <strong>COVID-19 pandemic</strong> hit, we didn’t take that as a sign of weakness. While we were not excited to temporarily pause our service, which was an important part of our customers’ daily lives, we didn’t let the pandemic bring us down. We didn’t stand down and simply wait for it to pass.</p><h3><strong>No.</strong></h3><p>We stopped, took a deep breath, re-calibrated and started to focus on how we could make this time <strong>invaluable</strong> and <strong>indispensable</strong> for us. We asked ourselves — what would have to be true for us to thrive in the new environment? What needed to be our new focus? How could we use focus and composure to adapt, and to capitalize on the opportunities created in the post-COVID-19 world?</p><h3>And that’s when we decided on a <strong>theme for Year 2 at Airlift</strong>.</h3><p>In Year 2, we’re going to be focusing on improvements and refinements. Things that’ll not only improve our current processes but also take us to the next level of scale.</p><p>In this regard, <strong>the people at Airlift led with focus, composure and speed</strong>. Individuals and teams came forward across all the functions, identified the major pain points and took the initiative to fix them. In just a span of three months since the lock down started, so many features and processes across so many different functions have been fixed. This simply would not have been possible had we been running operations on the side.</p><p>And from that attitude stemmed the growth of this blog; an opportunity for all individuals at Airlift to be involved in building a culture of writing across the team, as well as sharing this with the entire ecosystem in Pakistan. Airlift will always continue to lead the way for Pakistani startups by adopting global best practices and by introducing them to the local community. In doing so, our hope is that the impact of our work creates waves of innovation across the country.</p><p>Expect this blog to see a flurry of activity over the next few months. We can’t wait to share with the world what we’ve been working on and how exactly we make things happen!</p><p><em>Hussain works on the Technology Team as a Full-Stack Software Engineer. He loves to read, explore new technologies and has recently taken up writing as a hobby. He can be reached at </em><a href="/service/mailto:hussain@rideairlift.com"><em>hussain.ali@rideairlift.com</em></a><em>. </em><strong><em>To join our team</em></strong><em>, please click </em><a href="https://rideairlift.bamboohr.com/jobs/?source=other"><em>here</em></a><em> to visit our job board to browse open positions.</em></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=8211f43eab76" width="1" height="1" alt=""><hr><p><a href="https://medium.com/airlift-technologies/introducing-airlifts-engineering-and-leadership-blog-8211f43eab76">Introducing Airlift’s Engineering and Leadership Blog</a> was originally published in <a href="https://medium.com/airlift-technologies">Airlift Technologies</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Serverless: Reusing common configurations across functions]]></title>
            <link>https://medium.com/@HussainAliAkbar/serverless-reusing-common-configurations-across-functions-e17ee10e0645?source=rss-3805c8ea5f9------2</link>
            <guid isPermaLink="false">https://medium.com/p/e17ee10e0645</guid>
            <category><![CDATA[serverless]]></category>
            <category><![CDATA[aws]]></category>
            <category><![CDATA[serverless-framework]]></category>
            <category><![CDATA[aws-lambda]]></category>
            <dc:creator><![CDATA[Hussain Ali Akbar]]></dc:creator>
            <pubDate>Fri, 29 May 2020 04:57:06 GMT</pubDate>
            <atom:updated>2020-05-29T04:57:06.653Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*YDyJpHrHPbjP7Fm-rPFtUg.png" /></figure><p>In the final part of the series, We’ll build on top of our <a href="https://medium.com/@hussain94aa/serverless-breaking-a-large-serverless-yml-into-manageable-chunks-89fc909937c7">previous post</a> and look into how we can break down our template files further by reusing common configurations across multiple functions instead of defining them repeatedly.</p><p>Configurations such as VPCs, Cloudwatch Alarms, Layers, IAM Role Permissions (and much more) which can be common across functions can be reused for better and efficient management. This would help in case the configuration has to change. You could simply change the configuration in one place and it would be reflected in all the functions!</p><p>Lets see how we can do that with the <a href="http://www.serverless.com/">Serverless Framework</a>!</p><h3>Setting up the base code</h3><p>Since we’re continuing from where we left off last time, We can clone <a href="https://github.com/HussainAliAkbar/ServerlessPOCs/tree/master/Serverless:%20Breaking%20a%20large%20serverless.yml%20into%20manageable%20chunks/serverless-medium">this repository</a> from the last part. We should have the following project structure:</p><pre>- functions<br>    - randomGenerator<br>          - node_modules<br>          - package.json<br>          - package-lock.json<br>          - randomGenerator.js<br>          - serverless.yml <strong> </strong>  <br>    - uuidGenerator<br>          - node_modules<br>          - package.json<br>          - package-lock.json<br>          - serverless.yml<br>          - uuidGenerator.js<br>- .gitignore<br>- serverless.yml</pre><h3>Adding a VPC and Layer to the functions</h3><p>For the purpose of this post, I’ll be reusing an existing VPC and an existing layer but the same principle should be applicable on other common configurations as well.</p><p>So, go ahead and add the VPC and Layer configuration in the configs of both the functions <strong>randomGenerator</strong> and <strong>uuidGenerator</strong>:</p><pre>// functions/randomGenerator/serverless.yml</pre><pre>randomGenerator:<br>  /** previous config **/</pre><pre>  layers:<br>    - arn:aws:lambda:ap-southeast-1:XXXXXXXXXXXX:layer:lambda-commons-layer:21<br>  vpc:<br>    securityGroupIds:<br>      - sg-XXXXXXXXXXXXXXXXX<br>    subnetIds:<br>      - subnet-XXXXXXXXXXXXXXXXX</pre><pre>// functions/uuidGenerator/serverless.yml</pre><pre>uuidGenerator:<br>  /** previous config **/</pre><pre>  layers:<br>      - arn:aws:lambda:ap-southeast-1:XXXXXXXXXXXX:layer:lambda-commons-layer:21<br>    vpc:<br>      securityGroupIds:<br>        - sg-XXXXXXXXXXXXXXXXX<br>      subnetIds:<br>        - subnet-XXXXXXXXXXXXXXXXX</pre><p>With <strong>layers</strong>, we’re specifying the ARN of the Layer that we want to attach to our lambdas and with <strong>vpc, </strong>we’re specifying the security group and the subnet ids. (Be sure to replace the dummy ids with the correct ones in your AWS Account).</p><p>Run a serverless print and this point and save the console output somewhere. We’ll need this for future comparisons.</p><p>Next, Run serverless deploy and we should be able to see the layer and the vpc settings attached to both of our lambda functions.</p><p>While this approach is okay for a small project, Its not very efficient and manageable when you have hundreds of functions and therefore, <strong>re usability</strong> will help alot in the long run!</p><h3>Creating Common Configurations</h3><ol><li>Start by creating a folder “<strong>configurations</strong>” in the root project folder.</li><li>Create 2 sub folders inside: “<strong>vpc</strong>” and “<strong>layers</strong>”.</li><li>Create a file <strong>config.yml</strong> inside each of the sub folders.</li><li>Move the vpc related config from both the functions to <strong>configurations/vpc/config.yml.</strong></li><li>Move the layer related config from both the functions to <strong>configurations/layers/config.yml.</strong></li></ol><pre>// configurations/vpc/config.yml</pre><pre>vpc:<br>  securityGroupIds:<br>    - sg-XXXXXXXXXXXXXXXXX<br>  subnetIds:<br>    - subnet-XXXXXXXXXXXXXXXXX<br></pre><pre>// configurations/layers/config.yml</pre><pre>layers:<br>  - arn:aws:lambda:ap-southeast-1:XXXXXXXXXXXX:layer:lambda-commons-layer:21</pre><p>The above steps should result in the following project structure:</p><pre>- configurations<br>    - layers<br>          - config.yml <br>    - vpc<br>          - config.yml<br>- functions<br>   /* files inside the function folder */<br>- .gitignore<br>- serverless.yml</pre><p>6. Update the configs of both of our functions:</p><pre>// functions/randomGenerator/serverless.yml</pre><pre>randomGenerator:<br>  /** previous config **/</pre><pre>  layers: ${file(./configurations/layers/config.yml):layers}<br>  vpc: ${file(./configurations/vpc/config.yml):vpc}</pre><pre>// functions/uuidGenerator/serverless.yml</pre><pre>uuidGenerator:<br>  /** previous config **/</pre><pre>  layers: ${file(./configurations/layers/config.yml):layers}<br>  vpc: ${file(./configurations/vpc/config.yml):vpc}</pre><p>And with this, we’ve referenced the newly created configuration files in our function configs. We had to specify the configurations/ ARNs only once in our common configuration files and then reused those wherever needed.</p><p>Run serverless print at this point and compare the output with the previous one. Both of them should be the same. Run serverless deploy to verify the lambdas as well.</p><h3>Taking a step further with stages!</h3><p>We can take this a step further by splitting up our configuration based on different stages. For example, we might have different network settings for different environments and so on. So based on the stage that we’re deploying to, we can load the configurations relevant to that stage / environment.</p><p>To do this, we need to make some minor changes in our folder structure:</p><ol><li>create 2 new files in resources/vpc folder: <strong>config.dev.yml</strong> and <strong>config.staging.yml </strong>(or any other stage that you wish to deploy to)</li><li>Copy the contents of config.yml to config.dev.yml and config.staging.yml. Remove config.yml. We wont need it anymore.</li><li>Update the Security Groups and the Subnet Ids in the config files based on the environment.</li><li>Update the configs of both of our functions:</li></ol><pre>// functions/randomGenerator/serverless.yml</pre><pre>randomGenerator:<br>  /** previous config **/<br>  vpc: ${file(./configurations/vpc/config.${opt:stage, self:provider.stage}.yml):vpc}</pre><pre>// functions/uuidGenerator/serverless.yml</pre><pre>uuidGenerator:<br>  /** previous config **/<br>  vpc: ${file(./configurations/vpc/config.${opt:stage, self:provider.stage}.yml):vpc}</pre><p>Here, we’ve used Serverless variables “<strong>opt:stage</strong>” and “<strong>self.provider.stage</strong>” to reference our configuration file names.</p><p>If for example, “<strong>staging</strong>” is passed during deployment as the stage argument, the opt:stage variable would be set to “<strong>staging</strong>” and our respective config file (<strong>configurations/vpc/config.staging.yml</strong>) will be loaded. If no stage is provided then the default stage in self:provider.stage which is “<strong>dev</strong>” will be loaded and our dev configuration file will be referenced.</p><p>Run serverless print and serverless print -- stage staging to compare the outputs with the previous ones. The only difference you should see should be different vpc settings loading up for different stages. The rest of the output should be the same. Run serverless deploy and serverless print -- stage staging to verify the lambdas as well.</p><h3>And Thats it!</h3><p>This is yet again, a very simple concept but one thats extremely powerful, can be extremely useful in the long run as projects grow and easily portable to many other similar use cases.</p><h3>Conclusion</h3><p>This concludes my series of articles on the Serverless Framework. I had a wonderful time exploring the framework and an even better time documenting it! Feel free to reach out in case of any questions, concerns or feedback!</p><p><strong>PS</strong>: The Source Code for this part has been uploaded on <a href="https://github.com/HussainAliAkbar/ServerlessPOCs/tree/master/Serverless:%20Reusing%20common%20configuration%20across%20functions/serverless-medium">Github </a>for reference.</p><p><strong>This article is a part of my 5 Article Series on the Serverless Framework!</strong></p><p><a href="https://medium.com/@hussain94aa/serverless-managing-environment-variables-efficiently-with-stages-part-1-90c4465d2e64">Part 1: Serverless: Managing environment variables efficiently with stages</a></p><p><a href="https://medium.com/@hussain94aa/serverless-managing-config-for-different-environments-with-s3-and-bash-scripts-ebe6f959a7e9">Part 2: Serverless: Managing config for different environments with S3 and Bash Scripts</a></p><p><a href="https://medium.com/@hussain94aa/serverless-creating-light-and-lean-function-packages-3b382e865834">Part 3: Serverless: Creating light and lean function packages</a></p><p><a href="https://medium.com/@hussain94aa/serverless-breaking-a-large-serverless-yml-into-manageable-chunks-89fc909937c7">Part 4: Serverless: Breaking a large serverless.yml into manageable chunks</a></p><p><a href="https://medium.com/@hussain94aa/serverless-reusing-common-configurations-across-functions-e17ee10e0645">Part 5: Serverless: Reusing common configurations across functions</a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=e17ee10e0645" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Serverless: Breaking a large serverless.yml into manageable chunks]]></title>
            <link>https://medium.com/@HussainAliAkbar/serverless-breaking-a-large-serverless-yml-into-manageable-chunks-89fc909937c7?source=rss-3805c8ea5f9------2</link>
            <guid isPermaLink="false">https://medium.com/p/89fc909937c7</guid>
            <category><![CDATA[aws]]></category>
            <category><![CDATA[azure]]></category>
            <category><![CDATA[serverless-framework]]></category>
            <category><![CDATA[aws-lambda]]></category>
            <category><![CDATA[serverless]]></category>
            <dc:creator><![CDATA[Hussain Ali Akbar]]></dc:creator>
            <pubDate>Thu, 28 May 2020 18:24:55 GMT</pubDate>
            <atom:updated>2020-05-29T04:58:21.617Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*YDyJpHrHPbjP7Fm-rPFtUg.png" /></figure><p>The <a href="https://medium.com/@hussain94aa/serverless-creating-light-and-lean-function-packages-3b382e865834">Serverlerss Framework</a> uses a <strong>serverless.yml </strong>file to manage all the functions. All the configurations related to our environment, our functions and the Cloud Resources that the functions are dependent on reside in this <strong>serverless.yml</strong> file.</p><p>Because of this, the size of this file can quickly get out of hand even with a few functions thus making it <strong>unmanageable and prone to errors and merge conflicts in the long run</strong>.</p><p>Lets look at how we can break a large <strong>serverless.yml</strong> file into manageable chunks!</p><h3>Setting up the base code</h3><p>Clone <a href="https://github.com/HussainAliAkbar/ServerlessPOCs/tree/master/Serverless:%20Creating%20light%20and%20lean%20function%20packages/serverless-medium">this repository</a> from the <a href="https://medium.com/@hussain94aa/serverless-creating-light-and-lean-function-packages-3b382e865834">last part</a>. We should have the following project structure:</p><pre>- functions<br>    - randomGenerator<br>          - node_modules<br>          - package.json<br>          - package-lock.json<br>          - randomGenerator.js<br>    - uuidGenerator<br>          - node_modules<br>          - package.json<br>          - package-lock.json<br>          - uuidGenerator.js<br>- .gitignore<br>- serverless.yml</pre><p>And this should be the content of the <strong>serverless.yml:</strong></p><pre>service: serveless-medium<br><br>provider:<br>  name: aws<br>  runtime: nodejs12.x<br><br>functions:<br>  randomGenerator:<br>    handler: functions/randomGenerator/randomGenerator.handler<br>    package:<br>      individually: true<br>      exclude:<br>        - &quot;./**&quot;<br>      include:<br>        - ./functions/randomGenerator/randomGenerator.js<br>        - ./functions/randomGenerator/node_modules/**<br>        - ./functions/randomGenerator/package.json<br>  uuidGenerator:<br>    handler: functions/uuidGenerator/uuidGenerator.handler<br>    package:<br>      individually: true<br>      exclude:<br>        - &quot;./**&quot;<br>      include:<br>        - ./functions/uuidGenerator/uuidGenerator.js<br>        - ./functions/uuidGenerator/node_modules/**<br>        - ./functions/uuidGenerator/package.json</pre><p>You can see the size of this file with just 2 minimalist functions and 0 additional configurations. This size will only increase as our functions and configurations grow.</p><h3>Breaking the file</h3><p>Start by creating 2 new <strong>serverless.yml</strong> files in the function sub folders <strong>randomGenerator</strong> and <strong>uuidGenerator. </strong>And copy the configuration of each function to its respective <strong>serverless.yml</strong> file:</p><pre>// functions/randomGenerator/serverless.yml</pre><pre>randomGenerator:<br>  handler: functions/randomGenerator/randomGenerator.handler<br>  package:<br>    individually: true<br>    exclude:<br>      - &quot;./**&quot;<br>    include:<br>      - ./functions/randomGenerator/randomGenerator.js<br>      - ./functions/randomGenerator/node_modules/**<br>      - ./functions/randomGenerator/package.json</pre><pre><br>// functions/uuidGenerator/serverless.yml</pre><pre>uuidGenerator:<br>  handler: functions/uuidGenerator/uuidGenerator.handler<br>  package:<br>    individually: true<br>    exclude:<br>      - &quot;./**&quot;<br>    include:<br>      - ./functions/uuidGenerator/uuidGenerator.js<br>      - ./functions/uuidGenerator/node_modules/**<br>      - ./functions/uuidGenerator/package.json</pre><p>Once this is done, update the main <strong>serverless.yml </strong>in the root directory:</p><pre>functions:<br>  - ${file(./functions/randomGenerator/serverless.yml)}<br>  - ${file(./functions/uuidGenerator/serverless.yml)}</pre><p><strong>And thats it!</strong> We’ve successfully broken down our main<strong> serverless.yml</strong> template file into 3. The main file contains provider level settings and references to all the functions that we have. The remaining 2 sub files in the function sub folders contain only the configuration related to those respective functions.</p><p>We can keep on doing this for as many functions as we want. When deploying, Serverless will go through these references and compile the entire configuration itself.</p><p>Our folder structure at this point should be almost as similar as before with the addition of 2 new <strong>serverless.yml</strong> files:</p><pre>- functions<br>    - randomGenerator<br>          - node_modules<br>          - package.json<br>          - package-lock.json<br>          - randomGenerator.js<br>          - <strong>serverless.yml</strong></pre><pre>    - uuidGenerator<br>          - node_modules<br>          - package.json<br>          - package-lock.json<br>          - <strong>serverless.yml</strong><br>          - uuidGenerator.js<br>- .gitignore<br>- serverless.yml</pre><p>Doing a serverless print at this point should result in the exact same output that we had at the start of this post. Go ahead and run serverless print and then serverless deploy to verify the output and the lambda functions.</p><p>This might look trivial but is a very powerful concept that can be reused across multiple use cases in order to have maintainable folder structures and files.</p><h3>Whats next?</h3><p>In the <a href="https://medium.com/@hussain94aa/serverless-reusing-common-configurations-across-functions-e17ee10e0645">next part</a>, We’ll build on top of the same concept of breaking down serverless.yml template files and look into how we can reuse common configurations across multiple functions.</p><p><strong>PS</strong>: The Source Code for this part has been uploaded on <a href="https://github.com/HussainAliAkbar/ServerlessPOCs/tree/master/Serverless:%20Breaking%20a%20large%20serverless.yml%20into%20manageable%20chunks/serverless-medium">Github</a> for reference.</p><p><strong>This article is a part of my 5 Article Series on the Serverless Framework!</strong></p><p><a href="https://medium.com/@hussain94aa/serverless-managing-environment-variables-efficiently-with-stages-part-1-90c4465d2e64">Part 1: Serverless: Managing environment variables efficiently with stages</a></p><p><a href="https://medium.com/@hussain94aa/serverless-managing-config-for-different-environments-with-s3-and-bash-scripts-ebe6f959a7e9">Part 2: Serverless: Managing config for different environments with S3 and Bash Scripts</a></p><p><a href="https://medium.com/@hussain94aa/serverless-creating-light-and-lean-function-packages-3b382e865834">Part 3: Serverless: Creating light and lean function packages</a></p><p><a href="https://medium.com/@hussain94aa/serverless-breaking-a-large-serverless-yml-into-manageable-chunks-89fc909937c7">Part 4: Serverless: Breaking a large serverless.yml into manageable chunks</a></p><p><a href="https://medium.com/@hussain94aa/serverless-reusing-common-configurations-across-functions-e17ee10e0645">Part 5: Serverless: Reusing common configurations across functions</a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=89fc909937c7" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Serverless: Creating light and lean function packages]]></title>
            <link>https://medium.com/@HussainAliAkbar/serverless-creating-light-and-lean-function-packages-3b382e865834?source=rss-3805c8ea5f9------2</link>
            <guid isPermaLink="false">https://medium.com/p/3b382e865834</guid>
            <category><![CDATA[serverless]]></category>
            <category><![CDATA[aws-lambda]]></category>
            <category><![CDATA[azure-functions]]></category>
            <category><![CDATA[packaging]]></category>
            <category><![CDATA[serverless-framework]]></category>
            <dc:creator><![CDATA[Hussain Ali Akbar]]></dc:creator>
            <pubDate>Thu, 28 May 2020 09:00:51 GMT</pubDate>
            <atom:updated>2020-05-29T04:58:37.901Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*YDyJpHrHPbjP7Fm-rPFtUg.png" /></figure><p>Its a best practice throughout all Serverless Platforms to keep the function packages small, lightweight and lean in order to improve performance and cold start times. A general rule of thumb is to only include those dependencies in a package which are absolutely necessary and getting rid of all redundant items.</p><p>The <a href="https://www.serverless.com/">Serverless Framework</a> allows us to manage our Cloud Functions on AWS and Azure. However, by default, the framework packages everything together for all the functions which results in redundant dependencies being packaged together and increasing the package sizes. This can be fixed with just a few configurations!</p><h3>Setting up the base project!</h3><p>Create a project from scratch by running the serverless cli:</p><pre>serverless</pre><p>The CLI should set up the basic structure. I’ll be using <strong>Node.js</strong> and <strong>AWS</strong> for this post but feel free to change the provider and the runtime. Replace the contents of <strong>serverless.yml:</strong></p><pre>service: serveless-medium<br><br>provider:<br>  name: aws<br>  runtime: nodejs12.x<br>  region: ap-southeast-1<br>  role: arn:aws:iam::XXXXXXXXXXXX:role/lambda-role</pre><p>For this example, we wont use the default handler but create 2 of our own. <strong>randomGenerator.js</strong> and <strong>uuidGenerator.js</strong>:</p><pre><strong>//randomGenerator.js</strong></pre><pre>&#39;use strict&#39;;<br>const Chance = require(&#39;chance&#39;);<br>module.exports.handler = async event =&gt; {<br>  const chance = new Chance();<br>  return {<br>    statusCode: 200,<br>    body: chance.string()<br>  };<br>};<br></pre><pre><strong>//uuidGenerator.js</strong></pre><pre>&#39;use strict&#39;;<br>const <strong><em>uuid </em></strong>= require(&#39;uuid&#39;);<br>module.exports.handler = async event =&gt; {<br>  const generatedId = <strong><em>uuid</em></strong>.v4();<br>  return {<br>    statusCode: 200,<br>    body: generatedId<br>  };<br>};</pre><p>Lets add these function definitions at the end of our <strong>serverless.yml:</strong></p><pre>functions:<br>  randomGenerator:<br>    handler: randomGenerator.handler<br>  uuidGenerator:<br>    handler: uuidGenerator.handler</pre><p>Since we’re using some dependencies in our functions (<strong>chance.js in randomGenerator.js and uuid.js in uuidGenerator.js</strong>), we need to add these dependencies to our project as well. So create a <strong>package.json</strong> and do <strong>npm install:</strong></p><pre>//<strong>package.json</strong></pre><pre>{<br>  &quot;dependencies&quot;: {<br>    &quot;chance&quot;: &quot;^1.1.6&quot;,<br>    &quot;uuid&quot;: &quot;^3.3.3&quot;<br>  }<br>}</pre><p>The project structure should look like this:</p><pre>- node_modules<br>- .gitignore<br>- package.json<br>- package-lock.json<br>- randomGenerator.js<br>- serverless.yml<br>- uuidGenerator.js</pre><p>Run serverless deploy and we should be able to see our lambda functions created. This is what should be created:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*DdqR3vV1Tfc-C0dqu39SGA.png" /><figcaption>uuid generator function</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*iPg2Wwt43Szx6HVFn1f1ag.png" /><figcaption>random generator function</figcaption></figure><p>Whats wrong here is that even those dependencies / handler functions which are unrelated have been packaged together! <strong>the chance.js package and randomGenerator.js should not be packaged with the uuidGenerator function and vice versa</strong>. This bloating will increase even more as our functions and their dependencies increase.</p><p>Our aim is to separate out the dependencies of each function and only package the required dependencies.</p><h3>Separating the functions and their dependencies!</h3><p>In order to do this, we need to create a folder structure.</p><ol><li>Create a folder “<strong>functions</strong>”.</li><li>Create 2 sub folders “<strong>uuidGenerator</strong>” and “<strong>randomGenerator</strong>”.</li><li>Move <strong>randomGenerator.js </strong>inside the <strong>randomGenerator</strong> sub folder and move <strong>uuidGenerator.js</strong> inside the <strong>uuidGenerator</strong> sub folder.</li><li>Break the <strong>package.json</strong> file into two and add to the respective folders:</li></ol><pre>// functions/randomGenerator/package.json</pre><pre>{<br>  &quot;dependencies&quot;: {<br>    &quot;chance&quot;: &quot;^1.1.6&quot;<br>  }<br>}</pre><pre>// functions/uuidGenerator/package.json<br>{<br>  &quot;dependencies&quot;: {<br>    &quot;uuid&quot;: &quot;^3.3.3&quot;<br>  }<br>}</pre><p>5. Run <strong>npm install</strong> inside each subfolder to install the dependencies.</p><p>6. Delete <strong>node_modules</strong> from the root directory.</p><p>Our project structure should look like this now:</p><pre>- functions<br>    - randomGenerator<br>          - node_modules<br>          - package.json<br>          - package-lock.json<br>          - randomGenerator.js<br>    - uuidGenerator<br>          - node_modules<br>          - package.json<br>          - package-lock.json<br>          - uuidGenerator.js<br>- .gitignore<br>- serverless.yml</pre><p>Upon looking at the node_modules directory in each of the function folders, we can see that only those packages have been added which are relevant to that function.</p><h3>Updating serverless.yml</h3><p>All we need to do now is update the <strong>serverless.yml</strong> file in order to to tell the framework how we want to package our functions. Replace the functions block with the following:</p><pre>functions:<br>  randomGenerator:<br>    handler: functions/randomGenerator/randomGenerator.handler<br>    package:<br>      individually: true<br>      exclude:<br>        - &quot;./**&quot;<br>      include:<br>        - ./functions/randomGenerator/randomGenerator.js<br>        - ./functions/randomGenerator/node_modules/**<br>        - ./functions/randomGenerator/package.json<br>  uuidGenerator:<br>    handler: functions/uuidGenerator/uuidGenerator.handler<br>    package:<br>      individually: true<br>      exclude:<br>        - &quot;./**&quot;<br>      include:<br>        - ./functions/uuidGenerator/uuidGenerator.js<br>        - ./functions/uuidGenerator/node_modules/**<br>        - ./functions/uuidGenerator/package.json</pre><ol><li>Since our handler functions now exist inside a folder, we need to update the path. <strong>For example</strong>, in case of the randomGenerator function, the handler now becomes: <strong>functions/randomGenerator/randomGenerator.handler.</strong></li><li>We have added a new property called <strong>package </strong>which allows us to have more control over the packaging of our functions.</li><li><strong>individually: true </strong>tells the framework to package the functions individually.</li><li><strong>Exclude</strong> tells the framework to not package anything by default with the function. We might have entire test suites and other dev dependencies which are not needed in the package. With this, we can exclude all such items.</li><li>With <strong>Include, </strong>we individually and explicitly specify the files and folders that we want to package with a specific function. <strong>For example</strong>, in case of the randomGenerator function, we only want to package the handler file, the node_modules and the package.json.</li></ol><p>run serverless deploy and our lambda functions should be updated like this:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*rsAHi79sF-_f4nFwgZ7Pgw.png" /><figcaption>only the selected items have been packaged for randomGenerator function</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*xfa4xxOssen3JzrSDaP6yA.png" /><figcaption>only the selected items have been packaged for uuidGenerator function</figcaption></figure><p>We can see that as opposed to before, only the items that we selected have been packaged thereby making our package size as minimal as possible. Go ahead and try invoking the lambda functions with test events.</p><h3><strong>And that’s it!</strong></h3><p>These very simple configurations in the Serverless Framework can enable us to make our packages as lightweight as possible and only include the absolutely necessary items. No matter how many functions we have or how large our codebase grows, Our individual functions will always be light and lean.</p><p>In the <a href="https://medium.com/@hussain94aa/serverless-breaking-a-large-serverless-yml-into-manageable-chunks-89fc909937c7">next part</a>, We’ll see how we can break down the <strong>serverless.yml</strong> file into multiple chunks for better management!</p><p><strong>PS</strong>: The Source Code for this part has been uploaded on <a href="https://github.com/HussainAliAkbar/ServerlessPOCs/tree/master/Serverless:%20Creating%20light%20and%20lean%20function%20packages/serverless-medium">Github</a> for reference.</p><p><strong>This article is a part of my 5 Article Series on the Serverless Framework!</strong></p><p><a href="https://medium.com/@hussain94aa/serverless-managing-environment-variables-efficiently-with-stages-part-1-90c4465d2e64">Part 1: Serverless: Managing environment variables efficiently with stages</a></p><p><a href="https://medium.com/@hussain94aa/serverless-managing-config-for-different-environments-with-s3-and-bash-scripts-ebe6f959a7e9">Part 2: Serverless: Managing config for different environments with S3 and Bash Scripts</a></p><p><a href="https://medium.com/@hussain94aa/serverless-creating-light-and-lean-function-packages-3b382e865834">Part 3: Serverless: Creating light and lean function packages</a></p><p><a href="https://medium.com/@hussain94aa/serverless-breaking-a-large-serverless-yml-into-manageable-chunks-89fc909937c7">Part 4: Serverless: Breaking a large serverless.yml into manageable chunks</a></p><p><a href="https://medium.com/@hussain94aa/serverless-reusing-common-configurations-across-functions-e17ee10e0645">Part 5: Serverless: Reusing common configurations across functions</a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=3b382e865834" width="1" height="1" alt="">]]></content:encoded>
        </item>
    </channel>
</rss>