<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Arvind Naidu on Medium]]></title>
        <description><![CDATA[Stories by Arvind Naidu on Medium]]></description>
        <link>https://medium.com/@naiduarvind?source=rss-faacd45b333c------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/1*UM2T8slx5NQgg6_IL0yp7A.png</url>
            <title>Stories by Arvind Naidu on Medium</title>
            <link>https://medium.com/@naiduarvind?source=rss-faacd45b333c------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Thu, 19 Nov 2026 13:16:01 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@naiduarvind/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[First Quarter As An SRE]]></title>
            <link>https://medium.com/servicerocket-eng/first-quarter-as-an-sre-11117866a64e?source=rss-faacd45b333c------2</link>
            <guid isPermaLink="false">https://medium.com/p/11117866a64e</guid>
            <category><![CDATA[servicerocket]]></category>
            <category><![CDATA[mapping]]></category>
            <category><![CDATA[okr]]></category>
            <category><![CDATA[site-reliability-engineer]]></category>
            <dc:creator><![CDATA[Arvind Naidu]]></dc:creator>
            <pubDate>Mon, 07 Jan 2019 05:32:10 GMT</pubDate>
            <atom:updated>2022-04-28T02:27:55.587Z</atom:updated>
            <content:encoded><![CDATA[<h3>First Quarter as an SRE</h3><p>Ever since the Site Reliability Engineering (SRE) book was published in 2016, we have been improving our approach to operations by gradually adopting some tools mentioned in the book, e.g. defining Service Level Objectives (SLO), learning and preventing failures from incident reports, etc. Our selective adoption has met with limited success, mainly due to how an Operations Engineer was still being perceived as a firefighter. Before we were renamed to Operations Engineers, we were once called System Operators (SysOps).</p><p>Three months ago, we made the decision to commit holistically to the SRE <a href="https://landing.google.com/sre/sre-book/chapters/part2/">philosophy and principles</a>, by first unplugging our Operations Engineers from product engineering squads to form our first ever SRE squad. I was tasked to spearhead this initiative, along with my team member <a href="https://medium.com/u/cd43a6aa4531">Jayaseelan</a>.</p><p>The first thing was to revamp our job description by the JD framework in Engineering, which takes inspiration from <a href="https://www.amazon.com/Accountability-Experience-Poster-Linda-Galindo/dp/0470604786">Linda Galindo’s work on accountability</a>. A JD contains a brief introduction, measures of success, a set of responsibilities and accountabilities, along with principles to on-call duty, on-boarding processes, team mobility, etc.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/754/1*ar8cC9axgv9hRRDHe4vMsQ.png" /><figcaption>Snippet of the Site Reliability Engineering job description.</figcaption></figure><p>The next thing we did defined our 3 -month journey, which was to construct the Objective Key Results (<a href="https://rework.withgoogle.com/guides/set-goals-with-okrs/steps/introduction/">OKR</a>) for the quarter and set to work. OKR</p><p>Now that three months have passed, I am writing this to share my reflection and future plans for our SRE team.</p><h4>Leading as rookie</h4><p>I was on pins and needles when I was asked to re-build the JD for the role and hiring resources such as pre-screening and interview questions.</p><ul><li>The leaders I work with have great technical and leadership skills, while I had very little to none.</li><li>My teammates were all more experienced in their respective domains than I was (they still are and I am continuously learning from them).</li></ul><p>I am still by no means a leader by title or skills but so far it has turned out to work especially well.</p><h4>OKR</h4><p>We set a simple yet clear objective to begin with: form an SRE squad. I am very happy with the results of our work, done by just two people (and <a href="https://www.linkedin.com/jobs/view/1029669368/">we are hiring for the third member</a>!). There are more challenges ahead and imagining that makes me excited.</p><p><strong>1st Key Result: </strong>stabilize our platform in both private and public clouds. We started removing and improving components to reduce outages. While it is impossible to hit zero outages, our effort amounted to improve <em>signal to noise ratio </em>that had a significant reduction in downtime which directly resulted in increased engineer happiness.</p><ul><li>We removed all Puppetmodules which were baked into the container that was a direct result of creating a container image from a VM.</li><li>We moved away from an unstable version of <a href="http://supervisord.org/">supervisor</a> to <a href="https://cr.yp.to/daemontools/svscan.html">svscan</a> to monitor and control our processes.</li><li>We instrumented and profiled our application thoroughly which helped us correctly tune our JVM instances to near perfection.</li><li>We upgraded Tomcat from v7 to v9 and are thoroughly testing for an optimum tune, resulting in higher throughput and a fail-fast Tomcat (meow!).</li><li>We removed httpd from the container to give room for other processes and replaced AJP connector in Tomcat withNIO protocol.</li></ul><p>Some efforts above might seem small and straightforward but it was a direct consequence of the SRE squad formation. This relieved both of us from the constant pressure of tackling outages at wee hours with immediate remedies and product request but instead focus on long term solutions.</p><blockquote>Can you imagine the immense level of joy seeing all 12 Puppetmodules and a supervisor being removed from the container image which causes our application to restart at random time of the day in a week?</blockquote><p><em>Note: There will be follow up posts on some of the efforts above. Stay tuned!</em></p><p><strong>2nd Key Result: </strong>increase reliability of our platform through benchmarking. The work requires us to identify the availability of crucial components in the user execution flow.</p><ul><li>Define and establish <em>SLA</em> &amp; <em>SLO</em> for trace components of the flow.</li><li>Monitor and measure performance of the newly defined <em>SLA </em>&amp; <em>SLO</em>.</li></ul><p>This key result is still ongoing and it involves collaboration with our Product team in Santiago (Chile) and a good understanding of our end-user behaviors.</p><h4><strong>Near Future</strong></h4><p>We are currently exploring on-the-fly garbage collection (GC) analysis, collecting metrics on CPU &amp; memory profiles via <a href="https://perf.wiki.kernel.org/index.php/Main_Page">perf </a>and <a href="http://www.brendangregg.com/FlameGraphs/cpuflamegraphs.html">Flame Graphs</a>, and also benchmarking our application with different JVMs (e.g. <a href="https://www.graalvm.org/">GraalVM</a> and <a href="https://www.azul.com/">Zulu</a>). We are also exploring the adoption of a unique APM tool along with implementing OpenTracing to go with it.</p><p>In this quarter, we are looking to <a href="https://www.linkedin.com/jobs/view/1029669368/">hire new team mate</a>, to help building our vision of great operations, tooling, and culture.</p><h4>Personal Musing</h4><p>I have been musing on these 2 metaphors, and tinkering on the idea to broaden their adoption in our team:</p><p><strong>Argument by mapping.</strong> To arrive to the best decision in a discussion, I would like us to see arguments as proofs (as is common among the mathematicians), or even better, prove arguments with <a href="https://medium.com/wardleymaps">mapping</a>. Driving discussion with this degree of rational would harness better decisions with bigger impact and greater value for the business.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*UAYnfMQs_sLhkNl9dVFzTA.jpeg" /><figcaption>Mapping on developer visibility of the platform and its components against its evolution.</figcaption></figure><p><strong>Time is Money. </strong>With the new squad making headway for the SysOps engineers to be in SRE, most operational related tasks are now owned by product engineers instead. This provides us with ample time that otherwise would have been taken up to solve outages and write incident reports. As a result, the number of infrastructure-related outages has been significantly reduced, which in return gives back more time now for engineers to focus on what matters for the business.</p><p>If you would like to share how you run an SRE team effectively, or to implement OKR with a shared vision, please let me a comment here, or we could meet up over coffee to learn from each other.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=11117866a64e" width="1" height="1" alt=""><hr><p><a href="https://medium.com/servicerocket-eng/first-quarter-as-an-sre-11117866a64e">First Quarter As An SRE</a> was originally published in <a href="https://medium.com/servicerocket-eng">ServiceRocket Engineering</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Faris Amali Alis, Agile Developer, shares on his experience working at ServiceRocket]]></title>
            <link>https://medium.com/servicerocket-eng/faris-amali-alis-agile-developer-share-on-his-experience-working-in-servicerocket-73bb3a003375?source=rss-faacd45b333c------2</link>
            <guid isPermaLink="false">https://medium.com/p/73bb3a003375</guid>
            <category><![CDATA[servicerocket]]></category>
            <category><![CDATA[software-development]]></category>
            <category><![CDATA[kubernetes]]></category>
            <category><![CDATA[keratoconus]]></category>
            <category><![CDATA[programming-paradigms]]></category>
            <dc:creator><![CDATA[Arvind Naidu]]></dc:creator>
            <pubDate>Mon, 31 Dec 2018 02:51:18 GMT</pubDate>
            <atom:updated>2018-12-31T07:39:16.248Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*_Or02BiEQYZY7UoC-PPyNg.jpeg" /><figcaption>Faris Amali Alis — the one sitting down proudly showing off his basketball team.</figcaption></figure><p>Q: <strong>What is your name and how long have you been at ServiceRocket?</strong></p><p>A: My name is Faris Amali Alis. I work out of the Kuala Lumpur office and I have been a Rocketeer for roughly a year and a month.</p><p>Q: <strong>What is your current role in ServiceRocket?</strong></p><p>A: Officially, my title is Agile Developer and I am currently attached to the Reliability team where the current focus is to make the necessary changes throughout the application ecosystem for the big migration to Kubernetes.</p><p>Q: <strong>Why did you decide to join ServiceRocket? What did you do before?</strong></p><p>A: I was previously working for a startup selling t-shirts but a friend who got a job here told me about how awesome the Engineering team at ServiceRocket is. ServiceRocket had the reputation for its solid engineering practices despite being a relatively small development shop. Some of the interesting practices that got me hooked was 1) that they are big on functional programming paradigms 2) they adopt bleeding edge technologies fast 3) and architectural decisions are autonomous to teams.</p><p>Q: <strong>What makes working at ServiceRocket unique or different from your previous experiences?</strong></p><p>A: The uniqueness of any working experience, in my opinion, derives from the engineering challenges that a particular organization faces. ServiceRocket’s in-house products are currently scaling quite significantly, therefore poses very interesting engineering challenges both at application level as well as at operational level.</p><p>Q: <strong>What unique challenge or problem have you worked on at ServiceRocket?</strong></p><p>A: I’m particularly proud of shipping our own property-mapping rule engine for our authorization layer because it enables faster customer onboarding process and allowing greater flexibility on how identity data are captured and stored within the application. This project commenced earlier this year and completed just over a month.</p><p>Q: <strong>Have you had any transformational or inspirational moment from someone at ServiceRocket?</strong></p><p>A: Sure! I would attribute most of my transformational moments from the 1:1s with my tribe lead <a href="https://medium.com/servicerocket-eng/david-chui-engineering-tribe-lead-shares-how-he-inspire-engineers-at-servicerocket-1d9f2b857fb2">David Chui</a>, who has been very supportive and empowering to my personal growth.</p><p>I’ve always believed that 5 years of experience can be obtained in just a year if mentored and guided well, and without doubt I’ve avoided pitfalls and learnt to become <a href="https://www.kitchensoap.com/2012/10/25/on-being-a-senior-engineer/">mature</a> in my craft.</p><p>Q: <strong>Tell us about a moment your team mate helped you to overcome a challenge where time was of essence?</strong></p><p>A: We had a service outage for ~7 minutes for one of our customers due to a misconfigured deployment pipeline. During our post-deployment testing, I’ve identified that one of our customers was having login issues. I immediately called upon my (only) teammate to discuss the issue at hand and devise a recovery plan. We managed to recover from the outage and rectified the misconfiguration. I would credit the success of handling the situation to the professionalism that this company has ingrained in its culture.</p><p>Q: <strong>Have you had transformational or inspirational moments at ServiceRocket?</strong></p><p>A: To me, team dynamics are `status quo` that has to be earned. It requires time, effort and a whole lot of patience to making mutual concessions. Once achieved, not only does the working environment becomes more comfortable to work in, it will ultimately garner collective drive for success of the product.</p><p>Q: <strong>What are your hobbies? Or what can we find you doing outside of work?</strong></p><p>A: I have a lifelong desire to compose and produce music one day but in the meantime, you can find me on the court managing and coaching a community basketball team after after work and on weekends.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*tA5AkHHijns9tZ29Yiywaw.jpeg" /><figcaption>Faris seen coaching his basketball team.</figcaption></figure><p>Q: <strong>Is there something interesting about you that not a lot of people know?</strong></p><p>A: I recently underwent eye surgery to both of my eyes to correct Keratoconus. A condition that resulted from childhood eye scratching habit. I’m mentioning this here as a PSA for those with eye scratching habit out there. Stop scratching your eyes, use eye drops instead!</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=73bb3a003375" width="1" height="1" alt=""><hr><p><a href="https://medium.com/servicerocket-eng/faris-amali-alis-agile-developer-share-on-his-experience-working-in-servicerocket-73bb3a003375">Faris Amali Alis, Agile Developer, shares on his experience working at ServiceRocket</a> was originally published in <a href="https://medium.com/servicerocket-eng">ServiceRocket Engineering</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Making The Invisible Visible: Costs]]></title>
            <link>https://medium.com/servicerocket-eng/making-the-invisible-visible-costs-66f53c2b5a64?source=rss-faacd45b333c------2</link>
            <guid isPermaLink="false">https://medium.com/p/66f53c2b5a64</guid>
            <category><![CDATA[dashboard]]></category>
            <category><![CDATA[aws]]></category>
            <category><![CDATA[budget]]></category>
            <category><![CDATA[datadog]]></category>
            <dc:creator><![CDATA[Arvind Naidu]]></dc:creator>
            <pubDate>Sat, 26 May 2018 07:29:17 GMT</pubDate>
            <atom:updated>2018-05-31T15:57:09.027Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*ItSi4UZVJScQlzTPeELGGg.jpeg" /><figcaption>Datadog Dashboard for AWS Budgets</figcaption></figure><blockquote><strong>ServiceRocket (SR) Engineers are unable to make optimal decisions because they do not consider the true cost of services being used.</strong></blockquote><p>It is easy to see a price tag, but it is difficult to take into account all of the factors that make up a product’s true cost.</p><p>How can hidden costs be made visible? With a bit of effort, true monetary costs of many services used primarily on Amazon Web Services (AWS) can be calculated and made visible.</p><h4>Cost Allocation Tags</h4><p>AWS by default allows a tag to be assigned to an AWS resource, i.e EC2, RDS, etc. Tags can be used to organize resources, and cost allocation tags to track AWS costs on a detailed level.</p><p>AWS has two types of cost allocation tags, an <em>AWS generated tag</em> and <em>user-defined tags.</em></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*_sJSpvK28jAkGPTiwEUNcQ.png" /><figcaption>Activating Cost Allocation Tags in AWS</figcaption></figure><p>AWS defines, creates, and applies AWS generated tag, and user-defined tags can be defined, created and applied by the user. Both types of tags must be activated before they can appear on a cost allocation report or to be used for budgets in the Cost Allocation Tags page under <strong>Billing Management Console</strong>.</p><p>SR Engineering has two platforms, Learndot and Apps which is comprised of many teams that develop different products utilizing different environments. The image below illustrates the user-defined tags created and applied on AWS resources created by Engineering.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*xjSOt6z5liYXqMaETVfBGw.png" /><figcaption>User-defined Cost Allocation Tags for AWS Resources</figcaption></figure><p>All the teams in Learndot and Apps either use Terraform or Cloudformation with a Jenkins pipeline to provision their infrastructure and deploy their applications, thus implementing the user-defined tags above was fairly straightforward.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/738/1*JnXR4EDo5CTI05t0ROtz9Q.jpeg" /><figcaption>Cost Allocation Tags Implementation with Terraform</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/840/1*eoEOTMRn2NNIGiAZalXfOQ.jpeg" /><figcaption>Cost Allocation Tags Implementation with CloudFormation</figcaption></figure><h4>Budget on AWS</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*RkZG3vNPCF4MOdvZcz20kg.jpeg" /><figcaption>Budgets on AWS</figcaption></figure><p>After having the teams implement cost allocation tags as one of the Ops Missions, <a href="https://medium.com/u/c39f98f7fe46">Yuen-Chi Lian</a> created some budgets with guesstimated limits for services such as EC2, RDS, CloudWatch, and CDN based on cost usage by Engineering <em>(quarterly)</em> as a whole and by Learndot and Apps <em>(monthly)</em>.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*KMTk2-b-LQuWqVDtmjNEdw.png" /><figcaption>Budget Implementations for ServiceRocket Engineering</figcaption></figure><p>These budgets were implemented with notifications to the Engineering Operations (Ops) mailing group. Ops receives emails in regards to a service for a particular budget in AWS surpassing the limit set.</p><p>However, this did not provide the ability to visually understand if and when the budget needs to be realigned or if a service was costing way more than it was intended and requires attention. If needed to be done, it would require tremendous manual effort.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*I1jx2o4n55v2OxeWxktnZA.jpeg" /><figcaption>AWS Budget Email Notifications for Actual &amp; Forecasted Amount</figcaption></figure><h4>Dashboard on Datadog</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Cft-i1oCvJqLh6JXZoBATg.jpeg" /><figcaption>Datadog Dashboard for AWS Budgets (Night Mode)</figcaption></figure><p>Hence, the Datadog dashboard above was setup to display the budgets from AWS which provides better visibility on a monthly basis and over a longer period of time for each service and platform which otherwise could not have been done via the emails Ops received.</p><p>Additional continuous efforts include:</p><ol><li>Ops to analyze AWS dashboards on a quarterly basis and set close to realistic budget limits on all the AWS services and account.</li><li>Team to setup similar dashboard for products under Learndot (e.g Moo, Hamburger, and etc) and Apps (e.g. SFJC, SafetyOfficer, and etc) thus taking on cost ownership.</li></ol><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*neaImShxgliSx8MYu3eX3A.jpeg" /><figcaption>AWS Budget Datadog Dashboard for Moo Product under the Learndot Platform</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*xkcRydCLar4i0Hv4lkqwRg.jpeg" /><figcaption>AWS Budget Datadag Dashboard for Hambuger Product under the Learndot Platform</figcaption></figure><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=66f53c2b5a64" width="1" height="1" alt=""><hr><p><a href="https://medium.com/servicerocket-eng/making-the-invisible-visible-costs-66f53c2b5a64">Making The Invisible Visible: Costs</a> was originally published in <a href="https://medium.com/servicerocket-eng">ServiceRocket Engineering</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
    </channel>
</rss>