<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Lior Avni on Medium]]></title>
        <description><![CDATA[Stories by Lior Avni on Medium]]></description>
        <link>https://medium.com/@lior.avni?source=rss-2496de0b948f------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/1*l9bLTUSl147BjwMwYi-SLQ.jpeg</url>
            <title>Stories by Lior Avni on Medium</title>
            <link>https://medium.com/@lior.avni?source=rss-2496de0b948f------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Mon, 16 Nov 2026 18:19:17 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@lior.avni/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[Managing major incidents from home — What I learned from Mission Impossible]]></title>
            <link>https://medium.com/gett-engineering/managing-major-incidents-from-home-what-i-learned-from-ethan-hunt-cfc584715683?source=rss-2496de0b948f------2</link>
            <guid isPermaLink="false">https://medium.com/p/cfc584715683</guid>
            <category><![CDATA[incident-management]]></category>
            <category><![CDATA[mission-impossible]]></category>
            <category><![CDATA[tech-support]]></category>
            <category><![CDATA[devops]]></category>
            <category><![CDATA[root-cause-analysis]]></category>
            <dc:creator><![CDATA[Lior Avni]]></dc:creator>
            <pubDate>Fri, 10 Apr 2020 13:06:21 GMT</pubDate>
            <atom:updated>2020-04-23T12:50:17.039Z</atom:updated>
            <content:encoded><![CDATA[<h3>Managing major incidents from home — What I learned from Mission Impossible</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*hIzxWLisC65ep7F-" /></figure><p>Full disclosure — I am not Tom Cruise (Shocker, I know 🙃). What I am, is an Incident manager, who, not unlike Ethan hunt (portrayed by the mighty Tom), also has to deal with uncertainties in life and surprises. All the more in this day and age where COVID-19 forced us all to stay at home and be introduced to a new reality. In the following blog post, I’ll share the change our IM (Incident Management / Impossible Mission) process underwent to support working from home.</p><h3>Monitoring — tweaking the “false positives”</h3><p>The number one tool of an Incident Manager is an extensive monitoring system.</p><p>As you’ve read in my previous blog posts, we’ve set up a wide array of monitoring tools. COVID-19 effect on the world forced a change on the business metrics, bringing with it a lot of “false positive” alerts to the various business (and application) monitors.</p><p>While in normal times, “false positives” are a “once in a while” nuisance at best which can be dealt with as a task, nowadays, where time is a precious commodity, they are a major hassle that needs to be dealt with as soon as possible. During the last month, we carefully reviewed all our monitors, respective thresholds and business needs and applied the relevant changes to each tool. This way, we don’t have to respond to a non-existing incident during strange hours…</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/444/0*vP_pFfl6DRpCdRMh.jpg" /></figure><p>I also mentioned that by utilizing the out-of-the-box capabilities of most Application Performance Monitoring tools, you can also get a pre-warning on incidents (and save a lot of money for your business by doing so). Well, as Louis Pasteur said “Chance favors the prepared mind” (Or was it that bad guy from “<a href="https://www.imdb.com/title/tt0114781/">Under Siege 2</a>” 🤔… I get confused these days 🙂) — so we prepared and completed mapping all our external services and respective alerts.</p><p>Last item in this section is your 3rd party providers. We use several vendors for many of our services (Finance, Cloud etc…). If one of them has a major incident, it affects us immediately. We carefully reviewed all our current connections, reached out to all our vendors and synced on who is on-call for each of them.</p><p>I don’t need to quote good old Louis again, right? So let’s move on 🤓</p><h3>Transparency — Don’t keep secrets</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*zHyACaEhoPdNg63q" /></figure><p>These are uncertain times. We’re all at home, some of us not always aware of what’s going on in other teams, or production affecting incidents. What became a habit in our day to day work is now a core value when working from home on a major incident: <strong>Always</strong> share the data (be it a risky deployment you need to carry out as a developer, or maintenance work you need to carry out as a DevOps engineer).</p><p>Every incident has a root cause analysis (RCA) conducted and written, every RCA is shared with everyone in the company and all are welcomed to share inputs and suggestions for improvement. When you’re together in everything, you win big — even when the mission seems impossible (see what I did here? 😁)</p><h3>Communication when you can’t see anyone</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*bnlSEzTnAuJRoF1G" /></figure><p>In normal times, when we’ve faced a major incident, besides the usual notification channels we use to alert all stakeholders and relevant engineers, we always engage with them physically.</p><p>We all sit in the same open space and we make it a habit to work with the engineers who solve the problem, shoulder to shoulder. Today, that shoulder is a health risk, so we rely more and more on technology to help us in managing the crisis.</p><p>Let’s be honest — It’s not the same.</p><p>It can never be the same as the personal touch, the encouragement we take from our fellow engineers and the human interaction. So we need to be on our toes when alerting on an incident. Texting in Slack or Hangouts just won’t cut it anymore. Virtual “war rooms” should be set up within 10 minutes of the incident start time if the root cause is not clear yet, and <strong>actual</strong> phone calls to be made if management needs to know. It’s a whole new world out there and every second counts.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/730/0*Gh4u_qsrYAQ4FPgE" /></figure><p>That’s a good question, what is next?</p><p>Well, we live to fight another day… take each day as the challenge that it is, strive to do our best in a new reality and beat the living cr*&amp; out of any major incident that comes our way (pardon my language 😈).</p><p>This is a call to all my brothers and sisters who manage major incidents by day (and night) and manage their families in between: Hang in there, be brave, do what needs to be done and keep that smile on your face 😎 I have complete and utter faith in you all!</p><p>The day will come when we’re back at our NOC, safe and sound with all our monitors on the wall, so until then… this is your friendly incident manager encouraging you to stay calm and Don’t Panic… 🤡</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=cfc584715683" width="1" height="1" alt=""><hr><p><a href="https://medium.com/gett-engineering/managing-major-incidents-from-home-what-i-learned-from-ethan-hunt-cfc584715683">Managing major incidents from home — What I learned from Mission Impossible</a> was originally published in <a href="https://medium.com/gett-engineering">Gett Tech</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Show me the money! — Monitoring Production the “Jerry Maguire” way]]></title>
            <link>https://medium.com/gett-engineering/show-me-the-money-monitoring-production-the-jerry-maguire-way-20034829e107?source=rss-2496de0b948f------2</link>
            <guid isPermaLink="false">https://medium.com/p/20034829e107</guid>
            <category><![CDATA[apdex]]></category>
            <category><![CDATA[incident-management]]></category>
            <category><![CDATA[devops]]></category>
            <category><![CDATA[engineering]]></category>
            <category><![CDATA[system]]></category>
            <dc:creator><![CDATA[Lior Avni]]></dc:creator>
            <pubDate>Thu, 23 Jan 2020 15:44:32 GMT</pubDate>
            <atom:updated>2020-01-23T15:44:32.531Z</atom:updated>
            <content:encoded><![CDATA[<h3>Show me the money! — Monitoring Production the “Jerry Maguire” way</h3><p>Every Escalation engineer knows this simple truth — <em>“if you find it faster, you will solve it faster”</em> (“it” being the incident you want to avoid). To detect those pesky incidents, we invest a lot of money in various application performance monitoring (APM) tools, and most of us use the out-of-the-box functionality these tools offer.</p><p>But, what if I told you that you can save hundreds of thousands of yearly dollars if you take a step further and create transaction-specific alerts and be aware about third-party elements affecting your system. Would you like to know more? Than take the red pill and I’ll show you just how deep the rabbit hole goes… (Sorry, couldn’t help myself 😎)</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/640/0*_KZ4DYOIL4JG4I6t" /></figure><h3>“In the beginning, it is always dark…”</h3><p>So you’re starting your journey in managing a production environment. You implemented an APM solution from one of the leading vendors (full disclosure — I use <a href="http://www.newrelic.com/"><strong>NewRelic</strong></a> in my production environment, but there are other vendors to choose from and you should always do your market research before choosing a tool), and you start defining the basic alerts: Apdex and Error rate.</p><p><strong>Error rate</strong> is the percentage of errors generated by the monitored service over a set period of time.</p><p><strong>Apdex</strong> (Application Performance Index) is slightly more complex: it is an <a href="https://www.apdex.org/">open standard</a> for measuring the performance of software applications in the IT world (among other applications). Its purpose is to convert different measurements into insights about user satisfaction, by specifying an agreed upon approach to analyze and report the level to which measured performance meets user expectations</p><p>In simple terms — it tells you if your users are happy or dissatisfied with the service you provide them. Each of these basic alerts have a threshold that should be set when you configure these basic alerts and, as we say back home, “Andiamo” — you’re good to go. This is all well and good when you deal with a system that has 5,10 or even 15 applications to monitor. But what happens when your “house of cards” grows even bigger? Or to be more clear, when you start to break your monolith into dozens of Micro services which intertwine in a delicate fabric of dependencies with each other. The default monitors just aren’t enough. You need to take this a step further.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/402/0*ogY0gLzHp_sXCI7J" /></figure><h3>Crossing the threshold…</h3><p>Why do we need the additional monitors? The simple answer is that when you’re alerted to an Apdex disturbance of a service, you’re literally blind to which service / element is causing that disruption. You’ll need to dig deep into the service monitors to find out what caused it, and when dealing in real time incidents, every second counts when you’re racing to understand the cause.</p><p>You want to receive early notifications so you can</p><ul><li>target the correct engineer, who will eventually solve the issue, and</li><li>shorten your incidents MTTX (Mean time to <strong>D</strong>etect, <strong>U</strong>nderstand &amp; <strong>R</strong>esolve).</li></ul><p>There are 4 additional monitors you can apply out-of-the-box:</p><ul><li>Response time for Web transactions</li><li>Response time for non-web transactions</li><li>Throughput of Web transactions</li><li>Throughput of non-web transactions</li></ul><p>These act as an additional drill down to your respective service’s “vital signs” monitor as they present not the general behaviour of the service, but the various elements within the service which may trigger before the “general storm”. Web transactions are the application server’s transactions and the non-web are, well, everything else. 😇</p><p>But we can take this even further… come, take my hand, I don’t bite 🤡</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*uRpv2ftAIFHLsBEJ" /></figure><h3>Taking the road less traveled…</h3><p>This is not an easy task. It involves a thorough investigation into your system, knowing its internals and knowing which team is in charge of which service. It also involves a deeper understanding of your APM solution and knowing how to query the database it uses to maintain the data (NewRelic, for example, has its <a href="https://docs.newrelic.com/docs/query-data/nrql-new-relic-query-language/getting-started/nrql-syntax-components-functions">NRQL</a> for this purpose), but again — knowledge is power, and in this case, it can also save you a lot of money:</p><ul><li><strong>External services alerts: </strong>you can set a monitor on every external service that affects your mission critical services. By doing so, you can foresee an incident almost a full minute before it starts. Most built-in monitors take a minimum of 5 minutes before they trigger. If you set this additional safeguard, you receive a pre-alert and can preemptively track and fix the affected area, before the system is fully affected.</li><li><strong>Specific transaction Apdex</strong>: Each service has its critical transactions and the less critical transactions. Each of these transactions have an Apdex that can be monitored and an alert that can be triggered if it “fires up”. Once you query the database that holds the transaction data, you can save 2–3 minutes of precious alerting time and be aware in advance that “a storm is coming”.</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/480/0*rkQHllXSo2xuA0Ia" /></figure><h3>Where is the money you talked about???</h3><p>I did talk about saving money, right? So let’s get down to business — how is this going to save you money? Simple.</p><p>Downtime can cost companies $5,600 per minute and up to $300,000 per hour in web application downtime (according to a <a href="https://blogs.gartner.com/andrew-lerner/2014/07/16/the-cost-of-downtime/">2014 Gartner’s analysis</a>).</p><p>A typical company can suffer 1–3 incidents per week, ranging from partial downtime to full service disruption. By applying these additional monitors and alerts, you can potentially save 1–3 minutes of investigation per incident! That’s 3–10 minutes per week, 12–40 minutes per month, 36–120 minutes per Q, and 144–500 minutes per year! Or to put it in financial terms:</p><h3>± 200k$ savings per Q</h3><p>which translates into</p><h3>± 800K$ savings per Y!!!</h3><p>Imagine what you can do with all that money… 🤔</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/640/0*gBvSjOYXKD_nPVnB" /></figure><h3>This is only the beginning…</h3><p>There’s a whole world out there for you to monitor. The more you utilise the tools at your hand, the better understanding you’ll have of (and about) your system and the better your control will be when handling Incidents in real time.</p><p>It’s a never ending story my friends; So go, cross the rainbow and find the answer… (and if you understood the reference, than I’m not that old… 👨‍🦳)</p><p>Until we meet again, this is your friendly Gett Incident Management team signing off and reminding you, as always — Don&#39;t panic…🦉</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=20034829e107" width="1" height="1" alt=""><hr><p><a href="https://medium.com/gett-engineering/show-me-the-money-monitoring-production-the-jerry-maguire-way-20034829e107">Show me the money! — Monitoring Production the “Jerry Maguire” way</a> was originally published in <a href="https://medium.com/gett-engineering">Gett Tech</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Scientia potentia est — Knowledge is power]]></title>
            <link>https://medium.com/gett-engineering/gett-scientia-potentia-est-knowledge-is-power-5c210f8d2b8e?source=rss-2496de0b948f------2</link>
            <guid isPermaLink="false">https://medium.com/p/5c210f8d2b8e</guid>
            <category><![CDATA[knowledge-management]]></category>
            <category><![CDATA[knowledge]]></category>
            <category><![CDATA[incident-management]]></category>
            <category><![CDATA[engineering]]></category>
            <category><![CDATA[tech-support]]></category>
            <dc:creator><![CDATA[Lior Avni]]></dc:creator>
            <pubDate>Thu, 07 Nov 2019 12:35:31 GMT</pubDate>
            <atom:updated>2019-11-07T12:35:31.259Z</atom:updated>
            <content:encoded><![CDATA[<h3>Scientia Potentia Est — Knowledge Is Power</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*kajUwM4yovgQsoCp.jpg" /></figure><p>One of my favorite proverbs of all times is this:</p><p>“Give a man a fish and you feed him for a day. Teach a man to fish and you feed him for life” (Attributed to <em>Maimonides, 12th century A</em>.D).</p><p>Why do I like this proverb so much? Because it embodies a great truth in it that we should all implement in our day-to-day work as IT professionals: <strong>Share your knowledge</strong>.</p><p>The more you share, the better your business will become as more and more professionals will be able to operate both the system and problems that may occur (I do incident management for a living, so the more people I can train to operate the system, the more I get to sleep… 😇).</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/496/1*B7VY8e7PdbvpncUYfPh4uQ.png" /></figure><p>In the following blog post I’ll review how we share knowledge here at <a href="https://gett.com/">Gett</a>, what are the tools we use and how it improves our incidents MTTX (MTTD — Mean time to Detect, MTTU — Mean time to understand &amp; MTTR — Mean time to resolve).</p><h3>Tools of the trade</h3><p>When sharing knowledge, you want to think as efficiently as possible; i.e a knowledge sharing tool that is:</p><ul><li>Commonly used by your peers</li><li>Easy to operate</li><li>Doesn’t impose an additional cost to your company</li></ul><p>As we have 2 distinct, and very different, customers —R&amp;D from one side and our regional customer operations department from the other side, we use both <a href="https://www.atlassian.com/software/confluence"><strong>Confluence</strong></a> and <a href="https://www.salesforce.com/products/platform/products/force/?d=70130000000f27V&amp;internal=true"><strong>SalesForce</strong></a> knowledge modules as our KB engines.</p><p>We’ve set up our own space in <a href="https://www.atlassian.com/software/confluence">Confluence</a> where we regularly update technical challenges we face and how we solved them, as well as our very own FAQ section to help with our own onboarding process.</p><p>The SalesForce knowledge module has been set up for the benefit of our Customer Care department. When answering a call from a customer (be it rider or driver), The customer care representative will start typing the topic of the ticket and a list of knowledge articles will be presented adjacent to the ticket. The operator can then get to the bottom of the problem and either resolve it, or escalate it to the next level — Tier 2 (Global tech support).</p><blockquote>“I did then what I knew how to do. Now that I know better, I do better.”</blockquote><blockquote>― Maya Angelou</blockquote><h3>Sharing is caring</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Tdy5xa0x9wHaUF_9wVjz2g.png" /></figure><p>Now, you may think to yourself “<em>Why should I share my hard earned knowledge with everyone? How can it possibly benefit me in my day-to-day?</em>”. Well, knowledge sharing is <strong>the</strong> number 1 tool to improve your work efficiency. The more people know how to operate issues, the more potential “working hands” you have on it and the less work load falls on just a single person.</p><p>Let’s hypothesize — A recurring problem is escalated weekly to the tech support team. The tech support team solves this recurring problem daily and just returns it back to customer care as resolved. Does this sound like a healthy work procedure? We thought so as well, so we took the following steps to improve:</p><ul><li>Create a knowledge-base article on this specific problem (along with FAQs and screenshots)</li><li>Educate senior customer care operators with specific tools that Tech support use to eliminate minor problems</li><li>Create a 3 slides tutorial on how to solve this specific recurring issue and share it in new customer care representative onboarding process</li></ul><p>Additionally, we ran a full analysis on all the tickets escalated to Tech Support in the previous year. Each ticket that was solved with a common action, or that could have been solved with a simple database query, was labeled as “<em>Knowledge Base Needed</em>”. Each of these “KB” (Knowledge Base) topics received either a Database report that will assist customer care to quickly find the root cause of the customer’s problem, or a KB wizard with some guiding questions that will help the customer care representative get as much data as possible to assist the escalation engineer in solving the problem.</p><p>As I mentioned before, we use <a href="https://www.salesforce.com/products/platform/products/force/?d=70130000000f27V&amp;internal=true">SalesForce</a> as our ticket management tool when we work with the regional customer care. SalesForce has an amazing reports module and low and behold, within 2 quarters of implementing this knowledge process, we’ve managed to eliminate 20% of the tickets volume globally (and even more in specific regions).</p><p>Furthermore, since this implementation of knowledge sharing, our local customer have significantly reduced the time it takes to resolve a customer issue; the knowledge is present at their fingertips, allowing a faster MTTR, improving their workload, increasing customer satisfaction and generally making everyone more content — it’s a “win win” for all…</p><h3>When in doubt, try something new…</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/979/0*J8wVQOaqxfkIr3ww.jpg" /></figure><p>We’re always improving: there’s so much more you can do if you want to share your knowledge:</p><ul><li><strong>Periodic webinars</strong>: we routinely analyze the tickets escalated to tech support and when we find a proverbial “weak spot”, we create a 15 minute webinar where we explain the issue at hand in 5 slides (no more) and record a session about the issue and how to solve it. These webinars are then shared with all regional customer care departments to be delivered by the shift managers to all employees.</li><li><strong>Weekly sync meetings</strong>: In these meetings, Tier 1 &amp; Tier 2 support teams meet, go over weekly topics that were escalated and try to find areas that need enforcement. These areas are then put to a backlog in <a href="https://www.atlassian.com/software/jira">Jira</a> and are later converted to either Knowledge articles, Database reports and - most importantly - shared with all who can benefit from them.</li><li><strong>“If you see something, say something”</strong>: Don’t wait for the question, but rather be proactive and provide the answer in advance to issues you know your customers need to know.</li></ul><p>As the great Douglas Adams used to say: “<em>Human beings, who are almost unique in having the ability to learn from the experience of others, are also remarkable for their apparent disinclination to do so.</em>”</p><p>I urge you to go and prove Douglas wrong this time :-)</p><p>So long for now (and thanks for all the fish 🐟). Until we meet again…</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/800/0*q1O7haq_mKl7Nnks" /></figure><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=5c210f8d2b8e" width="1" height="1" alt=""><hr><p><a href="https://medium.com/gett-engineering/gett-scientia-potentia-est-knowledge-is-power-5c210f8d2b8e">Scientia potentia est — Knowledge is power</a> was originally published in <a href="https://medium.com/gett-engineering">Gett Tech</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[The importance of Proactive monitoring]]></title>
            <link>https://medium.com/gett-engineering/the-importance-of-proactive-monitoring-aa7c5194c3fe?source=rss-2496de0b948f------2</link>
            <guid isPermaLink="false">https://medium.com/p/aa7c5194c3fe</guid>
            <category><![CDATA[incident-management]]></category>
            <category><![CDATA[engineering]]></category>
            <category><![CDATA[proactive-support]]></category>
            <category><![CDATA[monitoring]]></category>
            <category><![CDATA[devops]]></category>
            <dc:creator><![CDATA[Lior Avni]]></dc:creator>
            <pubDate>Thu, 11 Jul 2019 10:52:40 GMT</pubDate>
            <atom:updated>2019-07-11T10:56:44.836Z</atom:updated>
            <content:encoded><![CDATA[<h3>The importance of Proactive Service Monitoring</h3><p>When I started my journey in the IT world, my badge bore the motto of the company that saw fit to grant me this opportunity (Comverse technologies, you are forever in my mind 😃): “<strong><em>Our goal is to meet or exceed our customers’ expectations</em></strong>”.</p><p>Take a moment and think about this motto. We all want to do a good job in service of our customers, but the thing that defines a successful service is when you foresee what your customer needs before they know it themselves. I’ve done my best to follow this motto during my career and to implement it in any workplace I’ve had the good fortune to work at, and Gett is no exception.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/480/0*LH8iWtXVBvKi5Yxt" /></figure><p>The primary customers of Gett’s Global tech support team are the front line support tier, the regional customer care operators who take the calls from the riders and drivers and do their best to resolve their predicaments. In cases of “Fire” incidents, they are the first line of defense to tackle the situation along side R&amp;D to resolve the issue as soon as possible.</p><p>Meeting the customer’s expectations means — resolving the incident as soon as possible. But what about <em>exceeding</em> their expectations? This is where our proactive monitoring mechanism comes into play, where we foresee an incident before it starts and proactively alert our frontline support to be ready: ”<em>There’s a Storm Coming</em>”.</p><p>In the following blog post, I will review how we created this array of proactive monitors, what are the tools we use for this and our roadmap for this project.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/680/0*4BL7TkOUb96r8Xzr.jpg" /></figure><h3>Monitor the world…</h3><p>Gett uses 3 types of monitors:</p><ul><li><strong>Applicative</strong>: we monitor each and every micro-service we deploy using <a href="https://newrelic.com/">NewRelic</a>. This monitoring includes (but is not limited to): latency, error rate, throughput and several other parameters that tell us if there’s something wrong with this service.</li><li><strong>Infrastructure</strong>: we monitor the health of the system infrastructure using <a href="https://www.datadoghq.com/">DataDog</a>. This monitoring includes (but is not limited to): Memory usage &amp; CPU of DataBases, Lag between replicas and master DBs, Error 500s recorded by the ELBs and more.</li><li>And last (and most relevant for this blog) — <strong>Business monitoring</strong>. Here we use a set of <a href="https://grafana.com/">Grafana</a> based monitors that enable us to detect anomalies in specific business flows based on predetermined thresholds.</li></ul><p>While we strive to be as proactive as possible in each monitoring aspect (i.e. alert our customers when we see an application flaw in a microservice, or a degradation in an infrastructure element, that can affect the business), it is that set of business monitors that is our primary aid in being proactive.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/402/0*43Vscpo7-yXKpJnJ" /></figure><h3>How did we get there?</h3><p>As with every good thing, the road was long… As you know, Gett is a ride hailing service first and foremost, so we huddled together with several business analysts and mapped what we call “the golden flows”:</p><ul><li>Login to the application (Rider and Driver).</li><li>Get a taxi as soon as possible.</li><li>Pay successfully for the ride (when using a credit card).</li><li>Receive the ride summary report by mail on time.</li><li>And a few more (the list is not that short… 😇)</li></ul><p>Each and every one of these golden flows is mapped to a specific part of the code and assigned an event that is triggered with success, or failure, of that action. These events are written to a database that can be queried and the set of collected results is presented in Grafana. This way, any anomaly is visible to even the untrained eye and can be used to proactively alert our customers on an impending production issue.</p><p>For example: if we detect that a timeout from one of our 3rd party vendors is exceeding the usual threshold and as a result, we fail to provide a critical service to our customers (for example deliver notifications on concluded transactions), we can assume something is wrong here, send a proactive notification to our users and advise that some disruptions in service may happen in that specific area. Or, if we detect a problem with a specific cellular carrier, we can provide our Tier 1 operators with a tool that will significantly minimize the MTTR of tickets raised by customers who use that specific cellular carrier.</p><p>As I always say, “<em>A picture’s worth a thousand words</em>” (well, maybe not an original saying of mine, but you get the point 😉), so below you can see examples of our Grafana graphs when a business related anomaly took place and we successfully alerted our customers before they started to “feel the pain”.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/880/0*fuK3kdstEymOHRfS" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/884/0*unBugDw8Zb9YgnbP" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/893/0*ms7fNb9PKxhTDBko" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/886/0*NaN3Df9VIhqtP6X9" /></figure><p>As you can see from these dashboards, an anomaly was detected, so we reacted and provided our first line of defence the means to be ready for the storm.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/225/0*QSJ1rTk1uhtZX3eV" /></figure><h3>So what’s next?</h3><p>Well, that’s a good question, with a very simple answer: <strong>Be as proactive as possible and as collaborative as possible</strong>. Each new service we deploy that is part of any “golden flow” comes equipped with the needed events to trigger a “<em>success</em>” or “<em>fail</em>” alert which we then add to our Grafana monitors.</p><p>Additionally, we share these monitors with our Tier 1 support so they get notified along with the tech support team (keeping them less surprised with production incidents). The ultimate goal is to automate this procedure and I’m sure we’ll get there eventually.</p><p>For now, this is your friendly Gett incident management team signing off and reminding you of the answer to life, the universe and everything…</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/458/0*iXX7yvd5kcMMca7e" /></figure><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=aa7c5194c3fe" width="1" height="1" alt=""><hr><p><a href="https://medium.com/gett-engineering/the-importance-of-proactive-monitoring-aa7c5194c3fe">The importance of Proactive monitoring</a> was originally published in <a href="https://medium.com/gett-engineering">Gett Tech</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[A “Fire” Incident case study]]></title>
            <link>https://medium.com/gett-engineering/a-fire-incident-case-study-73b0e5374100?source=rss-2496de0b948f------2</link>
            <guid isPermaLink="false">https://medium.com/p/73b0e5374100</guid>
            <category><![CDATA[devops]]></category>
            <category><![CDATA[real-life-stories]]></category>
            <category><![CDATA[incident-management]]></category>
            <category><![CDATA[tech-support]]></category>
            <category><![CDATA[engineering]]></category>
            <dc:creator><![CDATA[Lior Avni]]></dc:creator>
            <pubDate>Thu, 30 May 2019 11:37:39 GMT</pubDate>
            <atom:updated>2019-05-30T11:37:39.660Z</atom:updated>
            <content:encoded><![CDATA[<p>Here at <a href="https://gett.com/">Gett</a>, we have several degrees for production environment related issues:</p><ul><li><strong>🔨 Trivial</strong>: A small issue that affects the occasional user in a remote aspect of the application</li><li><strong>⚠️ Medium</strong>: slightly more inconvenient, but not business disrupting</li><li><strong>❗️ Critical</strong>: a potentially business affecting issue if not treated within the next few days</li><li><strong>🔥 Fire</strong>: A “<em>Drop everything you’re doing and fix this issue now</em>” type of situation. It has a critical effect on the business and users are currently experiencing severe degradation while using the service</li></ul><p>In the following blog post, I’ll review one such “<strong>Fire</strong>” incident we managed, how we analyzed it, what methodology we used to manage it, how it was resolved and what did we learn from it.</p><h3>It was a day like any other day…</h3><p>It started as a normal day, office spirit was high, it was mid-August and summer vacation was on with great weather all around. We were just about to go to lunch when the first call came from our Customer Care department: a customer called and complained that he can’t order a taxi as the app tells him he has an outstanding balance to his credit card.</p><p>As our internal monitors were not showing any signs of an anomaly, our first thought was that this is a temporary timeout between third-party components that could momentarily cause such glitches in service. Then, as the second, third and fourth calls came in, we realized something was up.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/968/0*kt0SG8pNQcOPYmiO" /></figure><h3>Initial investigation begins</h3><p>So we started gathering facts and create a timeline for this event as it unfolded:</p><ul><li>🕐 When did it start?</li><li>🔍 A common denominator for the complaints?</li><li>🌏 Is this a global issue or narrowed down to one region?</li><li>🤔 And most important - “what is the cause”?</li></ul><p>This is where the importance of the Incident Manager position comes into full effect. Our on-call incident manager took command of the incident and started executing our “Fire” protocol:</p><ul><li><strong>Issue a notification to all critical communication channels</strong>. Gett operates in 3 global regions and this turned out to be a global issue</li><li><strong>Issue a notification to upper management updating them on the current rolling issue</strong>. In situations like this, the most crucial thing to a regional manager is to be in the loop and to have answers ready if required by external parties.</li><li><strong>Assemble the correct engineers in one place</strong>. A direct result of the service mapping that was done during the past year was an immediate alert to the relevant engineering team(s). An impromptu “War Room” was setup where all teams were huddled and worked diligently to resolve this issue as soon as possible.</li></ul><h3>Root Cause ascertained</h3><p>It was a nerve wrecking hour or so until the root cause was discovered. As with all incidents, a tiny semicolon was placed in the wrong location (and really, show me one developer who didn’t miss a semicolon in production…), causing a chain reaction that created this specific issue.</p><p>The fix was multi faceted:</p><ul><li>Fixing the code and deploying the fix.</li><li>Fixing the damaged parts of the Database that were affected by this problem.</li><li>Dealing with the after effects of this incident on our customers.</li></ul><p>The first part was easy, while the second part required knowing exactly when the original bug was introduced to the system. Thankfully, every deploy is logged with full details, so it didn’t take too long to find the correct time, recover the data from the correct point in time and restore the system to operational state. The third part involved our customer care department that diligently went from customer to customer and verified that everything is now working as it should.</p><p>This was a defining moment in Gett’s incident handling methodology as this was an incident that affected many of our customers. It was like a successful space shuttle launch when we understood that the incident was behind us: cheers all around, hugs and claps on shoulders with the VP R&amp;D and division chiefs (that were all huddled in the war room together with the engineers, standing shoulder to shoulder and trying to assist in resolving the incident). This was a joint effort and we braved through it like a well oiled machine.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/600/0*wShqigYU9LUoosX-.png" /></figure><h3>We live, We learn, We move on</h3><p>The last piece of this puzzle, and also the most important aspect of the Incident Manager’s job, was the “post mortem” portion, where we derive the needed action items from this incident.</p><p>A meeting was assembled the day after where we:</p><ul><li>Reviewed the chain of events.</li><li>Understood what was missing from our proactive monitors.</li><li>Implemented these monitors.</li><li>Issued a company wide root cause analysis for this incident.</li><li>Added the needed internal safeguards to prevent this from happening again.</li></ul><p>Since that incident, we had additional incidents, but as we grew and matured, so did our <a href="https://medium.com/gett-engineering/gett-incident-management-minimizing-crisis-resolution-time-c802df3218f5">incident management process</a>. We now have extensive business monitors which allow us to proactively catch 90% of possible business affecting issues. We reduced our incident resolution time by 50% and our employees are all the more engaged in resolving these issues the minute they are triggered.</p><p>So, until we meet again, this is your friendly Gett Incident Management team signing off…</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*bryp6pxaq3fPUOBM" /></figure><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=73b0e5374100" width="1" height="1" alt=""><hr><p><a href="https://medium.com/gett-engineering/a-fire-incident-case-study-73b0e5374100">A “Fire” Incident case study</a> was originally published in <a href="https://medium.com/gett-engineering">Gett Tech</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Gett Global Technical support team — the Gatekeepers of the system]]></title>
            <link>https://medium.com/gett-engineering/gett-global-technical-support-team-the-gatekeepers-of-the-system-ffd42109b428?source=rss-2496de0b948f------2</link>
            <guid isPermaLink="false">https://medium.com/p/ffd42109b428</guid>
            <category><![CDATA[tech-support]]></category>
            <category><![CDATA[devops]]></category>
            <category><![CDATA[grafana]]></category>
            <category><![CDATA[engineering]]></category>
            <category><![CDATA[incident-management]]></category>
            <dc:creator><![CDATA[Lior Avni]]></dc:creator>
            <pubDate>Thu, 28 Feb 2019 14:40:52 GMT</pubDate>
            <atom:updated>2019-02-28T14:42:49.869Z</atom:updated>
            <content:encoded><![CDATA[<h3>Gett Global Technical support team — the Gatekeepers of the system</h3><p>Gett is a global operation driving scores of passengers daily around 3 geographical regions and providing service to hundreds of thousands of end users.</p><p>Did you ever stop to ask yourself what makes this great operation tick?</p><p>Besides the amazingly talented R&amp;D engineers that write the code and the fearless operators who handle the customers daily requests, there‘s a team of escalation engineers who tackle the not-so-usual issues, and who make sure Gett operation remains up and running.</p><p>These are the Global Technical Support team members, affectionately dubbed “<em>Tech Heroes</em>”.</p><p>In the following blog, I’ll share with you a “day in the life” of a Global Technical Support team member and show you what it takes to keep Gett’s operation alive and kicking.</p><p>Hint —It’s slightly more complicated than the image below… :-)</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/960/1*vkDgYQ7VkNvS6sPqbjgZHA.png" /></figure><h3>End users complaints</h3><p>The majority of the tech support team is comprised of solving escalated support tickets for riders and drivers who encounter issues in their day-to-day usage of the Gett Mobile App. These may include application operation queries, finance related questions, for both drivers and riders, on their respective applications.</p><p>This support is also directed at more than 1000 Gett employees as Tier 1, 2 and sometimes Tier 3 support for production environment related issues.</p><p>The team works with a dedicated ticketing system that enables monitoring the SLAs per each ticket’s priority, producing detailed analysis on tickets types and enabling the team to improve on specific domains.</p><h3>Production system tweaks</h3><p>Another important aspect of the team’s responsibility is implementing change requests. Gett’s backend services work in a <strong>Micro Services architecture</strong>, and as such are controlled by a myriad of system settings per feature. As guardians of the production system, it is the tech support team’s responsibility to modify any required system setting to a given value and verify that this change was successful. Change requests are usually sent from Product managers or local operation managers.</p><p>This role requires an ability to deep dive into every feature or service specification, capabilities and restrictions so that it will be possible to:</p><ol><li>Manage and test every change from end to end.</li><li>Gain a systematic view to understand the potential impact on other services once implemented.</li></ol><p>It also requires strong communication skills, understanding the business side of every change along with strong monitoring knowledge to be able to react fast in case of any problem</p><h3>Production E2E quality</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/620/0*xJ5N49kjFZ1b80Aj" /></figure><p>Global tech support is all about the quality of the production system. This includes running sanity tests on the system periodically, discovering bugs in production, recreating bugs in production (which can’t be tested in staging systems) and making sure these are resolved as soon as possible.</p><h3>System Monitoring</h3><p>“Knowledge is Power” — This is a constant in life (well, other than death and taxes… 😇) and as seekers of knowledge, the tech support team employs a number of APM tools, which I’ve covered <a href="https://medium.com/gett-engineering/gett-incident-management-minimizing-crisis-resolution-time-c802df3218f5">in my previous blog post</a>, to always be aware of what’s going on with the production system. Be it <a href="https://grafana.com/">Grafana</a> to monitor specific scenarios, <a href="https://newrelic.com/">New Relic</a> for system services health or <a href="https://www.datadoghq.com/">DataDog</a> for infrastructure components health, tech support agents “sleep with one eye opened” all the time, constantly watching the system.</p><h3>Application version management</h3><p>End users deal with the mobile (or web) application, and this area is also under the care of the tech support team. Each mobile version is uploaded to the relevant store (Google Play Store / Apple App Store) through a controlled process that requires the at most care. This is done both for the Driver App and for the Rider App.</p><h3>Incident Management</h3><p>As with every production system, the Gett system is not error-free and for that, Gett employs an organised incident management process to handle these unexpected faults.</p><p>For a full description on the Incident management process at Gett, you can follow <a href="https://medium.com/gett-engineering/gett-incident-management-minimizing-crisis-resolution-time-c802df3218f5">our previous post about our Incident Resolution process</a>.</p><p>Other than that, the Global tech support team members are part-time PMOs, Part time DBAs, Part time Dev Ops, Part time Psychologists and Full time “<em>Directors of getting things done</em>”.</p><p>We are the “Tech Heroes” — Hear us roar :-)</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/640/0*bCFjVAYH8rAj-z90.jpg" /></figure><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=ffd42109b428" width="1" height="1" alt=""><hr><p><a href="https://medium.com/gett-engineering/gett-global-technical-support-team-the-gatekeepers-of-the-system-ffd42109b428">Gett Global Technical support team — the Gatekeepers of the system</a> was originally published in <a href="https://medium.com/gett-engineering">Gett Tech</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Gett incident management — minimizing Crisis resolution time]]></title>
            <link>https://medium.com/gett-engineering/gett-incident-management-minimizing-crisis-resolution-time-c802df3218f5?source=rss-2496de0b948f------2</link>
            <guid isPermaLink="false">https://medium.com/p/c802df3218f5</guid>
            <category><![CDATA[devops]]></category>
            <category><![CDATA[grafana]]></category>
            <category><![CDATA[incident-management]]></category>
            <category><![CDATA[infrastructure]]></category>
            <category><![CDATA[engineering]]></category>
            <dc:creator><![CDATA[Lior Avni]]></dc:creator>
            <pubDate>Wed, 30 Jan 2019 08:00:00 GMT</pubDate>
            <atom:updated>2019-01-30T11:29:22.309Z</atom:updated>
            <content:encoded><![CDATA[<h3>Gett incident management — Minimizing Crisis resolution time</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*ga17FfFc5Q6jxbI7T7JJaA.jpeg" /></figure><p>Incident management is a crucial part in any service-providing company.</p><p>Its the process where a critical service disruption is managed from start to finish, all the while taking into account the following:</p><ul><li>In-time problem detection.</li><li>Alerting the correct service owners of the problem.</li><li>Coordinating the repair efforts.</li><li>Alerting stakeholders (of all levels) of the problem and its effects on the company’s business and clients.</li><li>Constantly updating upper management of the progress, keeping work transparent.</li><li>And last, but not least, Root Cause Analysis of the problem that will contain what caused the issue to happen and what can be done to prevent it from recurring</li></ul><p>In this post you’ll learn about the Incident Management process at <a href="https://gett.com/">Gett</a> through its different stages and also, what are our goals in this area.</p><h3>In-time problem detection (Proactive vs. Reactive):</h3><p>Our goal is to know about any major incidents from stage 0; i.e. detect the problem at its very beginning, and not when it’s reported by a customer. For this purpose we several APM (Application Performance Monitoring) tools:</p><ul><li>We use <a href="https://newrelic.com/"><strong>NewRelic</strong></a><strong> </strong>to monitor our micro services and different gateways.</li><li><a href="https://grafana.com/"><strong>Grafana</strong></a> is used to monitor flow-specific events that exceed a pre-configured threshold.</li><li>And finally, we use <a href="https://www.datadoghq.com/"><strong>DataDog</strong></a> to monitor the hardware components of our system (Databases, Load balancers, etc…)</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/500/1*5pIbXD_gL_RIdnfVGXRQrw.jpeg" /><figcaption>Gett’s array of APM tools</figcaption></figure><p>Additionally, each APM tool is linked to a number of notification channels (Slack, PagerDuty, Email etc…), so each member of our incident team is notified of any anomalous events.</p><blockquote>“Our goal is to reach a quarterly proactive coverage rate of 90% of possible major incidents, and we’re almost there.”</blockquote><h3><strong>Alerting the correct service owner of the problem</strong></h3><p>We use a micro-services strategy at <a href="https://gett.com/">Gett</a>, and we’re talking about <em>a lot</em> of services. No single person can remember all service owners, definitely not when you wake up in the middle of the night to an alert 😉.</p><p>For this purpose, we’ve created an internal catalog of all service owners in the company along with backups, in cases of emergency.</p><p>This alone minimized the incident resolution time <strong>by 20%</strong>.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*9wulAr46Ybj-5oRd5iqWxg.jpeg" /></figure><h3><strong>Coordinating the repair efforts</strong></h3><p>An incident with no designated person leading the efforts is a recipe for an ongoing disaster. <a href="https://gett.com/">Gett</a> employs a Global Technical Support Team where each of its members is trained in managing an incident from start to finish. The team is dispersed across our regions of operations and runs 24/7 on-call shifts to cover all un-anticipated incidents.</p><p>Additionally, the Global Incident Manager acts as an escalation point for major incidents, is accountable to upper management for ongoing updates and provides a final Root Cause Analysis report per each major incident.</p><h3>To recap</h3><p>The incident management process comprises of:</p><ul><li>Reaching the correct service owner per issue.</li><li>Coordinating the fix efforts.</li><li>Making the tough decisions during the incident, and;</li><li>Updating executive levels in a “what they need to know, when they need to know it” manner.</li></ul><p>When the incident is completed and the system is restored to normal operation, the RCA (Root Cause Analysis) process begins.</p><p>Our tool of choice for managing incident reports is <a href="https://www.atlassian.com/software/jira">Jira</a>. We use the <a href="https://www.atlassian.com/software/jira/ops/handbook">Incident module</a> and have added several fields to it:</p><ul><li><strong>Initial RCA</strong>: A high-level description of the problem’s cause (DB issue, Infrastructural issue, Code issue, etc…) provided by the technical support agent that managed the incident (to be completed in a 24hr timeframe, post incident resolution)</li><li><strong>R&amp;D RCA</strong>: This is the most important part of the RCA process and is completed up to 48 hours post incident resolution (We also have Jira widgets in place to alert the relevant Team Lead if this wasn’t filled in time). It includes a full technical root cause analysis of the issue, why it happened, what was done to fix it, why it wasn’t detected (missed by QA, no relevant monitor, etc…), what is the probability of this issue to reoccur and what can be done to prevent it from happening again.</li><li><strong>Action items</strong>: Once preventative measure for this incident are ascertained, they are created as sub-tasks directly from the incident, with the incident left open until all sub-tasks are completed. This motivates the relevant R&amp;D team to complete the related issue as it “dirties up” their Kanban board with an open incident.</li></ul><blockquote><em>“It is the Global incident manager’s responsibility to make sure this process is enforced and completed after every incident.”</em></blockquote><p>So now you know how we deal with incident management here at Gett! What is the incident management procedure in your company? We’d be happy to hear, receive feedback and mutually learn from each other.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=c802df3218f5" width="1" height="1" alt=""><hr><p><a href="https://medium.com/gett-engineering/gett-incident-management-minimizing-crisis-resolution-time-c802df3218f5">Gett incident management — minimizing Crisis resolution time</a> was originally published in <a href="https://medium.com/gett-engineering">Gett Tech</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
    </channel>
</rss>