<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Italo Santos on Medium]]></title>
        <description><![CDATA[Stories by Italo Santos on Medium]]></description>
        <link>https://medium.com/@italux?source=rss-a639b32b2f2f------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/1*Apq2WIRfPpyfJOXMjYiWbg.jpeg</url>
            <title>Stories by Italo Santos on Medium</title>
            <link>https://medium.com/@italux?source=rss-a639b32b2f2f------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Tue, 17 Nov 2026 00:09:22 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@italux/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[Join me at NoDegree Engineer]]></title>
            <link>https://italux.medium.com/join-me-at-nodegree-engineer-e19bbb88610a?source=rss-a639b32b2f2f------2</link>
            <guid isPermaLink="false">https://medium.com/p/e19bbb88610a</guid>
            <dc:creator><![CDATA[Italo Santos]]></dc:creator>
            <pubDate>Wed, 24 Jan 2024 10:12:02 GMT</pubDate>
            <atom:updated>2024-01-24T10:17:33.848Z</atom:updated>
            <content:encoded><![CDATA[<h4>Moving to Substack under a new domain name</h4><blockquote>TL;DR; I am transitioning to Substack, where I will continue to publish under a new name <a href="https://www.nodegree.engineer/"><strong>No Degree Engineer</strong></a>.</blockquote><p>Why Substack, you might wonder? In the spirit of making my content more accessible, this move is driven by a desire to ensure that my insights and stories are available to everyone, unrestricted by the limitations faced by non-members on Medium.</p><p>I invite you to join me at the <a href="https://www.nodegree.engineer/">nodegree.engineer</a> on Substack by <a href="https://www.nodegree.engineer/subscribe">subscribing to the newsletter</a></p><p>Thank you for being a part of this enriching journey on Medium.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=e19bbb88610a" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Leaving a job smoothly]]></title>
            <link>https://medium.com/swlh/leaving-a-job-smoothly-b8c0619c7976?source=rss-a639b32b2f2f------2</link>
            <guid isPermaLink="false">https://medium.com/p/b8c0619c7976</guid>
            <category><![CDATA[leadership]]></category>
            <category><![CDATA[quitting]]></category>
            <category><![CDATA[management]]></category>
            <category><![CDATA[job-offer]]></category>
            <dc:creator><![CDATA[Italo Santos]]></dc:creator>
            <pubDate>Fri, 30 Jun 2023 12:38:49 GMT</pubDate>
            <atom:updated>2023-06-30T12:38:49.393Z</atom:updated>
            <content:encoded><![CDATA[<h4>An offboarding process practical guide</h4><p>There are s<a href="https://resources.workable.com/tutorial/employee-onboarding">everal articles</a> about a new employee onboarding in the company to ensure that the newly hired person reaches maximum performance as quickly as possible, <strong>but how about when someone is leaving the company?</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*u7ZA1Uno900iI7OjCCsNUw@2x.jpeg" /><figcaption>Photo by Wilhelm Gunkel on Unsplash</figcaption></figure><p>I’ve said goodbye to several people during my career, besides the emotional part I also had to <strong>deal with the side effects of a coworker leaving</strong>, which include, but are not limited to, critical integrations done using personal credentials, a manual process done by the living person forgotten, undocumented stuff, etc…</p><p>Recently I’ve decided to leave my job and I started to write down everything I’ve done to avoid the issues mentioned before, which I’d like to share since I couldn’t find good material about this topic.</p><h3>Deciding to leave</h3><p>During my 15 years of experience in the Tech industry, <strong>I’ve changed jobs only three times </strong>(<em>until now</em>).</p><p>At the beginning of my career, I spent <strong>almost 10 years in the same company</strong>, mainly because was my first job in the tech industry. I learn a lot working with amazing people, but in the last 2 years of my experience, I felt that I reached a learning plateau and started looking for a new job. That wasn’t a hard decision to make</p><p>I use to say, that <strong>people should always be doing the interview process even if they’re happy with their actual job</strong> (<em>including people from the teams I lead</em>). This is important for several reasons, during an interview process you will know new people, expand your networking, also learn new things with the interviewers, and with that you’ll be able to have a good picture of the market direction by analyzing the job titles, description, and expectations.</p><p>From my point of view, <strong>doing an interview process</strong>, appropriately, <strong>doesn’t mean that you’re unhappy</strong> or looking for a new job, the interview process could be good for the above reasons or to <strong>ensure that you’re happy where you are</strong>.</p><p>There are several signs that can show you that it’s time to leave your job — <a href="https://hbr.org/2022/02/6-signs-its-time-to-leave-your-job"><strong>6 Signs It’s Time to Leave Your Job</strong></a><strong> — </strong>that you can pay attention to, and once you make the decision there are some other facts to consider after that.</p><h3>Succession planning</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*ZcunyF77KF61MPFr" /><figcaption>Photo by <a href="https://unsplash.com/@morganhousel?utm_source=medium&amp;utm_medium=referral">Morgan Housel</a> on <a href="https://unsplash.com/?utm_source=medium&amp;utm_medium=referral">Unsplash</a></figcaption></figure><p><strong>Succession</strong> isn&#39;t something that you should worry about only when you decide to leave your current job, it&#39;s an ongoing job that starts earlier.</p><p>Will Larson wrote a good <a href="https://lethain.com/succession-planning/">succession planning article</a> which was a good guide, but in my experience, <strong>I spend almost a year and a half</strong> building a team to a possible transition.</p><p>I start bringing senior leaders closer and sharing part of my daily-basis job, like asking them to attend some meetings with me or to hand over some meetings and the goal of that was to expose them to topics that aren’t on their daily basis to understand how I handle those topics but also to create their own opinion about that topics, but also to share opinions and knowledge after the sessions (<em>this is so important as bring people together with you</em>) — meetings are the boring part because spend too much time and cognitive effort, but I see them as the “<em>hands-on part</em>” of the management role.</p><p>Besides the &quot;<em>hands-on part</em>&quot;, talking frequently about what I was envisioning for the future of the team allow me to have smooth conversations on how to build a clear roadmap to reach that vision in the long term, and also help me to get more feedback about the organizational structure, what needs to be changed, what isn&#39;t working well, etc… creating a communication channel which allows them to put their thoughts in practice.</p><p><strong>Being transparent with the team</strong> about my future intentions by doing all these things was essential to <strong>keep trust and avoid fear</strong> (<em>kind of</em>).</p><h3>Announcing you&#39;re leaving</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*6I5apHBPmGjJywK66z3lGg.png" /><figcaption>Photo by <a href="https://unsplash.com/@brett_jordan?utm_source=unsplash&amp;utm_medium=referral&amp;utm_content=creditCopyText">Brett Jordan</a> on <a href="https://unsplash.com/photos/rxZS3kCTyJI?utm_source=unsplash&amp;utm_medium=referral&amp;utm_content=creditCopyText">Unsplash</a></figcaption></figure><p>It shouldn’t be a surprise to most of the people you trust and share your plans with while building a succession plan, including the direct leader (<em>in my case the CTO</em>). I tried to be transparent about my future intentions earlier, even before I made the decision, and was important.</p><p>So when the time comes, be aware that your leader can try to change your mind but it’s important to hold to your plan. Enter the conversation announcing your decision, the reason (<em>assuming you’ve been transparent, up to this point, shouldn&#39;t be something new to say</em>), and the transition plan</p><ul><li><strong>Departure timeline:</strong> Since I got a job offer that interested me, I’ve shared a plan that worked for me but that had some room to wiggle.</li><li><strong>The next steps will be:</strong> Assume a new role in another company (<em>in my case</em>), take time to rest, sabbatical, study, whatever in your case.</li><li><strong>Recommended transition plan:</strong> Who should step into the role — <em>on that I&#39;d suggest 3 options a peer, a reportee, or new hire which I could help to hire if need</em>—, where to announce the leaving (<em>meeting, slack, email, etc…) </em>but be open to a more structured communication plan together with HR (<em>what happen in my case</em>)</li></ul><h3>Transition period</h3><p>Since I was in the company for a long period —<em> 5 years</em> — after the announcement many people reach out to talk about my motivations to leave, to be grateful for the time we spend working together, to ask for advice, etc… was something really surprising and meaningful to me mainly because I was able to understand the real reach of my work as a leader.</p><p>At this point, there was already a replacement for my role, so I was acting more as a consultant making recommendations, rather than leading the function on a daily basis.</p><h4>Practical checklist</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*EsQwsLfcSuN-LI99seQFuQ.png" /><figcaption>Photo by <a href="https://unsplash.com/@lunarts?utm_source=unsplash&amp;utm_medium=referral&amp;utm_content=creditCopyText">Volodymyr Hryshchenko</a> on <a href="https://unsplash.com/photos/x_OKGbxB3fA?utm_source=unsplash&amp;utm_medium=referral&amp;utm_content=creditCopyText">Unsplash</a></figcaption></figure><p>As I mentioned before there are some very tiny things that I’ve done to reduce the amount of side effects after I leave the company and my credentials be deactivated.</p><ul><li>I’ve used <a href="https://www.linkedin.com/posts/fernandapiresbichuette_modelo-de-handover-activity-7003833494799814656-YaCm?utm_source=share&amp;utm_medium=member_desktop">Fernanda’s vacation template</a> to <strong>share everything I conducted daily with the team</strong> (<em>sheets update, recurring checkups, etc…</em>)</li><li>Ensure that all <strong>approval workflows</strong> (<em>finance, HR, etc…</em>) were changed</li><li>Set the team&#39;s OKRs, Goals, KPIs, and career development plans</li><li>Aggregate into a shared drive the most important documents I’ve written(<em>plans, guidelines, presentations, etc…</em>)</li><li>Ask the team to <strong>deactivate your user account before leaving</strong> in definitive as a test to understand what could break</li><li>Ensure that all email groups have more people than me, as owners</li><li>Change the recurring <strong>calendar events ownership</strong></li><li>Ensure that all file shares have more people them me, as owners</li><li>Replace me from the on-call rotation platform</li><li><strong>Email redirect for a specific period</strong> and auto-reply email telling that you are a former employee like below:</li></ul><blockquote>Subject: Important Notice</blockquote><blockquote>Thank you for your message. I’d like to inform that I’m no longer working at [Company Name] as of April, 21st 2023 and unfortunately, I can no longer answer your email. Please direct all future inquiries to Jane Doe, at jane.done@company.com where he will be able to assist you.</blockquote><blockquote>Please note that your email will not be forwarded automatically.</blockquote><blockquote>Sincerely,</blockquote><h4>See also</h4><p><a href="https://lethain.com/leaving-the-executive-job">Deciding to leave your (executive) job.</a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=b8c0619c7976" width="1" height="1" alt=""><hr><p><a href="https://medium.com/swlh/leaving-a-job-smoothly-b8c0619c7976">Leaving a job smoothly</a> was originally published in <a href="https://medium.com/swlh">The Startup</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Observability, Monitoring, Alerting]]></title>
            <link>https://italux.medium.com/observability-monitoring-alerting-43ef79a9309?source=rss-a639b32b2f2f------2</link>
            <guid isPermaLink="false">https://medium.com/p/43ef79a9309</guid>
            <category><![CDATA[observability]]></category>
            <category><![CDATA[monitoring]]></category>
            <category><![CDATA[sre]]></category>
            <category><![CDATA[reliability]]></category>
            <dc:creator><![CDATA[Italo Santos]]></dc:creator>
            <pubDate>Wed, 25 Jan 2023 23:15:29 GMT</pubDate>
            <atom:updated>2023-01-27T12:58:17.104Z</atom:updated>
            <content:encoded><![CDATA[<h4>Do you know what’s going on with your application?</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*PuWL3bIQGeeANg7P" /><figcaption>Normal question to be asked when you&#39;re on-call</figcaption></figure><p>Over the past years, we’ve heard a lot about observability, after Twitter’s engineers wrote a blog post called <a href="https://blog.twitter.com/engineering/en_us/a/2013/observability-at-twitter.html">Observability at Twitter</a>, in September 2013. That was one of the first times the term observability was used in the context of software systems.</p><p>A few years later, <a href="https://twitter.com/peterbourgon">Peter Bourgon</a> attended the 2017 Distributed Tracing Summit which discussed more the definition and scope of how tracing helps provide observability, and in his blog post called <a href="https://peter.bourgon.org/blog/2017/02/21/metrics-tracing-and-logging.html">Metrics, tracing, and logging</a>, he describes the thoughts that could probably map out the domain of instrumentation.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/613/0*OBGvqOd4i6EOzgma" /><figcaption><a href="https://peter.bourgon.org/blog/2017/02/21/metrics-tracing-and-logging.html">Peter Bourgon — Metrics, tracing, and logging</a></figcaption></figure><p>Several discussions started about the theme and in 2020 the industry was aligned with the thinking that <strong>observability isn’t a feature</strong> that you can install or a service you can subscribe to, or simply the deployment and collection of three types of data.</p><p>But the observability concept isn&#39;t something new, Hungarian-American engineer Rudolf E. Kálmán introduced it in 1960.<a href="https://en.wikipedia.org/wiki/Observability#cite_note-1">[1]</a><a href="https://en.wikipedia.org/wiki/Observability#cite_note-2">[2]</a></p><blockquote><em>&quot;Observability is a measure of how well internal states of a system can be inferred from knowledge of its external outputs.&quot;</em></blockquote><p>Even though observability might mean different things to different people, however, the heartwood remains the same —<strong> bringing better visibility into systems.</strong></p><p>Considering that observability can be obtained with<strong> the set of logs, metrics, and traces</strong>, based on Rudolf&#39;s theory, those are outputs of a system’s internal observation, which is defined as<a href="https://en.wikipedia.org/wiki/Telemetry"> <strong>telemetry data</strong></a><strong> </strong>that are used to have a better understanding of the systems and are considered the observability pillars<strong> </strong>for modern systems.</p><p><strong>Observability is data, </strong>which is produced, collected, and stored to produce insights that can lead to answers, but the process of knowing what information to expose and how to examine the evidence (<em>aka observations</em>) still requires a good understanding of the system’s domain, as well as a good sense of intuition.</p><h3><strong>mon-i-tor</strong></h3><blockquote>/ˈmänədər/<br><em>verb<br> </em>gerund or present participle: monitoring</blockquote><blockquote>observe and check the progress or quality of (something) over a period of time; keep under systematic review.</blockquote><p>Despite the observability that looks much similar, monitoring is the ability to collect, process, aggregate, and display real-time quantitative data about a system to mainly understand the system’s health. While monitoring looks at the systems from a more static perspective, observability can be more dynamic, providing much more information about <strong>unpredictable failure modes.</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/940/0*vru1Y5XkgA6XUAt-.png" /><figcaption>Observability from <a href="http://www.stochasticgeometry.ie/2021/02/05/observability/">Stochastic Geometry</a></figcaption></figure><h3>Observability Infrastructure</h3><p>In the microservices world, a good monitoring platform is essential, but also a hard problem. Several tools can solve multiple problems using the same solution <a href="https://newrelic.com/">Newrelic</a>, <a href="https://www.datadoghq.com/">Datadog</a>, <a href="https://www.dynatrace.com/">Dynatrace</a>, <a href="https://lightstep.com/">Lightstep</a>, etc… but this can cost a lot depending on how big your application ecosystem is.</p><p>While we’re <a href="https://partiu.loggi.com/spraying-our-bike-shed-with-some-new-colors-c5f4fbc5847f">Spraying our bike shed with some new colors</a> at Loggi, we’ve decided to adopt <a href="https://istio.io/latest/about/service-mesh/">Istio’s service mesh</a> as one of the core infrastructure layer components, which allows you to transparently add capabilities like observability, traffic management, and security, without adding them to your code.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1000/0*0w4LnuYOZQOzL9zR.png" /><figcaption>Istio Architecture Overview</figcaption></figure><p>Istio is capable of generating detailed <strong>telemetry data for all service</strong> communications within a mesh, <strong>providing <em>observability</em></strong> of the service’s behavior throughout the following types of telemetry:</p><ul><li><a href="https://istio.io/latest/docs/concepts/observability/#metrics"><strong>Metrics</strong></a> → Istio generates a set of service metrics based on the <strong>four “golden signals”</strong> of monitoring (latency, traffic, errors, and saturation)</li><li><a href="https://istio.io/latest/docs/concepts/observability/#distributed-traces"><strong>Distributed Traces</strong></a> → Istio generates <strong>distributed trace spans</strong> for each service</li><li><a href="https://istio.io/latest/docs/concepts/observability/#access-logs"><strong>Logs</strong></a> → As traffic flows into service within a mesh, Istio can generate a <strong>full record of each request</strong>, including <strong>source</strong> and <strong>destination</strong> metadata</li></ul><p>Although it’s not possible to get all the internal data from the applications using only Istio, we come up some libraries that are capable of doing this, like <a href="https://micrometer.io/docs/ref/jvm">Micrometer</a> for JVM apps and<a href="https://github.com/korfuri/django-prometheus"> Django Prometheus</a>. These libraries together with <a href="https://opentelemetry.io/docs/concepts/what-is-opentelemetry/">Opentelemetry</a> expose internal framework data and are easily customizable to generate other metrics we deem necessary, building an ecosystem of tools to help you to observe your systems.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/515/0*g_HqQFC-PGDEGSlW" /><figcaption>Observability Ecosystem Tools</figcaption></figure><h3>Using the observability data</h3><p>Observability can produce a huge amount of data and it could become a mess so it&#39;s important to understand how to use it.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/500/0*kIto1jlIjPyXnXXC" /><figcaption>Monitoring Hierarchy — Inspired by<a href="https://medium.com/@alexwking?source=post_page-----78775c021b0e--------------------------------"> Alex King</a>’s <a href="https://medium.com/faun/devops-meets-observability-78775c021b0e">article</a></figcaption></figure><p>The pyramid above proposes how the collected data can be used, which I call the <strong>Monitoring Hierarchy.</strong></p><p>The <strong>bottom of the pyramid</strong> represents the <strong>telemetry data</strong> from the several systems, whereas the <strong>top of the pyramid</strong> contains the notifications which a human being will read about a <strong>problem that requires immediate attention</strong>, while<strong> </strong>the middle layer allows <strong>insights into operating efficiency</strong>.</p><p>We can also divide the pyramid into two categories, <strong>metrics</strong> (<em>data to be observed</em>) and <strong>diagnostics</strong> (<em>root cause analysis</em>)</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/800/0*Ae1vJc4sshY5jD0a" /></figure><p><strong>Metrics</strong> aims to show an overview of the health of a system and looks at the current moment, while <strong>diagnostic</strong> data analyzes past data, such as logs to identify the reason for the failure</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/800/0*NXR1OY4am0ggDTZ6" /></figure><p>For this reason, your monitoring system should address two major questions: <strong><em>“What’s broken, and Why?”</em></strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/800/0*x_boHCKlCtviUKk4" /></figure><p>Again… <strong>There is no silver bullet!</strong></p><p>But there are some methods to help you to address these questions.</p><ul><li><a href="https://sre.google/sre-book/monitoring-distributed-systems/#xref_monitoring_golden-signals"><strong>Four Golden Signals</strong></a> → user-centric, monitoring system from the <strong>user perspective</strong></li><li><a href="https://grafana.com/blog/2018/08/02/the-red-method-how-to-instrument-your-services/"><strong>RED</strong></a> → Focused on <strong>microservices</strong>, check the system from the <strong>request perspective</strong>, this will answer which service is broken</li><li><a href="https://www.brendangregg.com/usemethod.html"><strong>USE</strong></a> → More focus on <strong>infrastructure</strong> to identify <strong>performance</strong> issues and <strong>bottlenecks</strong></li></ul><h3>Conclusion</h3><blockquote>“Observability Isn’t a Panacea”</blockquote><p>In Greek mythology, <strong>Panacea</strong> was a goddess of healing and the term “<em>panacea</em>” is also widely used with the meaning of <strong>healing for all males</strong>.</p><p><strong>We can’t look at observability as purely an operational concern</strong>. An observable system isn’t achieved by plainly having monitoring infrastructure in place or by having an observability team, <strong>observability is a feature that needs to be embedded into a system since the design.</strong></p><h3>References &amp; Good articles</h3><ul><li><a href="https://peter.bourgon.org/blog/2017/02/21/metrics-tracing-and-logging.html">Peter Bourgon · Metrics, tracing, and logging</a></li><li><a href="https://istio.io/latest/docs/tasks/observability/distributed-tracing/overview/">Istio Distributed Tracing Overview</a></li><li><a href="https://rakyll.medium.com/sampling-in-observability-db0142cdda5b">Sampling in Observability</a></li><li><a href="https://github.com/korfuri/django-prometheus/">Export Django monitoring metrics for Prometheus.io</a></li><li><a href="https://micronaut-projects.github.io/micronaut-micrometer/2.0.0/guide/index.html#metricsAndReportersPrometheus">Micronaut Micrometer &amp; Prometheus</a></li><li><a href="https://learning.oreilly.com/library/view/distributed-systems-observability/9781492033431/">Distributed Systems Observability by Cindy Sridharan</a></li><li><a href="https://copyconstruct.medium.com/logs-and-metrics-6d34d3026e38">Logs and Metrics. At last Wednesday’s Go meetup, one of…</a></li><li><a href="https://medium.com/faun/devops-meets-observability-78775c021b0e">Come Meet the Pyramid of Happiness! | by Alex King | FAUN</a></li><li><a href="https://landing.google.com/sre/sre-book/chapters/monitoring-distributed-systems/#xref_monitoring_golden-signals">The Four Golden Signals</a></li><li><a href="https://medium.com/thron-tech/how-we-implemented-red-and-use-metrics-for-monitoring-9a7db29382af">How we implemented RED and USE metrics for monitoring</a></li><li><a href="https://docs.google.com/document/d/199PqyG3UsyXlwieHaqbGiWVa8eMWi8zzAn0YfcApr8Q/edit?usp=sharing">My Philosophy on Alerting by Rob Ewaschuk</a></li><li><a href="https://www.humio.com/blog/observability-redefined/">Observability (Re)defined</a></li></ul><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=43ef79a9309" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[How much is your Salary?]]></title>
            <link>https://medium.com/swlh/how-much-is-your-salary-6794a4b22a47?source=rss-a639b32b2f2f------2</link>
            <guid isPermaLink="false">https://medium.com/p/6794a4b22a47</guid>
            <category><![CDATA[salary]]></category>
            <category><![CDATA[stock-options]]></category>
            <category><![CDATA[bonus]]></category>
            <category><![CDATA[compensation]]></category>
            <category><![CDATA[carreer]]></category>
            <dc:creator><![CDATA[Italo Santos]]></dc:creator>
            <pubDate>Wed, 03 Mar 2021 13:38:32 GMT</pubDate>
            <atom:updated>2021-03-04T01:38:35.338Z</atom:updated>
            <content:encoded><![CDATA[<h4>Understanding the total compensation</h4><p>While reading <a href="https://erikbern.com/2020/06/08/how-to-set-compensation-using-commonsense-principles.html">Erik Bernhardsson&#39;s article</a> I realized that thinking about the <strong>Total Compensation</strong> makes more sense instead of salary as the single income source and I&#39;ll try to explain my thoughts throughout this article.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/300/1*5rY_Iwl2XUrC0-Audko1XQ.png" /></figure><p><strong>Salary is only one component</strong> of your total compensation, which includes the value of any benefits received in addition to your salary. Some of the benefits that are most commonly provided within your salary include:</p><ul><li>Paid time off (vacation days, sick days, and holidays)</li><li>Insurance (medical, dental, disability, and/or life)</li><li>Gym memberships</li><li>Childcare assistance</li><li>Retirement plans</li><li>Employee assistance programs that offer legal advice, counseling, and other services</li><li>Etc…</li></ul><p>All these, in addition to your salary, will, in some way, reduce your life cost because you&#39;ll don&#39;t need to spend your base salary with any of those and this is something to consider as compensation.</p><h3>Total Compensation</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*kDi7IeitfxdfBYIV" /><figcaption>Photo by <a href="https://unsplash.com/@sharonmccutcheon?utm_source=medium&amp;utm_medium=referral">Sharon McCutcheon</a> on <a href="https://unsplash.com/?utm_source=medium&amp;utm_medium=referral">Unsplash</a></figcaption></figure><p>For a while, let&#39;s put apart the benefits mentioned above and talk about the major incomes which compose the total compensation.</p><p>Total compensation is <strong>gross income on an annual basis</strong> and<em> </em>commonly will be composed by:</p><ol><li>Base salary</li><li>Bonus</li><li>Equity</li></ol><blockquote>When looking at the total compensation, <strong>always consider the annual values</strong> because some of the components are calculated using this period and helps with the big picture.</blockquote><h4>Calculating your total compensation</h4><p>Using the major components mentioned above, below the calculation formula proposed:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/858/1*WEkMiRHg6mN5XQ95VkTqUg.png" /><figcaption>Total compensation formula</figcaption></figure><ul><li><strong>Base salary</strong> is obviously part of this formula.</li><li>The <strong>bonus</strong> should only be a part of it if it’s the “roughly guaranteed” type of bonus. In some companies, this is a component attached to the performance evaluation and usually a multiplier factor of the base salary. (<em>with or without a cap value</em>)</li><li><strong>Equity</strong> is the complex part of the formula, but in summary, is the <strong>number of grants</strong> (<em>which is the total amount divided by the grant value</em>), <strong>multiplied by the grant value</strong> and <strong>divided by the vesting time</strong>, and using the original value of the grant, not the present value.</li></ul><blockquote>If you have Stock Options you should subtract the strike price - <a href="https://smartasset.com/investing/stock-options-vs-rsu">Stock Options vs RSU</a>)</blockquote><h4>Spreadsheet calculator</h4><p>I&#39;ve built a public spreadsheet to help with the total compensation calculation. The image below is the <a href="http://bit.ly/compensation-sheet-usd"><strong>USD sheet</strong></a>, since equity plans use USD as currency I also build a <a href="http://bit.ly/compensation-sheet-brl"><strong>BRL sheet</strong></a> converting the equity value.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/928/1*9ATFK_QVsq26nGN45iGxVA.png" /><figcaption>Total Compensation Calculator spreadsheet</figcaption></figure><p>There are several variables consider in the total compensation but this article aims to help you to look at your income from another perspective when evaluating a job offer or future positions in your current work.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=6794a4b22a47" width="1" height="1" alt=""><hr><p><a href="https://medium.com/swlh/how-much-is-your-salary-6794a4b22a47">How much is your Salary?</a> was originally published in <a href="https://medium.com/swlh">The Startup</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[The CI / CD odyssey]]></title>
            <link>https://faun.pub/the-ci-cd-odyssey-1cc058088feb?source=rss-a639b32b2f2f------2</link>
            <guid isPermaLink="false">https://medium.com/p/1cc058088feb</guid>
            <category><![CDATA[continuous-integration]]></category>
            <category><![CDATA[site-reliability-engineer]]></category>
            <category><![CDATA[continuous-deployment]]></category>
            <category><![CDATA[continuous-delivery]]></category>
            <category><![CDATA[devops]]></category>
            <dc:creator><![CDATA[Italo Santos]]></dc:creator>
            <pubDate>Tue, 05 Jan 2021 13:18:53 GMT</pubDate>
            <atom:updated>2023-01-26T02:28:44.334Z</atom:updated>
            <content:encoded><![CDATA[<h4>A brief history of the concepts</h4><p>W ho never heard the term CI/CD? <strong>Continuous Integration</strong> and <strong>Continuous Delivery</strong> are common practices widely used by most technology companies as part of the development cycle to make product development faster.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/650/0*PLiBaqSeKHbiMplL.jpg" /><figcaption>Image by <a href="https://devops.com/author/breselman/"><em>Roelbob</em></a> — <a href="https://devops.com/ci-cd/">DevOps.com</a></figcaption></figure><p>The most common development cycle done in most technology companies is that developers make all code changes in their local machines, after checking out a copy of the code from the main branch in a remote and centralized repository. After making all changes, check if the code is up to date with the main branch remotely, running tests locally (<em>and pass</em>) everything is ready to be committed and pushed to the main branch into the remote repository.</p><blockquote>That’s the <strong>happy path</strong>…</blockquote><p>It’s hard to guarantee that everyone run all the tests (<em>and pass</em>) before pushing the code to the main branch and, even more, in a company with multiple developers, there is no way to ensure that everyone was running the same version of code and also that all pieces of code will work well together.</p><h3>The beginning of the Continuous era…</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/400/1*f5qyh8IEQHg1z66m0YU8Mg.png" /><figcaption>Image from <a href="https://imgs.xkcd.com/comics/git.png">XKCD</a></figcaption></figure><p>In 2006 <a href="https://martinfowler.com/aboutMe.html">Martin Fowler</a> (<em>probably you know who is he don’t you? Believe me, you know him</em>) wrote <a href="https://martinfowler.com/articles/continuousIntegration.html">an article</a> advocating that the development cycle should <strong><em>continuously integrate</em></strong> the developer’s code and also run the test suite automatically to reduce the number of bugs detecting then quickly.</p><p>As mentioned in <a href="https://circleci.com/blog/a-brief-history-of-devops-part-iii-automated-testing-and-continuous-integration/">CircleCI’s article</a>, this practice becomes popular in 2008’s with the growth of the <a href="https://en.wikipedia.org/wiki/Hudson_(software)">Hudson CI</a> which was late replaced by <a href="https://www.jenkins.io/">Jenkins</a> in 2011 after Oracle announced its plans to trademark and commercialize it.</p><h3>Putting code in production</h3><p>Martin Fowler, in 2010, forwards the <a href="https://martinfowler.com/books/continuousDelivery.html"><strong>Continuous Delivery</strong></a> book written by Jez Humble, David Farley which says…</p><blockquote>“Software that’s been successfully integrated into a mainline code stream still isn’t software that’s out in production doing its job.”</blockquote><p>However, <strong>it’s common to confuse continuous delivery with continuous deployment</strong> and CircleCI’s <a href="https://circleci.com/blog/a-brief-history-of-devops-part-iv-continuous-delivery-and-continuous-deployment/">blog post</a> defines…</p><blockquote><strong>Continuous delivery</strong> is the practice of ensuring that software is <strong>always ready to be deployed</strong></blockquote><p>This means that all code into the main branch should be already tested (ran unit tests, integration tests, end-to-end, etc…) and packaged (using Docker, Helm, etc…) which <strong>requires</strong> continuous integration.</p><h3>What about automating this?</h3><p>On the other hand, <strong>Continuous Deployment</strong> can be interpreted as the automation of continuous delivery which takes care of the last-mile of the development cycle automatically.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/550/1*g4vou4obVCJ-4-ffN_TILQ.png" /></figure><p>The main objective of the continuous deployment is to ship small changes in production as quickly as possible (attention to the “<em>small changes</em>” sentence) and it’s important to mention that a big change is made by small changes and this culture will not be created with a tool or a process.</p><blockquote>First of all, there’s the somewhat pedantic fact that deployment does not imply release.</blockquote><p>That was mentioned by <a href="https://medium.com/u/25ce121a07f2">Jez Humble</a> in their <a href="https://continuousdelivery.com/2010/08/continuous-delivery-vs-continuous-deployment/">article explaining the difference between concepts</a>. What makes continuous deployment special is deploying every change that passes the automated tests (and optionally a short QA gate) to production.</p><blockquote>Continuous deployment might seem extreme and dangerous</blockquote><figure><img alt="" src="https://cdn-images-1.medium.com/max/500/1*2YfVVL8USdc-lBQxYy9mbg.png" /></figure><p>Teams usually need some breathing room between development and deploy, especially with database migrations changes when it’s crucial to have a clear safe point to rollback in case of failure and ensure that the code and the database are backward compatible, and so on. And because of that only a few companies properly run this practice nowadays…</p><h4>It’s hard but not impossible…</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/486/0*u8vaaYy412XEFnwN.jpg" /></figure><p>There some deployment strategies like <a href="https://martinfowler.com/bliki/BlueGreenDeployment.html">Blue / Green</a>, <a href="https://martinfowler.com/bliki/CanaryRelease.html">Canary deployment</a>, or <a href="https://launchdarkly.com/blog/why-leading-companies-dark-launch/">Dark Launch</a> that can help when doing continuous deployment. These techniques will replicate some traffic to the new production version with or without real user traffic allowing you to observe the system behavior before moving forward.</p><p><a href="https://medium.com/u/13a36f31dadc">Davi de Castro Reis</a> explains these and other techniques in the <a href="https://medium.com/@daviws/techniques-and-limitations-for-experiments-be0c318f369e"><em>Techniques and limitations for experiments</em></a><em> </em>article. Also, <a href="https://medium.com/u/7a5687feeca3">Felix Bauer</a> from <a href="https://medium.com/3yourmind">3yourmind</a> wrote a very nice article about <a href="https://medium.com/3yourmind/keeping-django-database-migrations-backward-compatible-727820260dbb"><em>Keeping Django database migrations backward compatible</em></a> which can help to handle the partial deployment strategies mentioned before which need to leave with two or multiple database schemas.</p><p><strong>Feature Flags</strong> is also another approach to split release from the deployment making it safer, according to <a href="https://martinfowler.com/articles/feature-toggles.html">Martin Fowler</a>. <a href="https://medium.com/u/1bc37f6b0eb7">Nassos Michas</a> also wrote a nice article about <a href="https://medium.com/swlh/feature-flags-how-to-decouple-code-from-features-b36b59792e0c">How to Decouple Code from Features</a></p><h4>Putting everything in a single image…</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/752/1*N5L6dRRsBwb-l75F6mU5Ng.png" /><figcaption>Image by <a href="https://www.atlassian.com/continuous-delivery/principles/continuous-integration-vs-delivery-vs-deployment">Atlassian</a></figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/727/0*Piks8Tu6xUYpF4DU" /></figure><p>👋 <a href="https://faun.dev/join"><strong>Join FAUN today and receive similar stories each week in your inbox!</strong></a><strong> </strong>️ <strong>Get your weekly dose of the must-read tech stories, news, and tutorials.</strong></p><p><strong>Follow us on </strong><a href="https://twitter.com/joinfaun"><strong>Twitter</strong></a><strong> </strong>🐦<strong> and </strong><a href="https://www.facebook.com/faun.dev/"><strong>Facebook</strong></a><strong> </strong>👥<strong> and </strong><a href="https://instagram.com/fauncommunity/"><strong>Instagram</strong></a><strong> </strong>📷 <strong>and join our </strong><a href="https://www.facebook.com/groups/364904580892967/"><strong>Facebook</strong></a><strong> and </strong><a href="https://www.linkedin.com/company/faundev"><strong>Linkedin</strong></a><strong> Groups </strong>💬</p><figure><a href="https://www.faun.dev/join?utm_source=medium.com/faun&amp;utm_medium=medium&amp;utm_campaign=faunmediumbanner"><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*_cT0_laE4iPcqW1qrbstAg.gif" /></a></figure><h4>If this post was helpful, please click the clap 👏 button below a few times to show your support for the author! ⬇</h4><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=1cc058088feb" width="1" height="1" alt=""><hr><p><a href="https://faun.pub/the-ci-cd-odyssey-1cc058088feb">The CI / CD odyssey</a> was originally published in <a href="https://faun.pub/">FAUN.dev() 🐾</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[The Release Manager model]]></title>
            <link>https://medium.com/loggi-product/the-release-manager-model-7af93f9f499f?source=rss-a639b32b2f2f------2</link>
            <guid isPermaLink="false">https://medium.com/p/7af93f9f499f</guid>
            <category><![CDATA[site-reliability-engineer]]></category>
            <category><![CDATA[continuous-delivery]]></category>
            <category><![CDATA[sre]]></category>
            <category><![CDATA[logistics]]></category>
            <category><![CDATA[technology]]></category>
            <dc:creator><![CDATA[Italo Santos]]></dc:creator>
            <pubDate>Tue, 29 Sep 2020 14:02:26 GMT</pubDate>
            <atom:updated>2020-11-08T04:54:30.923Z</atom:updated>
            <content:encoded><![CDATA[<h4>Scaling monolith deployment process with continuous delivery</h4><p>Nowadays the <strong>Continuous Integration</strong> and <strong>Continuous Delivery</strong> (i.e.: CI/CD) practices are widely used by most of the technology companies and the goal of these practices is to speed up product development.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/400/1*LoDytpscZyG5AqgJQBasIw.png" /><figcaption>Deployment pipeline by {<a href="http://turnoff.us/geek/deployment-pipeline/">turnoff.us</a>}</figcaption></figure><p>To understand how the <strong>Release Manager model</strong> fits into these practices we may take a look at the path until getting into the proposed model.</p><h3>Talking about history…</h3><p>Back in 2017, Loggi’s main application deployment process was made based on pull requests and you should be asking yourself, what does that mean? This means that if you merge something into the main branch, it was “<em>recommended</em>” to ship the code to production right after merging the code.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*rrrwgWod1EQPEeE-Qs6XBw.png" /><figcaption>Loggi’s main development slack channel topic message (PT-BR)</figcaption></figure><p>This can be defined as something <strong>between</strong> <a href="https://circleci.com/blog/a-brief-history-of-devops-part-iii-automated-testing-and-continuous-integration/"><strong><em>continuous delivery </em>and<em> continuous deployment</em></strong></a> since every new commit was shipped to production with a manual process.</p><h4>Let’s talk about the problems…</h4><p>Let’s imagine a scenario where someone wants to put some code in production, the code is merged into the main branch and the deploy process was initiated…</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/500/0*dMR0mFWz3a5qRYg8.jpg" /></figure><p>However, someone else “<em>accidentally</em>” merged another pull request before and didn’t ship it to production (<em>no issues until now</em>), and the previously merged code contained a bug that caused a serious incident.</p><p>The first action is to revert the last known pull request,<strong> which doesn’t solve the problem</strong> and the next actions will be: <strong>start calling people and digging into the last merged pull requests</strong> as a try to identify the buggy code to do the right revert.</p><p>The above scenario highlights a few problems:</p><ul><li>No explicit recovery point in case of failure</li><li>No release or changelog mechanism</li><li>Lack of protection for human error</li><li>Error-prone &amp; Miscommunication issues</li></ul><h4>The schedule</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/572/1*HZbNhQq-STmhT2uORr2rkw.png" /><figcaption>Deploy calendar notification on Slack channel</figcaption></figure><p>As an attempt to better organize deploys, a <strong>shared calendar was created with scheduled deployment windows</strong> where anyone from the engineering team could simply pick a free time to deploy their code to production</p><p>The scheduled windows were divided into 4 windows:</p><ul><li><strong>Morning</strong>: 7:00 AM to 8:00 AM BRT</li><li><strong>Lunch</strong>: 1:00 PM to 2:00 PM BRT</li><li><strong>Late Afternoon</strong>: 6:00 PM to 7:00 PM BRT</li><li><strong>Early Morning</strong>: 2:00 AM to 5:00 AM BRT</li></ul><p>This addresses the visibility issue but still has a communication overhead and starts to slow down the engineering team since the team starts growing and becomes hard to find a spot on the deployment windows.</p><h4>What about Continuous Deployment?</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/550/1*g4vou4obVCJ-4-ffN_TILQ.png" /></figure><blockquote>First of all, there’s the somewhat pedantic fact that <a href="https://continuousdelivery.com/2010/08/continuous-delivery-vs-continuous-deployment/">deployment does not imply release</a>.</blockquote><p>But what makes continuous deployment special is deploying every change that passes the automated tests (and optionally a short QA gate) to production.</p><blockquote>Continuous deployment might seem extreme and dangerous</blockquote><p>Teams usually need some breathing room between development and deploy, especially with database migrations changes when it’s very important to have a clear safe point to rollback in case of failure and ensure that the code and the database are backward compatible and so on. And because of that only a few companies properly run this practice nowadays…</p><p>As an attempt to solve the productivity issue we started some tests with what we called “<strong><em>continuous deployment for real</em></strong>”… In the first design, we tried to automate the current deployment strategy since the development cycle already uses branching to split environments.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/800/1*3XCwFtLjU-WrWez9w9hf5w.png" /><figcaption>Continuous deployment draft back 2018</figcaption></figure><p>The initial implementation was made with the <strong>staging environment, </strong>which already had its own branch and was a safe place to fail, and definitely improve the engineering productivity because everyone was able to just push the code to the branch and get it ready into a real-world environment…</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/275/0*88lxR8UUKAHm1Hkq" /></figure><p><strong>But</strong> (<em>this world is so cruel</em>) we face some problems, which in fact could be very harmful in the production environment. One of the challenges comes with database migrations, in the first place it wasn’t so hard to automate the execution of the migration if it exists, but to scale the process we’ll also need to create a lot of tooling to safeguard failure scenarios and create a stable recovery point in case of a rollback.</p><blockquote>To be Continued…</blockquote><p>Using the branching approach in mind, we also tried to use the<a href="https://www.atlassian.com/git/tutorials/comparing-workflows/gitflow-workflow"> Gitflow</a> workflow and couple each branch to an infrastructure environment (<em>i.e.: development, staging, and production</em>)</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/658/0*3SQ_2iHMxTHlkg_e" /></figure><p>However, the <strong>GitFlow is a bit complex</strong> which should take too much effort to automate the entire process. Furthermore, <strong>we face an organizational issue</strong> where we have multiple teams working in several features that can have different release dates and <strong>would be very hard to align everyone to make a single release.</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/500/1*kCUdXXgB4jiq4s8zg2ziYw.png" /><figcaption>Automating GitFlow</figcaption></figure><h3>Evolving for stability</h3><p>All previous approaches were driven by speed and productivity, but this philosophy becomes a bottleneck when the <a href="https://partiu.loggi.com/how-we-hired-200-software-engineers-in-one-year-41151c50cd54">engineering team starts to grow</a> which makes that the “deployment queue” becomes bigger and bigger, which <strong>doesn’t leave too much time to observe the system in production.</strong></p><p>During the 2018’s Black Friday we identified that we need to change the way we ship changes to production because we’ve identified that we <strong>were moving so fast that was hurting us</strong>, mainly<strong> </strong>because we were <strong>shipping bad code to production</strong> and increasing the number of outages.</p><figure><img alt="" src="https://cdn-images-1.medium.com/proxy/0*qrdEaMPgSkdRfjt8" /><figcaption>Loggi’s 2018 Outage Main Reasons</figcaption></figure><p>As the company grows the business becomes more critical and less susceptible to failures. That is a very common path for growing companies, the Facebook is an example of that, where Zuckerberg said in 2014 company’s F8 conference, referring to the “<a href="https://mashable.com/2014/03/13/facebook-move-fast-break-things/"><em>Move Fast and Break Things</em></a>” philosophy</p><blockquote>“What we realized over time is that it wasn’t helping us to move faster because we had to slow down to fix these bugs and it wasn’t improving our speed.”</blockquote><p>But slowness isn’t good to a Startup, and we put ourselves to think about <strong>how can we keep moving fast and stable</strong>…</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/500/1*WXIN3INKrjat2lcI28KDbA.png" /></figure><h4>Changing (a bit) the development cycle</h4><p>Since we’ve dropped the idea to go with the GitFlow + Continuous Deployment approach, mainly for the production environment, we’ve decided to get the best from <a href="https://www.atlassian.com/git/tutorials/comparing-workflows/feature-branch-workflow">Feature Branch Workflow</a> and Google’s SRE <a href="https://landing.google.com/sre/sre-book/chapters/release-engineering/#branching-nqs9tMud">Release Engineering Branching strategy</a></p><blockquote><strong>The main</strong> <strong>branch is the single source of truth</strong> and code into this branch is considered <strong>stable and ready for deploy</strong></blockquote><p>And, with that said we keep the development cycle very similar to the Feature Branch where all feature development should take place in a dedicated branch instead and merged back in the main branch</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/500/1*hKO6em7OEYwc-w1HLHUJGg.png" /><figcaption>Image by <a href="https://medium.com/u/7c993c792d0">Kasun Siyambalapitiya</a></figcaption></figure><p>Google’s SRE releasing strategy said that projects don’t release directly from the mainline, instead, a new branch is created from the mainline and named with a specific revision version and become read-only.</p><p>We’ve dropped the GitFlow strategy because of the huge number of branches coming and going, we decided to <strong>use </strong><a href="https://git-scm.com/book/en/v2/Git-Basics-Tagging"><strong>git tag</strong></a><strong> as a version marker</strong> which is like a branch but <strong>doesn’t change</strong> allowing us to keep the atomicity.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/500/1*Le79K_lbzxKHq6PZE95lQg.png" /></figure><p>Even after all the previous work we still have the organizational issue mentioned before and still an open question:</p><blockquote>What is a release for us? And when we should do it?</blockquote><p>At the end of 2018’s holidays, we introduce the concept of the<strong> daily deploy </strong>which was <strong>our definition of a release</strong> and also <strong>when it will happen</strong>.</p><h4>The Red / Black deployment</h4><p>Since our main goal was to increase the stability we started the year using a more reliable deployment approach that we called “<strong><em>Push do Macunha”</em></strong><em> </em>in honor of its creator.</p><p>The design uses a similar approach of the Canary Deployment but we rename it to <strong>“Red /</strong> <strong>Black” </strong>(<em>believe it or not, we already had the blue/green terminology in our platform</em>)</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/500/1*_VIL-Mbm72uG_mQC2ZDMOw.png" /></figure><p>On a daily basis scheduled time, a new tag is automatically generated, and the pipeline is started leaving the code ready to be deployed, <strong>waiting to be approved.</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/990/1*bsovPpJjsApezSzCYf1BhA.png" /><figcaption>Loggi’s “canary-like” deployment overview</figcaption></figure><h3>The Release Manager</h3><p>Ben Treynor advocates on your talk about<a href="https://youtu.be/n4Wf14e2jxQ?t=2136"> Keys for SRE</a> that the SRE team needs to keep the <strong>developer teams into the production loop and share around 5% of the operation work</strong> and, during the 2018’s holidays (<em>including Black Friday</em>) the deployment process was being handled by the SRE team which increase the operational load and moreover, transforming the deploy process in a “<em>mystic creature</em>”.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/860/0*P4Tylckc59Uq3OKU.png" /><figcaption>Image from <a href="https://me.me/i/588966146550488e8065982c34008aaf">me.me</a></figcaption></figure><p>SRE team doesn’t run the production alone, opposite of that, we believe in a shared responsibility environment that everyone on the engineering team can run Loggi’s main application deployment process and, inspired by<a href="https://landing.google.com/sre/sre-book/chapters/release-engineering/"> Google’s Release Engineer</a> discipline we’ve created the <strong>Release Manager.</strong></p><p>The main objective of the Release Manager role is to spread the knowledge, the operational load across the engineering team <strong>scaling the deployment process</strong> in multiple teams.</p><p>The Release Manager<em> </em>is responsible for the following actions:</p><ul><li>Coordinate the Loggi’s main application deployment process.</li><li>Monitor and take the decision to move forward or rollback the code.</li><li>In case of a rollback, it will be in charge of identifying and reverting the offending PR.</li></ul><p>We encourage everyone to be part of the rotation, but there is no obligation to be part of the process and, with that said we’ve also created a <strong>Release Manager Rotation, </strong>which is a list with the names of everyone which applied to be a release manager.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/780/1*6XKBxV_qcu-pB02o09AXMg.png" /><figcaption>The first version of the Release Manager rotation sheet</figcaption></figure><blockquote>The first list born with 15 names and <strong>today we have more than 40 release managers</strong></blockquote><figure><img alt="" src="https://cdn-images-1.medium.com/max/350/0*ZZ9pBG6x7SyO-sgn.jpeg" /></figure><h4>The deployment process isn’t linear</h4><p>Even with the process running for a long time we realized that the new release manager’s confidence was low, mostly because of the number of decisions which need to be taken during the deployment mainly in failure cases</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/500/1*GeAT3uTfSbrSUd7U3YInDw.png" /><figcaption>Deployment workflow overview</figcaption></figure><blockquote>To reach the 40+ release managers we have today, we needed to address this issue in some way, which is also an issue for new SRE members.</blockquote><p>So one of the new SRE members that were recently joined the team had attended<a href="https://partiu.loggi.com/como-recebemos-novos-loggers-de-forma-100-remota-bd852eeed51f"> Loggi’s Xboarding</a>, and also the SRE’s, realized that it could use the same methodology to give more context to new release managers in a more practical way, and the <strong>Release Manager Onboarding</strong> was born.</p><p>The process was born to be self-regulated creating a continuous learning environment and for that, we’ve created a classification list based o the previous experience as a release manager (<em>RM for short</em>)</p><p><strong>Full RM </strong>— The experienced release manager that has already done several deploys and is able to “<em>spread the word</em>” to the other members.</p><p><strong>Assistant RM</strong> — A less experienced release manager which already attended some deploys as shadow and has absorbed sufficient context to run an entire deploy by themselves paring with a Full RM.</p><p><strong>Shadow RM</strong> — Since the process born to <em>“spread the word”</em> and transfer the knowledge across the engineering team, <strong>anyone can be part of the deploy process</strong> as a Shadow (<em>another inspiration from our</em><a href="https://partiu.loggi.com/como-funciona-nosso-processo-seletivo-de-engenharia-de-software-8c493722feeb"><em> hiring process</em></a>), which is an observer.</p><figure><img alt="" src="https://cdn-images-1.medium.com/proxy/0*Xo-044Go2AkTfJSY.png" /><figcaption>The face of the Continous Learning approach</figcaption></figure><p>Everything is done through a conference call allowing everyone to participate and the <strong>Full RM</strong> should follow the normal flow of the deployment, but the difference is that it will share its screen with everyone to share their knowledge and experience</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*VOyG-execheYXmiOSUAH9g.png" /></figure><h3>References</h3><ul><li><a href="https://landing.google.com/sre/sre-book/chapters/release-engineering/">Google SRE book — Release Engineering</a></li><li><a href="https://www.atlassian.com/continuous-delivery/principles/continuous-integration-vs-delivery-vs-deployment">Continuous integration vs. continuous delivery vs. continuous deployment</a></li><li><a href="https://circleci.com/blog/a-brief-history-of-devops-part-iii-automated-testing-and-continuous-integration/">A Brief History of DevOps, Part III: Automated Testing and Continuous Integration</a></li><li><a href="https://circleci.com/blog/a-brief-history-of-devops-part-iii-automated-testing-and-continuous-integration/">A Brief History of DevOps, Part IV: Continuous Delivery and Continuous Deployment</a></li><li><a href="https://circleci.com/blog/building-strong-distributed-teams-one-pixel-at-a-time/">Continuous deployment without downtime</a></li></ul><p>If you are looking for a new job opportunity, we are hiring new Loggers to work in Brazil and Portugal, check out our open positions <a href="https://bit.ly/3gQtTPX">here</a>!</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=7af93f9f499f" width="1" height="1" alt=""><hr><p><a href="https://medium.com/loggi-product/the-release-manager-model-7af93f9f499f">The Release Manager model</a> was originally published in <a href="https://medium.com/loggi-product">Loggi</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Celery + RabbitMQ common issues]]></title>
            <link>https://italux.medium.com/celery-rabbitmq-common-issues-a741a3800b30?source=rss-a639b32b2f2f------2</link>
            <guid isPermaLink="false">https://medium.com/p/a741a3800b30</guid>
            <category><![CDATA[django]]></category>
            <category><![CDATA[celery]]></category>
            <category><![CDATA[sre]]></category>
            <category><![CDATA[rabbitmq]]></category>
            <dc:creator><![CDATA[Italo Santos]]></dc:creator>
            <pubDate>Sun, 12 Apr 2020 17:08:14 GMT</pubDate>
            <atom:updated>2020-04-12T17:25:16.523Z</atom:updated>
            <content:encoded><![CDATA[<h4>Best practices to improve your broker infrastructure reliability</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*AX33WfpBkugfrNrupp00HA.png" /></figure><p>After while running Celery + RabbitMQ in production, you face several issues, which make you learn more about both technologies. In this article, I’ll try to share some os the most common issues and some best practices that can help with those issues.</p><h3>What’s Celery?</h3><p>According <a href="http://www.celeryproject.org/">Celery Project website</a>…</p><blockquote>Celery is an asynchronous task queue/job queue based on distributed message passing. It is focused on real-time operation, but supports scheduling as well.</blockquote><figure><img alt="Celery basic architecture visual" src="https://cdn-images-1.medium.com/max/1024/0*HvZk015GZdvL1tyv.png" /><figcaption>Basic Celery architecture visualization by <a href="https://blog.imaginea.com/scale-part-i-task-queues/">Imaginea</a></figcaption></figure><p>Celery <a href="https://docs.celeryproject.org/en/latest/getting-started/brokers/"><em>supports several brokers</em></a> as backends, including <a href="https://www.rabbitmq.com/"><em>RabbitMQ</em></a><em>, </em>which is widely used across multiple companies as a message broker solution and is the same described in this article.</p><h3>What’s RabbitMQ?</h3><p>According <a href="https://www.cloudamqp.com/blog/2015-05-18-part1-rabbitmq-for-beginners-what-is-rabbitmq.html">Cloud AMQP blog post</a>…</p><blockquote>RabbitMQ is a message-queueing software also known as a <em>message broker</em> or <em>queue manager.</em> Simply said; it is software where queues are defined, to which applications connect in order to transfer a message or messages.</blockquote><figure><img alt="" src="https://cdn-images-1.medium.com/max/800/0*b86dt4OsIMTSsNvY.png" /><figcaption>RabbitMQ overview architecture by <a href="https://www.cloudamqp.com/blog/2015-05-18-part1-rabbitmq-for-beginners-what-is-rabbitmq.html">Cloud AMQP blog</a></figcaption></figure><h3>Common Issues</h3><p>The most common issues I’ve experienced were related to <strong>broker connection, memory usage and also with long-running tasks consuming too many resources</strong>. So, let’s talk about these issues and the proposed solutions…</p><h4>Connection to broker lost</h4><p>RabbitMQ uses AMQP as a messaging protocol, which can be defined as an application and network layer protocol that allows client applications to talk to the server and interact. Cloud AMQP has a very detailed article about <a href="https://www.cloudamqp.com/blog/2019-11-21-what-is-amqp-and-why-is-it-used-in-rabbitmq.html"><em>What is AMQP and why is it used in RabbitMQ?</em></a> which is a good read recommendation.</p><p>Some time ago, I’ve faced several connection errors between Celery workers and the RabbitMQ. Those errors happened in a very random way, without any explicit errors… causing the application to stop consuming messages and things start breaking. The only workaround available was to do a <strong>restart on the message broker service</strong> to force workers to reopen their channels.</p><p>After some time digging into the logs I found the error below, that happens every time the application stops to consuming messages:</p><pre>consumer: Connection to broker lost. Trying to re-establish the connection...<br><br>ConnectionError: Bad frame read<br>consumer: Cannot connect to amqp://username:**@broker.rabbitmqhost.com:5672//: Error opening socket: a socket error occurred.<br>Trying again in 2.00 seconds...</pre><pre>consumer: Cannot connect to amqp://username:**@broker.rabbitmqhost.com:5672//: Error opening socket: a socket error occurred.<br>Trying again in 4.00 seconds...</pre><pre>consumer: Cannot connect to amqp://username:**@broker.rabbitmqhost.com:5672//: Error opening socket: a socket error occurred.<br>Trying again in 6.00 seconds...</pre><pre>Connected to amqp://username:**@broker.rabbitmqhost.com:5672//</pre><p>The problem with that is, sometimes the automatic connection retry doesn’t work and the workaround describe previously need to be executed.</p><p>Even with a bad workaround described, this gave some time to research more about the issue. Looking into the <a href="https://github.com/celery/celery/issues"><em>Celery’s GitHub issues</em></a> I’ve found some recommendations to <strong>replace the </strong><a href="https://pypi.org/project/librabbitmq/"><strong>librabbitmq</strong></a><strong> by </strong><a href="https://pypi.org/project/PyAMQP/"><strong>pyamqp</strong></a><strong> </strong>as we can see on <a href="https://github.com/celery/librabbitmq/issues/54"><em>issue #54</em></a><em>. </em>So, after replace the connection string to use <em>pyamqp</em> helps with the issue…</p><h4>Missed heartbeat messages</h4><p>During a normal workday, you can face a lot of log messages like below in your production worker environment.</p><pre>missed heartbeat from celery@worker-01<br>missed heartbeat from celery@worker-02<br>missed heartbeat from celery@worker-03<br>missed heartbeat from celery@worker-04<br>missed heartbeat from celery@worker-05<br>missed heartbeat from celery@worker-06<br>missed heartbeat from celery@worker-07<br>missed heartbeat from celery@worker-08</pre><p>Those messages describe that Celery workers weren’t able to get a keepalive message from the broker. When a missed heartbeat happens several times, the worker can try some reconnection using <a href="https://docs.celeryproject.org/en/stable/userguide/configuration.html#broker-failover-strategy">BROKER_FAILOVER_STRATEGY</a> which can drive us to the previously described issue. Also, it&#39;s important to say that celery has his own heartbeat implementation which <strong>sends a keepalive message every 2 seconds</strong> as described by <a href="https://docs.celeryproject.org/en/stable/userguide/configuration.html#broker-heartbeat-checkrate">BROKER_HEARTBEAT_CHECKRATE</a> , with that, we can face &quot;<em>flood</em>&quot; behavior. This was described in this <a href="https://stackoverflow.com/a/21038204"><em>StackOverflow discussion</em></a><em>…</em></p><p>In summary, the defaults Celery’s configuration sends a lot of “<em>unnecessary</em>” messages to the broker. To avoid this flood behavior you can <strong>add some parameters like </strong><a href="http://docs.celeryproject.org/en/latest/reference/celery.bin.worker.html#cmdoption-celery-worker-without-heartbeat"><strong>--without-heartbeat</strong></a><strong> as followed by </strong><a href="http://docs.celeryproject.org/en/latest/reference/celery.bin.worker.html#cmdoption-celery-worker-without-gossip"><strong>--without-gossip</strong></a><strong> and </strong><a href="http://docs.celeryproject.org/en/latest/reference/celery.bin.worker.html#cmdoption-celery-worker-without-mingle"><strong>--without-mingle</strong></a> on the worker&#39;s startup command to decrease the number of messages and avoid unexpected reconnection behaviors.</p><p>These configurations are recommendations from several articles like <a href="https://www.cloudamqp.com/blog/2016-01-25-identify-and-protect-against-high-cpu-and-memory-usage.html#python-celery">CloudAMQP blog post</a>, <a href="https://www.cloudamqp.com/docs/celery.html#commandline-arguments">CloudAMQP doc</a> and also in this issue from the <a href="https://github.com/celery/celery/issues/4980#issuecomment-413536165">celery repository</a>.</p><h4>High memory usage and a large number of queues</h4><p>Celery worker has the ability to send a message whenever some event happens and these events can be captured by tools like Flower to <a href="http://docs.celeryproject.org/en/v4.0.0/userguide/monitoring.html#events"><em>monitor the cluster</em></a><em>.</em></p><p>When you have <a href="https://docs.celeryproject.org/en/stable/userguide/configuration.html#events">events enabled</a> and you have a high tasks throughput you’ll have a lot of events messages and if you haven’t expiration rules configured, you can reach a scenario like this:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*-e9YK2zSLu1HT8fwEpQSjQ.png" /><figcaption>RabbitMQ celery event queue size</figcaption></figure><p>To avoid this, add two expiration settings, one to thee event queues <a href="https://docs.celeryproject.org/en/stable/userguide/configuration.html#event-queue-expires">CELERY_EVENT_QUEUE_EXPIRES</a> and the second to event messages <a href="https://docs.celeryproject.org/en/stable/userguide/configuration.html#event-queue-ttl">CELERY_EVENT_QUEUE_TTL</a>. More information can be found on <a href="https://github.com/celery/celery/issues/2387"><em>celery/celery #2387</em></a><em> </em>issue.</p><h4>Tasks time limits</h4><p>When you have long-running tasks is a good practice to add <a href="https://docs.celeryproject.org/en/stable/userguide/configuration.html#task-soft-time-limit">CELERY_TASK_SOFT_TIME_LIMIT</a> and also<a href="https://docs.celeryproject.org/en/stable/userguide/configuration.html#task-time-limit">CELERY_TASK_TIME_LIMIT</a> to protect the broker from long-running tasks being executed forever consuming a lot of resources from your infrastructure.</p><h3>Conclusion</h3><p>Celery’s default configuration isn’t optimal to all environments and you’ll need to make adjustments to give your application workload the best performance and for that, Celery has a very good documentation with some optimizing recommendations, see <a href="https://docs.celeryproject.org/en/stable/userguide/optimizing.html"><em>User Guide — Optimizing</em></a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=a741a3800b30" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[PostgreSQL Out Of Memory]]></title>
            <link>https://italux.medium.com/postgresql-out-of-memory-3fc1105446d?source=rss-a639b32b2f2f------2</link>
            <guid isPermaLink="false">https://medium.com/p/3fc1105446d</guid>
            <category><![CDATA[django]]></category>
            <category><![CDATA[database]]></category>
            <category><![CDATA[postgresql]]></category>
            <dc:creator><![CDATA[Italo Santos]]></dc:creator>
            <pubDate>Tue, 25 Feb 2020 21:44:18 GMT</pubDate>
            <atom:updated>2020-05-29T03:13:26.291Z</atom:updated>
            <content:encoded><![CDATA[<h4>Database connections uses huge amount of memory</h4><p>Imagine you, in a beautiful sunny day, running your production environment when suddenly…</p><h3><strong>Your database SHUTDOWN!</strong></h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/816/0*PFfeDdk3VmA1lirw.png" /></figure><p>First thing to do is put everything up &amp; running again and, after that you start the diagnostic job. The first place to look is to the logs, when you start seek the messages you face with the message bellow:</p><pre>[Fri Oct 25 14:11:39 2019] <strong>Out of memory: Kill process 2591 (postgres)</strong> score 225 or sacrifice child<br>[Fri Oct 25 14:11:39 2019] Killed process 2591 (postgres) total-vm:34872820kB, anon-rss:2776kB, file-rss:0kB, shmem-rss:33416816kB</pre><pre>[...]</pre><pre>2019-10-25 14:12:15 UTC [2589]: [6] user=,db=,client=,app= LOG:  <strong>database system is shut down</strong></pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/512/0*RqFlPJzvB4r-HJd3" /></figure><p>That surprises you, once the database server have a lot of RAM which make no sense hit an OOM issue. With that in mind the next step is look to resources metrics (<em>i.e.: memory, CPU, Network, etc…</em>) with the expectation to find some leak which can explain an increasing memory usage, but, another surprise, nothing changes until the OOM issue time. The memory metrics was stable and suddenly<strong> free memory goes to zero </strong>causing the database shutdown.</p><p>So you start to seek for tuning recommendations, and the official PostgreSQL documentation has a good one about how <a href="https://www.postgresql.org/docs/9.6/kernel-resources.html"><em>Managing Kernel Resources</em></a><em>.</em> The first change done was the <em>Linux Memory Overcommit:</em></p><pre>sysctl -w vm.overcommit_memory=2</pre><p>The second was manually adjust the PostgreSQL process score to avoid kernel (<em>i.e.: </em><a href="https://lwn.net/Articles/761118/"><em>OOM Killer</em></a>) to kill the database process:</p><pre>echo -1000 &gt; /proc/self/oom_score_adj</pre><h4>After this optimizations the expectation is that everything is solve right? <strong>NO!</strong></h4><p>Some hours after, another database outage happen and this start to happen several times during the day, even out of working hours, and the logs show that the database doesn&#39;t shutdown anymore, but initiate to <strong>repeatedly restart</strong>…</p><pre>2019-10-25 21:15:57 UTC [122914]: [2] user=readonly,db=production-db,client=0.0.0.0,app=[unknown] FATAL:  the database system is in recovery mode</pre><pre>2019-10-25 21:16:31 UTC [89367]: [7] user=,db=,client=,app= LOG:  all server processes terminated; reinitializing</pre><p>While debug you can identify a lot of heavy queries and start doing a lot of optimizations and the next step was <strong>decrease the </strong><strong>work_mem</strong> configuration to use less memory on complex sorting queries.</p><p>But you still seen database restart caused by <em>Out of Memory</em> errors into the log messages:</p><pre>2019-11-01 13:31:54 UTC [12954]: [8] user=readonly,db=production-db,client=0.0.0.0,app=[unknown] ERROR:  out of memory<br>2019-11-01 13:31:55 UTC [13144]: [16] user=readonly,db=production-db,client=0.0.0.0,app=[unknown] ERROR:  out of memory</pre><pre>[...]</pre><pre>2019-11-01 13:33:45 UTC [69292] LOG:  server process (PID 84886) exited with exit code 127</pre><p>Besides query optimizations, you change more kernel params. The next ones is about the <em>Shared Memory</em>, by increasing it to leave more memory resource to the database process, something like bellow, following the <a href="https://www.postgresql.org/docs/9.6/kernel-resources.html#SYSVIPC"><em>documentation formula</em></a><em> </em>described below:</p><blockquote>The default maximum segment size is 32 MB, and the default maximum total size is 2097152 pages. A page is almost always 4096 bytes except in unusual kernel configurations with “huge pages” (use getconf PAGE_SIZE to verify).</blockquote><pre>sysctl -w kernel.shmmax=17179869184<br>sysctl -w kernel.shmall=4194304</pre><p>Even after all this modifications, nothing changes, and the database still restarting over and over…</p><p>So you&#39;ll need to understand more about how PostgreSQL uses memory to try to identify the possible cause of the issue.</p><p>One of the first things to do is try to understand how PostgreSQL’s memory allocation works, and, for that, <a href="https://severalnines.com/database-blog/architecture-and-tuning-memory-postgresql-databases"><em>severalnines has a nice post</em></a> about PostgreSQL memory architecture, explaining the differences between <em>Local / Shared</em> memory areas and for what each one is used. Below the image to illustrate how the things works:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/584/0*UI-ufEiVX-IEEEAU.png" /><figcaption>Basic PostgreSQL memory architecture</figcaption></figure><p>Alright, this helps but you&#39;ll need more deep known about this topic and <a href="https://blog.pgaddict.com/posts/introduction-to-memory-contexts"><em>PostreSQL Addict writes a very nice article</em></a> giving a very good understand about <strong>MemoryContexts</strong>, which are, basically, groups of allocated pieces of memory, making it easier to manage lifecycle. are organized in a tree, roughly matching the execution plans. It&#39;s main objective is to to minimize malloc calls/book-keeping, maximize memory reuse, and never really frees memory.</p><p>After all this reading you are able to understand more about memory allocation but doesn’t clarify the possible cause of the OOM issue.</p><p>So you&#39;ll keep researching and, <a href="https://medium.com/u/9644bc03a055"><em>Brandur</em></a> shows a big picture how <a href="https://brandur.org/postgres-connections#concurrency-limits">PostgreSQL process are forked and allocate memory</a> as we can see on the image below:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*0Px3r5KxytoyGGwdQaqrqQ.png" /><figcaption><em>A simplified view of Postgres’ forking process model.</em></figcaption></figure><p>Poring over the reading you face with the below quote:</p><blockquote>Each of these “backend” processes starts out at around 5 MB in size, but may grow to be much larger depending on the data they’re accessing</blockquote><p>With that, something come up into your mind: <em>&quot;If the connections are leaking?&quot;. </em>Since PostgreSQL is very mature code you wasn&#39;t able to find any memory leak bug, but, as said on <a href="https://dba.stackexchange.com/a/222815">Stack Exchange</a>, long-live connections can use a lot of memory:</p><blockquote>So if you have hundreds of thousands or millions of these things, and a <strong>long-lived connection</strong> will eventually touch each one of them, then <strong>their memory usage will continuously grow</strong>.</blockquote><figure><img alt="" src="https://cdn-images-1.medium.com/max/474/1*F56h0PYPIjcvYPuUjy3fsQ.png" /></figure><p>OK…! Things starts to make sense, since your application uses <a href="https://docs.djangoproject.com/en/1.11/ref/databases/#persistent-connections"><em>Django Persistent Connection</em></a> with database and you realize that you&#39;ve set the <strong>CONN_MAX_AGE</strong> to a very big number (i.e.: <strong>600</strong>) which should eating the server memory.</p><p>So, what is the right value? As per <a href="https://stackoverflow.com/a/37625753"><em>Stack Overflow question</em></a>, there&#39;s no <em>&quot;right value&quot;</em> since you need to evaluate your traffic, so the general recommendation is use <strong>CONN_MAX_AGE = 60 </strong>and watch.</p><h4>After apply this setting…</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/883/1*Y5zTBoE8CfA3-_gAE6m9AQ.png" /><figcaption>PostgreSQL percentage of memory usage after reduce CONN_MAX_AGE</figcaption></figure><h3>Conclusion</h3><p>The most common cause of out of memory issue happens when PostgreSQL is unable to allocate the memory required for a query to run. This is defined by <a href="https://www.postgresql.org/docs/current/runtime-config-resource.html#GUC-WORK-MEM"><strong>work_mem</strong></a><strong> </strong>parameter, which sets the maximum amount of memory that can be used by a query operation before writing to temporary disk files.</p><p>As said by <a href="https://www.citusdata.com/blog/2018/06/12/configuring-work-mem-on-postgres/"><em>Citus</em></a>, give too much memory to a query operation can cause some collateral effects like OOM issue…</p><blockquote>When you consume more memory than is available on your machine you can start to see out of out of memory errors within your Postgres logs, or in worse cases the OOM killer can start to randomly kill running processes to free up memory. An out of memory error in Postgres simply errors on the query you’re running, where as the the OOM killer in linux begins killing running processes which in some cases might even include Postgres itself.</blockquote><p><a href="https://pganalyze.com/docs/log-insights/server/S5"><em>PGAnalyse also describes</em></a> some characteristics and recommendations about OOM issues and configuration tuning.</p><p>But that isn&#39;t the only cause, as described, connection needs to be treat as a resource, as the same CPU, Memory, Network, etc… Although, there are no much documentation about connection behavior as we were able to see on the narrative.</p><p>There some mechanisms to protect the database infrastructure from this application <em>&quot;misbehaviour&quot;, </em>which is the <a href="https://www.postgresql.org/docs/9.6/runtime-config-client.html#GUC-IDLE-IN-TRANSACTION-SESSION-TIMEOUT"><strong>idle_in_transaction_session_timeout</strong></a> configuration…</p><blockquote>Terminate any session with an open transaction that has been idle for longer than the specified duration in milliseconds.</blockquote><p>This means that the database control the expiration for long-live connection opened from application.</p><p>Also, another approach is use some connection pool solution like <a href="https://www.pgbouncer.org/"><em>PgBouncer</em></a> or <a href="https://www.pgpool.net/mediawiki/index.php/Main_Page"><em>PGPool-II</em></a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=3fc1105446d" width="1" height="1" alt="">]]></content:encoded>
        </item>
    </channel>
</rss>