<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Boris Cherkasky on Medium]]></title>
        <description><![CDATA[Stories by Boris Cherkasky on Medium]]></description>
        <link>https://medium.com/@cherkaskyb?source=rss-665b8f6a764a------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/1*syQQrHELXakjdChP6je1IQ.jpeg</url>
            <title>Stories by Boris Cherkasky on Medium</title>
            <link>https://medium.com/@cherkaskyb?source=rss-665b8f6a764a------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Wed, 18 Nov 2026 07:51:25 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@cherkaskyb/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[A 10 Year Retrospective of a Passionate Software Engineer]]></title>
            <link>https://cherkaskyb.medium.com/a-10-year-retrospective-of-a-passionate-software-engineer-2d0cd7b3d836?source=rss-665b8f6a764a------2</link>
            <guid isPermaLink="false">https://medium.com/p/2d0cd7b3d836</guid>
            <category><![CDATA[programming]]></category>
            <category><![CDATA[software]]></category>
            <category><![CDATA[software-development]]></category>
            <category><![CDATA[software-engineering]]></category>
            <category><![CDATA[careers]]></category>
            <dc:creator><![CDATA[Boris Cherkasky]]></dc:creator>
            <pubDate>Mon, 11 Nov 2024 08:02:26 GMT</pubDate>
            <atom:updated>2024-11-11T08:02:26.381Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*gQI7p1HWxZjKEcV6" /><figcaption>Artwork by <a href="https://unsplash.com/@glenncarstenspeters">@glenncarstenspeters</a> on <a href="https://unsplash.com/photos/person-writing-bucket-list-on-book-RLw-UC03Gwc">unsplash</a></figcaption></figure><p>Since year 9 of my career I have been waiting to write this post. I was waiting to retrospect and look back at my path, and a decade of engineering was sounding like a nice point in time to do so.</p><p>I am now well into year 13 of my career, so as any software project — I am late to deliver.</p><p>Why am I writing this? <br>I mentor junior and mid level engineers, and one common theme that many of them mention is the lack of clear “progression path”, or what is the expected level at each point in the career timeline. <br>I see this article as a way to benchmark on how my career timeline looked like.</p><p>Every engineer does a lot in each year of programming, but I tried to distill <strong>one main theme</strong> of each year of my career progression.</p><p>It’s important to mention that in no way do I think my path is the right path, or the best path (I actually quite certain it isn’t), but it’s the path this humble engineer went through. It is also not the full one, but rather a gist of important milestones.</p><h3>Year 1 — Learn to code</h3><p>I’ve graduated from a good university, and what was clear when I began actual engineering work is that the coding level of fresh graduates is quite low. Most of our degree is focused on learning how to learn (and a lot of calculus), but our fingers didn’t see enough coding mileage.</p><p>So year one was just thousands of coding hours.</p><p>The code level was probably quite poor, but I’ve learned to deliver features, and just found my own phase of coding.</p><p>In the yearly review my manager’s main feedback was that the code I deliver wasn’t up to par with the quality standard of the team, and was buggier than expected 🙁. I doubled down on getting better.</p><p>🏁 If I had to sum it up to one achievement — I was able to deliver <strong>working features</strong>.<br>⏱️most time spent on <strong>coding<br>⬆️</strong>progress made throughout this year: <strong>minor</strong></p><h3>Year 2 — Processes, People, being part of a team</h3><p>I started to <em>“feel”</em> how the company operates, although most of my time was still coding, I was a more integral part of a team, I had more opinions and I was starting to matter on a team level.</p><p>I was working more with people, mostly through requirements, reviews, etc.</p><p>🏁 If I had to sum it up to one achievement — I was able to become <strong>a confident, pleasant team member</strong> to work with.<br>⏱️most time still spent on <strong>coding </strong>with more emphasis on <strong>code quality</strong> (less bugs, more tests)<br><strong>⬆️</strong>progress made throughout this year: <strong>medium</strong></p><h3>Year 3 — Complexity</h3><p>The organization is now sure I can deliver, and more responsibility is passed down to me, I was leading quite large software components with more complex interfaces.</p><p>If until now each interface was “given to me”, and I just assumed it to be the absolute truth, now I was the one designing the interface, and maybe even questioning the design of the existing ones.</p><p>Through this more complex kind of work, I was in more contact with the product organization and customers, and it became clear that <strong>customers and engineers speak completely different languages</strong>. Each word used is nuanced with different expectations on each side.</p><p>I’ve decided to understand the customer better, and eventually <strong>learned to ask better questions</strong>, making sure what the customer wants, and what I understand and intend to implement, are the same things.</p><p>I’ve become quite a good engineer — customer middleman, I’ve became a <strong>domain expert</strong>.</p><p>It’s also become clear that <strong>I have a mentor that is helping me</strong>, and has a significant impact on my level. He was one of the veteran engineers in the organization, we had a click in one of the projects I was working on. Looking back — the impact on his mentorship (and of others) was a differential factor in leveling up.</p><p>This year had one of the pivotal moments of my career — <strong>Learning to unblock myself at any cost.</strong></p><p>I was waiting for something from a colleague that was preoccupied by more urgent tasks.</p><p>I decided that even though this is not my discipline — there’s no reason I can’t do some basic electrical circuits. We had a short sync, I took a tool box, built a simple resistor voltage divider, and unblocked myself.</p><p>It will later be a major theme in how I operate — <strong>full ownership</strong>, learning even outside my discipline, and pushing features even when they seem to be blocked.</p><p>🏁 If I had to sum it up to one achievement — I was able to <strong>design and integrate</strong> mid-large software systems<br>⏱️most time still spent on <strong>integration and design<br>⬆️</strong>progress made throughout this year: <strong>Large</strong></p><h3>Year 4 — Dunning Kruger, Doubling down on learning</h3><p>In year 4 I’ve hit hard in the face by the <a href="https://en.wikipedia.org/wiki/Dunning%E2%80%93Kruger_effect">Dunning Kruger effect</a> and I’ve acknowledged the hard truth: <br><em>I am thinking I am a much better developer than I really am</em>.</p><p>My big red flag for it, was that I moved to a new project, and when I began working on the new project, I was “feeling” the codebase level isn’t great, <strong>but I didn’t have the means to express why, not did I have a clear path on how to fix it</strong>. <br>It was just a hunch, I was lacking domain language and expertise. I started leveling up!</p><p>I read <a href="https://www.amazon.com/Clean-Code-Handbook-Software-Craftsmanship/dp/0132350882">clean code</a>, I started going to meetups and conferences, and most importantly — I <strong>started nudging and taking more face time with our tech leads</strong> — asking them how they would address the problem I am trying to solve. <br><strong>I can’t stress this enough — quality time with senior engineers is a huge stepping stone</strong>.</p><p>My tasks went from <em>“just get it done”</em> to <em>“learn all you need to get it done </em><strong><em>right</em></strong><em>”</em>.</p><p>🏁 If I had to sum it up to one achievement — <strong>exponential grown in the art of coding<br></strong>⏱️most time back being spent on <strong>coding</strong>, now emphasizing on the art of code.<br><strong>⬆️</strong>progress made throughout this year: <strong>exponential</strong></p><h3>Year 5 — Leading, Learn to “glue”, The art of coordination</h3><p>I am getting things done, I’m proficient in a programming language (at the time it was C#), It’s time for some management skills.</p><p>I got my first team and project to lead.</p><p>We were 3 engineers, working on the project. It wasn’t the most challenging one, but it was ok.</p><p>I learned to motivate people enough, learned to delegate (which is a key skill of a good manager or leader), I learned to put checkpoints and milestones without micromanaging.</p><p>I was required to coordinate multiple parties, hold a lot of context in my head, and make sure nothing falls through the cracks.</p><p>Eventually I’ve really sharpened one of my (at the time) weakness — <em>coordination and tracking. </em>I now consider this to be one of my strong suits. <br>It became clear I was <a href="https://www.youtube.com/watch?v=KClAPipnKqw">the glue</a> for the projects I worked on, even those I am not managing.</p><p>I look back 7 years since, and it’s clear to me that communication, coordination and project management are important skills that no one thinks of when thinking of the “engineer’s path” — but it is one of the major roles of a senior engineer.</p><p>🏁 If I had to sum it up to one achievement — <strong>letting go, and delegating<br></strong>⏱️most time spent in <strong>coordination</strong>, ticket management and talking to colleagues.<br><strong>⬆️</strong>progress made throughout this year: <strong>large</strong></p><h3>Year 6 — Having Fun</h3><p>Although it wasn’t the first time I enjoyed engineering (I enjoy it most of the time), in year 6 the stars aligned, and one of my best friends joined my engineering org, and we were staffed to work together on a project. It was crazy fun.</p><p>On the more professional ladder vertical — it was quite similar to year 5. Not much growth happened. Which led me to pivot and switch a workplace.</p><p>⏱️most time spent in <strong>coordination </strong>and<strong> documentation<br>⬆️</strong>progress made throughout this year: <strong>minor</strong></p><h3>Year 7 — SaaS, Cloud and Scale</h3><p>I was eager to learn how Facebook and Google handle their massive scale, so I went to work for a hyper growth SaaS startup.</p><p>I switched technology stack, and was diving deep into cloud (AWS), infrastructure, Devops and how scale works. To be more precise, I was doubling down on how scalable systems are being built (I started from reading <a href="https://www.amazon.com/Martin-Kleppmann/e/B00Q43XKD6/ref=dp_byline_cont_book_1">Martin Kleppmann</a>’s <em>Designing Data-Intensive Applications</em> which I highly recommend).</p><p>It was my first encounter with Microservices with strict SLAs and multi-instance systems, which drove me to learn patterns of building resilient systems (some of which I blogged about in the <a href="https://medium.com/riskified-technology/engineering-for-failure-f73bc8bc2e87">engineering for failure</a> post).</p><p>Again, I had a colleague who informally became a mentor, and had a major impact on my professional growth.</p><p>🏁 If I had to sum it up to one achievement — <strong>learning patterns of resilient systems and devops<br></strong>⏱️most time spent in <strong>coding<br>⬆️</strong>progress made throughout this year: <strong>medium</strong></p><h3>Year 8 — Observability, Observability, Observability + Devops and SRE</h3><p>It was my first time doing OnCall, and responding to real time production incidents.</p><p>I got intrigued, then inspired, and finally fell head over heels for Devops and SRE.</p><p><a href="https://betterprogramming.pub/the-path-of-getting-comfortable-in-production-c88456146f00">I started to dig into the infrastructure stack</a>, and how our systems were operating in production, anything from CI/CD pipelines to the configurations and description objects of our production deployments.</p><p>From there the path to observability was clear, and I got hooked on monitoring, building dashboards, alerts, tuning production while trying to balance production stability and the oncall fatigue.</p><p><strong>It became clear that being a developer that “speaks” and understands infrastructure is a super power, and a differentiating factor. <br></strong>Do yourselves a favor and read <a href="https://sre.google/sre-book/table-of-contents/">Google’s SRE book</a>.</p><p>🏁 If I had to sum it up to one achievement — <strong>gained real expertise in monitoring and observability<br></strong>⏱️most time spent in <strong>observability platforms and Kubernetes YAMLs<br>⬆️</strong>progress made throughout this year: <strong>Exponential</strong></p><h3>Year 9 — Business metrics and Cost efficiency</h3><p>I was now quite experienced with building systems, even complex ones, I was never required though to be on budget.</p><p>My projects were either:</p><ul><li>expensive by design (to support the phase of hyper growth startup)</li><li>or cheap due to small scale and usage.</li></ul><p>I was never required until now to think of the cost of the software, or its unit economics. Year 9 was all about building the same systems, but cheaper, taking cost into design considerations.</p><p>This was the year of large cost optimization projects (you can read more <a href="https://medium.com/riskified-technology/over-pay-as-you-go-for-your-datastore-11a29ae49a8b">this</a> and <a href="https://medium.com/riskified-technology/500mb-of-memory-saved-us-60-in-our-dynamodb-bill-abd5424137f9">this one</a>).</p><p>We were talking about operational cost in design reviews now. It was much more challenging building <strong>the same reliable systems</strong> but only under budget.<br>At one point we scratched the “optimal solution” for a good enough, 10x cheaper solution. <strong>Engineering is all about tradeoffs</strong>.</p><p>It wasn’t only about money, I started seeing, and asking more and more about the business incentives, goals, and KPIs.</p><p>I became super business centric.</p><p>It became a key part of my day to day, to the point that I was able to convince to take on a project (or reject it) by simply by asking <em>“how will this project increase our KPI of focus”</em>. From here the path was short to start taking more interest in the work of product managers, and I did some leveling up on that front too (I recommend you read <a href="https://www.amazon.com/INSPIRED-Create-Tech-Products-Customers-ebook/dp/B077NRB36N">inspired</a> very much. Yes, even if you are a developer).</p><p>🏁 If I had to sum it up to one achievement — <strong>Better focus and understanding of business goals and product metrics (KPIs)<br></strong>⏱️most time spent in <strong>observability platforms<br>⬆️</strong>progress made throughout this year: <strong>medium</strong></p><h3>Year 10 — Year of initiative</h3><p>It became clear to me that I don’t want to pursue a career in management, nor do I want to code all day long. It’s the mixture of the two that I enjoy most, plus solving complex problems and driving difficult changes throughout the organization.</p><p>It set me on a Senior IC career path, taking initiative in my engineering organization, looking for hard challenges to solve, and trying to help level up the people around me.</p><p>As I had mentors along the way — I became one for others.</p><p><strong>I really believe that leveling up, and creating other senior engineers around you is one of the fundamental requirements of a senior engineer</strong>, and making others better is part of my way to make an <a href="https://dariusforoux.com/prices-law/">asymmetrical impact</a>.</p><p>It was also the year of achieving a personal goal of mine of giving my <a href="https://youtu.be/HqrI1zLYOvo?si=3bVotrPyzejlW0aN">first talk</a> at a tech conference 🤘</p><p>🏁 If I had to sum it up to one achievement — <strong>mentoring other engineers<br></strong>⏱️most time spent in <strong>collaboration<br>⬆️</strong>progress made throughout this year: <strong>medium</strong></p><h3>Year 10+</h3><p>This will wait for the 20 year retro, but generally I’ve set myself on the Senior IC path, working hard on technical leadership.</p><p>I’ll update in 7 years how that went ;)</p><h3>Timeless truths</h3><p>Looking back on the path taken, I think I can point out few truths:</p><ol><li>No one is responsible for <strong>your</strong> career path but you. <br>Your managers might care enough to do some of that for you, but it’s your responsibility to drive it forward.</li><li>Having a mentor is a force multiplier! <br>It’s literally a means to learn faster, it’s a shortcut! It’s a way to see first-hand how more experienced engineers think and operate. <br><strong>Find yourself a mentor</strong>.</li><li>No employer will ever think less of anyone who takes initiative and wants to do more. <br>It’s totally within your reach to be that person.</li><li>Learning through daily tasks is not enough for becoming a top tier engineer (and I’m not saying I am one). <br>The craft and technology it’s just too complex and requires a lot of passion and time. At least this is my observation.</li><li><strong>Nothing </strong>— and I mean it — <strong>Nothing</strong>! is more important than being a decent human being, a pleasant colleague, and a pragmatic engineer. <br>Nothing is more important than making your colleagues feel comfortable and safe working with you. This becomes even more important as you hold more senior positions.</li></ol><h3>Final thoughts</h3><p>I think this is the most personal post thus far.</p><p>It was fun looking back, retrospecting, and trying to distill a whole year worth of work to a single achievement, or main occupation. It made me think where I could do better, and where I really did my best.</p><p>If you’re a senior engineer — I’d be thrilled to hear your thoughts on my path, or hear your path.</p><p>If you’re a junior — mid, start thinking about your career path, ask the seniors around you how they’ve gotten to where they are, take control over your career path.</p><p>As always, thoughts and comments are welcome on twitter at <a href="https://twitter.com/cherkaskyb">@cherkaskyb</a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=2d0cd7b3d836" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Creeping Bytes, Costing Dollars — Creeping Cloud Costs Explained]]></title>
            <link>https://itnext.io/creeping-bytes-costing-dollars-creeping-cloud-costs-explained-a0a9ba0ab8f6?source=rss-665b8f6a764a------2</link>
            <guid isPermaLink="false">https://medium.com/p/a0a9ba0ab8f6</guid>
            <category><![CDATA[finops]]></category>
            <category><![CDATA[devops]]></category>
            <category><![CDATA[cloud-computing]]></category>
            <category><![CDATA[cloud-cost-management]]></category>
            <category><![CDATA[software-engineering]]></category>
            <dc:creator><![CDATA[Boris Cherkasky]]></dc:creator>
            <pubDate>Wed, 17 Apr 2024 10:11:14 GMT</pubDate>
            <atom:updated>2024-04-20T08:12:45.728Z</atom:updated>
            <content:encoded><![CDATA[<h3>Creeping Bytes, Costing Dollars — Creeping Cloud Costs Explained</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*-9H21wNT93wTfKeCRKhiig.png" /><figcaption>artwork by <a href="https://unsplash.com/@equeen">@equeen</a> on <a href="https://unsplash.com/@equeen">unsplash.com</a></figcaption></figure><p>Managing cost efficiency in a modern production cloud environment is a challenging task, especially when you’re only starting your development effort — everything is cheap on a small scale — the 3$ / Day in storage costs seems meaningless — It’s only 3 dollars!</p><p>This was our case too, but as our operation grew I’ve started noticing a pattern in those costs that were “just 3$” only 5 months ago, we’ve nicknamed this “creeping cloud costs” — It’s when your costs are slowly but surely rise, with no seasonality, and little correlation to additional business.</p><p>This article will cover which cloud services are prone to those creeping costs, how to identify them, and lastly how to mitigate them.</p><p>The chart used in this article are from the <a href="https://www.finout.io/">Finout</a> platform (full disclosure — I am one of the engineers building it).</p><h3>What Are “Creepers”?</h3><p>“Creepers” are usually monotonously increasing costs, with no seasonality (seasonality may affect the rate of increase, but it’ll be there regardless), and very little decreases, and generally unrelated to business growth.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*LxK-Tf_Q7fqlm3BS" /></figure><p>Good examples will be Container Repositories (such as Amazon ECR or Google Cloud Registry) — Regardless of application traffic, or seasonality, each GB of stored images will be billed monthly.</p><p>The rate at which developers create images is usually unrelated to business (and probably correlates more to the size of the team)</p><p>The above snapshot is our ECR cost which triggered a low-hanging cost optimization effort and this article.</p><h3>Identifying “Creepers”</h3><p>It’s relatively easy to identify Creepers:</p><ul><li>They have no seasonality</li><li>They are monotonically increasing over the course of a long period of time</li></ul><p>Just look for line resources with cost trending upwards on a constant basis.</p><h3>Cloud Services That Are Known To Have “Creeping” Potential</h3><p><strong>Pay-per-use storage — based services</strong> are usually the immediate offenders in creeping costs cases — each byte we store today, will be billed monthly as long as it exists.</p><p>In AWS for example the immediate suspect is S3, but there are many other services that have a pay-per-use storage component such as DynamoDB, ECR, <a href="https://docs.aws.amazon.com/managedservices/latest/userguide/log-customize-retention.html">CloudWatch logs</a> and others.</p><p>This is obviously not limited to AWS and happens in all other cloud providers.</p><h3>How’s That Different From Any Other Cloud Resource?</h3><p>Unlike other services, the storage component of pay-per-use services is <strong>accumulative</strong>.</p><p>Your RDS has a ~<em>fixed price</em> (according to the size of the instance):</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*fVI6yRVHaDDZdwZo" /></figure><p>Your lambda / ALB transfer cost is <em>correlated to traffic / input</em>:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*LjpG1qCIvy3d5ONS" /></figure><p>S3 storage on the other hand — you’ll pay monthly, recurringly, for each byte stored for as long as it exists! This cost will accumulate over time, won’t be seasonal, and won’t trend downwards without deleting objects explicitly:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*wxJt0cybRHxZVzVk" /></figure><h3>Handling creeping costs</h3><p>Obviously the mitigation of these creepers is to set TTL / Expiration / Life Cycle policy (LCP) (depending on the cloud service).</p><p>It’s important to note, that most cloud services have no TTL / Expiration / Life Cycle policy by default, therefore <strong>each newly created resource is a </strong>potential<strong> “creeper” in the making</strong> unless proactively treated!</p><h3>Handling Creepers and Cost to Business Correlation</h3><p>Generally after applying a Life Cycle Policy on the relevant service we should achieve a “<strong>steady state</strong>” where the overall storage cost is <strong>relatively constant</strong>. Moreover, after applying LCP <strong>the cost becomes correlated to the business</strong>, this is because an increase in business (traffic / volume) will cause the “steady state” storage to trend upwards in a proportional manner until achieving a new steady state.</p><p><strong>This is every Finops’s wet dream — correlation of costs to business growth!</strong></p><p>As a real life example, after applying LCP on our ECR we were able to achieve a cost reduction + steady state:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*rV8TsDXK7WcNa9MK" /></figure><h3>Stop The Rat Race — Use Automation and Safeguards</h3><p>I used to bucket those resources into 3 groups, and tag them accordingly:</p><ul><li><strong><em>Lifecycled resources</em></strong> — resources with LCP / TTL applied to them. <br>They are bounded, and you can probably forget about them. You are relatively safe here.</li><li><strong><em>Unbounded resources</em></strong> — those that we don’t have and can’t have a LCP for. Resources such as your main s3 storage or your main operational DynamoDB table for example.<br><strong>Those are expected to be very expensive, and are with high correlation to the business</strong>. You just need to keep an eye on them so they don’t get too out of hand.</li><li><strong><em>Cheap unbounded resources</em></strong> — those are unbounded resources, but you don’t expect them to be expensive, therefore tagging them, and adding an alert that they don’t cross some boundary seems enough here.<br>Example for such a resource — an ECR repository where you need the full history of revisions and you can’t trim them.</li></ul><p>By tagging resources into the above bucket, alerts can be set that things are kept within expected cost ranges.</p><p>As you can probably imagine, monitoring or knowing about each and every cloud resource that is created is just infeasible without proper automation and tooling, therefore we can leverage policy agents such as <a href="https://www.openpolicyagent.org/">OPA</a> or <a href="https://kyverno.io/">Kyverno</a> to deny the introduction of resources without LCP / TTL (or on the other hand explicitly tagging them as unbounded resources).</p><h3>Closing thoughts</h3><p>Each resource created should have some kind of Life cycle policy attached to it, and not only due to its implicit costs — each stored byte requires governance, it has the risk of including PII, and being relevant to regulations such as GDPR and CCPA.</p><p>We should aim for a limited lifespan of each byte we process, and if that’s not possible — that byte should be of great importance to the business.</p><p>As always, thoughts and comments are welcome on Twitter at <a href="https://twitter.com/cherkaskyb">@cherkaskyb</a>.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=a0a9ba0ab8f6" width="1" height="1" alt=""><hr><p><a href="https://itnext.io/creeping-bytes-costing-dollars-creeping-cloud-costs-explained-a0a9ba0ab8f6">Creeping Bytes, Costing Dollars — Creeping Cloud Costs Explained</a> was originally published in <a href="https://itnext.io/">ITNEXT</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[The Worst Test Suite-Testing Anti Patterns Experienced In Real Life]]></title>
            <link>https://itnext.io/the-worst-test-suite-testing-anti-patterns-experienced-in-real-life-24fd13ee3ddd?source=rss-665b8f6a764a------2</link>
            <guid isPermaLink="false">https://medium.com/p/24fd13ee3ddd</guid>
            <category><![CDATA[programming]]></category>
            <category><![CDATA[software-engineering]]></category>
            <category><![CDATA[software-testing]]></category>
            <category><![CDATA[software-development]]></category>
            <category><![CDATA[testing]]></category>
            <dc:creator><![CDATA[Boris Cherkasky]]></dc:creator>
            <pubDate>Mon, 04 Dec 2023 08:34:45 GMT</pubDate>
            <atom:updated>2023-12-14T22:28:45.018Z</atom:updated>
            <content:encoded><![CDATA[<h3>The Worst Test Suite—Testing Anti Patterns Experienced In Real Life</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*Glf_ceZ1EV7dJsM1" /><figcaption>artwork by <a href="https://unsplash.com/@antoinegiret">@antoinegiret</a> on <a href="https://unsplash.com/@equeen">unsplash.com</a></figcaption></figure><p>I don’t think there’s an introduction needed here, I am going to tell you how the worst test suite I encountered looked like, and how it was working around it.</p><p>I warn you, this article will make your stomach turn, your skin scratch, and your engineering skills cry.</p><p>The Antipatterns I’ll cover will be ranked with 🤮s. <br>From puke score of 🤮 Being bad, to puke score of 🤮🤮🤮 being completely unforgivable. <br>All of the antipatterns are terrible, but some of them I have never seen elsewhere, and I thought ranking them will emphasize how obscured they were.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/640/0*7lNs5U-S2IUrikuM" /><figcaption>you, while reading this article</figcaption></figure><h3>The Suite</h3><p>The whole suite was about 10 test files big. 9 of them were “reasonable”, the tenth was a nightmare: It had roughly 3000 lines of code, that combined a total of ~80 test cases. About 95% of the test suite was this one file.</p><p>In addition to the tests, All the utilities the test uses were scattered around this file.</p><h4>Mixing Test Types</h4><p>Puke score: 🤮</p><p>The file included many tests of different types — from unit, to component, to e2e, and there were tests for which it was extremely difficult to figure out what the test type — there were unit tests who tested a specific small functionality by running the major part of the system, while in other cases there were major end-to-end tests that were executed by calling a sequence of internal capabilities of the system</p><h4>Why Are The Tests Numbered?</h4><p>Puke score: 🤮🤮🤮</p><p>All test names were numbered! All tests were of the form TestXX_what-the-test-checks</p><p>It was a major red flag that I didn’t fully grasp its magnitude when I onboarded the project. After about two months of work it became apparent that the tests were numbered because they were failing in any other execution order! The numbers were there for force ordering. <strong>The tests were co-dependant</strong>.</p><p>It goes without saying that I’ve learned this the hard way when I moved some tests around to make that huge file a bit cleaner.</p><p>If I wanted to add a test, <strong>it was critical that it be ordered correctly with other tests</strong>. It created an outrageous situation where I’d need to name the test correctly for it to run in the correct order. So if I want a test to run between Test34_XXX and Test34_YYY I’d had to name my test Test341_ZZZ for the lexicographical order to hold 🤯</p><h4>Anonymous Tests</h4><p>Puke score: 🤮</p><p>One more thing about test names — some of them were anonymous — tests that didn’t say anything about what they actually cover, tests like: test19_requirement_59_passesor the all time favorite <em>test87_process_works</em>.</p><p>There were tests I learned what they test only when I have introduced changes that made them fail, which forced me to do investigative work to figure out what they do.</p><h4>Assertions? They are mere recommendations</h4><p>Puke score: 🤮🤮🤮</p><p>Some tests didn’t end with an <strong>assertion</strong>. You are rightfully asking what a test without an assertion is doing?</p><p>In those tests there was a comment at the top of the test that was instructing the “user” to go and do something. Usually it was something like “go to the log file and check that there’s a log message of the format X — Y — Z”.</p><p>It goes without saying that this didn’t state where the log file was, and what to do if there’s more than one (due to log rotation configuration). Also — in some cases the instructions were obsolete, and the log message changed since the “test” and its instructions were written.</p><p>Those tests always passed obviously, no one told me about those during project handoff, and I discovered those when I’ve added a functionality, and by sheer chance noticed there’s a test that’s name says it tests the obsolete opposite of what I’ve implemented. It obviously passed (since it had no assertion).</p><p>I’ve deleted all those tests, and never looked back.</p><h4>Complex and Obscure Input</h4><p>Puke score: 🤮</p><p>The system’s test input was quite complex, and most tests were based on a single input file. It was quite obscure what it included other than “just enough input for the tests to pass”. No one really remembered how this test file was created.</p><p>To get each test to the relevant state, scattered around the 3000 lines of code, there were utility methods that manipulated the input file. None of them explained what they did, and they were usually called something like <em>prepare_for_test78</em>.</p><p>Anytime we needed to change the input we cried a bit 🥲</p><h4>Dragged Shared State</h4><p>Puke score: 🤮🤮</p><p>Not only were the tests co-dependant, the system under test itself was dragging some internal state in between tests.</p><p>Instead of recreating the system before each test, the original authors added a set of methods that enabled the test driver to nullify the internal state.</p><h4>Unclear Entry Point</h4><p>Puke score: 🤮🤮</p><p>There were few tests where internal functions were called in a specific order — It was literally unclear what the entry point of the test was. It turned out that all those called functions that were scattered across different classes affected some internal shared state which was eventually tested for its expected final state.</p><h3>Final Thoughts</h3><p>Yes, this is a true story.<br>Yes, it’s a bit exaggerated, but only a bit.</p><p>I don’t know how this test suite got to that rotten state that was handed off to me, <strong>but I assume only good intentions</strong>.</p><p>The project was in POC stage for a long time, and wasn’t in anyone’s focus or priority until a late stage, that can explain it, but I don’t know.</p><p>Luckily enough — the system under test was a relatively simple one. Other than the complex input, the logic itself wasn’t too difficult to understand and reason about. This made it possible to work under these conditions.<br>Had the system been a bit more complex — it would have been probably even worse.<br>That being said, maybe the fact that the system was simple enough is what enabled the test suite to rot that far?</p><p>We did our best to make things better, but we mostly tried not to do worse. Anything new developed was held to higher standards, and we decided to add as little as possible to the “one file of doom”.</p><p>This experience was an amazing school for me though — I’ve learned what not to do, and what’s the impact of doing it. It’s like getting a really bad case of Pneumonia to learn you shouldn’t run outside in a T-shirt during a freezing winter.</p><p>This post is based on a <a href="https://twitter.com/cherkaskyb/status/1679419348417380354">Twitter thread</a> (published in Hebrew) that was written in the spur of the moment when I remembered this traumatic experience.</p><p>As always, thoughts and comments are welcome on Twitter at <a href="https://twitter.com/cherkaskyb">@cherkaskyb</a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=24fd13ee3ddd" width="1" height="1" alt=""><hr><p><a href="https://itnext.io/the-worst-test-suite-testing-anti-patterns-experienced-in-real-life-24fd13ee3ddd">The Worst Test Suite-Testing Anti Patterns Experienced In Real Life</a> was originally published in <a href="https://itnext.io/">ITNEXT</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[I Don’t Alert on Apdex. It Confuses Me]]></title>
            <link>https://cherkaskyb.medium.com/i-dont-alert-on-apdex-it-confuses-me-5e639242e5db?source=rss-665b8f6a764a------2</link>
            <guid isPermaLink="false">https://medium.com/p/5e639242e5db</guid>
            <category><![CDATA[software]]></category>
            <category><![CDATA[observability]]></category>
            <category><![CDATA[software-engineering]]></category>
            <category><![CDATA[devops]]></category>
            <category><![CDATA[programming]]></category>
            <dc:creator><![CDATA[Boris Cherkasky]]></dc:creator>
            <pubDate>Mon, 14 Aug 2023 06:42:31 GMT</pubDate>
            <atom:updated>2023-08-14T06:42:31.631Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*I0r_GO3PrpAL92it" /><figcaption>artwork by <a href="https://unsplash.com/@redaquamedia">@redaquamedia</a> on <a href="https://unsplash.com/@equeen">unsplash.com</a></figcaption></figure><p>There are many ways to build your observability and alerting posture for your production systems. It doesn’t matter if you build it yourself, use open source tools, or leverage one of the many observability vendors out there, you’ll have to make decisions — what to monitor, what to alert on, and how.</p><p>It’s common to use out of the box mechanisms offered by vendors, or open standards. One of those standards is the <a href="https://en.wikipedia.org/wiki/Apdex">Apdex</a> which is used to monitor <strong>application performance</strong>.</p><p>This is my <em>personal take</em> on something that is considered standard that <em>I just don’t understand.</em> So here we go — the Apdex, what it is, and why I don’t use it!</p><h3>What is an Apdex</h3><p>The <a href="https://en.wikipedia.org/wiki/Apdex">Apdex</a> is a single number that <em>tries</em> to represent the user’s “satisfaction” with the system.</p><p>It is a standard for measuring performance. It looks at systems from the end user’s perspective, bucketing the requests served by a system into three buckets — Satisfied / Tolerated / Frustrated, weighing them all into a single numerical value called “the Apdex score”.</p><p>It does that by defining a threshold called the “Tvalue”, which is the maximal latency that is considered “good”, and the bucketing works as follows:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*YlvkV0P9ApaugQO1" /></figure><p>It is also implicitly inferred that failed requests are considered Frustrated (regardless of latency).</p><p>Sounds quite good right? <strong>I don’t like it</strong>.<br>The rest of this post I’ll do my best to explain why.</p><h3>1. No one knows what Apdex is</h3><p>First and foremost, <strong>No one knows what the definition of Apdex is</strong>.</p><p>I literally learned about it at 3am while the Apdex score went below its alert threshold. If observability in general is a domain that is hard to master, then the Apdex is usually very far down the learning syllabus.</p><p>Furthermore, even when you know its definition — <strong>The Apdex score is not tangible</strong>! What does a score of 0.87 mean?! Can you imagine it? Can you plot a chart of system health and Apdex — what will be the correlation between them? <strong>Given a Score can you draft a mental model of your system’s state</strong>?</p><p>It’s an abstraction, or a summary that simplifies nothing, it just adds confusion!</p><p>As <a href="https://twitter.com/nukemberg">@nukemberg</a> said when we discussed it: “people don’t have intuition for it, there’s no concrete example, there’s no simulator”.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*JpKVqIyN3K1Bp7gN" /><figcaption>Apdex score as shown in the <a href="https://newrelic.com/blog/how-to-relic/new-relic-apm-troubleshoot-performance">New Relic APM view</a></figcaption></figure><h3>2. It’s hard to set meaningful alert threshold</h3><p>Since the numeric value of the score is hard to imagine and understand, how exactly will we set the alert threshold? Will it be a score under 0.9? 0.8? I find the alert threshold to be an <strong>arbitrary value</strong>! (Which we all know is the absolute best way to get your team into alert fatigue).</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/500/0*-E8kqa-Ad8VuSi3L" /></figure><h3>3. It “feels” arbitrary</h3><p>It tries to simplify the complexity of understanding latency distribution. It takes a distribution, which isn’t a trivial concept to begin with, and breaks it down into 3 over-simplified-bucket <strong>which always feel arbitrary to me</strong> (Why is 4*T is frustrated? Why not 1.5 or 8?).</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*0Wxvn63lUejQC4Ci" /><figcaption>Simplified bucketing</figcaption></figure><h3>4. It’s a “default” not an optimum</h3><p>It feels like a mechanism to add simple, default visibility into complex systems — it treats all requests the same, all errors the same, while reality is far more complex than that.</p><p>Moreover — different use cases or customers might have different T values. I.e. different definitions of what are satisfied and Frustrated requests. For example, satisfaction for your freemium users can be 1 second response time, while for your enterprise users it can be 400ms.</p><p><strong>I consider it to be an attempt to create “one suite” that actually fits none</strong></p><h3>5. It has multiple dimensions</h3><p>Lastly, and here it’s <strong>completely my personal preference</strong>, It has multiple dimensions — latency and error rate. They are summed into a single number so getting alerted on a low Apdex score doesn’t say what exactly the issue is — Is it Latency? or errors?</p><p>I prefer alerts to be with a single dimension — this way it’s very clear what’s the source of the alert.</p><h3>If not Apdex, What should we alert on?</h3><p>Appdex has 2 dimensions — Errors and Latency, so I just explicitly alert on those…it’s much simpler to grasp and understand — the alert discloses what’s going on.</p><p>It is possible to create an effective SLO that combines multiple dimensions — but this is done by defining consistent thresholds and not ambiguous scores. For example — % of requests with latency above 500 ms or failure rate. This is similar to leveraging the Satisfied or Frustrated Apdex buckets only.</p><h3>Final Thoughts</h3><p>In user facing applications the Apdex score can have some value as a type of <em>“hunch engineering”</em> — getting a general <em>“feeling”</em> of the system’s performance.</p><p>In service to service applications (i.e. systems that aren’t interacting directly with a human user) I really don’t see any point in using Apdex — there’s no notion of “satisfaction” — the SLA is either met or not. No middle grounds.</p><p>As with my personal experience as a <em>not-user-facing engineer</em> — every “low Apdex score” alert I’ve gotten, confused me! It was a struggle to understand how the system behaves, and what exactly are the symptoms.</p><p>Some credits:<br><a href="https://twitter.com/nukemberg">@nukemberg</a> who i’ve discussed the ideas behind this post and helped sharpen them. <br><a href="https://twitter.com/Eladleev">@Eladleev</a> who’s “<a href="https://leevs.dev/the-profound-problem-with-terraform/">the profound problem with terraform</a>” post inspired me to put in writing what I think of Apdex (which was sitting in the drawer for more than a year).</p><p>As always, thoughts and comments are welcome on Twitter at <a href="https://twitter.com/cherkaskyb">@cherkaskyb</a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=5e639242e5db" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Data Puppy — Shrinking Data Dog Costs | Part III]]></title>
            <link>https://cherkaskyb.medium.com/data-puppy-shrinking-data-dog-costs-part-iii-4701a7d70ff6?source=rss-665b8f6a764a------2</link>
            <guid isPermaLink="false">https://medium.com/p/4701a7d70ff6</guid>
            <category><![CDATA[software-engineering]]></category>
            <category><![CDATA[devops]]></category>
            <category><![CDATA[finops]]></category>
            <category><![CDATA[programming]]></category>
            <category><![CDATA[software-development]]></category>
            <dc:creator><![CDATA[Boris Cherkasky]]></dc:creator>
            <pubDate>Tue, 25 Jul 2023 06:37:04 GMT</pubDate>
            <atom:updated>2023-07-25T06:41:33.492Z</atom:updated>
            <content:encoded><![CDATA[<h3>Data Puppy — Shrinking Data Dog Costs | Part III</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*d1ULogRNz8_fQXCn" /></figure><p>The Datadog cost series:</p><ul><li><a href="https://cherkaskyb.medium.com/the-costly-elephant-in-the-room-is-a-data-dog-part-i-d690ad3cbee4">Part I — The costly elephant In the room is a (Data)dog</a></li><li><a href="https://cherkaskyb.medium.com/the-magic-that-is-datadog-pricing-part-ii-5fa56cd8476e">Part II — The magic that is Datadog pricing</a></li><li><a href="https://cherkaskyb.medium.com/data-puppy-shrinking-data-dog-costs-part-iii-4701a7d70ff6">Part III — Data Puppy, shrinking Datadog costs</a></li></ul><p>Welcome back to the series on hacking the Datadog pricing model. In the <a href="https://cherkaskyb.medium.com/the-magic-that-is-datadog-pricing-part-ii-5fa56cd8476e">previous post</a> we explored the Datadog pricing model. In this part, we’ll discover the key factors to consider and unlock the hidden potential for optimization.</p><p>Without further adieu, let’s dive into what can be done.</p><h3>Use Committed Prices</h3><p>You already decided Datadog is your observability platform of choice (which is a good decision), so you need to be able to manage spend and commitments with your Datadog account manager</p><p>The easiest saving opportunity is to analyze your usage, and adjust your commitments or base fee accordingly — this can help you save anywhere between 10% to 30%, without any engineering involvement.</p><p>Once you set your commitment levels, You have to have the tools to monitor it on a daily basis, alert on anomalies, and proactively identify any threat to those commitment levels.</p><p>Build a process of cost government — when you understand their pricing model and how your production system behaves — it is achievable even in products as complex as DataDog.</p><p>This process is an ongoing effort and it needs to be repeated monthly / annually (depending on your plan).</p><h3>Transition from End of Month “Bill Shock” to Push-Based Daily Cost Monitoring</h3><p>One of the main pain points using the Datadog platform is not that it’s expensive, it’s the looming bill shock. The billing date might blind side you with unexpected usage.</p><p>That being said, the earlier you catch anomalies and cost spikes, the earlier you can address their root cause, and the more predictable your end of the month payment will be.</p><p>There are many reasons for cost spikes in the wild:</p><ul><li>A developer enabling debug logs in production by mistake.</li><li>Unknowingly adding a high cardinality tag to a custom metric.</li><li>Spinning up a testing environment and forgetting it.</li><li>A new feature is released to production that you didn’t foresee it’s usage increase</li></ul><p>So keep an eye on your usage, and preferably set alerts! Within the „atadog platform this can be achieved by setting Monitors on the <a href="https://docs.datadoghq.com/account_management/billing/usage_metrics/">estimated usage metrics</a> (note that those are usage and not cost metrics), or alternatively leveraging an anomaly engine such as <a href="https://www.finout.io/solution-anomaly-detection">Finout</a> (full disclosure — I work @ Finout).</p><h3>Unit Economics of Your Observability Platform</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/656/0*0lwjwts7EErM-tKg" /></figure><p>Datadog is designed to scale with your business and infrastructure and it does that amazingly well, just keep an eye on the unit economics of the observed system.</p><p>In practice, make sure your observability spend is within a “reasonable” ratio to the infrastructure spend. Anything between 2%-6% of your infra cost is probably “healthy”.</p><p><strong>Monitoring the trend</strong> here is key, don’t get caught with your devs adopting Datadog without an ability to verify that the increase makes business sense.</p><h3>Reduce Waste and Idle Monitoring</h3><p>This is the fun part — eliminate waste.</p><h4>Logs</h4><p>Everything in my <a href="https://itnext.io/log-warn-this-log-will-cost-you-on-log-cost-optimization-5c6fd8b234e9">post on logging optimization</a> is also applicable to Datadog logging products too. Make sure your debug logs are turned off at any scalable system, and only log what you need, and with the correct retention.</p><p>Datadog offers a cool platform of “logs as metrics”. This can be leveraged to reduce the volume of <strong>indexed</strong> logs, while still keeping some of their visibility as metrics (Note that you’ll still pay for the log ingestion)</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*90VT5yPCXqG0EMiS" /><figcaption><em>Debug log costs by service from the Finout Platform</em></figcaption></figure><h4>No, you don’t need all those Custom Metrics!</h4><p>We love metrics, and especially custom metrics. A code review ending with “well you are sending too many metrics” is rare.</p><p>We are educated to feel that “ you could never have enough monitoring”, and that we need each and one of these metrics in case a production incident happens and this specific metric will save my Friday night.<br>Three months later we are left with tons of idle metrics that are being used only by the Datadog billing systems, so clean up your infrastructure and remove any unused custom metrics.</p><p>I’ve played around with <a href="https://gist.github.com/cherkaskyb/576f492582c82b6ca976e46c9fe23d2e">this python script</a> to identify unused custom metrics that do not<em> exist in any Dashboard or Monitor</em>, I encourage you to take it out for a test.</p><p>Just Delete them. You can let go of them.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/570/0*jQSLp3D3bZKmfnsJ" /></figure><h4>Metrics cardinality — a blessing in disguise</h4><p>You’ve deleted all your old, stale, unused custom metrics. You are left with only what is necessary to keep production alive, but what about the tags cardinality? Make sure the cardinality of sent tags is reasonable! Custom metrics product is <a href="https://docs.datadoghq.com/account_management/billing/custom_metrics/?tab=countrate#counting-custom-metrics">billed</a> by the metric type used (Gauge / Histogram / Counter) and the cardinality of tags sent.</p><p>A good starting point will be leveraging the <a href="https://docs.datadoghq.com/api/latest/metrics/#list-active-tags-and-aggregations">list-active-tags-and-aggregations</a> API, to get a general understanding of your stance.</p><p>Finally, use the <strong>right</strong> observability primitives for each metric — Histograms and Distributions are more expensive than Counters and Gauges, who use them only when you really going to use their added value.</p><h4>Synthetic Tests</h4><p>Make sure that your Synthetic tests are actually used and alerted on by at least one <strong>active</strong> Monitor! Silenced Monitors tends to be forgotten while the Synthetic test is still running and being billed. In addition — verify that it is scheduled at reasonable intervals.</p><p>Make sure you don’t have multiple tests for a single endpoint — I know this is obvious, but in a large organization with a lot of staff changing roles — these things tend to happen.</p><h4>Host Monitoring and APM</h4><p>This is a tricky one, but in general make sure that you only monitor hosts you care about <strong>and not all your fleet by default.</strong> Take into account that a highly containerized environment inflicts additional container costs, and in highly volatile environments such as batch processing workloads — the container costs can with the same order of magnitude as the infra cost itself.</p><p>Consider using cheaper open source tools for “general monitoring”, and use the full extent of the Datadog platform for your most valuable hosts and services (see my <a href="https://www.youtube.com/watch?v=mAiO8wfcT0E"><em>“I Have An APM Addiction”</em></a> talk [Hebrew] that covers exactly this issue.)</p><p>One of the patterns I’ve used in the past is to create internal low maintenance observability solutions for non-critical workloads (such as dev and staging environments) by leveraging open source tools with very short retentions.</p><h3>Encourage Effective Monitoring State of Mind</h3><p>Many engineers, especially less experienced ones, do not necessarily understand the cost of observability — for them it’s simply a single line of code, and <strong>lines of code are free</strong>!</p><p>Create an environment where the engineering organization is aware of observability costs, and consider any log line or metric added or change, as a change with cost implications.</p><p>Focus on custom metric tag cardinality and log throughput.</p><h3>End Note</h3><p>I hope this series gave you some food for thought on how to address your Datadog cost, and position yourself better in your next pricing negotiations.</p><p>Datadog is an amazing observability platform (although I am more of an open source monitoring kind of a guy 😉), but its cost can get out of hand quite easily and become a hassle for your Finops or Engineering/Devops manager.</p><p>It doesn’t have to be.</p><p>As always, thoughts and comments are welcome on twitter at <a href="https://twitter.com/cherkaskyb">@cherkaskyb</a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=4701a7d70ff6" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[The Magic That Is Datadog Pricing | Part II]]></title>
            <link>https://cherkaskyb.medium.com/the-magic-that-is-datadog-pricing-part-ii-5fa56cd8476e?source=rss-665b8f6a764a------2</link>
            <guid isPermaLink="false">https://medium.com/p/5fa56cd8476e</guid>
            <category><![CDATA[software-engineering]]></category>
            <category><![CDATA[finops]]></category>
            <category><![CDATA[devops]]></category>
            <category><![CDATA[programming]]></category>
            <category><![CDATA[software-development]]></category>
            <dc:creator><![CDATA[Boris Cherkasky]]></dc:creator>
            <pubDate>Tue, 25 Jul 2023 06:36:34 GMT</pubDate>
            <atom:updated>2023-07-25T06:41:01.690Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*pxVPq5xpXJ3FJERt" /></figure><p>The Datadog cost series:</p><ul><li><a href="https://cherkaskyb.medium.com/the-costly-elephant-in-the-room-is-a-data-dog-part-i-d690ad3cbee4">Part I — The costly elephant In the room is a (Data)dog</a></li><li><a href="https://cherkaskyb.medium.com/the-magic-that-is-datadog-pricing-part-ii-5fa56cd8476e">Part II — The magic that is Datadog pricing</a></li><li><a href="https://cherkaskyb.medium.com/data-puppy-shrinking-data-dog-costs-part-iii-4701a7d70ff6">Part III — Data Puppy, shrinking Datadog costs</a></li></ul><p>Welcome back to the series on Datadog pricing. <a href="https://cherkaskyb.medium.com/the-costly-elephant-in-the-room-is-a-data-dog-part-i-d690ad3cbee4">In the previous post</a> we’re covered some of the reasons why you should care about your Datadog costs. In this post, we’re going to cover how in general Datadog products get billed, and what causes this end-of-month invoices to be a bit, let’s say, unexpected.</p><p>As in the previous post, a disclaimer first — Datadog is an <strong>AMAZING </strong>product. And although throughout this series I’m covering how costly it is, it’s also <strong>very valuable</strong>, and the observability gained by it is exceptionally good. You pay a lot, but also get quite a lot in return.</p><h3>Datadog Products in High Level</h3><p>Generally, we can break down Datadog product pricing into 3 major buckets:</p><ul><li><strong>Host</strong> based pricing</li><li><strong>Volume</strong> based pricing</li><li><strong>User</strong> based pricing</li></ul><p>Some of those pricing models usually have exceptions, and other addons, but the general scheme of things is the above three.</p><p>To best understand the end-of-the-month invoice, it’s important to understand each of those, and how they can be optimized, or forecasted.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/800/0*6nN1dRuyarOtYb-g" /></figure><p>Let’s dive into each one of those.</p><h3>Host Based Products</h3><p>In host base products, such as <a href="https://www.datadoghq.com/pricing/?product=apm--continuous-profiler#apm--continuous-profiler">APM</a>, <a href="https://www.datadoghq.com/pricing/?product=infrastructure#infrastructure">infrastructure</a> and <a href="https://www.datadoghq.com/pricing/?product=cloud-security-management#cloud-security-management">CSM</a>, Datadog offers a monitoring platform for hosts (instances), with various capabilities.</p><p>In those products pricing model — the customer <strong>commits </strong>to a number of monitored hosts on a monthly basis, with what seems to be a very reasonable price of a few tens of dollars, but the interesting and what might be the surprising part, is that Datadog bills by the <strong>hourly</strong> concurrently monitored hosts (concurrently active host in a 5 minute intervals, averaged by the hour). So generally, when you’re committing to X monitored hosts, you’re actually committing to an X concurrently active hosts in each monthly hour.</p><p>Why can that be surprising? Because our systems are usually elastic, and might (auto) scale — therefore the 100 hosts you’ve committed to can turn 170 in the peak hours of your system’s traffic, and 70 in the low point of your traffic, but you have to put a fixed number on your commitment. And here it becomes a game of dynamic optimization — commit to just enough hosts to not be slaughtered on on-demand rates during your peaks, while not overcommitting in your low hours.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*mcTgCUyrWFZlDHBw" /></figure><p>Moreover Datadog pushes you to commit up front in this pricing model by applying quite an aggressive method: If you’re not committing to any quantity — <strong>they are billing by 99th percentile used hour and NOT by the actual usage. I.e. the number of hosts active in your 99th percentile most active hour will be your end-of-month invoice. </strong>And as engineers, we know how unpredictable our 99th percentile can be.</p><p>On the engineering side — I can understand why this pricing model was chosen — when Datadog has no estimation for your usage, they need to over-provision their system to support any load, thus having larger operational cost. Does this justify the p99 billing point — it’s up to you to decide.</p><h3>Volume Based Products</h3><p>In volume base products, such as <a href="https://www.datadoghq.com/pricing/?product=log-management#log-management">Logs</a>, <a href="https://www.datadoghq.com/pricing/?product=synthetic-monitoring#synthetic-monitoring">Synthetics</a> you pay per use — as simple as that.</p><p>You send Datadog 100GB of logs, you’ll pay for 100GB of logs, You execute 1M synthetic checks — you’ll pay for 1M synthetic checks.</p><p>The commitment model here is quite simple too — you commit to Volume, get a reduced price, and when you use all your committed volume, you get billed the on-demand rates.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*MPdmZAfTGRQTJ47J" /></figure><p>This is the basic SAAS business model, and Datadog is not different from many other vendors.</p><p>Logs, since being a very popular Datadog product, varies a bit from other volume based products and is worth diving a bit deeper into.</p><p>It’s important to understand that in the Logs product sending logs to the Datadog platform has multiple billings applied to it:</p><ul><li><strong>Ingestion</strong> — You pay for the GB volume of logs you send (usually it’s a fixed price of 0.1$ / GB of data)</li><li><strong>Indexing</strong> — You pay for the amount (count) of logs you store on the Datadog platform, and the pricing varies upon the retention for which the data is saved (the longer you save the data, the more expensive that is)</li><li><strong>Rehydration</strong> — if for any reason, you want to make logs that passed their retention period available again, you need to rehydrate them, and pay for it.</li></ul><p>Up until this point, no surprises — you send data to Datadog, therefore you pay for it, totally makes sense.</p><p>The interesting part about logs is that you pay <strong>regardless of usage</strong> — i.e. even if you never opened the Datadog log search webpage, you still pay those amounts. That makes total sense business wise for Datadog, since they have to process, store, and index all these logs anyhow, and pay themselves for processing it. But as a user, you might be paying for something you never <strong>actively</strong> use, or use very rarely. I don’t have numbers to support that, but if I had to guess by my personal experience, I’d say that less than 0.1% of the indexed logs are actually being used.<strong> </strong>The bottom line here is to <strong>think before you log </strong>(I’ve written in the past on <a href="https://itnext.io/log-warn-this-log-will-cost-you-on-log-cost-optimization-5c6fd8b234e9">log cost optimization</a>).</p><p><strong>Logs aren’t free, and aren’t cheap.</strong></p><p>It’s important to mention, that in usage-based products it is <strong>very easy to over-use commitments</strong> and get to on-demand pricing which is <strong>&gt;150%</strong> of the committed price (by default).</p><p>All it takes is to make a synthetic API test to run every 5 minutes instead of 10 to <strong>double </strong>the cost. Or launch a new service to production that takes another 20% chunk of logs, those are things that developers can do (and are doing) without applying much thought to it.</p><h3>Active User Based Products</h3><p>In user based products, such as <a href="https://www.datadoghq.com/pricing/?product=incident-management#incident-management">Incident management</a> and <a href="https://www.datadoghq.com/pricing/?product=ci-visibility#ci-visibility">CI Visibility</a> you pay per active users — as simple as that.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*7RD_YtxbRr604cgb" /></figure><p>And to be honest, in this case I have no complaints against Datadog — they are quite gentlemen about defining what “active” users are.</p><h3>Buy One Get Some for Free</h3><p>The last piece of the puzzle are products such as containers, custom metrics, index spans.</p><p>For those products, you get a “free amount” for some of the “main” Datadog products you use.</p><p>For example, for each monitored host, you’ll get (for enterprise account):</p><ul><li>4 monitored containers</li><li>200 custom metrics</li><li>1M APM spans</li></ul><p>Those products will generally behave as host based / usage based products mentioned above.</p><p>These products are the hardest to understand, reason about, and get their const under control. I won’t get to the dirty details of why they are so difficult to manage, but I’ll try to summarize the general theme bellow.</p><p><strong>The Challenge of controlling container monitoring cost</strong></p><p>In highly containerized and dynamic environments such as Kubernetes, each host (i.e. cloud provider instance), can have a huge amount of <strong>unique</strong> active containers, causing the host-monitoring cost to skyrocket. In our case it increased our cost by ~50%.</p><p>To be fair — Datadog tries to <a href="https://docs.datadoghq.com/account_management/billing/containers/#frequently-asked-questions">average out outliers</a>, but in the age of Kubernetes — containers come and go and incline additional Datadog costs.</p><p><strong>The Challenge of controlling custom metrics cost</strong></p><p>The Custom Metrics product in general is <a href="https://docs.datadoghq.com/account_management/billing/custom_metrics/?tab=countrate">billed with respect of the cardinality</a> of the tag value of your metrics — the more <strong>unique combinations</strong> you report — the more expensive it is.</p><p>Since the cost is per <strong>unique combinations</strong> it means the cost CAN grow non-linearly when adding additional tags! (it depends greatly on the cardinality of each tag and whether they are codependent).</p><p>In addition <strong>using the right observability primitive</strong> is critical! Distribution / histogram custom metrics are billed <strong>at least</strong> <a href="https://docs.datadoghq.com/account_management/billing/custom_metrics/?tab=histogram">5 times higher</a> than Gauges / Counters.</p><p>If you ever built modern complex micro service based systems, you can probably understand how the cardinality of those tags can get out of hand quite easily.</p><h3>The Complexity of the Datadog Pricing Mental Model</h3><p>Datadog has about 20–30 products, sub products, and additional fine-print add ons, all of them behave roughly in the above mentioned pricing models (with some variations), some price per GB, some price per 1M event, some 1K events, and some for 10K events, some per host-month, some per host-hour.</p><p>See the <a href="https://www.datadoghq.com/pricing/?product=synthetic-monitoring#synthetic-monitoring">synthetics pricing page</a> as example for how fine lined the pricing model is:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*16xWu__2CL6WusYW" /></figure><p>The <em>Browser tests</em> are not 2.4 times more expensive than <em>API tests</em>, they are <strong>24 times</strong> more expensive.</p><p>Or the part about non-committed hosts priced by the p99 usage hour, it isn’t mentioned in the main infrastructure/APM pricing pages, it’s mentioned in the <a href="https://docs.datadoghq.com/account_management/billing/">billing FAQ</a>, a page that took me 5 minutes to find while writing this post, even though I <strong>know it exists</strong> and knew what i was looking for.</p><p><strong>Someone needs to hold this mental model in their head</strong> to understand, provision, and commit to a budget.</p><p>It took me more than two weeks to get my head around all those bits and bytes and create usage to cost translation for our application. Those are two weeks that I was in <strong>full</strong> <strong>focus</strong> on solving that problem, and I already have previous experience with many other observability platforms.</p><p>A week later I forgot half of the things I learned. Expecting anyone doing this commitment budgeting once a year (at best) to understand those bits and bytes is quite an unreasonable expectation.</p><p><strong>This is hard on the Finops engineers of the company</strong> — although they probably understand these pricing models, they usually don’t know the actual technical behaviour of the system, and aren’t involved in the technical roadmap (thus don’t know how many new services will be spawned or scrapped — which affects usage).</p><p><strong>This is hard on the Dev team too</strong> — doing budgeting and cost allocation isn’t their core competence, and even for Devs it’s hard to understand and forecast volumes.</p><p>The bottom line is that <strong>this is a guessing game</strong> until you get experience with forecasting usage, translating it to cost, and figuring out the right cost-effective commitment. <br><strong>It shouldn’t be.</strong></p><h3>Next up</h3><p>I hope I was able to shed some light on how the Datadog pricing model works, and help with your understanding of what you’re paying for, and maybe where you start looking for optimization opportunities or adjusting your commitments to better suit your needs.</p><p>In the next part we’ll cover what can be done to optimize Datadog costs</p><p>As always, thoughts and comments are welcome on twitter at <a href="https://twitter.com/cherkaskyb">@cherkaskyb</a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=5fa56cd8476e" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[The Costly Elephant In The Room Is a (Data)dog | Part I]]></title>
            <link>https://cherkaskyb.medium.com/the-costly-elephant-in-the-room-is-a-data-dog-part-i-d690ad3cbee4?source=rss-665b8f6a764a------2</link>
            <guid isPermaLink="false">https://medium.com/p/d690ad3cbee4</guid>
            <category><![CDATA[programming]]></category>
            <category><![CDATA[finops]]></category>
            <category><![CDATA[software-engineering]]></category>
            <category><![CDATA[software-development]]></category>
            <category><![CDATA[devops]]></category>
            <dc:creator><![CDATA[Boris Cherkasky]]></dc:creator>
            <pubDate>Tue, 25 Jul 2023 06:35:57 GMT</pubDate>
            <atom:updated>2023-07-25T06:41:08.494Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*_nxqUFpz_1GDoVCz" /></figure><p>The Datadog cost series:</p><ul><li><a href="https://cherkaskyb.medium.com/the-costly-elephant-in-the-room-is-a-data-dog-part-i-d690ad3cbee4">Part I — The costly elephant In the room is a (Data)dog</a></li><li><a href="https://cherkaskyb.medium.com/the-magic-that-is-datadog-pricing-part-ii-5fa56cd8476e">Part II — The magic that is Datadog pricing</a></li><li><a href="https://cherkaskyb.medium.com/data-puppy-shrinking-data-dog-costs-part-iii-4701a7d70ff6">Part III — Data Puppy, shrinking Datadog costs</a></li></ul><p>Unless you’re living under a rock, you’ve probably heard of <a href="https://www.datadoghq.com/">Datadog</a>, one of the most loved and used observability platforms currently available.</p><p>Recently, we at <a href="https://www.finout.io/">Finout</a>, set out on the journey to shine a bright light on the Datadog costs, and allow organizations to better understand them, prepare for the end of month invoice, and optimize their bottom line.</p><p>The main reason for us to go down this rabbit hole, is that the Datadog platform lacks observability on its out cost (yep, I see the irony here), making it quite challenging to reason about its cost, optimizing it, and being alerted on ominous increase.</p><p>In the upcoming series of posts, I’ll cover</p><ol><li>In this post — why you take an interest in your Datadog costs</li><li>In the second part — How Datadog pricing works, and the lessons learned as a Datadog user</li><li>In the third part — What can we reduce the Datadog end of month blll.</li></ol><p>This article, the first one in the series will try to answer the basic question of them all; Why you should care about your Datadog costs.</p><p>Firstly, a disclaimer — Datadog is an <strong>AMAZING </strong>product. And although throughout this series I’m going to cover how costly it is, it’s also <strong>very valuable</strong>, and the observability gained by it is exceptionally good. You pay a lot, but also get quite a lot in return.</p><p>I feel it’s important to explicitly mention it before we dive in.</p><h3>Datadog Cost 101</h3><p>Datadog has numerous products, for various different use cases, and most of them have addons with additional functionality, capabilities and cost. But the general rule of thumb you need to understand is — the more your use, the more you pay:</p><p>You are billed on the hosts to monitor or profile, You are billed on the volume (GB) of logs you are sending and indexing, etc.</p><p>If you know, or have a hunch how much you’re going to use — you can:</p><ol><li><strong>Commit</strong> to a usage for relevant Datadog products, and get a decent <strong>discount.</strong></li><li><strong>Prepay</strong> a “Base fee” — An upfront payment, then get <strong>discounted</strong> prices for any used products, and then any usage gets billed from that prepaid amount. It reminds debit cards in a way.</li></ol><p>But the key here, you need to know how much to commit to, and then basically wait and pray you won’t over use it.</p><h3>Why Should We Even Care About Datadog Costs?</h3><p>If you’re asking that, you are either:</p><ol><li>Very optimized — and therefore a real kudos for you! I know how complex achieving this is.</li><li>Still at a small scale, therefore your Datadog costs are limited.</li><li>You never used Datadog, and therefore don’t know how easy it is to get to a very large invoice.</li></ol><p>Let’s break it down, and understand what makes datadog different for your main cloud provider.</p><h3>It’s Not Your Core Competence</h3><p>First, let’s agree that a typical DD invoice can easily get to 4–10% of your cloud provider invoice — and since we know how expensive your cloud provider is, you can now understand how expensive Datadog is too.</p><p>Datadog is NOT your core competence, it’s a supporting tool — a very good one, and a very important one, but a supporting tool non the less. It’s not the EC2 instances that keeps your business available, or the underlying storage that stores the data that gives you competitive edge, it’s an observability platform that helps your operations organization keep the lights on and the engine going.</p><p>You are paying, <strong>because</strong> it’s not your core competence — you’re paying because you want to focus on building your core value.</p><p>That being said, we should really be vigilant on how much we pay on services that are outside our core competence too, and since there’s always room for “more monitoring”, and more monitoring means more cost, where do we draw the line?</p><h3>Scalable — Throughput and $ Wise</h3><p>It’s crazy scaleable, therefore anything you’ll throw at it will be ingested, stored, available for you to query and then alerted on. Sounds amazing right?</p><p>Well it’s indeed amazing, and then — anything you throw at it, also gets billed.</p><p>And that’s what makes it so so easy to get out of hand — launching a new service that sends too many logs can easily be the difference between $100 a month to $1500 a month. This can be due to developer error, or just since the service you’ve just launched serves so much traffic that even the minimal necessary logging causes excessive costs (and this happened to me, and I’ve written about our <a href="https://itnext.io/log-warn-this-log-will-cost-you-on-log-cost-optimization-5c6fd8b234e9">log cost optimization effort</a>).</p><h3>Implicitly Increases With Infrastructure Scaling</h3><p>As the business grows you add more infrastructure instances (EC2 instances, serverless function etc), you implicitly add additional resources to be monitored, with in turn increase the Datadog billed amount.</p><p>This is the expected behaviour — you just need to keep you unit economics at check, i.e. the increase in monitoring cost is indeed in linear correlation to the business and infrastructure costs.</p><h3>Difficult To Forecast</h3><p>Lastly, it’s quite hard to provision usage, and therefore forecast the expected cost.</p><p>Ask your average developer what’s their services’s throughput, and what’s the volume of logs each request generates — I doubt they could give you the number, And even if they knew — there’s still the process of translating usage to estimated cost.</p><p>Or alternatively — what’s the expected number of hosts we’re going to use to support the black Friday sale, or the Superbowl half time? And should we commit to this number, or our average regular usage?</p><p>What’s the expected additional services we’re about to develop — how many compute resources, serverless, log and metrics they are going to use?</p><p>And since those numbers are hard to come up by, the commitment we buy from Datadog is also hard to come by. Most of the time, the de-facto way to do it, is to start using the product with some (or no commitment), and adjust the commitment according to the actual usage (which is obviously forever changing).</p><h3>So Why Not Just Use Datadog Usage Dashboards?”</h3><p>It’s true, Datadog offers <a href="https://docs.datadoghq.com/account_management/billing/usage_details/">usage metering</a>, where you can have a general understanding of your stance and usage.</p><p>But does the average developer / product owner understand what 4TB of 7 days retention indexed logs means in terms of cost and operability of the service?</p><p>Can we do with 3.5TB of 3 days retention?</p><p>And how much will we save if we’ll go for it?</p><p>Do they understand how this can be optimized? and should this even be?</p><p>And do they understand how much it costs, which portion of it is committed up-front, and which gets billed on a pay-per use pricing model?</p><h3>To Care Or Not To Care</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/582/0*LDmdLy_9LzRnGeUS" /></figure><p>We pay for Datadog so someone else will handle the headache of managing such a complex platform, but <strong>when the Datadog expense gets to be a headache of its own</strong> — this is the time to regroup and rethink your strategy.</p><p>Through right usage analysis that turns into usage optimizations and better commitment allocation, it’s possible to save a large chunk of the end-of-month bil. A chunk that can be in the ranges of tens and even hundreds of thousands of dollars a year.</p><p>And since either way when your bill gets painful enough you’re going to do the maths and calculate your commitments from time to time — why not do it better, and save more?</p><h3>Next up</h3><p>This was an initial post on why it is hard, and why we should care.</p><p>In the <a href="https://cherkaskyb.medium.com/the-magic-that-is-datadog-pricing-part-ii-5fa56cd8476e">next post</a> we’ll talk about the Datadog pricing models and their pitfalls.</p><p>As always, thoughts and comments are welcome on twitter at <a href="https://twitter.com/cherkaskyb">@cherkaskyb</a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=d690ad3cbee4" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[What Is That Change That Is The Source Of All Instability]]></title>
            <link>https://cherkaskyb.medium.com/what-is-that-change-that-is-the-source-of-all-instability-c3eb03c5fdc3?source=rss-665b8f6a764a------2</link>
            <guid isPermaLink="false">https://medium.com/p/c3eb03c5fdc3</guid>
            <category><![CDATA[engineering]]></category>
            <category><![CDATA[reliability-engineering]]></category>
            <category><![CDATA[devops]]></category>
            <category><![CDATA[software]]></category>
            <category><![CDATA[software-engineering]]></category>
            <dc:creator><![CDATA[Boris Cherkasky]]></dc:creator>
            <pubDate>Wed, 28 Jun 2023 07:22:10 GMT</pubDate>
            <atom:updated>2023-06-28T07:22:10.597Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*9-cdENYPviKvGMUD" /><figcaption>Artwork by <a href="https://unsplash.com/@rossfindon">@rossfindon</a> on unsplash.com</figcaption></figure><h3>What Is That Change Which Is The Source Of All Instability?</h3><p>It is a known saying in the SRE world that change is the source of all instability. And it is indeed so — change is what usually disrupts service, and takes applications down.</p><p>Engineers usually think of “change” as code change, but the ugly truth is that there are many other types of change that are as dangerous to the production environment.</p><p>Unexpected changes will surely take your system down, so in this post I’ll share some hard learned lessons of such unexpected changes that took my production systems down.</p><p>I hope that reading about such less-expected types of change will make you stop for a moment and ponder upon what are the risks your production env is prone to.</p><p>This post is inspired by <a href="https://twitter.com/nukemberg">@nukemberg</a>’s talk <a href="https://www.youtube.com/watch?v=dEjeS2avtis">The Reboot, Reloaded</a> Talk (Hebrew)</p><h4>Putting Types of Change on a Scale</h4><p>I like to put “change” on a scale of <strong>explicitly</strong> — where an <strong>explicit</strong> change is one where there’s at least one human actor (usually dev/ops person) who are fully aware that a change is happening — they might not fully understand the change or it’s implications, but they know a change is in progress.</p><p>While on the other hand <strong>implicit</strong> change is one where no human is aware that it is happening, or no human is in interaction with the system or production environment while it’s happening.</p><p>There are a lot of ways to put change on a scale, but I found the scale of explicity to be a very good way to express the complexity of systems, and how large is the varsity of changes that modern systems goes through.</p><p>Let’s dive in and cover in detail some common types of change</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*W1M-6Qm51ZLw2cLk" /><figcaption>change types on a scale of implicity</figcaption></figure><h4>Code and Configuration</h4><p>This is the most obvious one — changing code or configuration is a major source of instability, this is the most obvious one, and the most explicitly type of change our systems undergo</p><h4>Load Change</h4><p>The load our systems support and serve is also changing, and obviously affects our systems.</p><p>Load is trickier than code though. It might be an explicit planned change such as black Friday for example — load is going to increase drastically, everyone knows about it, and is prepared for it.</p><p>But load can change much more implicitly — a DDOS attack, a halfway break in the Rugby world cup that you are not aware of, or just onboarding a new large customer that the engineering team is unaware of (obviously, auto — capacity tuning is the answer, but organization take time to get there).</p><p><strong>Unexpected load is probably the number one factor for system failure after code and configuration changes.</strong></p><h4>Population Change</h4><p>Population change is a nuanced topic, it can be considered as a “load change”, but I like to think of it as a separate type due to its more unexpected nature.</p><p>Population means the mixture of request you’re system is serving, some examples</p><ul><li>The ratio of GET / POST / PUT requests</li><li>The throughput ratio between endpoints the system serves</li><li>The throughput ratio between customers the system serves</li></ul><p>Population change means the ratio of requests served changes — for example — a ratio of 20–80 POST-GET requests can change to 40–60 POST-GET — the system gets much more write heavy.</p><p>This is entirely implicit in most cases — it’s a change in how <strong>users interact with the system</strong>, or even a change in who the users are.</p><p>This is extremely difficult to predict, and even identifying this change is not always trivial.</p><p>What can population change cause? For instance, as in the POST-GET example above, it can change the nature of the system, driving it from being a read intensive system, to a write intensive system — those systems have completely different design characteristics!</p><p>Another example might be that the requests in the system becomes “heavier” — for example, if you’re serving 1K RPS with median latency of 50ms, onboarding a new customer might change the population causing you to serve 1.1K RPS with a median latency of 200ms — a 10% change in throughput causing a 400% impact on median latency.</p><p>Maybe the requests sent by this new customer are larger in size, or require heavier computations.</p><p>These things happen! They are hard to identify, and we need to be aware of them!</p><h4>Time Changes</h4><p>Time changes, and we don’t even think of it!</p><p>That means any time-based logic is prone to errors and bugs that will be manifested only in specific times or dates.</p><p>I once saw a if request.created_at &gt; “2023–04–22: return BAD_REQUESTin a production system. I didn’t know why someone expected the end of the world to happen in April 2023, but this was a time bomb waiting to happen.</p><p>We found it by sheer chance before it triggered.</p><p>The operating system / infrastructure applications might have jobs scheduled for specific times, during which the system behaves differently. For example — when the Postgres VACUUM job triggers — performance might be impacted severely.</p><p>It is a known issue in general purpose scheduling systems that human generated jobs tend to be scheduled on a round hour — that means your production infrastructure will have spiky load at those times.</p><h4>Long Running Apps and Instances</h4><p>On one of my first vacations on the job, the system went down just because we stopped deploying daily, and an instance ran for 3 days straight for the first time. It had a misconfigured log rotation configuration, causing its disk to fill up and crash.</p><p>The fact that your instance hasn’t been reset for a long period of time is a change by itself! It might be the first time it has been running for such a long time.</p><p>Generalizing the above example — a large scale system has the tendency to reach various saturation points when running over long periods of time without the proper maintenance.</p><p>Some examples i’ve encountered in the wild:</p><ul><li>DB that gets to some limits and starts to slow down</li><li>Reaching the point where you get to the limit set by your improper log rotation configuration</li><li>Getting to the point where the memory leaks you have lurking finally reach their critical point</li></ul><p>The only way to check this is to actually run the system for a long time, under load.</p><h3>Final thoughts</h3><p>The amount of such changes that can take a production system down is huge, and I’m not even pretending to be able to cover them all.</p><p>Thinking and covering all those possibilities is a time consuming effort, and until you’ve seen production systems in the wild go down on a 500 jobs scheduled at the same moment at midnight, or an instance with a memory leak that ran for the first time for a week — it’s hard to think of these scenarios.</p><p>I hope this post, and my personal experience, helped open your eyes to different ways your production system can fail, and nudge you to block some time and think about what’s relevant in your production environment.</p><p>As always, thoughts and comments are welcome on Twitter at <a href="https://twitter.com/cherkaskyb">@cherkaskyb</a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=c3eb03c5fdc3" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Improving Incident Recovery By using SLI Pyramid]]></title>
            <link>https://medium.com/better-programming/improving-incident-recovery-by-using-sli-pyramid-94fe906a5df8?source=rss-665b8f6a764a------2</link>
            <guid isPermaLink="false">https://medium.com/p/94fe906a5df8</guid>
            <category><![CDATA[engineering]]></category>
            <category><![CDATA[devops]]></category>
            <category><![CDATA[product-management]]></category>
            <category><![CDATA[software-development]]></category>
            <category><![CDATA[software-engineering]]></category>
            <dc:creator><![CDATA[Boris Cherkasky]]></dc:creator>
            <pubDate>Wed, 19 Apr 2023 06:45:36 GMT</pubDate>
            <atom:updated>2023-04-21T23:56:25.388Z</atom:updated>
            <content:encoded><![CDATA[<h3>Improving Incident Recovery By using The SLI Pyramid</h3><h4>Improving system reliability and uptime by prioritising SLIs</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*EfdPUByZWF9kALFX" /><figcaption>Image by <a href="https://unsplash.com/@lazycreekimages">@lazycreekimages</a> on <a href="https://unsplash.com/">Unsplash</a></figcaption></figure><p>Let me take you back to one of my most painful production incidents: one of our main datastores failed during what we thought to be a simple maintenance act.</p><p>The datastore was a main component in our system at that time and turned out to be a single point of failure. While it was down, we couldn’t process any requests.</p><p>Full recovery was estimated to take a few hours, which seemed like a lifetime.</p><p>About ten minutes into the recovery process, a few talented engineers from the incident response team came up with an idea that might get our system operating: disconnecting the logic which uses the datastore from the main process by toggling a “dooms-day” configuration setting we had.</p><p>Sounds amazing! The first rule in the incident response rule book — <em>“Stop the bleeding! Get the system back up.”</em> Well, it turned out not to be that simple. Rule one in the rule book has some vague implications we were about to uncover.</p><p>In this article, I’ll explain that although “getting the system back up” should be our first priority, to do so safely, we first need to very carefully define what “up” means.</p><h3>The Incident — And The Response</h3><p>Circling back to my story, our system was processing messages from a message bus using a microservice dependent on the failing datastore.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*0_2zCIvCc25rIVfe" /></figure><p>The first thing we did when we identified the issue was to scale down our workers that pulled requests from one of our message buses.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*07wTCHv8z950HsAW" /></figure><p>By doing this, we limited the noise in the system and allowed optimal recovery performance for the impacted services. So the current state is that requests were queued in our message bus, waiting to be processed.</p><p>Circling back to the configuration setting, we debated whether to toggle it.</p><p>Toggling it meant we could turn off the system using the datastore, thus “unblocking” the system, scale it back up, and start processing the (already huge) backlog of requests. But the microservice using the failing datastore was there for a reason! It performed an important role in our system — can we provide valid results without it?</p><p>This debate lasted about an hour and involved various people from the organization:</p><ul><li>The engineering team was eager to toggle it on and get the system back running</li><li>One of the business liaisons on the incident response team found out that toggling it will impact our performance in a way that will cause a beach SLA with few of our top customers</li><li>Next, the head of engineering mentioned it might cause issues downstream when other “offline processes” would try and process partially processed requests (partially = without the input from the failing microservice)</li></ul><p>We couldn’t decide what to do; each alternative had pros and cons. Eventually, it escalated to the company’s executives, who decided the SLA impact was unacceptable, and we couldn’t toggle it.</p><p>We sat there waiting for the recovery process to finish. Although this was the correct decision, it was still an awful feeling.</p><p>The bottom line is that it’s “easy” to mitigate an incident when the mitigation process fixes the underlying issue and returns the system to its correct state. But when the mitigation process puts your system in a different bad state, it’s much harder to decide whether to go down this mitigation path.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/660/0*r19C3TUq6BY92-Ug" /><figcaption>Our alternatives in mitigating the incident</figcaption></figure><h3>The SLI Pyramid</h3><p>First, it’s important to mention that the method I propose is not always possible and is deeply dependent on the system you have at hand, your understanding of the business and technology, and your company’s business stance.</p><p>The issue that made our decision so difficult was that “<em>getting the system back up</em>” was infeasible at that point. We could only choose between two different bad states.</p><p>I suggest addressing this by slightly redefining what “up” means. Instead of a single state, consider it an “SLI Pyramid,” a visual representation of <a href="https://sre.google/sre-book/service-level-objectives/">SLI</a> prioritization. It means we decide, during system design, or system lunch review, or any other point in time, what our most important SLIs are, and order them by importance. An example pyramid can be something like this:</p><figure><img alt="top to bottom: mobile platform availability, API latency, data loss, data quality" src="https://cdn-images-1.medium.com/max/1024/0*sIkpXWCcbfJd1ADt" /></figure><p>This pyramid means that SLIs at the top of the pyramid can be sacrificed if we can secure the SLIs at the bottom of the pyramid — when forced to choose between bad states, we now have a way of defining which are preferred.</p><p>Consider it to be the <a href="https://simplypsychology.org/maslow.html">Maslow pyramid</a> of your system.</p><p>In the above pyramid example, when an incident escalated to a state where both the mobile platform was in a degraded state and we had a data quality issue — if by turning the mobile platform off, we could fix the data quality issue — we were authorized (and expected) to do it!</p><p>The pyramid should be defined with the product owner, business, and whoever can sign off on such an important decision.</p><p>It’s important to mention that it doesn’t always work, and making this discrete prioritization is impossible. But in most cases I’ve encountered, it’s completely doable (even if it means slightly more complex SLIs). For example, it could be something like this:</p><figure><img alt="top to bottom: data loss, API latency, all other customers error rate &lt; 0.1%, top-tier customers error rate &lt; 0.01% and p99 latency &lt; 500ms" src="https://cdn-images-1.medium.com/max/1024/0*Ac7q7YRt09Qeazug" /></figure><p>The pyramid doesn’t have to cover 100% of use cases, incidents, and SLIs, but I am certain some of your SLIs are so clear they can easily be prioritized.</p><h3>A Word About Analysis Paralysis</h3><p>In my story, the incident response team suffered from <a href="https://en.wikipedia.org/wiki/Analysis_paralysis">analysis paralysis</a> — being unable to decide due to overanalyzing the problem.</p><p>It took us hours of debating, weighing alternatives and outcomes, and even then, only escalation to a company executive enabled us to decide.</p><p>This was quite a frustrating scenario; it’s frustrating to the response team, who’s helplessly waiting for the recovery process to finish, and frustrating to the creative engineer, who came up with an alternative mitigation path since we weren’t progressing with it.</p><p>The SLI Pyramid tries to “pre-analyze” the most important SLIs of the system and make the decision-making during outages a little easier.</p><h3>Closing Thoughts</h3><p>I believe that applying this method can really help with reducing incident friction and maintaining the psychological safety of your responders. This is also a great way to think of production drills or “game days” where we can simulate choosing between sub-optimal mitigation steps and track how our response team operates under these tough conditions.</p><p>This article is based on my talk <em>“The (ir)rational Incident Response: How Psychological Biases Affect Incident Response”</em> and is available in <a href="https://www.youtube.com/watch?v=HqrI1zLYOvo">English</a> and <a href="https://www.youtube.com/watch?v=mrYfZVsjXFs">Hebrew</a>.</p><p>As always, thoughts and comments are welcome on Twitter at <a href="https://twitter.com/cherkaskyb">@cherkaskyb</a>.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=94fe906a5df8" width="1" height="1" alt=""><hr><p><a href="https://medium.com/better-programming/improving-incident-recovery-by-using-sli-pyramid-94fe906a5df8">Improving Incident Recovery By using SLI Pyramid</a> was originally published in <a href="https://betterprogramming.pub/">Better Programming</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Leveraging Mutation Testing To Onboard Legacy Projects]]></title>
            <link>https://cherkaskyb.medium.com/leveraging-mutation-testing-to-onboard-legacy-projects-e968eb5d0e62?source=rss-665b8f6a764a------2</link>
            <guid isPermaLink="false">https://medium.com/p/e968eb5d0e62</guid>
            <category><![CDATA[software]]></category>
            <category><![CDATA[software-engineering]]></category>
            <category><![CDATA[testing]]></category>
            <category><![CDATA[programming]]></category>
            <category><![CDATA[software-dev]]></category>
            <dc:creator><![CDATA[Boris Cherkasky]]></dc:creator>
            <pubDate>Mon, 13 Feb 2023 08:31:16 GMT</pubDate>
            <atom:updated>2023-02-13T08:31:16.752Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1000/0*Wj132AQxzqm11rD8" /><figcaption>image source <a href="https://unsplash.com/photos/NMk1Vggt2hg">unsplash.com</a> by <a href="https://unsplash.com/@dallehj">@dallehj</a></figcaption></figure><p>You are an accomplished engineer, your skill set is vetted, and you are trusted with a challenging task — doing some important maintenance work on a legacy service you know nothing about.</p><p>The original developers of the service are long gone, you have enough business context for what it does but limited context for the technical implementation of the service.</p><p>Just understanding the architecture and code of the project is complex enough, but you need to figure out how to test the change, and then assess its risk and commit to effort estimations based on very little context and knowledge.</p><p>Wouldn’t this be amazing to have a way to reduce the unknown, and relevant risks that come with it?</p><p>In this post I’ll cover a pattern I’ve found useful when onboarding an unfamiliar codebase (doesn’t have to be legacy) that works quite well in boosting confidence before starting the actual coding phase.</p><h3>Challenges With Onboarding to Legacy Projects</h3><p>One of the greatest challenges with unfamiliar codebases, is lack of context — the lack of “tribal knowledge” that was gathered by the developers of the code while working on it, for example:</p><ul><li>what are the complex parts of the code</li><li>all those “gotchas”</li><li><em>The “Why”</em> behind some of the design decisions and patterns</li><li>Knowledge of which features were developed under pressure and are lacking test coverage and code quality, and which were developed under better patterns.</li></ul><p>At least for me, when I start onboarding a new codebase — I start with a short exploration of the production environment to understand the moving parts.</p><p>Then I go to read the main code flows (the ones relevant for my change).</p><p>Then I skim through the tests to have a general sense of their quality, And while skimming them I run them. Hoping for the test suite to turn out green ✅.</p><p><strong>Luckily, they passed!</strong> But what exactly are they testing? Are they worth anything? Will my changes be caught by the existing test suite?</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*fqXdgzXIWS-hbF23" /><figcaption>“all my tests passed”</figcaption></figure><p>How awesome would that be if you could know that the code you’re about to change is covered by <strong>thorough tests</strong>! That any change you’ll make will break a test, and you’ll have an assertion that’ll show you the impact of your change? How safe will you feel? How simpler time estimations for this project will be?</p><p>What if I told you you can!</p><p>You can have an assessment for how well the code you’re about to touch is covered by tests? Wouldn’t this be awesome?</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/525/0*9N9QTyBeEo3guMoF" /></figure><h3>Mutation Testing In A Nutshell</h3><p>In a nutshell, <a href="https://en.wikipedia.org/wiki/Mutation_testing">mutation testing</a> is the act of applying small changes to a program and asserting that the program’s test suite catches those changes — i.e. for each change, at least one test is failed.</p><p>Some examples for what are those “small changes”, called <em>“mutants”</em>, are:</p><p><strong>Changing branch conditions </strong>— if the code has if x &lt; 10 the mutant might be if x &gt; 10 or if True</p><p><strong>Changing literals</strong> — changing the values of literals such as strings / integers / Booleans so if your code is x = 2; return x * 10; the mutant might be x = 10; return x * 10;</p><p>Actual example in Scala from <a href="https://stryker-mutator.io/">stryker-mutator</a> project:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*6PSM9J_MIrcVefAH" /><figcaption>An example for an acutal “mutant”</figcaption></figure><p>The teal circle is an actual mutation that was applied on the code — changing the s.trim.nonEmpty line 13 to s.trim.isEmpty</p><p>In my opinion, by running mutation testing on a codebase you gain two things:</p><ul><li>Understanding which parts of the code are lacking test coverage or test quality</li><li>A “feeling” of how safe it is to introduce changes to the program</li></ul><h3>Using Mutation Testing For Blindspot Analysis</h3><p>The intro for this post happened to me — I was assigned to introduce a change to a system I wasn’t very familiar with.</p><p>It had, what seemed to be, a fair test suite of a few hundred unit tests, but their quality wasn’t known to me and from initial skimming of their code I was afraid they weren’t good enough — they seem to do a lot of things, with quite long tests — which I know from experience to be a code-smell, and a bug red flag 🚩.</p><p>I was looking for a way to assess whether the changes I am about to introduce would have been covered by the test suite. Sounds exactly like what mutation testing promises no?</p><p>By sheer luck, I stumbled upon the <a href="https://stryker-mutator.io/">stryker-mutator project</a> just when I was about to start my work. After ~10 minutes of configuration I had an HTML report of all changes introduced to my code base that the existing test suite failed to catch.</p><p>Or in simpler words — it showed me <strong>which parts of the code base aren’t covered well enough by tests</strong> — It shined a very bright light on where it’ll be difficult, risky and time consuming to apply changes.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*cu-z3IhfyxviI19_" /></figure><h3>Caveats</h3><p>It’s not all perfect, when I’ve executed the tests, many of the mutants that weren’t caught by the test suite were non-functional requirements that usually aren’t tested — for example:</p><ul><li>Mutants that changed log messages — we don’t test for those.</li><li>Mutants that changed monitoring metrics values such as incrementing a request counter by 2 instead of 1 — we don’t have tests for those.</li></ul><p>So while the initial report had quite a low Signal — Noise Ratio, by configuring the test not to change string literals for example, I was able to get better results that were actually actionable.</p><p>In addition — the mutants tested obviously do not cover 100% of possible changes, so a “green” execution doesn’t guarantee change safety, while a “red” execution is a definite problematic place, where code changes should be carefully introduced.</p><h3>Isn’t Test Coverage Enough?</h3><p>You are right to think that coverage can give you similar insights, BUT — coverage only checks for whether executing a test “steps” on a code path, <strong>it doesn’t check whether changing that path will cause a test to break</strong> — thus it might create a false sense of safety. From my experience, indeed, a high coverage rate isn’t an indicator for anything.</p><h3>Final Thoughts</h3><p>Getting comfortable in an unfamiliar codebase is always a challenge, let alone in a legacy codebase.</p><p>Running a mutation testing suite when onboarding the project can really ease the process!</p><p>It can shine a light on the problematic parts of the codebase.<br>It can serve as a warning sign to where you should be more careful introducing code changes, and where additional tests will be required.</p><p>As always, thoughts and comments are welcome on twitter at <a href="https://twitter.com/cherkaskyb">@cherkaskyb</a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=e968eb5d0e62" width="1" height="1" alt="">]]></content:encoded>
        </item>
    </channel>
</rss>