<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Pavel Kilovatiy on Medium]]></title>
        <description><![CDATA[Stories by Pavel Kilovatiy on Medium]]></description>
        <link>https://medium.com/@pavelkilovatiy?source=rss-5da964da12da------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/0*L-K8l3GzQ3wKkD81.jpg</url>
            <title>Stories by Pavel Kilovatiy on Medium</title>
            <link>https://medium.com/@pavelkilovatiy?source=rss-5da964da12da------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Wed, 18 Nov 2026 00:49:35 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@pavelkilovatiy/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[How to make technical debt everyone’s priority]]></title>
            <link>https://medium.com/hacking-talent/how-to-make-technical-debt-everyones-priority-316b28e619e3?source=rss-5da964da12da------2</link>
            <guid isPermaLink="false">https://medium.com/p/316b28e619e3</guid>
            <category><![CDATA[engineering]]></category>
            <category><![CDATA[agile]]></category>
            <category><![CDATA[technical-debt]]></category>
            <category><![CDATA[prioritization]]></category>
            <category><![CDATA[engineering-mangement]]></category>
            <dc:creator><![CDATA[Pavel Kilovatiy]]></dc:creator>
            <pubDate>Thu, 06 Jan 2022 13:43:30 GMT</pubDate>
            <atom:updated>2022-01-06T15:33:43.576Z</atom:updated>
            <content:encoded><![CDATA[<p><em>Addressing technical debt should not be put on the back burner. But in a busy workplace with competing deadlines, how can we give technical debt the attention it deserves?</em></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*uRaskre3nZ_3OPgLQga3HQ.jpeg" /></figure><p>One of the biggest challenges in growing a B2B software product business is prioritising technical debt. This is perhaps no surprise, given the constant pressure to deliver new features and generate new revenue. In this article, we’ll explore some of the ways that Beamery Engineering prioritises technical work when the workplace becomes a whirlwind of competing pressures.</p><p>Disclaimer: my experience is based on delivering B2B SaaS products in eCommerce/multi-variate testing and talent management categories over the last 20 years.</p><h3>TLDR</h3><ol><li>You need a sales pitch. Focus on the speed of value release, or a measurable reduction in maintenance costs, in order to get buy-in from your main stakeholders.</li><li>You should formulate a hypothesis and have operational evidence to support it (especially for costly preventative maintenance or large refactoring). Focus on acquiring data points (i.e product/service observability) if evidence is lacking</li><li>Prioritise product and engineering backlog during the same planning session &amp; align tech debt (refactoring/automation/observability/NFR testing) to product priorities as much as you can.</li></ol><h3>What do we mean by tech debt?</h3><p>Before we get stuck in, it’s important to explain what exactly we’re referring to. We’ll be looking at tech debt that originates from features put on ‘life-support’ by the organisation causing either regular or occasional disruptions to product delivery. We’ll also cover tech debt that arises from features that are dependent on tightly coupled services / components.</p><p>Moreover, there are other types of technical debt originating from such as evolved requirements; CI/CD inefficiencies; lack of testing automation, both end-to-end functional and non-functional testing such as performance, load, stress, burnout or security; and ‘short-term’ debt accumulated due to the need to validate value hypotheses or get customers live.</p><h3>The tech debt problem</h3><p>In the flurry of day-to-day demands, engineers are the invisible heroes who keep the well-oiled machine ticking along. But the problem with being invisible is that it’s hard to show the value of your work.</p><p>Technical debt has been covered in <a href="https://martinfowler.com/bliki/TechnicalDebt.html">great detail</a> over the last few years, and there is no shortage of advice on the importance of ‘paying the capital,’ not just ‘servicing the tech debt.’ But what’s less obvious is how to prioritise, or ‘sell’, the tech debt to the wider organisation. This can be a significant challenge, especially for B2B software companies experiencing sustainable revenue growth of 2x (or higher) year on year, and where the cost of quarterly prioritisation decisions is measured in hundreds of thousands of dollars per <a href="https://www.theguardian.com/technology/2018/apr/24/the-two-pizza-rule-and-the-secret-of-amazons-success">two-pizza sized team</a>.</p><h3>It’s a sales pitch</h3><p>Consider tech debt prioritisation as a sales pitch to the business. Not every engineer is a natural salesperson, but you don’t need a pinstripe suit and clipboard in order to make your case effectively. Instead, think about leading with a problem statement, the language you use, and be empathetic to other users with a stake in prioritisation, including end-users! Specifically, avoid using technical terms, and remember to explain abbreviations where possible.</p><p>We all have competing priorities in the workplace, and it’s understandable that commercially minded users don’t always recognise the principle of ‘de-couple services, improve architecture’. To flip this on its head, why not focus on speed of value delivery or the cost of reactive work ) to smoothen out your sales pitch?</p><p>For example, one of the challenges we had at Beamery was to break the legacy monolith that drives the longevity of the major release cycle (3 months) due to the expensive regression cycle (2 weeks) for features, depending on the monolith. For comparison, for features using an independent feature release cycle, we ship value bi-weekly, or as in the case of the Horizon platform, daily, using Continuous Delivery.</p><p>We knew we should disassemble monoliths as they are the source of cross-team execution dependency, not to mention an expensive regression suite. But the scope of doing so would be substantial, and providing an accurate estimate to decompose all services deployed inside the monolith is very time consuming, with uncertain outcomes. At Beamery, the approach we took was to retire the monolith by extracting each feature one by one and accelerating the value for such features from months to weeks/days.</p><p>This is where you need a Product Manager as well as operational data.</p><h3>Getting buy-in from the Product Manager</h3><p>Product managers (who care) understand the importance of fast value release, which shortens the feedback loop. If the Product Manager does not jump on this, you’re in trouble!</p><p>At Beamery, we’re focusing on reducing the release cycle across our feature estate, which means shrinking our release cycle from quarterly to bi-weekly (or faster) for features under active development.</p><p>In the process we defined clear feature ownership boundaries and aimed to reduce cross-team execution dependencies as much as possible.</p><p>For more background reading, check out the definition of <a href="https://medium.com/@pavel.kilovatiy/independent-feature-releases-definition-of-done-3ca2201ba130">Independent Feature Releases</a>, which can be used as one of the key initiatives to drive architecture evolution and autonomous execution.</p><h3>Getting buy-in from senior stakeholders</h3><p>This is more challenging but with the right preparation, is certainly doable.</p><p>You can help your cause by bringing clear problem statements, factual evidence to support your hypothesis (more on this below), as well as an estimated timeline resources needed, and impact on value release — ideally both short term (0–3 months) and long term (3 months +). It’s a good idea to reduce timeline granularity (i.e. use months or quarters) and use time ranges proportionate to the level of uncertainty you face. Familiarity with the <a href="https://en.wikipedia.org/wiki/Cone_of_Uncertainty">Cone of Uncertainty</a> concept might help you explain your estimate rationale.</p><p>If technical uncertainty is too high and you’re unable to commit to a reasonable timeline, one tactic is to give a ‘date for a date’. In other words, the date by which you anticipate you can agree to another date. Rather than kicking things into the long grass, it’s important to understand that the only output of the ‘date for a date’ exercise must be a delivery plan which the team can commit to.</p><h3>Formulate a hypothesis (and have operational evidence to support it)</h3><p>You can’t argue with the facts, even if some people try! Collecting evidence is a fundamental part of prioritising any technical debt. Often, production incidents trigger a strong emotional response from engineers, especially if the occurrence of a similar incident is less than quarterly. Sounds familiar? Yet we can’t rely on memory alone, so our best friends here are the facts recorded in production logs, incident management systems and post-mortems.</p><p>Start with formulating a hypothesis. What factors contributed to the incident? Was it a change in user behaviour due to a recent feature release? A growing number of concurrent users during a specific time interval? A change in underlying user data distribution? There can be a lot of hypotheses, yet investing in detailed incident/bug analysis and brainstorming during post-mortems gives engineers the best ROI on their time. Be careful not to cut corners here!</p><p>Bear in mind that your Product Manager may ask how likely it is that a similar type of incident will reoccur, particularly if it will take several weeks or more to estimate the remediation cost. The best way to assess the likelihood of this is to have operational data points (especially if this is the first occurrence, or if an incident is rare).</p><p>If you lack operational evidence (such as response time distribution, throughput distribution, error rates distribution) then before scheduling any remediation work, you should invest in observability for a given feature or set of components — below you’ll find more details on where to look for evidence.</p><p>If it’s still difficult to assess the recurrence likelihood it may be worth asking if your business or customers can afford to suffer from the same issue again, and estimate the impact.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*GYZoz_IVFXlNkKWW89zmJA.jpeg" /></figure><h3>Where to look for data points</h3><p><strong>Why production data matters</strong></p><p>Production data (such as data volumes, growth rate, throughput and performance) should be the first port of call if you’re looking to tackle backlog items related to service operations.</p><p>What’s more, production data can help with prioritisation (even if incident velocity is slow or inconclusive). What is the severity of the problem? Does it cause downtime? How many tenants are impacted? What was the service time to recovery? What is the total customer revenue impacted? Estimating incident impact on customers can be very persuasive.</p><p>Please also see <a href="https://medium.com/hacking-talent/beamery-engineering-operational-ownership-52f7b9d70455">Operational Ownership</a> article to see how Beamery implements service observability to acquire production data.</p><p><strong>Using Issue Tracking System</strong></p><p>An incident or bug tracker system, such as JIRA, is another source of data you can implement to support tech debt prioritisation.</p><p>It’s important to highlight trends regarding the frequency of incidents, the severity, and impact on other Sprint commitments (issues lead and cycle times metrics can be used to quantify impact on productivity), which can help to justify prioritisation decisions. This is another way to prioritise engineering work that is not aligned with product priorities.</p><p>Clearly, your case will be as accurate as the data in the tracking system, so it’s critical for engineers to have good tracking system ‘hygiene’. This means you should always track incidents, ensure the correct issue has been allocated for each feature or component, issue status transitions or the resolution status, and make sure the affected version and fix version attributes are accurate.</p><p>Bear in mind that each major severity incident requiring application fix likely to cost company at least 1 man week or more, not to mention cost of context switching which equates to $2&#39;000 on the optimistic side.</p><p><strong>Process &amp; CICD Metrics</strong></p><p>Use build, testing and deployment timings, and failure rates, as key evidence to justify the investment needed into the automating R&amp;D manufacturing line or testing. You also need deployment to production as key validation of faster value release.</p><p>Such investment metrics help address engineering waste and can impact productivity, so you would also need to track process flow metrics like lead time, cycle time and throughput.</p><h3>Make technical debt visible in the backlog</h3><p>Is your Product Manager on board by now? Great. Next, technical debt needs to be converted into an executable plan with defined milestones (as scale dictates). For each milestone, there should be stories written and added to the team backlog in order to figure out the size (or prioritisation) of each task.</p><h3>Prioritise product and engineering backlog as part of the same planning session</h3><p>Avoid separate planning sessions for product and engineering backlogs. Why? There is a natural tendency to maximise resources towards feature delivery, which makes any engineering backlog item a ‘scope creep’ by definition, when it should be part-and-parcel of the planning. The inverse is equally true — if you start with the engineering backlog prioritisation session, you would not have any capacity left for product deliverables.</p><h3>Tactics: Align technical debt with product priorities</h3><p>You should aim to align tech debt work with current product priorities. For example, you should include the cost of needed refactoring for services and components that constitute a prioritised feature.</p><p>It’s much easier to negotiate extended delivery timelines (provided it’s backed up by operational evidence) for such features compared to reworks that are perceived to be a lesser priority.</p><h3>Tactics: Focus on objectives, not % of time allocation</h3><p>Often, teams tend to block a percentage of time to work on technical items. But it’s arguably better to formulate 1–3 OKRs (stack ranked) for the next planning cycle and derive a prioritised backlog from those objectives. This helps to focus minds and also prevents attempts to claw back allocated time.</p><h3>Tactics: Avoid the temptation to generalise the solution</h3><p>When an engineer tries to gain support for their point of view (or solution), one achilles heel is the tendency to generalise the solution to a particular problem. Sometimes the engineer simply assumes that everyone will recognise the benefits of the solution, and doesn’t make the case effectively. Frequently, the audience loses interest, and worse, they may not clearly understand what the engineer means.</p><p>To counter this, try to be as specific with the problem and solution as you can, and use data points to support the importance or scale of the problem you want to solve.</p><h3>Conclusion</h3><p>Time is the only currency any startup/scale-up really has. And engineering time is perhaps the most important (certainly the most expensive) commodity of all. This is true for any business experiencing sustained growth over a prolonged period of time, where engineering time is often the most constrained resource.</p><p>So, when you need to prioritise any work driven by engineering needs, do not expect business just to follow your gut feel or strong conviction. Try and see where your commercially-minded colleagues are coming from, and once you strike the right balance, they’ll be forever in your (technical) debt.</p><h3>Interested in joining our Engineering, Product &amp; Design Team?</h3><p>We’re looking for Software Engineers Front &amp; Back Mid/Senior, SRE Platform Engineers, Engineering Managers, Tech Leads, Product Operations/Managers/Designers across all levels and even more roles across London, Remote &amp; Berlin! to join us — <a href="https://careers.beamery.com/global-teams/engineering-product-and-design/?utm_medium=Organic&amp;utm_source=MediumBlog&amp;utm_campaign=PavelMediumBlog&amp;utm_term=Careers&amp;utm_content=January6">apply here</a>!</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=316b28e619e3" width="1" height="1" alt=""><hr><p><a href="https://medium.com/hacking-talent/how-to-make-technical-debt-everyones-priority-316b28e619e3">How to make technical debt everyone’s priority</a> was originally published in <a href="https://medium.com/hacking-talent">Beamery Hacking Talent</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Independent Feature Release — Definition Of Done]]></title>
            <link>https://pavelkilovatiy.medium.com/independent-feature-release-definition-of-done-3ca2201ba130?source=rss-5da964da12da------2</link>
            <guid isPermaLink="false">https://medium.com/p/3ca2201ba130</guid>
            <category><![CDATA[engineering]]></category>
            <dc:creator><![CDATA[Pavel Kilovatiy]]></dc:creator>
            <pubDate>Thu, 25 Nov 2021 13:18:03 GMT</pubDate>
            <atom:updated>2021-11-25T13:18:03.418Z</atom:updated>
            <content:encoded><![CDATA[<h3>Independent Feature Release — Definition Of Done</h3><h3>Brief</h3><p>An Independent feature release is a strategic initiative at Beamery Engineering. It allows us to manage ever-growing technology complexity, evolve platform architecture, and tackle technical debt.</p><p>The main advantage of such an approach is the easier articulation of technology evolution through value release speed instead of architecture evolution/service decomposition. Architecture evolution should come as a natural artefact of</p><p>(a) Fast value release</p><p>(b) Spawning a new team when the cognitive load overwhelms the team and hinders team output (team-first approach to architecture evolution, pls see <a href="https://teamtopologies.com/">https://teamtopologies.com/</a> for more details)</p><h3>Definitions</h3><blockquote><strong><em>Product Feature Boundary</em></strong> — full set of services and components required for a feature to function</blockquote><blockquote><strong><em>Team Boundary</em></strong><em> — </em>set of services and components team owns</blockquote><blockquote><strong><em>Service Boundary</em></strong><em> — </em>set of external interfaces exposed by service to clients and upstream dependencies</blockquote><h3>IFR — Definition of Done</h3><p>Following conditions should be met for a <em>Product Feature </em>to be independently releasable:</p><p>Feature (or service ) is independently released if and only if all of the following conditions met:</p><ol><li><a href="https://medium.com/@pavel.kilovatiy/feature-boundary-diagram-f7a84788d509"><em>Product Feature Boundary</em></a> is clearly defined and documented.</li><li><em>Product Feature Boundary</em> is a subset of a <em>Team Boundary </em>OR (if <em>Product Feature Boundary</em> overlaps multiple <em>Team Boundaries) </em><strong>team interactions abstracted </strong>through <strong>API contracts</strong></li><li><em>Product Feature </em>can be built, configured, deployed, tested, and enabled independently of other product capabilities.</li><li><em>Product Feature</em> has a feature flag and can be enabled in a controlled fashion for selected tenants or across multiple tenants for selected user groups</li></ol><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=3ca2201ba130" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Beamery Engineering — Operational Ownership]]></title>
            <link>https://medium.com/hacking-talent/beamery-engineering-operational-ownership-52f7b9d70455?source=rss-5da964da12da------2</link>
            <guid isPermaLink="false">https://medium.com/p/52f7b9d70455</guid>
            <category><![CDATA[operational-management]]></category>
            <category><![CDATA[software-development]]></category>
            <category><![CDATA[engineering]]></category>
            <category><![CDATA[operational-excellence]]></category>
            <category><![CDATA[software-engineering]]></category>
            <dc:creator><![CDATA[Pavel Kilovatiy]]></dc:creator>
            <pubDate>Mon, 08 Feb 2021 09:40:11 GMT</pubDate>
            <atom:updated>2021-02-10T13:17:28.977Z</atom:updated>
            <content:encoded><![CDATA[<h3>Beamery Engineering — Operational Ownership</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*mpyrgqwMjfclV2oN1U2VIA.jpeg" /><figcaption>Photo by <a href="https://unsplash.com/@lukechesser?utm_source=unsplash&amp;utm_medium=referral&amp;utm_content=creditCopyText">Luke Chesser</a> on <a href="https://unsplash.com/s/photos/monitor-dashboard?utm_source=unsplash&amp;utm_medium=referral&amp;utm_content=creditCopyText">Unsplash</a></figcaption></figure><h3>What does Operational Ownership Mean?</h3><p>Operational Ownership is a critical principle in Beamery Engineering. It means that a delivery team is able to be fully aware of the current production state for a service or component, allowing the team to be accountable for our operational performance and issue troubleshooting.</p><p><em>Hopefully, this article enables your team to define your roadmap and acquire operational awareness and ownership of a feature by providing guidance and a </em><strong><em>definition of done (DoD)</em></strong><em>.</em></p><p>The dashboards below are the outcome of evolving our operational ownership principles in our Enigma delivery team — they show the operational state of the Beamery Serverless Horizon platform (full operational coverage is much wider). Horizon is an integration layer between Beamery Core SaaS and Application Tracking Systems (such as WorkDay/SAP/Greenhouse). The dashboards are used as the primary source of operational awareness by Enigma on a 24x7 basis.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*uLNq6Camq50x2v5K99j1sA.png" /><figcaption><em>Beamery Serverless Horizon Platform — data processing success rate/latency. You can see </em><a href="https://status.cloud.google.com/incident/container-engine/21001"><em>here</em></a><em> the Horizon incident caused by this GKE issue</em></figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*oLjILqUx08nNpS3s7BsysQ.png" /><figcaption><em>Beamery Serverless Horizon Platform — data entity distribution per tenant</em></figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*HXrEJdBvMWhaaKx7vH7IxQ.png" /><figcaption><em>Beamery Serverless Horizon Platform — ETL errors distribution</em></figcaption></figure><h3>So, why should a team aspire to achieve Operational Ownership?</h3><ul><li>It demonstrates a team’s focus on the business outcome (i.e., how feature adopted/performs) as opposed to operational output (i.e., how frequently we deploy to prod)</li><li>It demonstrates a team’s empathy for a customer’s experience.</li><li>It increases team independence and autonomy while enhancing the visibility of useful metrics for data-driven decision making.</li><li>It’s a relatively uncommon and valuable skill to acquire.</li><li>It helps teams to identify and address issues quicker.</li><li>The team that develops the product/feature knows how to best monitor it.</li><li>Teams can gather valuable data and present it in a consumable, human-readable format to express a feature&#39;s value to the business using numbers.</li></ul><h3>How can my team use this write up?</h3><p>Your team can use the following to define its own path to operational ownership maturity. For each feature, this document provides a “<strong>definition of done</strong>”, helping you to build out your roadmap and gain Operational Ownership.</p><p>The following questions may also help to create a roadmap:</p><ol><li>Do we have team buy-in that Operational Ownership is important for feature X? If yes — why is feature X prioritised ahead of feature Y?</li><li>Are we doing this today for components/services of which feature X comprises?</li><li>Have we defined the “normal” operational parameters of the component/feature; have an Operational Dashboard showing service <em>telemetries</em>? How do we know the service is operational and performs as expected?</li><li>What needs to be true for our team to become operationally aware?</li><li>Do we have a skill set in the team for instrument monitoring/alerting/escalation? What is missing? How do we acquire such skills? Do we ask the Platform team to do this for us?</li><li>Do we have access to the necessary tools and production services/monitoring systems?</li><li>Do we have the capacity to staff 24x5/24x7 rota?</li><li>How do we ingrain operational service state reviews into team rituals so that it becomes a habit?</li></ol><p>Our recommendation is to start with services/components known to produce regular operational disruptions, as this is where ROI may be higher. However equally, all-new features need to be developed with observability from the get-go.</p><h3>Operational Ownership — Definition of Done (scope: production environment)</h3><h3>Service telemetries logged and monitored.</h3><ol><li>Performance metrics such as response time</li><li>Current throughput expressed in requests per second</li><li>Any other suitable metric (i.e., data/traffic volume processed). “Suitable” implies information useful to spot potential problems/irregularities with service behaviour/performance.</li></ol><h3>Service error rates logged and monitored.</h3><ol><li>HTTP or gRPC service errors rates</li><li>Internal service errors (i.e., validation) and arrival rates</li></ol><h3>Service logs reviewed (i.e., daily) for new errors and rates.</h3><ol><li>A habit of reviewing the state of production services within the zone of Ownership ingrained in the team’s daily/weekly routines</li><li>Service logs contain sufficient and relevant information for problem troubleshooting.</li><li>Service log verbosity ensures no operational fatigue/ignorance developed due to false-positive entries/volume of log information.</li></ol><h3>The altering and escalation process managed by the team &amp; 24x5 or 24x7 support rota operational</h3><ol><li>Service warning/critical thresholds are defined and alerted on</li><li>Clear escalation path defined &amp; operational for all alert types.</li><li>Easy to find information which person is currently on operational duty within a team at any point in time given agreed support schedule (24x5 / 24x7)</li></ol><h3>The team uses operational knowledge to drive the evolution of service architecture/work prioritization.</h3><ol><li>Service operational state used to project future performance jointly with business plans (i.e., revenue growth)</li><li>Service operational state used to support prioritization decisions during release/iteration planning.</li><li>Service operational state used to support architectural choices.</li></ol><blockquote>Want to work alongside some great humans and inspiring leaders to solve big problems? Engineering is hiring! Click here to join the <a href="https://careers.beamery.com/teams/engineering-product-and-design/">#BeamTeam</a> and change the future of work.</blockquote><p><strong>Credits</strong>: special thank you, <a href="https://medium.com/u/8f6b91f5d7f">Kunal Golani</a>, <a href="https://medium.com/@tom.gallacher_57882">Tom Gallaher</a>, Marco Pierleoni, and <a href="https://medium.com/u/4b6c4ce3b1a9">Sultan Saidov</a>, for your contributions and feedback!</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=52f7b9d70455" width="1" height="1" alt=""><hr><p><a href="https://medium.com/hacking-talent/beamery-engineering-operational-ownership-52f7b9d70455">Beamery Engineering — Operational Ownership</a> was originally published in <a href="https://medium.com/hacking-talent">Beamery Hacking Talent</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Beamery Engineering — Operational Ownership]]></title>
            <link>https://pavelkilovatiy.medium.com/beamery-engineering-operational-ownership-cb13ccae64da?source=rss-5da964da12da------2</link>
            <guid isPermaLink="false">https://medium.com/p/cb13ccae64da</guid>
            <category><![CDATA[operational-excellence]]></category>
            <category><![CDATA[engineering]]></category>
            <category><![CDATA[operations]]></category>
            <category><![CDATA[beamery]]></category>
            <dc:creator><![CDATA[Pavel Kilovatiy]]></dc:creator>
            <pubDate>Wed, 03 Feb 2021 16:16:52 GMT</pubDate>
            <atom:updated>2021-03-08T15:53:00.293Z</atom:updated>
            <content:encoded><![CDATA[<h3>Beamery Engineering — Operational Ownership</h3><h3>What does Operational Ownership Mean?</h3><p>Operational Ownership is a critical principle in Beamery Engineering. It means that a delivery team is fully aware of the current production state for a service or component and accountable for operational performance and issue troubleshooting.</p><p><em>Hopefully, this blog article enables a team to define a roadmap to acquire operational awareness and ownership for a feature by providing guidance and definition of done.</em></p><p>The operational dashboards below are the outcome of operational ownership evolution in Enigma delivery team — these show operational state for the Beamery Serverless Horizon platform (full operational coverage is much wider). Horizon is an integration layer between Beamery Core SaaS and Application Tracking Systems (such as WorkDay/SAP/Greenhouse). The dashboards are used as the primary source of operational awareness by Enigma on a 24x7 basis.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*uLNq6Camq50x2v5K99j1sA.png" /><figcaption><em>Beamery Serverless Horizon Platform — data processing success rate/latency. You can see the Horizon incident caused by this GKE issue: </em><a href="https://status.cloud.google.com/incident/container-engine/21001">https://status.cloud.google.com/incident/container-engine/21001</a></figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*oLjILqUx08nNpS3s7BsysQ.png" /><figcaption><em>Beamery Serverless Horizon Platform — data entity distribution per tenant</em></figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*HXrEJdBvMWhaaKx7vH7IxQ.png" /><figcaption><em>Beamery Serverless Horizon Platform - errors distribution</em></figcaption></figure><h3>Why should a team aspire towards Operational Ownership?</h3><ul><li>It demonstrates the team’s focus on the business outcome (i.e., how feature adopted/performs) as opposed to operational output (i.e., how frequently we deploy to prod)</li><li>It demonstrates the team’s empathy for a customer’s experience.</li><li>Increases team independence and autonomy while enhancing the visibility of useful metrics for data-driven decision making.</li><li>It’s a relatively uncommon and valuable skill to acquire.</li><li>It helps a team to identify and address issues quicker.</li><li>The team that develops the product/feature knows best how to monitor it.</li><li>The team gathers valuable data and present it in a consumable human-readable format to express feature value to the business using numbers.</li></ul><h3>How can my team use this write up?</h3><p>A team can use the following guidelines to define a path to achieve operational ownership maturity. For each feature, this document provides a “definition of done” (DoD) and can help build a roadmap to gain Operational Ownership.</p><p>The following questions may also help to create a roadmap:</p><ol><li>Do we have team buy-in that Operational Ownership is important for feature X? If yes — why is feature X prioritised ahead of feature Y?</li><li>Are we doing this today for components/services of which feature X comprises?</li><li>Do we defined the “normal” operational parameters of the component/feature; have an Operational Dashboard showing service <em>telemetries</em>? How do we know the service is operational and performs as expected?</li><li>What needs to be true for our team to become operationally aware?</li><li>Do we have a skill set in the team for instrument monitoring/alerting/escalation? What is missing? How do we acquire such skills? Do we ask the Platform team to do this for us?</li><li>Do we have access to the necessary tools and production services/monitoring systems?</li><li>Do we have the capacity to staff 24x5/24x7 rota?</li><li>How do we ingrain operational service state reviews into team rituals so that it becomes a habit?</li></ol><p>The recommendation is to start with services/components known to produce regular operational disruptions, as this is where ROI may be higher. But equally, all-new features need to be developed with observability from the get-go.</p><h3>Operational Ownership — Definition of Done (scope: production environment)</h3><h3><strong><em>Service telemetries logged and monitored</em></strong>.</h3><ol><li>Performance metrics such as response time</li><li>Current throughput expressed in requests per second</li><li>Any other suitable<em>*</em> metric (i.e., data/traffic volume processed).</li></ol><p>*<em>“Suitable” implies information useful to spot potential problems/irregularities with service behaviour/performance.</em></p><h3><strong><em>Service error rates logged and monitored</em></strong>.</h3><ol><li>HTTP or gRPC service errors rates</li><li>Internal service errors (i.e., validation) and arrival rates</li></ol><h3><strong><em>Service logs reviewed (i.e., daily) for new errors and rates.</em></strong></h3><ol><li>A habit of reviewing the state of production services within the zone of Ownership ingrained in the team’s daily/weekly routines</li><li>Service logs contain sufficient and relevant information for problem troubleshooting.</li><li>Service log verbosity ensures no operational fatigue/ignorance developed due to false-positive entries/volume of log information.</li></ol><h3><strong><em>The altering and escalation process managed by the team &amp; 24x5 or 24x7 support rota operational</em></strong></h3><ol><li>Service warning/critical thresholds are defined and alerted on</li><li>Clear escalation path defined &amp; operational for all alert types.</li><li>Easy to find information which person is currently on operational duty within a team at any point in time given agreed support schedule (24x5 / 24x7)</li></ol><h3><strong><em>The team uses operational knowledge to drive the evolution of service architecture/work prioritization</em></strong>.</h3><ol><li>Service operational state used to project future performance jointly with business plans (i.e., revenue growth)</li><li>Service operational state used to support prioritization decisions during release/iteration planning.</li><li>Service operational state used to support architectural choices.</li></ol><p>ps Beamery Engineering is hiring — join our crew here: <a href="https://careers.beamery.com/jobs/?department=Engineering&amp;location=#results">https://careers.beamery.com/jobs/?department=Engineering</a></p><p><strong>Credits</strong>: special thank you, <a href="https://medium.com/u/8f6b91f5d7f">Kunal Golani</a>, <a href="https://medium.com/@tom.gallacher_57882">Tom Gallaher</a>, Marco Pierleoni, and <a href="https://medium.com/u/4b6c4ce3b1a9">Sultan Saidov</a>, for your contributions and feedback!</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=cb13ccae64da" width="1" height="1" alt="">]]></content:encoded>
        </item>
    </channel>
</rss>