<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Dimitri Koubaroulis on Medium]]></title>
        <description><![CDATA[Stories by Dimitri Koubaroulis on Medium]]></description>
        <link>https://medium.com/@dimkoubaroulis?source=rss-d4e647a1ff06------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/1*cg8jg-A45aZgFkKvwER0BA.png</url>
            <title>Stories by Dimitri Koubaroulis on Medium</title>
            <link>https://medium.com/@dimkoubaroulis?source=rss-d4e647a1ff06------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Sun, 22 Nov 2026 10:24:16 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@dimkoubaroulis/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[PACE: a developer productivity framework]]></title>
            <link>https://medium.com/mitti-engineering/pace-a-developer-productivity-framework-7fab68eabaaa?source=rss-d4e647a1ff06------2</link>
            <guid isPermaLink="false">https://medium.com/p/7fab68eabaaa</guid>
            <category><![CDATA[software-engineering]]></category>
            <category><![CDATA[team-collaboration-tool]]></category>
            <category><![CDATA[developer-productivity]]></category>
            <category><![CDATA[leadership]]></category>
            <dc:creator><![CDATA[Dimitri Koubaroulis]]></dc:creator>
            <pubDate>Fri, 23 Sep 2022 01:11:56 GMT</pubDate>
            <atom:updated>2022-09-23T02:46:23.343Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*w491aF5uWImuhQfT.png" /></figure><p>Developer productivity engineering (DPE) can mean different things to different people. Although definitions vary across organisations, there is a permeating theme across all of them — building better software, faster, in a sustainable way.</p><p>Since the mid-2010s, world class software organisations of the likes of Google, Netflix and Microsoft, all have elevated DPE to be a first class citizen in their organisation structure. They did that because they recognise DPE as a key contributor to organisational success, directly linked to both business outcomes and job satisfaction. The size of their investment is indicative of the importance of the function. For example, <a href="https://www.youtube.com/watch?v=PqgbJSgvqj0">Netflix</a> at a time had 120 DP engineers supporting a total 2000 engineering staff. <a href="https://www.youtube.com/watch?v=pjNMPmdcLh4">LinkedIn</a> went even further, a ratio of 1 DPE to every 10 engineers.</p><p>At SafetyCulture, engineering leaders place a great emphasis on developer productivity. At the time of writing in 2022, there has not been a greater need for continuous improvement in the productivity space as the company scales rapidly and with the additional complexities of our hybrid ways of working.</p><p>Earlier this year, we introduced the PACE developer framework at SafetyCulture with the following main aims:</p><ol><li>The publication of a handful of productivity metrics that align well with the engineering priorities of SafetyCulture at this point in time. We wanted these to be based on seminal industry frameworks such as DORA and SPACE.</li><li>A cultural shift within engineering that makes productivity metrics available to and a concern of everyone in engineering, not just managers. Decoupling it from performance management.</li><li>Tooling &amp; guidance to make it easier for SafetyCulture teams to measure and optimise developer productivity consistently, learning from each other.</li></ol><blockquote><em>A cultural shift within engineering that makes productivity metrics available to and a concern of everyone in engineering, not just managers</em></blockquote><p><strong>Who is PACE for?</strong></p><p>Any engineering team that delivers software. Firstly for engineers. We wanted everyone at SafetyCulture to understand how investing in the chosen productivity metrics can help them have more good, productive days at work.</p><p>Secondly, for engineering managers. When everyone understands how to measure their team’s productivity in a way that is consistent with the rest of the organisation, it becomes easier to optimise it.</p><p><strong>What is PACE not?</strong></p><ul><li>A software application that you can download and install. You will have to implement it (like we did) or configure similar metrics in a paid tool like <a href="https://linearb.io/">LinearB</a> or <a href="https://www.multitudes.co/">Multitudes</a>.</li><li>A performance management framework. Team performance necessarily includes business outcomes which are outside the scope of the PACE framework. On top of that, people understandably dislike ‘being tracked’. To avoid any misunderstandings, the metrics we chose are intentionally team-level, rather than individual ones.</li><li>A way to compare teams. The aim is for teams to learn from each other, rather than compete.</li></ul><p><strong>The PACE metrics</strong></p><p>PACE is an adaptation of the SPACE framework (2021) which in turn is an extension of <a href="https://services.google.com/fh/files/misc/state-of-devops-2019.pdf">DORA framework</a> (2019).</p><p>Many world-class organisations use variants of the four so-called DORA metrics:</p><ul><li>Cycle time</li><li>Deployment frequency</li><li>Mean time to recover (from failure)</li><li>Change failure rate</li></ul><p>A 2021 extension of the DORA framework, led by some of the people who developed DORA is the <a href="https://queue.acm.org/detail.cfm?id=3454124">SPACE framework</a>. SPACE is a more holistic set of quantitative and qualitative productivity indicators that adds the dimensions of collaboration, communication and wellbeing to the four DORA metrics.</p><ul><li><strong>S</strong>(atisfaction &amp; wellbeing)</li><li><strong>P</strong>(erformance)</li><li><strong>A</strong>(ctivity)</li><li><strong>C</strong>(ommunication &amp; collaboration)</li><li><strong>E</strong>(fficiency &amp; flow)</li></ul><p>A team may choose to track a subset of the 5 SPACE areas. A minimum of 3 areas is recommended. The organisation can decide on the exact metrics to invest in independently depending on what is important for the organisation at the time. SPACE is already used by organisations such as Netflix and GitHub (listen to them talk about it <a href="https://techbeacon.com/app-dev-testing/continuous-improvement-metrics-lessons-6-software-teams">here</a>).</p><p>Adapting SPACE for SafetyCulture, we formed PACE by choosing the following 4 areas that could be easily automated:</p><ul><li><strong>P</strong>erformance (change failure rate, a function of raised defects and incidents over all production deployments in the reviewed period)</li><li><strong>A</strong>ctivity (number of production deployments)</li><li><strong>C</strong>ollaboration (time to first code review comment)</li><li><strong>E</strong>fficiency (median cycle/elapsed time of a JIRA ticket from in progress to done)</li></ul><p>It is easy to tweak these metrics for your organisation. For instance, you may care more about the time to merge a core review than the time to first comment.</p><p>To visualise these metrics, we collect relevant data from JIRA and GitHub for the past 7 days and make them available to everyone in Engineering using a <a href="https://grafana.com/">Grafana</a> dashboard.</p><blockquote><em>It is easy to tweak these metrics for your organisation</em></blockquote><p>Here is an example snapshot of the PACE dashboard for one of our teams:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Pp5sLH09ZHDWm9NoNwy9sw.png" /><figcaption>A team’s PACE dashboard</figcaption></figure><p>Additionally, the dashboard provides a helpful breakdown of each PACE area of interest. For example, all JIRA tickets and GitHub code changes that resulted in the reported production deployments under Activity. This allows anyone to drill down for more insights. Here is an example:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*YfzVZya1AkrM0sjI.png" /><figcaption>A drill-down view of all production deployments in the last 7 days</figcaption></figure><p>Each team agrees on what metric range represents “healthy” for them. We strive for consistency, except where special circumstances apply. My team reviews the PACE dashboard on a weekly basis at our planning session and we ensure the weekly plan takes action where required for a healthy PACE.</p><blockquote><em>we ensure the weekly plan takes action where required for a healthy PACE</em></blockquote><p><strong>Some key learnings</strong></p><p>A fully automated PACE dashboard that is readily available to everyone in the organisation is a big time saver and a great way to encourage consistency where appropriate.</p><p>We have not seen gaming of PACE metrics in a year at SafetyCulture. We think this is due to the transparency and a good understanding of the benefits of PACE by all engineers.</p><p>When metrics go into unhealthy territory we discuss and adjust our plan as a team. For example, we may decide to focus on resolving more defects than usual this week, as last week there was a spike. Similarly, we take action to resolve roadblocks when cycle time goes out of range.</p><p>We found the “change failure rate” to be the trickiest metric to formulate and make sense of. Some of the defects raised are a result of historical changes and not of the deployments that happened in the reviewed period. It is still valuable as it is though, and we will keep looking for ways to improve it in future. Do send us any tips or suggestions!</p><p><em>A very special thanks to the the world class engineering team at SafetyCulture that made PACE a reality!</em></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=7fab68eabaaa" width="1" height="1" alt=""><hr><p><a href="https://medium.com/mitti-engineering/pace-a-developer-productivity-framework-7fab68eabaaa">PACE: a developer productivity framework</a> was originally published in <a href="https://medium.com/mitti-engineering">Mitti Engineering</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[3 easy team delivery metrics with ActionableAgile]]></title>
            <link>https://medium.com/mitti-engineering/3-easy-team-delivery-metrics-with-actionableagile-ce02ef9c1209?source=rss-d4e647a1ff06------2</link>
            <guid isPermaLink="false">https://medium.com/p/ce02ef9c1209</guid>
            <category><![CDATA[agile]]></category>
            <category><![CDATA[engineering]]></category>
            <category><![CDATA[delivery]]></category>
            <category><![CDATA[metrics]]></category>
            <category><![CDATA[productivity]]></category>
            <dc:creator><![CDATA[Dimitri Koubaroulis]]></dc:creator>
            <pubDate>Tue, 17 Aug 2021 09:13:05 GMT</pubDate>
            <atom:updated>2021-08-17T09:13:05.151Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*TgepyIhQRpwI2zcsFUQi4A.jpeg" /><figcaption>Photo by <a href="https://unsplash.com/@erdaest?utm_source=unsplash&amp;utm_medium=referral&amp;utm_content=creditCopyText">Erda Estremera</a> on <a href="https://unsplash.com/s/photos/delivery-box?utm_source=unsplash&amp;utm_medium=referral&amp;utm_content=creditCopyText">Unsplash</a></figcaption></figure><p>In a previous post, I discussed how <a href="https://medium.com/safetycultureengineering/optimising-engineering-team-delivery-using-the-cycle-time-metric-2ba3a7c1c025">cycle time plays a key role in optimising the delivery health</a> of engineering teams at SafetyCulture. In this post, I will dive a little deeper into how we measure specific delivery metrics in minutes using JIRA and the <a href="https://marketplace.atlassian.com/apps/1216661/actionableagile-for-jira-agile-metrics?tab=overview&amp;hosting=cloud">ActionableAgile JIRA plugin</a>.</p><p>Three metrics we have been tracking this way are:</p><ul><li>Cycle time</li><li>Deployment frequency (customer-facing changes per week)</li><li>Deployment frequency (non-customer facing changes per week)</li></ul><p>Although the process described above is rather manual, once the initial setup is done it usually takes no more than a couple of minutes to get the metrics needed. Combined with JIRA’s and ActionableAgile’s flexibility around filtering, trend visualisation, and deep-dive insights, we have found this approach preferable compared to more automated but less flexible methods we have tried or the option of building and maintaining such tooling ourselves.</p><blockquote>it usually takes no more than a couple of minutes to get the metrics needed</blockquote><p>The process consists of a one-time setup step, followed by a simple metric visualisation step.</p><p>Part 1 — Setup</p><p><strong>Installing ActionableAgile</strong></p><p>We have purchased and installed the ActionableAgile JIRA plugin on our JIRA cloud subscription from <a href="https://marketplace.atlassian.com/apps/1216661/actionableagile-for-jira-agile-metrics">here</a>. Once installed, it automatically updates to the latest version.</p><p><strong>Setting up a cross-team JIRA board</strong></p><p>Our teams each use a different JIRA project and board. To be able to see metrics across different teams we have created a JIRA board with a special JQL filter that combines all relevant team projects. The filter looks like this:</p><blockquote>project in (TEAMA, TEAMB, TEAMC) AND labels not in (outlier) AND updated &gt;= -24w AND status was not in (“Won’t Do”) ORDER BY updated DESC, Rank ASC</blockquote><p><strong>Setting up the team’s project</strong></p><p>We want to exclude tickets closed as “Won’t Do”. “Next gen” JIRA projects do not support a “Won’t Do” resolution so we use JIRA Classic projects.</p><p>To be able to quickly see metrics from different teams on the same ActionableAgile dashboard we label every ticket with the team name. We do that so that we can then filter by this label later on. We give these labels consistent names like <em>inspections-team</em>, <em>actions-team</em> etc. so that it is visually easier to select them later on.</p><p>We automate adding the team label to all new tickets created by going to Project Automation in our JIRA project and adding a rule as shown in the image below:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*tdeHKBIMsCHvnzRJXO7nzQ.png" /></figure><p>For projects with an existing backlog, we also run a bulk issue update once to apply the team label to all existing tickets.</p><p><strong>Excluding particular types of work</strong></p><p>Sometimes we need to exclude certain types of work from the metrics calculation. For instance, tickets that capture a long-running background monitoring process or other non-delivery-related tickets for which the metrics described here are not suitable. We do that by manually labeling such tickets in JIRA with a label of our choice (we use “outlier”). This allows us to filter them out from the board later (as shown in the JIRA filter earlier).</p><p><strong>Dealing with “blocked time”</strong></p><p>When work in progress gets blocked for a non-trivial amount of time (say half a day) we add a flag and a comment to the JIRA ticket to indicate. Similarly, we remove the flag when the work can be resumed. This not only raises visibility of blockages on the JIRA board, it also allows us to control whether to include or exclude blockage time from the collected metrics. There are useful insights in either case. Note that even though we do not work on weekends we do not treat weekends as “blocked time” so that cycle time reported more closely resembles the customer experience.</p><p><strong>Classifying work</strong></p><p>By convention, we raise Story and Bug tickets for changes with direct customer-facing impact such as a UI change. Otherwise, we use Task. This helps collect metrics for customer-facing changes only as required.</p><p>Part 2— Tutorial</p><p>In the following section, I will walk you through how we answer some key delivery questions at SafetyCulture.</p><p>To get started:</p><p>On your JIRA project page, on the left-hand side menu panel click “ActionableAgile” to open the plugin. The ActionableAgile main page loads.</p><p>On the top right-hand corner, click the folder icon and “Load a board”.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/324/1*Nr4C5G2pkcRL6iLfCATv0Q.png" /></figure><p>Select the board you created earlier to combine all projects of interest. If there are too many tickets you can opt to only load tickets after a particular date to save on loading time.</p><p>Continue.</p><p>Select all ticket fields when prompted. You can trim fields if you would like to optimise the time it takes to load the dashboard in the future.</p><p>Continue.</p><p>On the right-hand side open the filter settings and:</p><ul><li>select the ticket types you want to track. We track Bug, Story, and Task tickets</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/202/1*SmIbiBOeTXpz9RCTOsRlZg.png" /></figure><ul><li>select your team’s label from the list of labels</li><li>select the “Done” item from the “Resolution” column to exclude “Won’t do”</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/848/1*YSHhO5EmMzfL1Jhm69-rpA.png" /></figure><p>Close the filter settings window to go back to the ActionableAgile main page.</p><p>On the right-hand menu panel:</p><ul><li>untick “todo” and/or “backlog” stage from the “Workflow stages” (unless you explicitly want to measure lead time instead of cycle time)</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/434/1*hgVI4HsmDNc1LcWhyxQP-A.png" /></figure><ul><li>choose whether to “include blocked time”</li><li>tick “Average (Beta)” to track average cycle time</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/362/1*nY0-l1CCTvpTYK4gAIKLeA.png" /></figure><p>NB. ActionableAgile does not refresh automatically each time the page is reloaded. To get the latest project data: Go to the “folder icon” at the top right and “Check for updates to &lt;your board’s name&gt;”</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/300/1*bG2zfYTf61gYT4YiQrPI9A.png" /><figcaption>Refreshing to get latest data</figcaption></figure><p>Now we are ready to collect our metrics.</p><p><strong>What was the average cycle time last week?</strong></p><p>From the bottom panel, drag to select the time period e.g. the last 7 days as required. The average cycle time will be shown as a dotted line on the scatterplot above. If not enough tickets were completed in the selected period, extend the time period to include earlier tickets until a reliable average can be computed.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*7dbL84De69oUb21gbumrhQ.png" /><figcaption>Cycle time for a selected date range</figcaption></figure><p><strong>How many customer-facing and non-customer-facing releases were made in the last 7 days?</strong></p><p>Here we assume that each ticket that is not labeled as an “outlier” (as described above) results in one deployment/release.</p><p>To find the number of customer-facing releases, on the right-hand side open the filter settings and:</p><ul><li>select the customer-facing ticket types: Bug and Story only</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/226/1*9z7duBdz-Rc8HhvgqCUrlg.png" /><figcaption>Customer-facing changes only filter</figcaption></figure><ul><li>open “Summary Statistics”</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/336/1*gE3WMZvHWlgBsqDyPnQnNA.png" /></figure><ul><li>see how many customer-facing releases were made in the chosen date range</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/432/1*sUfpfYeJxrpR2xmxZj7FTQ.png" /></figure><p>To see all releases, including non-customer-facing ones, repeat the same process above after you have also ticked the “Task” ticket type at the first step above.</p><p>If you want to review the releases individually in more detail, you can hover over the dots on the cycle time scatterplot and deep dive into each JIRA ticket.</p><p>I hope you found this guide useful. Let us know if you have any related tips or a better approach! Even better, come join us, <a href="https://safetyculture.com/jobs/">we are hiring</a>!</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=ce02ef9c1209" width="1" height="1" alt=""><hr><p><a href="https://medium.com/mitti-engineering/3-easy-team-delivery-metrics-with-actionableagile-ce02ef9c1209">3 easy team delivery metrics with ActionableAgile</a> was originally published in <a href="https://medium.com/mitti-engineering">Mitti Engineering</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Blameless estimation for healthy engineering teams]]></title>
            <link>https://medium.com/mitti-engineering/blameless-estimation-for-healthy-engineering-teams-52b483924ee9?source=rss-d4e647a1ff06------2</link>
            <guid isPermaLink="false">https://medium.com/p/52b483924ee9</guid>
            <category><![CDATA[team-culture]]></category>
            <category><![CDATA[engineering]]></category>
            <category><![CDATA[noestimates]]></category>
            <category><![CDATA[estimations]]></category>
            <category><![CDATA[project-management]]></category>
            <dc:creator><![CDATA[Dimitri Koubaroulis]]></dc:creator>
            <pubDate>Thu, 13 May 2021 23:52:12 GMT</pubDate>
            <atom:updated>2021-05-15T11:20:00.725Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*6NtSssfv62SdciCT" /><figcaption>Photo by<a href="https://unsplash.com/@carissaweiser?utm_source=unsplash&amp;utm_medium=referral&amp;utm_content=creditCopyText"> Carissa Weiser</a> on <a href="https://unsplash.com/s/photos/highway-block?utm_source=unsplash&amp;utm_medium=referral&amp;utm_content=creditCopyText">Unsplash</a></figcaption></figure><p>Estimation of engineering work plays an important role on what commitments a team makes and how to keep them in order to build and maintain trust towards a sustainable, healthy team culture. In this post I will talk about some lessons I’ve learnt about estimation in product engineering teams and the impact of the estimation culture on team health.</p><p>My position here is aligned with “agile” thinking: it is more important to manage expectations all along and to show business value being created regularly than to “hit” an estimated project date. Having said that, I strongly believe that trying to optimise for the former will help with the latter too. The opposite does not hold true, however.</p><blockquote>it is more important to manage expectations all along and to show business value being created regularly than to “hit” an estimated project date</blockquote><p><strong>Effects of unhealthy estimation culture on team health</strong></p><p>The problems I’ve seen with estimation in my career seem to have consistent themes. Engineers feeling frustrated when asked to give estimates on limited information, or worse when their “gut feel” estimates based on limited information are used as a delivery commitment. Product managers feeling “let down” by the team “missing targets” and losing trust in engineers when estimates are inaccurate (hint: they almost always are). Finally, engineering managers feeling pressured to hold engineers accountable for hitting particular dates, bringing down team morale, accepting undesirable trade-offs in product quality, increasing stress and the likelihood of burnout. I certainly have been in situations like this in the past, experienced from all sides of that story. If you are reading my post I am guessing you have too.</p><p>So how do we avoid that type of culture? What does healthy look like? Here is how we go about it at SafetyCulture.</p><p><strong>Avoiding pitfalls</strong></p><p>Let’s start with some definitions for context. An estimate is a guess, an educated one at best about how much time or effort a piece of work will take.</p><blockquote>An estimate is a guess</blockquote><p>The reliability of this guess depends heavily on whether</p><p>a) the domain and solution are well understood e.g. the team has done this work before</p><p>b) the environment (people, product, process &amp; dependencies) is stable for the duration of the project</p><p>c) the implied trade-offs of the project management triangle (scope, time, quality) are well understood and aligned by all stakeholders</p><p>These conditions are often not true in fast evolving, agile organisations working on complex problems. How can a team estimate a solution involving a technology they haven’t used before? What will be the velocity of people who haven’t yet started? etc. Teams typically assign “buffer” estimates to account for these unknowns but that’s trying to solve a problem by introducing another instance of the same problem.</p><p>If the solution to a problem is not known already at project kick off then estimating the solution at any level of confidence requires doing at least some part of the work itself. At SafetyCulture we call this a “spike” which can range from a few days to a few weeks of work. This will typically be a time-boxed effort with an option to extend its duration or pivot at any time as knowledge is created. Resist the urge to call this time-box an “estimate for an estimate”. The estimate is not the goal here. The goal is to learn what we need to learn in order to make valuable decisions as soon as possible. After 2–3 days or weeks of work it may well become apparent that continuing down the proposed direction does not make sense at all!</p><blockquote>The goal is to learn what we need to learn in order to make valuable decisions as soon as possible</blockquote><p>In the high performing teams I am privileged to work with at SafetyCulture, product managers understand innately that asking “what if” questions too frequently can disrupt the flow of the team. Similarly, engineers I work with don’t hesitate to provide effort size “gut feel” estimates when needed, ensuring that they include best and worst case scenarios where possible while recommending options and related experiments or spikes to validate hypotheses. In the past, for example, I helped steer the direction of a team away from an “a few weeks” option being considered by saying that solving a customer problem of that size would take 6 months or more. It took a lot more than 6 months in the end. However, the important thing here is that my “gut feel” estimate helped avoid wasting important time.</p><p><strong>Communicating a delivery date</strong></p><p>With the need for agility well understood, engineering managers still need to communicate a delivery date for their projects in order to manage business and customer expectations. How do we do that in environments with a high degree of uncertainty?</p><p>Engineering culture is very important here. First and foremost, build a culture where engineers feel comfortable providing “gut feel” guesses of the size of work without feeling like they will be “locked in” or “judged” for not delivering to those dates. Rely more on validating hypotheses early. Actively manage expectations on the delivery date of the project on a weekly basis without a sense of being “punished” for any delays or decisions to pivot.</p><blockquote>build a culture where engineers feel comfortable providing “gut feel” guesses of the size of work without feeling like they will be “locked in”</blockquote><p>Senior management plays a big part in fostering this culture. At SafetyCulture, “frontline” engineers see senior management as their extended team sharing both the accountability and the risk together with them. This closes the divide between “the team” and “management” and is very much aligned with SafetyCulture’s “better as a team” value.</p><p>At some point engineering managers only have “gut feel” estimates, but still need to communicate a date. We have the option to spend time doing extensive requirements analysis and comprehensive engineering design reviews, hoping that we covered every single bit of scope. This may take a long time during which we will not be able to provide a date. We also know that the results may still not be exhaustive if we did that.</p><p>At SafetyCulture we go ahead and share this “gut feel” estimate as the delivery date. We do it in the form of a range, a month, rather than a specific day. This makes the focus of the team visible to other teams and invites feedback. The benefits of sharing plans and work in progress early are nicely covered in<a href="https://en.wikipedia.org/wiki/Creativity,_Inc."> Creativity Inc.</a> by Ed Catmull. In order to manage expectations effectively here, it is important to communicate a level of confidence together with a date. Be explicit about the steps that will be taken to reduce uncertainty. Finally, follow up and show how confidence is increasing as new knowledge is being generated.</p><blockquote>it is important to communicate a level of confidence together with a date</blockquote><p>Here is an example of communicating a delivery date. “<em>With Team A we are working on scaling the system to 10 times the projected peak load. If it all goes well this will be achieved in April. Depending on the answer to a couple of open questions about complex database migrations this may be delayed by a month or more. We will update the delivery date as soon as the full scope of work is better understood. An update will be provided regardless, by the end of this week. As far as we know no other teams will be impacted by a delay at this point and we are highly confident on the advertised business impact of this change</em>”. At that point, in a healthy engineering organisation such as SafetyCulture we expect engineering managers to ask how they can assist with answering the open questions sooner in order to build confidence on the advertised delivery date rather than pressurising the team to meet it at all costs.</p><p><strong>Managing performance</strong></p><p>If delivery dates are not seen as a commitment, how do engineering managers manage performance then? Well, commitment is still an important part of managing performance both at the team and at the individual level. Just not with respect to hitting a particular deadline per se.</p><p>At SafetyCulture we expect people to strive to meet the requirements of their respective roles and to embody the company values. In practical terms, I expect engineers in my teams to commit to a weekly goal while keeping the larger project goal in sight. I don’t want them to feel ashamed for missing a weekly goal. At the same time I want everyone to feel accountable for the weekly goals they set by asking them to add a retrospective comment on the reasons for any delay. In most cases we all learn from these notes and become better as a team when we plan the following week.</p><blockquote>expect people to strive to meet the requirements of their respective roles and to embody the company values</blockquote><p>Further expectations arise with reference to the 3 conditions for effective estimation at the top of this post. It is up to all of the team, in particular engineering managers, to identify gaps in domain knowledge, misalignment in trade-offs and accounting for changes to the environment when planning. The expectation here being that the team strives to fill in any gaps as early as possible to reduce the chance of surprises late in the project.</p><p>Together with the weekly commitment I want engineers to be accountable for sharing a plan for their work, including specific dates where possible. I make it clear to them that the dates may change, I only expect them to let the team know promptly when they do and why. I find it’s a great boost to the team morale to see these mini plans progress and eventually completed with our engineers instead of a manager at the driving seat. Here is an example of such a mini plan. Note the confidence call-out:</p><ol><li>Migrate reporting service — Deployed on 16/03</li><li>Migrate inspection listing requests — 12/04 (medium confidence)</li><li>Migrate public API — 15/04</li><li>Migrate tooling — 20/04</li></ol><blockquote>it’s a great boost to the team morale to see these mini plans progress and eventually completed with our engineers instead of a manager at the driving seat</blockquote><p>For larger pieces of work spanning multiple months I expect lead engineers to communicate a sense of percent completion of the work, relying on their proven intuition and experience to complement progress indications from tools like JIRA. For instance a few JIRA tickets may be incomplete, with JIRA showing 70% of the project done. However, the lead engineer knows that the work remaining is low risk and around 90% of the project is done at least as far as risk is concerned. This is extremely valuable information for the engineering manager when managing expectations.</p><p>I want to expand a little more on the importance of seniority in engineering when it comes to delivery excellence. Kathryn Koehler, Director of Productivity Engineering at Netflix strongly resonated with me by saying that their “senior” engineering teams “don’t use [fine-grained] estimates at all”, instead they rely on their culture of “freedom with responsibility” and their engineers’ experience. They only ensure the team is moving towards a direction that “feels right” using “order of magnitude” estimates like t-shirt sizes, checking regularly that the solution “solves the customer problem”. On the other hand, she argues that for less senior teams estimates help evaluate if they “are delivering on what they sign up to”. The relevant quote is<a href="https://youtu.be/tOpJcLeAbZ4?t=2094"> here</a> and I recommend watching the whole talk.</p><p>At the team level at SafetyCulture, instead of scrutinising delivery dates we manage performance by holding ourselves accountable to maintaining a steady cycle time of 3–5 days for customer-facing value-adding changes as I describe in my older post<a href="https://medium.com/safetycultureengineering/optimising-engineering-team-delivery-using-the-cycle-time-metric-2ba3a7c1c025"> here</a>. We also maintain a track record of business value created by the work of the team and the rate of it. Remember, what matters at the end of the day is the sustained rate of business impact and team health, not whether a particular date was hit.</p><blockquote>what matters at the end of the day is the sustained rate of business impact and team health, not whether a particular date was hit</blockquote><p><strong>Where to from here?</strong></p><p>With our focus on the team engine rather than estimation accuracy at SafetyCulture we keep iterating on our team cycle time metrics to make them even more objective, automated and aligned with world class standards such as those by<a href="https://www.devops-research.com/research.html"> DORA</a>.</p><p>Although I think estimating has its place at various stages of engineering, I am also personally interested in the emerging “no estimates” movement. “No estimates” proponents promote <em>forecasting</em> based on throughput over humans estimating pieces of work with limited information. In simple terms, forecasting works like this:</p><p>a) break down the work into user stories</p><p>b) count the stories</p><p>c) run a statistical estimation process like Monte-Carlo analysis to predict the likelihood that this number of stories will be completed in a given amount of time based on historical throughput of the team.</p><p>Using methods like forecasting can reveal that a project will be completed on this date with a probability of 25% or on that date with a probability of 80%.</p><blockquote>forecasting can reveal that a project will be completed on this date with a probability of 25% or on that date with a probability of 80%</blockquote><p>The key point here is that a completion date can be provided together with a confidence level without having to estimate any tasks. No guesswork required. No estimation poker. No philosophical debates about what is the difference between 1, 3 and 5 points. No anchoring bias such as when the estimate of a senior engineer is hard to challenge by more junior people. Less chance of ignoring “fat-tail” type of work that is conveniently left out as an “edge case” during initial estimation. Finally, no need to re-estimate the whole backlog each time new scope is identified.</p><p>Of course, the success of forecasting depends on our ability to break down the work into pieces of similar type (eg user stories or epics), without necessarily going through a sizing exercise. Like estimation, this ability is also subject to all 3 estimation conditions listed at the top of this post. Project management does not become a free lunch with forecasting.</p><blockquote>Project management does not become a free lunch with forecasting</blockquote><p><strong>Exploring further</strong></p><p>For the benefits of cycle time over estimation I recommend <a href="https://www.norberhuis.nl/how-i-started-believing-in-cycle-time-over-estimation/">this blog article</a> by Steffan Norberhuis. For more on “no estimates” and forecasting see a 30-minute<a href="https://www.youtube.com/watch?v=yXXz05TC1wc"> presentation by Vasco Duarte</a> on the topic. An in-depth analysis can be found in <a href="https://www.amazon.com.au/Actionable-Agile-Metrics-Predictability-Introduction-ebook/dp/B013ZQ5TUQ">Daniel Vicanti’s book</a>. Download Daniel’s <a href="https://actionableagile.com/">ActionableAgile plugin</a> we use at SafetyCulture if you’d like to try it yourself. More probabilistic methods of forecasting are listed in <a href="https://medium.com/swlh/why-story-pointing-needs-to-die-e60a775f9d37">this blog article</a> by Ron Jeffries the creator of “story points” who interestingly now also advises against using them, in favour of forecasting.</p><p>How does our estimation and planning culture resonate with you? Would you enjoy working in a team like ours? We are <a href="https://safetyculture.com/careers/">hiring</a>.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=52b483924ee9" width="1" height="1" alt=""><hr><p><a href="https://medium.com/mitti-engineering/blameless-estimation-for-healthy-engineering-teams-52b483924ee9">Blameless estimation for healthy engineering teams</a> was originally published in <a href="https://medium.com/mitti-engineering">Mitti Engineering</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Optimising engineering team delivery using the Cycle Time metric]]></title>
            <link>https://medium.com/mitti-engineering/optimising-engineering-team-delivery-using-the-cycle-time-metric-2ba3a7c1c025?source=rss-d4e647a1ff06------2</link>
            <guid isPermaLink="false">https://medium.com/p/2ba3a7c1c025</guid>
            <category><![CDATA[engineering]]></category>
            <category><![CDATA[delivery]]></category>
            <category><![CDATA[agile]]></category>
            <category><![CDATA[cycle-time]]></category>
            <dc:creator><![CDATA[Dimitri Koubaroulis]]></dc:creator>
            <pubDate>Fri, 18 Sep 2020 02:45:42 GMT</pubDate>
            <atom:updated>2020-09-18T02:45:42.387Z</atom:updated>
            <content:encoded><![CDATA[<p>Delivery health is a key element of a healthy engineering team. At SafetyCulture we are not simply looking for delivery health. We aspire for world-class level excellence in how we deliver value to customers.</p><p>This is particularly important to me as an engineering manager accountable for product delivery. I need methods and tooling that make it easy to support multiple cross-functional teams building some of the most critical components of our flagship <a href="https://safetyculture.com/iauditor">iAuditor</a> product. I recently had the opportunity to experiment with exactly that. The experiment was successful and I hope my learnings are useful to you.</p><p>In a fast paced agile environment such as SafetyCulture the <em>flow</em> of a team’s work was identified as a key productivity indicator reflecting delivery health. In simple terms, you can think about flow as the average rate of getting things done as a team. Flow is the rhythm or pulse of the team’s delivery engine. For example, at SafetyCulture our product teams aim to deliver customer value every 3–5 days on average.</p><blockquote>Flow is the rhythm or pulse of the team’s delivery engine</blockquote><p>Optimising for flow is better than other agile measurements such as velocity based on story points as it works in the language of the customer (wall clock time). Other benefits include improved predictability, streamlined team operations, boosted team morale and the ability to be truly agile when process or behavioural change is needed. For a detailed analysis of the benefits of flow versus other delivery health metrics I highly recommend Daniel Vacanti’s book “<a href="https://www.amazon.com/Actionable-Agile-Metrics-Predictability-Introduction/dp/098643633X"><em>Actionable Agile Metrics for Predictability: An Introduction</em></a><em>”</em>.</p><p><strong>How do we measure “flow”?</strong></p><p>At SafetyCulture our teams manage work using Scrum, Kanban and all variations of styles in-between. We needed a metric that was agnostic to the iteration style of the team.</p><p>The metric we chose to measure flow at SafetyCulture is the team’s <em>cycle time</em>. In simple terms, cycle time is the time between when someone starts working on a task and the time the task is done. Our goal is not to minimise the cycle time, rather it is to keep the cycle time stable within an agreed time range. The time range is defined by the team depending on the nature of the tasks undertaken. We found for most product teams in our environment a 3–5 days is a reasonable goal to aim for at this time.</p><blockquote>cycle time is the time between when someone starts working on a task and the time the task is done</blockquote><p>We wanted to be able to answer questions like “what was our team’s cycle time in the last 7 days?” and “how stable was our team’s cycle time over a time period?”. Additionally we wanted a way to quickly visualise the cycle time trend and be able to drill down to the supporting data to derive actionable insights at any time.</p><p>As we use <a href="https://www.atlassian.com/software/jira">JIRA</a> for managing work, we first experimented with the built-in <a href="https://support.atlassian.com/jira-software-cloud/docs/view-and-understand-the-control-chart/">JIRA cycle time reports</a>. Using JIRA’s built-in cycle time scatterplot went some way into answering the “what is our cycle time” question. However, we needed more powerful tooling to answer all our other questions above. We found that the <a href="https://marketplace.atlassian.com/apps/1216661/actionableagile-for-jira-agile-metrics?hosting=cloud&amp;tab=overview">“Actionable Agile” plugin for JIRA</a> gave us most of what we needed. See some examples below for how we made the most of it.</p><p>Here is an example of one of our team’s cycle time trend over time. The different colours indicating different types of work items like stories, bugs and tasks.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*0ghhYdG8imGX2nP4" /><figcaption>A team’s cycle time trend over a year</figcaption></figure><p>At a glance we can see the average cycle time was 5.3 days and 64% of tasks were completed within the 5.3 days average. Productive conversations can be had immediately by looking at the graph around what it would take to have more work items completed within the desired cycle time. Or how we could break down work better so that some items that took a bit longer than our desired range could fit within it.</p><blockquote>Productive conversations can be had immediately by looking at the graph</blockquote><p>The trend on that graph is fairly stable except for some time periods which insightfully correspond to challenging times for that team such as losing key team members and on-boarding new ones.</p><p>Looking at such graphs in retrospect we can learn lessons about how to avoid similar spikes. Most importantly, looking at such graphs in real time helps identify spikes before they happen and take corrective action to optimise flow.</p><p><strong>How do we optimise “flow”?</strong></p><p>The first step to optimising flow is to measure and visualise it and spend some time with the team understanding it. If the cycle time is not stable the team will typically discuss how to make it stable. Once stable, if the cycle time is longer than what the team thinks is healthy, the team will discuss what needs to be done to bring it within the desired time range. I have found all of these very productive conversations that always yield positive corrective actions where required.</p><p>Here are a couple of examples from the many interesting findings:</p><p>In one situation, Android pull requests were taking up to 12 days to complete. This was identified by the team’s cycle time trending upwards. We asked some more Android developers to help with code reviews, cycle time was restored and our Android delivery cadence was healthy again.</p><p>In another example, one of our engineers was leaving stories in progress for two or three weeks at a time, waiting to deploy them to production until after all sub-tasks had been completed. We discussed ways to optimise for incremental delivery to production, they adjusted their way of breaking down work and had a very positive impact on team delivery shipping value every few days to production!</p><p>The result of corrective actions can be seen almost immediately, within a few days or weeks. Here is an example of another of our teams which optimised their cycle time from a spike of more than 20 days down to 5 days recently</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*sETneXPgECvwKoQ9" /><figcaption>Optimising flow over 4 weeks</figcaption></figure><blockquote>The result of corrective actions can be seen almost immediately, within a few days or weeks</blockquote><p><strong>Wish list</strong></p><p>One thing I wished we had with Actionable Agile is the ability to share a link to a filtered view of a team’s analytics. A built-in ability to email us an updated cycle time report for each team at regular intervals would be greatly welcome too!</p><p><strong>Closing remarks</strong></p><p>Tracking team delivery health by using the team’s cycle time to measure and optimise the flow of work has been rewarding for our product teams at SafetyCulture. Cycle time conversations are becoming more and more part of our team culture. Every team member can easily understand how their work contributes to the team’s delivery rhythm. Work is flowing better than ever before with less streams of work being blocked and more incremental delivery across the board.</p><p>As useful as they are, cycle time scatterplots and trends are only the beginning of our exploration into team delivery metrics. Stay tuned for more updates on this topic. Or even better, come work with us, <a href="https://safetyculture.com/careers/">we are hiring!</a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=2ba3a7c1c025" width="1" height="1" alt=""><hr><p><a href="https://medium.com/mitti-engineering/optimising-engineering-team-delivery-using-the-cycle-time-metric-2ba3a7c1c025">Optimising engineering team delivery using the Cycle Time metric</a> was originally published in <a href="https://medium.com/mitti-engineering">Mitti Engineering</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Event sourcing and CQRS at SafetyCulture]]></title>
            <link>https://medium.com/mitti-engineering/event-sourcing-and-cqrs-at-safetyculture-9684f9263c53?source=rss-d4e647a1ff06------2</link>
            <guid isPermaLink="false">https://medium.com/p/9684f9263c53</guid>
            <category><![CDATA[go]]></category>
            <category><![CDATA[microservices]]></category>
            <category><![CDATA[grpc]]></category>
            <category><![CDATA[event-driven-systems]]></category>
            <category><![CDATA[event-driven-architecture]]></category>
            <dc:creator><![CDATA[Dimitri Koubaroulis]]></dc:creator>
            <pubDate>Tue, 18 Jun 2019 00:06:01 GMT</pubDate>
            <atom:updated>2019-06-18T00:06:01.030Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*2ywO4Bnt8MGIWBY7mVoiDA.jpeg" /><figcaption>Photo by <a href="https://unsplash.com/@eklektikum?utm_source=unsplash&amp;utm_medium=referral&amp;utm_content=creditCopyText">Iva Rajović</a> on <a href="https://unsplash.com/search/photos/stacking-weight?utm_source=unsplash&amp;utm_medium=referral&amp;utm_content=creditCopyText">Unsplash</a></figcaption></figure><p>At SafetyCulture, our team of engineers have had the opportunity to work with the latest cloud based technologies including, <a href="https://medium.com/safetycultureengineering/the-safetyculture-journey-to-kubernetes-ac7f9615710f">Kubernetes</a>, <a href="https://medium.com/safetycultureengineering/rebuilding-for-scale-521c0632f40e">gRPC</a>, <a href="https://medium.com/safetycultureengineering/event-driven-team-collaboration-49c8c624d999">event-driven architecture</a> and <a href="https://medium.com/safetycultureengineering/iot-getting-it-right-e17d38b08870">IoT</a>. As many of these fields are relatively new as mainstream startup tech choices, sharing our learnings is important for other teams thinking about going down similar paths.</p><p>In this post, I will introduce you to our first attempt at <a href="https://martinfowler.com/eaaDev/EventSourcing.html">event sourcing</a> (ES) with <a href="https://martinfowler.com/bliki/CQRS.html">Command-Query Responsibility Segregation</a> (CQRS) and our experience with it so far.</p><p>My first deep-dive into ES for cloud services was at a <a href="https://www.youtube.com/watch?v=aweV9FLTZkU">talk by Martin Fowler</a> in Sydney back in 2016. My main takeaway at the time was that ES made great sense for designing reliable, scalable web services but teams had to consider the pros and cons carefully.</p><p>The opportunity to build a fully event sourced system came up in late 2018 as my team was starting a new exciting project. We were not constrained by legacy infrastructure and the business case could be nicely described as a “workflow” comprised of a set of well understood business events. The natural extensibility of an event sourced system with stream processing for real-time insights and machine learning in future made the idea even more attractive.</p><h3>The implementation</h3><p>Under tight time constraints we had to be pragmatic. For simplicity we opted for infrastructure we were already using at SafetyCulture. We chose <a href="https://kafka.apache.org/">Apache Kafka</a> as the event store and encoded messages using <a href="https://developers.google.com/protocol-buffers/">protocol buffers</a> for schema validation as well as backwards and forwards compatibility. For our read model (data projection) database we used <a href="https://aws.amazon.com/rds/postgresql/">AWS RDS (PostgreSQL)</a>.</p><p>We considered “<a href="https://gist.github.com/eulerfx/11227933">command sourcing</a>” but opted against implementing a command stream and command processor initially. Instead, we published all API incoming requests into an event source stream that doubled as a command stream. Only stateless validation was performed at the API layer before producing the events. The stream consumer performed simple stateful validation against the read model database.</p><p>To ensure we maintained flexibility for future changes we decided to treat the event source stream private to the implementation and generate a second stream for integrations with other business domains. See my post on <a href="https://medium.com/safetycultureengineering/event-driven-team-collaboration-49c8c624d999">event-driven team collaboration</a> for details on how we do that.</p><p>Following Ben Stopford’s advice in his book “<a href="http://www.benstopford.com/2018/04/27/book-designing-event-driven-systems/">Designing Event-Driven Systems</a>”, we designed our event source stream to capture “delta” events that matched our gRPC service calls which in turn represented the user intent. Our integration stream on the other hand captured “full fact” (or as we call them “snapshot”) events so that downstream services were kept simple.</p><p>The final system looks like this</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*vo3NdLFlyTJd3D1CHa7Haw.png" /><figcaption>Event sourcing &amp; CQRS architecture</figcaption></figure><p>In the diagram above, a data projection from another domain (Domain 4) is utilised for “enriching” read requests with additional data and for performing stateful validation. For example, Domain 1 may only store IDs of user accounts internally but when retrieving details, the response is enriched with the user name as well from the user data projection. Domains 2 and 3 consume the integration event stream to create their own data projections.</p><p>Six months since launching in production, I decided to look back at the journey it has been for the team.</p><h3>Wins</h3><p>Has event sourcing given us a reliable, scalable system that is easy to reason about and extend? Yes, definitely.</p><p>Our <a href="https://en.wikipedia.org/wiki/Event_storming">event storming</a> sessions with the team were very insightful about modelling the business domain not only in “space” but also in “time”. Limiting the number of events and naming them carefully was key to coming up with a successful event model.</p><p>Given the set of agreed events, product and engineering can now speak a ubiquitous language when talking about system capabilities. The business event “assign to user X” no longer has to be translated to an “HTTP PUT request” and back again. The product requirement “can this be done each time this other thing has happened” can be easily understood in the context of the system, even by a non-experienced engineer. In hindsight, having a common language, is one of the greatest benefit of ES in my opinion. If you have ever pondered <a href="https://stackoverflow.com/questions/630453/put-vs-post-in-rest">how best to design a RESTful “update entity”</a> API endpoint you will know what I am talking about.</p><p>When diagnosing errors, there is no need to try to decipher “what happened” by sifting through noisy service text logs. The “log” is now the primary data source, containing everything that changed. Since ever! If you’ve ever tried to debug or replay complex system state changes involving multiple micro-services over time using service logs, you will again know how important this is.</p><p>Separating the writes (Kafka) and reads (RDS) via CQRS has been a big win for us in terms of better performance, scalability, data governance and general ease of reasoning and extensibility of the system. Knowing we can recreate our read model at any time as well as add any number of additional read models (eg an <a href="https://www.elastic.co/">ElasticSearch</a> data projection) to suit our evolving business needs is really powerful.</p><p>Two other business domains managed by different teams are already successfully consuming our integration stream, storing only the information they need in the data projections that make sense for them (S3 and RDS in this case).</p><h3>Challenges</h3><p>There certainly is no “free lunch” with ES/CQRS. We quickly confirmed that using Kafka as the event store introduced a number of challenges, mainly around <a href="https://medium.com/@pierreprinetti/event-sourcing-in-go-the-event-handler-29f9438c58f0">computing the “current state” of an entity</a> to perform stateful validation. We were aware of advice against using <a href="https://medium.com/serialized-io/apache-kafka-is-not-for-event-sourcing-81735c3cf5c">Kafka as an event store</a> and did consider putting <a href="https://medium.com/@gwenshapira/the-case-for-database-first-pipelines-f86240c69863">database in front of Kafka</a> or using a <a href="https://medium.com/@pierreprinetti/event-sourcing-in-go-the-event-handler-29f9438c58f0">different type of event store</a>. To keep things simple and truly event sourced we decided to stick with Kafka and evolve the architecture if and when that became necessary. We haven’t needed to yet. We apply all updates to the read-model, simply log and ignore certain types of errors and tune our reads to ignore data that would have been rejected by a validator. Not ideal, however it has worked for us so far.</p><p>Another challenge we faced was dealing with duplicate events in our event source. In the absence of a command stream and stateful validation we updated the event stream consumer to deal with invalid events such as duplicates by ignoring them.</p><p>A few months ago we realised one of our consumers had a bug and the data in our data projection was partially corrupted. Being able to simply blow away our data projection and re-create it by “replaying” the patched consumer over the event source stream was straightforward. Of course, fixing data corruption in the actual event source stream would be more complex given the stream is immutable. Thankfully we haven’t seen that yet, largely due to the use of protocol buffer schema validation.</p><p>What about <a href="https://en.wikipedia.org/wiki/Eventual_consistency">eventual consistency</a> inherent in ES/CQRS? Well, it has not been a big issue for us so far. We understand <a href="https://hackernoon.com/1-year-of-event-sourcing-and-cqrs-fb9033ccd1c6">eventual consistency is there regardless</a> and it’s better to embrace it than ignore it. We work around it by optimistic updates in our client app UIs. That means we assume the change succeeded and surface errors later as needed. We also had to build more sophisticated automated API tests to take eventual consistency into account.</p><h3>Conclusion</h3><p>Our particular business case, engineering culture and people helped make our first event sourced/CQRS attempt a success. Overall, we traded extra complexity for an architecture that is easier to reason about, scale, extend and maintain.</p><p>It is still early days on our event sourcing journey and no doubt there is a lot more to learn. I am looking forward to see how the system evolves in future.</p><p>One thing is for sure, building a fully event sourced/CQRS system has been a truly rewarding experience for the team, for SafetyCulture engineering and for our customers.</p><p>Sounds exciting? Get in touch, we are <a href="https://safetyculture.com/jobs/?department=engineering&amp;location=choose">currently hiring</a>.</p><p><strong>References</strong></p><p>Ben Stopford’s book gave us great overview of event sourcing and helped us decide between “delta” and “full fact” events, especially the importance of capturing the “user intent”</p><p><a href="http://www.benstopford.com/2018/04/27/book-designing-event-driven-systems/">Book: Designing Event Driven Systems - ben stopford</a></p><p>This talk was helpful when deciding options on stateful validation options in eventual consistent systems</p><iframe src="https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Fwww.youtube.com%2Fembed%2Ftlbr7i1_blA%3Ffeature%3Doembed&amp;url=http%3A%2F%2Fwww.youtube.com%2Fwatch%3Fv%3Dtlbr7i1_blA&amp;image=https%3A%2F%2Fi.ytimg.com%2Fvi%2Ftlbr7i1_blA%2Fhqdefault.jpg&amp;key=a19fcc184b9711e1b4764040d3dc5c07&amp;type=text%2Fhtml&amp;schema=youtube" width="854" height="480" frameborder="0" scrolling="no"><a href="https://medium.com/media/187367d8bd94af4bea9385218f84b131/href">https://medium.com/media/187367d8bd94af4bea9385218f84b131/href</a></iframe><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=9684f9263c53" width="1" height="1" alt=""><hr><p><a href="https://medium.com/mitti-engineering/event-sourcing-and-cqrs-at-safetyculture-9684f9263c53">Event sourcing and CQRS at SafetyCulture</a> was originally published in <a href="https://medium.com/mitti-engineering">Mitti Engineering</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Event-driven team collaboration]]></title>
            <link>https://medium.com/mitti-engineering/event-driven-team-collaboration-49c8c624d999?source=rss-d4e647a1ff06------2</link>
            <guid isPermaLink="false">https://medium.com/p/49c8c624d999</guid>
            <category><![CDATA[software-development]]></category>
            <category><![CDATA[teamwork]]></category>
            <category><![CDATA[event-driven-architecture]]></category>
            <category><![CDATA[event-driven-systems]]></category>
            <dc:creator><![CDATA[Dimitri Koubaroulis]]></dc:creator>
            <pubDate>Wed, 01 May 2019 09:59:52 GMT</pubDate>
            <atom:updated>2019-05-01T09:59:52.720Z</atom:updated>
            <content:encoded><![CDATA[<p>Engineering Leads at SafetyCulture are constantly looking for ways to improve how our cloud service teams can be more autonomous and at the same time work better together.</p><p>We are specifically looking to avoid teams being tightly coupled to design assumptions made by other teams and to reduce the risk of one team’s services being unavailable due to failures in the services of another team.</p><p>In the past, we’ve tried different ways to achieve loose coupling between services built by different teams. One of them involved modelling a <a href="https://www.no.lol/2018-10-28-exploring-the-strongly-typed-graph/">domain graph</a> using a GraphQL API.</p><p>Our traditional peer-to-peer API-call based micro-services architecture got us to 90 services and a dozen product teams but we felt we needed looser coupling between teams in order to <a href="https://medium.com/safetycultureengineering/rebuilding-for-scale-521c0632f40e">rebuild for higher scale</a>.</p><p>Here, I am going to share a high level overview of how using <a href="https://martinfowler.com/articles/201701-event-driven.html">event-driven architecture</a><em> </em>to share data between team services has helped our teams collaborate better and what we have learnt so far.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/500/1*589RnXSr-dcx1hHj1vQYkQ.jpeg" /><figcaption>Photo by <a href="https://unsplash.com/photos/3CS0Ne4wBPw?utm_source=unsplash&amp;utm_medium=referral&amp;utm_content=creditCopyText">Nathan Dumlao</a> on <a href="https://unsplash.com/search/photos/pour-coffee?utm_source=unsplash&amp;utm_medium=referral&amp;utm_content=creditCopyText">Unsplash</a></figcaption></figure><h3>Collaborating with APIs</h3><p>Let’s look at a simple scenario involving two services belonging to two different teams.</p><p>The Users team has a service that gives access to user profile data. The Actions team needs user profile data to enrich Action reports with user profile data each time a user generates an Action report.</p><p>In the past, we would have implemented the above collaboration scenario as an API request from the Actions service to the Users service, fetching the user profile data each time it was needed.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/603/1*Ud7UI30iOWkvd6uYuQ8QKQ.png" /><figcaption>Cross-team collaboration using API requests</figcaption></figure><p>API-based collaboration is ubiquitous and has many benefits but also a few drawbacks, which can become significant impediments at scale mainly due to tight coupling across teams.</p><p>Besides the extra delays of network hops and data transformations, with the added dependency of the Actions service, the Users team service would need to scale to accommodate the anticipated (guessed at this stage) rate of requests of the Actions team’s service.</p><p>Once operational, the Actions team would depend on not only the Users team service to always be available but also on the Users service’s own dependencies to be always available too.</p><p>That’s a lot of coupling between teams for a single API request scenario between two services!</p><h3>Collaborating with event streams and data projections</h3><p>To avoid the above complexities, our teams have been building more and more services that do not depend on API requests for cross-team collaboration.</p><p><strong>How does it work?</strong></p><p>Instead of API requests, when the Actions team needs data from the Users team’s service it accesses the data from a <em>data projection,</em> which is built using data from an <em>event stream</em> made available by the Users team.</p><p>A <em>data projection</em> is a data store local to the Actions team’s service. Its main benefit is that it contains exactly the subset of data content and exactly the data format that the Actions team needs from the Users team. In more complex scenarios the data projection may also blend data from other teams to avoid having to do that transformation or complex SQL queries on the fly.</p><p>The data projection is built and kept up to date by an event processing service called the <em>consumer</em>.</p><p>At SafetyCulture we use <a href="https://kafka.apache.org/">Apache Kafka</a> for our event streams and a variety of data stores to build data projections, including <a href="https://www.postgresql.org/">PostreSQL</a> and <a href="https://www.elastic.co/">ElasticSearch</a>.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/667/1*-wIK7rY-oYSDZa4xk_-WMQ.png" /><figcaption>Cross-team collaboration using an event stream and a data projection</figcaption></figure><p>Our event streams are modelled using <a href="https://developers.google.com/protocol-buffers/">Protocol Buffers</a>. This means that the Users team can keep changing its event stream data model in a way that is backward- and forward-compatible with all its consumers, avoiding complex consumer contract management infrastructure such as <a href="https://docs.pact.io/">Pact</a>.</p><p><strong>Benefits</strong></p><p>With this event-driven approach, there are no more network hops from the Actions team’s service to the Users team’s service and its dependencies. No more extra work by the Actions team’s service to transform user data each time it is fetched on the fly. No more reliance on Users team’s assumptions for structuring their APIs in a way that is practical for the Actions team. If the Users service went down, the Actions team’s service keeps working with existing data from its data projection, albeit some data may be stale until the Users service is back up again.</p><p>The Actions team may choose to blow their data projection away at any time and rebuild it to suit their needs or to fix data corrupted by a bug. In fact, we did run into that exact situation a few times already and rebuilding the data projection was a straightforward way to restore health in the system not just for new data but for all historical data as well!</p><p><strong>Trade-offs</strong></p><p>One trade off we had to make for the above conveniences is the extra work to setup and manage the event stream, the consumer and the data projection. As this is multiplied by the number of event streams processed, for services that have many dependencies that can add up to a lot of infrastructure to run and maintain. The flexibility of the approach means as we can always fall back to API-based collaboration when it makes more sense to do so.</p><p>For services that manage large amounts of data, the storage and maintenance cost of the data projection may also be a consideration.</p><p>The pattern has worked well for us consuming data from another domain. When it comes to changing (mutating) data across domains we prefer to use API calls. Typically immediate feedback is required by the caller to know if the update succeeded or failed eg. in order to retry.</p><p>The <a href="https://en.wikipedia.org/wiki/Eventual_consistency">eventual consistency</a> inherent to this pattern has not been an issue for us so far. That may not be the case in other business domains.</p><p>We always do a cost-benefit analysis against our business requirements and the ability of a team to support the extra infrastructure. We have also been exploring patterns like <a href="https://kafka.apache.org/documentation/streams/">Kafka Streams</a> in order to automate and standardise the process across our teams.</p><p>Have you done something similar with your teams? Let us know!</p><p><strong>References</strong></p><p><a href="https://www.confluent.io/designing-event-driven-systems">Designing Event-Driven Systems | Confluent</a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=49c8c624d999" width="1" height="1" alt=""><hr><p><a href="https://medium.com/mitti-engineering/event-driven-team-collaboration-49c8c624d999">Event-driven team collaboration</a> was originally published in <a href="https://medium.com/mitti-engineering">Mitti Engineering</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
    </channel>
</rss>