<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Elli Engineering on Medium]]></title>
        <description><![CDATA[Stories by Elli Engineering on Medium]]></description>
        <link>https://medium.com/@elli-engineering?source=rss-664565f11d82------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/1*O4bHymQSwZy81tLUVW7d5w@2x.png</url>
            <title>Stories by Elli Engineering on Medium</title>
            <link>https://medium.com/@elli-engineering?source=rss-664565f11d82------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Wed, 18 Nov 2026 18:39:45 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@elli-engineering/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[Game Days at Elli]]></title>
            <link>https://elli-engineering.medium.com/game-days-at-elli-fe123e823ebb?source=rss-664565f11d82------2</link>
            <guid isPermaLink="false">https://medium.com/p/fe123e823ebb</guid>
            <category><![CDATA[technology]]></category>
            <category><![CDATA[sre]]></category>
            <category><![CDATA[engineering]]></category>
            <category><![CDATA[ellis]]></category>
            <category><![CDATA[programming]]></category>
            <dc:creator><![CDATA[Elli Engineering]]></dc:creator>
            <pubDate>Fri, 12 Apr 2024 14:36:04 GMT</pubDate>
            <atom:updated>2024-04-12T14:36:04.397Z</atom:updated>
            <content:encoded><![CDATA[<h4>Let the Games Begin</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Mq6XGD-MgtKUR4j-NY2s-g.png" /><figcaption>Let the Game Days Begin! — Illustration by Lukas Hanke</figcaption></figure><p>In this post, we’ll explore Game Days — what they are, why our software engineering department dedicates two days per year to them, and their benefits. By the end, you’ll understand the concept well enough to start your journey. Note that this isn’t a detailed tutorial; since organisations vary you’ll need to tailor your approach.</p><h3>Journey</h3><p>Following, we will share more about our experience with hosting the Game Days.</p><h4>Game Days</h4><p>Game Days are exercises where engineers fix simulated incidents, enhancing their skills, improving our systems, and refining our processes.</p><p>Next, we will guide you through our motivation and the key considerations when designing incidents and organising Game Days. This includes experiments we conduct, how we structure the day, and how team composition can be adjusted. Finally, we’ll summarise the benefits for our engineers and Elli by running Game Days.</p><h4>Motivation</h4><p>At Elli, we’re dedicated to developing robust EV charging infrastructure. Our commitment to reliability constantly drives us to enhance our systems, processes, and skills. Game Days have become a key part of this journey starting from before our services went live. Having completed three iterations, and with our team and services expanding over the past four years, we’re preparing for the next Game Days.</p><h4>You build it, you run it</h4><p>It’s important to share our <em>modus operandi</em> to understand how Game Days fit us. At Elli, we embrace the <a href="https://medium.com/elli-engineering/introduction-to-elli-engineering-our-guiding-principles-cb0c9a1b907b">“You Build It, You Run It”</a> mentality. Every backend engineer:</p><ul><li>Writes application code</li><li>Writes infrastructure code</li><li>Creates pipelines</li><li>Works on observability</li><li>Performs database migrations</li><li>Fixes security vulnerabilities</li><li>Is on-call during business hours</li></ul><h4><strong>With great ownership, comes great responsibility</strong></h4><p>What Werner, Amazon’s CTO, shared in 2006 brings more insights into this mentality, and we have adapted it to our needs.</p><blockquote>Each service has a team associated with it, and that team is completely responsible for the service — from scoping out the functionality, to architecting it, to building it, and operating it.</blockquote><p>There is another lesson here:</p><blockquote>Giving developers operational responsibilities has greatly enhanced the quality of the services, both from a customer and a technology point of view. The traditional model is that you take your software to the wall that separates development and operations, and throw it over and then forget about it. Not at Amazon. You build it, you run it. This brings developers into contact with the day-to-day operation of their software. It also brings them into day-to-day contact with the customer. This customer feedback loop is essential for improving the quality of the service.</blockquote><p><em>— Werner Vogels, Chief Technology Officer at Amazon.</em></p><p>Effectively, Game Days aim to improve the quality of ownership.</p><h4>Incident Design</h4><p>In designing incidents, we focus on three aspects: likelihood, learnings, and setup. We’ll discuss each and provide an example and an anti-example.</p><ul><li><strong>Likelihood</strong>: What’s the chance of the incident occurring in production?</li><li><strong>Learnings</strong>: What key insights can be gained? Ideally, the incident should illuminate one or two technical or procedural areas.</li><li><strong>Setup: </strong>How easily can we replicate the incident within time constraints, ensuring it functions as intended during a Game Day?</li></ul><p><strong>Example</strong>: Consider a query endpoint with high latencies due to a missing database index. The scenario is likely when the table grows and query performance is not monitored.</p><p>The learning point is understanding indexes and how to introduce them. Setting up this incident is straightforward: remove the index and make requests to the endpoint.</p><p><strong>Anti-example</strong>: A malicious actor gained access to our Kubernetes nodes to mine cryptocurrency. While plausible, it’s improbable when using managed services and following best security practices. The setup is complex, requiring the removal of security features and other modifications. On a side note, one valuable practice we adopted regarding security is the <a href="https://www.microsoft.com/en-us/download/details.aspx?id=20303">Elevation of Privilege</a> game, but that’s a topic for another post.</p><p>For effective incident design, consult with experienced team members across different domains. Analyse old post-mortems for hidden lessons — initially, stage simpler incidents to target specific domain knowledge and learning objectives.</p><h4>Scheduling</h4><p>Scheduling depends on the desired hosting approach and the organisation’s constraints. In our scenario, we allocated two days for all engineering teams to create a festive and educational atmosphere. We experimented with two different formats.</p><ol><li><strong>Two full consecutive days</strong>: All engineers engaged in solving incidents continuously over two days.</li><li><strong>Two split days over a month</strong>: We divided the event into two separate days within a month, weeks apart. On the first day, we focused on disaster recovery (DR) practices. On the second day, we worked on incident handling. This approach allowed us to use the DR session to create environments to “break” during the incident handling day. We used the time between the two events to finalise the environments that we would need to use for Incident Handling.</li></ol><p>For incident handling, we structured the day as follows:</p><ul><li>5–6 incidents for each team in a day</li><li>1h per incident</li><li>20&#39; buffer</li><li>1h lunch break</li><li>30&#39; intro explaining the whole process</li><li>30&#39; for wrap-up and feedback form</li></ul><p>For DR, we allocated the entire day to the practice, allowing teams to prioritise work based on their services’ criticality. It’s important to note that teams owning multiple services focused only on the most critical due to time constraints.</p><h4>Teams Composition</h4><p>We experimented with two approaches, working with the original team at the Disaster Recovery (DR) and mixing teams during incidents. Each one comes with distinct advantages.</p><p><strong>Original Team: </strong>Established teams are effective in their domains. This rationale supports using original teams for DR tasks; they navigate complex aspects of their services with expertise. Furthermore, the teams can independently manage DR practices and create follow-up stories based on identified improvements. The knowledge acquired remains within the team. However, we do not prefer this setup for handling incidents due to the team’s high competency in their services — a simple incident is too straightforward and lacks challenge.</p><p><strong>Mixed Team: </strong>The mixed-team setup was successful, supported by the similarity of our tech stack across the organisation. It allowed engineers from different domains to collaborate on incident resolution, expanding their internal network, and understanding of other domains and technologies (e.g., different frameworks). Nonetheless, the productivity of mixed teams may decline if their tech stacks are vastly different.</p><p>By integrating team members with varying levels of seniority, we ensured a balanced distribution of experience. Each team included an Incident Response Team member to support the incident response process and a mix of seasoned engineers who could support newer members. We recommend a collaborative approach for each incident, similar to a mob programming session, with about five people working on the same issue.</p><h4>Facilitators</h4><p>A few engineers volunteered during the planning days to become facilitators. This role involves designing an incident and playing it during the Game Days. Facilitators are responsible for preparing and hosting the incident, and guiding engineers as they work to resolve it throughout the day. In our context, they played each incident 5 to 6 times.</p><h3>Payoff and Results</h3><p>In the following sections, we detail our outcomes.</p><h4>Upskilling Engineers</h4><p>To ensure our engineers effectively manage real incidents under stress, we practise in Game Days.</p><p>These simulations require engineers to understand the incident’s impact, triage, communicate with stakeholders, resolve the issue, and write a postmortem. By this, engineers develop and sharpen crucial skills, and familiarise themselves with lesser-handled, yet critical components of our services — areas that are not touched often due to their reliability.</p><h4>Systems Improvement</h4><p>Looking at our backlog from 2021 and 2022, we have created and worked on more than 20 stories. We have made significant improvements in the following areas:</p><ul><li>Streamlined and updated our <strong>documentation</strong> for enhanced clarity, comprehensiveness, and accessibility.</li><li>Improved our <strong>terraform setup</strong>, coming closer to a terraform apply to bring up services.</li><li>Enhanced <strong>backup</strong> monitoring and refined our recovery practices for better reliability.</li><li>Removed outdated or unnecessary <strong>code and configurations</strong>, streamlining our systems.</li><li>Updated our <strong>logging</strong> to ensure that is useful and removed what is not needed.</li></ul><h4>Feedback</h4><p>After each Game Day, we collected feedback from our engineers. Here, we share a summary of the feedback we received from our latest Game Days.</p><p><strong>The Good</strong></p><ul><li>Opportunity to explore and understand our systems and tech stack.</li><li>Cross-team collaboration and networking.</li><li>Hands-on approach expanding our thinking when facing incidents, and thus; gaining confidence.</li></ul><p><strong>The Bad</strong></p><ul><li>Difficulty identifying the impact of an incident.</li><li>Writing the post-mortem and stakeholder communication was cumbersome.</li><li>Incident handling was challenging.</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*35PwbuYne7a0lm7RFpJoJA.png" /><figcaption>Illustration by Lukas Hanke</figcaption></figure><h3>Wrap-up</h3><p>In wrapping up, Game Days at Elli has contributed to our continuous quest for excellence. By embracing the “You Build it, You Run It” philosophy and reinforcing it through Game Days, we improve the quality of our ownership — we upskill our engineers and boost the reliability and robustness of our EV charging infrastructure. These events foster an environment for learning, collaboration, and continuous improvement, significantly contributing to our culture. The positive results and feedback we’ve gathered affirm the value of the practice.</p><p>We would love to hear from you.</p><ul><li>Do you implement Game Days in your organisation? How do they look?</li><li>If not, what is preventing you from doing so?</li><li>Do you have further questions or need further insights? We are happy to share our experiences. Leave us a comment with your questions.</li></ul><h3>Learn More</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*2Z5CUnbFGj7XVSdI.png" /><figcaption>Follow us on Medium! (illus. by Jane Kim)</figcaption></figure><p>If you are interested in finding out more about how we work, please subscribe to the <a href="https://elli-engineering.medium.com/">Elli Medium blog</a> and visit our company’s website at <a href="http://elli.eco/">elli.eco</a>! See you next time!</p><p><strong>About the author</strong></p><p><strong><em>Thanos Amoutzias</em></strong><em> is a Software Engineer, he develops Elli’s Charging Station Management System and drives SRE topics. He is passionate about building reliable services and delivering impactful products. You can find him on </em><a href="https://www.linkedin.com/in/thanosam/"><em>LinkedIn</em></a><em> and in the 🏔️.</em></p><p><em>Credits: Thanks to all my colleagues who reviewed and gave feedback on the article!</em></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=fe123e823ebb" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[How to be happier and faster after a scale down]]></title>
            <link>https://elli-engineering.medium.com/how-to-be-happier-and-faster-after-a-scale-down-b98c4ec88cf4?source=rss-664565f11d82------2</link>
            <guid isPermaLink="false">https://medium.com/p/b98c4ec88cf4</guid>
            <dc:creator><![CDATA[Elli Engineering]]></dc:creator>
            <pubDate>Mon, 05 Feb 2024 16:01:08 GMT</pubDate>
            <atom:updated>2024-02-05T16:01:08.029Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*0qUwdHEijQ2igyHV" /><figcaption>Illustration by Lukas Hanke</figcaption></figure><p>The core message of this blog post is about slicing your work into small increments. Why do I write a blog post about probably the oldest topic in lean development? There is plenty of published literature proving the benefits of smaller batches and I have never heard someone actively arguing for large batch sizes. As a result of this common knowledge, one would expect that working in small increments happens automatically, because everybody believes in it and should strive towards it. However, this was not always the case for our team.</p><p>When reading this blog post I hope you can take something away on:</p><ol><li>Why we had large stories in the first place and what we did to change that.</li><li>Which benefits we got out of slicing our user stories small, apart from the classic “small increments, fast feedback, less waste”.</li><li>What happens when you overdo it with the slicing, which happened during our transition from big to small slices.</li></ol><p>For this, I compare a period of several months of two subsequent years, to include the same seasonal capacity fluctuations due to vacations, in both periods. Between these periods our team scaled down by 25% and at the same time took up more responsibilities. The data sample for the “slicing small” period contains over 73 stories while the “slicing big” sample contains 43, which is a result of looking at the same amount of time but having the stories scoped differently.</p><h3>The change to smaller user stories</h3><p>Our team grappled with large user stories for a long time. The main reason for the large user stories was that we wanted to keep the scope of our user stories at a level where the completion of a story would deliver value to the user. Therefore, our efforts to slice stories into what we deemed the smallest units resulted in often larger than ideal Stories. This led to planning difficulties, high work-in-progress (WIP) and spillage across sprints. The result of that was dissatisfaction and persistent cognitive strain. Despite attempts to address these issues with retrospectives, the situation did not improve significantly. It needed a radical change.</p><p>From now on features are the high-level definitions of our product that should deliver obvious user value, while stories represent smaller pieces in the process of delivering this user value.</p><p>This made it possible to slice stories to a tiny level, a level that we previously rejected because it had no immediate user value. It is very important to note that putting the extra effort into slicing our batches small did not result in additional meeting slots for refinement and planning. We just asked ourselves more often if we could slice the parts even smaller.</p><h3>Smaller team, same throughput, less mental strain</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*rbGpXQzd3oqmTR3v" /><figcaption>Illustration by Lukas Hanke</figcaption></figure><p>Let’s look at a couple of flow metrics and connect them to the perception that team members reported in retros and 1on1s. The following bar chart compares the big-batch-phase as the green baseline with the phase after we started slicing smaller in the purple bars. In the next paragraphs I describe the meaning of the bars from left to right.</p><p>The average cycle time of user stories went down by 60% from 27 to 11 working days. The change in cycle time is mainly driven by us doing less on a story. So we do less in less time. That is not impressive at all. It simply shows that we actually do what we intended to do, slice smaller.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*R-7qUJNquMS_20tB" /></figure><h3>Measuring code changes per developer</h3><p>For throughput and WIP I use the changes in the code that we made per developer. “Per developer” means that the data is normalised by the team size. This is important to make data comparable before and after the scale down.</p><p>Code changes are a difficult metric, I am well aware of that. Of course, we did not do exactly the same changes, so we have to take it with a grain of salt. In this case I believe it is fine to use it as we worked in similar code bases on similar topics. It would be best to compare delivered business value but we have no single metric to directly compare between our “big batch” and our “small batch” phase.</p><h3>Higher throughput</h3><p>Our average code change throughput per week relative to the team size increased by over 27%. This does not account for the additional benefit of avoiding waste and re-work as a classic result of the fast feedback cycle and adapting what actually is built. I attribute this gain in throughput to more focus and clarity by using smaller stories. We also delivered an important feature on time, which boosted morale, as we were initially unsure if we could make it.</p><h3>Reduction of WIP</h3><p>Being faster is great but being faster and with less strain is even better. Therefore, the two right bars show the average code changes that are “active” on any given day relative to the team size. This means on the story-level, all the changes that were necessary to complete this story are counted for all the days that this story was in progress. This shall illustrate the size and complexity of the story that we have to keep in our heads while working on it.</p><p>On a story-level, the average WIP went down by over 60%. The story-level is especially important because it was one of the points that caused high cognitive load and confusion in daily work. With the smaller stories, we complete them faster and get them out of our heads, moving on to the next one. The smaller stories are usually also better defined, as there is less uncertainty in smaller stories.</p><p>Looking at the task level, the WIP dropped by over 20%. There are two factors here:</p><ol><li>Slicing the stories smaller also led to an effort of slicing tasks smaller.</li><li>We put an additional focus on the “stop starting, start finishing” principle in our everyday work, which is not directly related to slicing our work smaller. Therefore, 20% could be largely unrelated to “slicing small” but rather a “finish first” result.</li></ol><p>Something that I learned from Donald Reinertsen’s “The Principles of Product Development Flow” and from the resistance of the developers was to not have a fixed limit of WIP. Limiting WIP to a certain number of active stories was something I had firmly believed in. However, I saw that having story cycle time and throughput as additional metrics, we could actually control WIP. At times we could increase it and keep story cycle time constant but increase our throughput compared to a fixed WIP limit. If cycle time started increasing, it was a sign to reduce WIP again.</p><h3>Less risk for outliers means better plan-ability</h3><p>We have already seen the increased throughput in the previous blog post but working in smaller batches also reduces the overall uncertainty in the work, something that Donald Reinertsen points out very well in “The Principles of Product Development Flow” (page 95 ff).</p><h3>Cutting down the unexpected story explosions</h3><p>The following graph shows that the distribution of story cycle time changed drastically after our change in slicing. Green is the phase of initial big batches and purple current phase of slicing smaller. You can clearly see the shift of the distribution towards smaller cycle times and the reduction of stories that explode, having cycle times of several weeks or even months. You can also clearly see that we are still far from perfect with the occasional explosion happening.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*9F87aoY6X5kL3-02" /></figure><p>But the most important part is that we reduce the tail of the distribution and the risk for outliers.</p><h3>The effect of cutting the distribution tail on a year’s quarter of work</h3><p>Now what happens when you need to do a certain amount of changes in your system and you do them either in small stories or in large stories?</p><p>To illustrate this, I took the cycle time and the amount of changes for each story in the two phases. As the baseline I chose the amount of changes that we achieved in one quarter in our “slicing big” phase. Then I created 5000 scenarios by taking random samples of stories that had the combined total of changes equivalent to this baseline. There are three cases to compare:</p><ul><li>Big Stories, with 27 days per user story on average.</li><li>Small Stories, with 11 days per user story on average.</li><li>Our new target of having less than 6 days per user story on average.</li></ul><p>The target case is created by taking our current slicing, the “Small Stories” case, and reducing the cycle time and the amount of changes for each story by half.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*uH5qehKobWIZiYZT" /></figure><p>The bar plot shows the 95th percentile of working days, required for the changes of one quarter. The purple line represents 90 days, which we would expect these changes to take.</p><p>What happens when we slice the same work into smaller increments? We reduce the outliers to the top (exploding stories) and with that increase our predictability significantly. This is important for several reasons:</p><ul><li>As a team we gain confidence in our ability to deliver value towards our customers and stakeholders in time, reducing the frustration of stories blowing up.</li><li>Our stakeholders see us as more reliable partners and we can manage their expectations better.</li><li>Often uncertainty is covered by buffers. This is a problem because it increases cost estimates and feedback cycles unnecessarily. Reducing the uncertainty reduces the need for buffers.</li></ul><p>These benefits are not as quickly visible as a reduction of WIP or a reduction in cycle time. They are a result of working in smaller increments over a sustained period of time and staying in this working mode takes dedication.</p><h3>Can we slice too small? — Yes we can!</h3><p>This transformation did not happen without hick-ups. Especially I wanted to really swing our way of working from one extreme to the other and pushed towards such small stories that we ended up with very small parts and the following issues:</p><ul><li><strong>Interdependent Micro-Stories:</strong> The overly small slices resulted in stories that were highly interdependent, resulting in effort to track numerous tiny stories and their progress.</li><li><strong>Transactional Inefficiency:</strong> Some stories were so tiny that the time spent on refining, discussing dependencies, and planning exceeded the actual work involved (Essentially the same issue as with async pull-requests described <a href="https://medium.com/elli-engineering/dont-do-small-pull-requests-2b9f8793e4e8">here</a>).</li><li><strong>Unintended Duplication:</strong> Working on a small story sometimes went so smoothly that additional work was done as a drive-by. This resulted in confusion as completed work overlapped with items still present in the backlog, causing unnecessary redundancy.</li></ul><p>However, these initial problems lasted only two weeks. We addressed them together by re-planning and merging some of the stories.</p><h3>Conclusion</h3><p>Going back to what I wanted you to take away from this:</p><ul><li>Why we had large stories in the first place and what we did to change that.</li></ul><p>We did not further improve because we stuck to a convention of what our smallest increment should deliver, which was actually limiting us more than helping us. Questioning that made our improvements possible.</p><ul><li>Which benefits we got out of slicing our user stories small apart from the classic “small increments, fast feedback, less waste”.</li></ul><p>We managed to compensate for the 25% scale down by increasing throughput per developer by over 25%. Even more importantly, we have a massive reduction of what we need to keep in our heads every day, as small stories close faster, reducing story-level WIP. Our discussions are more focused and there is more alignment in the team on what we do. This part is to me by far the most important gain. Further, we reduced uncertainty in our delivery, which increases our confidence to deliver what we plan and the confidence of our stakeholders in us.</p><ul><li>What happens when you overdo it with the slicing, which happened during our transition.</li></ul><p>Early on we ran into unwanted dependencies between tiny stories. Also, we got into the transactional cost vs. batch size trap by making discussion time in refinements and plannings longer than actual work to be done. Fortunately, this was only a short transition phase.</p><p>Overall, this is not the perfect experiment. We just cannot perfectly control the team situation, fix all variables and only adjust the story size. However, I feel this is as close as I could get for a significant period of time. Let me know what you think and if you have done similar experiments.</p><p><strong>About the author</strong></p><p><a href="http://www.linkedin.com/in/matthias-foerth-60b68a173"><strong><em>Matthias Förth </em></strong></a><em>is</em> <em>a Product Owner and former Software Engineer focused on backend development. His current interests are the streamlining of development and pushing data-driven decision making at Elli.</em></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=b98c4ec88cf4" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Navigating the Evolution of Stream-Aligned Teams: Lessons from Our CI/CD Journey]]></title>
            <link>https://medium.com/elli-engineering/on-communities-of-practice-c7a24f563ab0?source=rss-664565f11d82------2</link>
            <guid isPermaLink="false">https://medium.com/p/c7a24f563ab0</guid>
            <category><![CDATA[continuous-delivery]]></category>
            <category><![CDATA[engineering]]></category>
            <category><![CDATA[ci-cd-pipeline]]></category>
            <dc:creator><![CDATA[Elli Engineering]]></dc:creator>
            <pubDate>Thu, 03 Aug 2023 14:56:39 GMT</pubDate>
            <atom:updated>2024-01-15T11:16:23.448Z</atom:updated>
            <content:encoded><![CDATA[<h4>When they fly like a bird and when they hit the unexpected window hard</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*VkyR7UqU6tF1yoWzqexxJA.png" /><figcaption>(illustration made by Lukas Hanke)</figcaption></figure><h3>Introduction and Scope</h3><p>At Elli we aim for “stream-aligned teams”. According to the book <em>“Team Topologies”</em> — by<em> Manuel Pais</em> and <em>Matthew Skelton</em>, we focus on optimising our workflow when it comes to feature delivery and minimising the number of handovers between teams. To reduce handovers, these teams have end-to-end ownership of their services and infrastructure. Ownership means freedom and independence but also a high cognitive load. To reduce this load and upskill teams in cross-team topics like infrastructure or continuous integration and delivery (CI/CD), we have various Communities of Practice (CoPs). We are proud of our CoPs and our culture of continuous improvement. It has served us well but recently we hit some barriers we cannot push through with our current setup. Facing these difficulties we see the need for having dedicated teams to also drive such improvements across the organisation.</p><p>In this blog post, I want to share with you the story of how we got here, what our CoPs have done well and where they ended up coming short. To illustrate my points I will use one CoP as an example — the CI/CD CoP. But we see similar issues also in other CoPs.</p><h3>Status Quo of CI/CD</h3><p>Before we start, CI/CD at Elli is in a good state and it is consistently improving. Our services deploy multiple times per week or even per day as we deploy every merged change. Deployments usually take less than 30 minutes (median is around 25 minutes). The pie chart shows the 2023 survey results for the statement <em>“I am happy with the state of CI/CD in my team”</em>. In 2023, over 50% strongly agree, which is an over 20% improvement to 2022. But we noticed a couple of difficulties with our approach of solely relying on CoPs for topics like CI/CD. This blog post is about these difficulties and barriers, which we face on our way to turn the pie chart fully green.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*HAPG67gspYrzwDVysNFEpg.png" /></figure><h4>The early achievements and the following decline</h4><p>We have around 30% of our working time dedicated to technical and operational excellence (TOX) in software development. Most of this budget is spent on our <a href="https://elli-engineering.medium.com/to-tox-or-not-to-tox-6a089d2e75a2">TOX work in each of our feature teams</a>. But it also includes the time we spend in our CoP’s one-hour weekly meetings and bi-weekly four-hour workshops. Reserving a time budget aside from feature delivery work has reaped many benefits across the entire organisation: We went from triggering releases with tags once or twice per story to releasing every merged change. The median deployment frequency is about 3–4 times per week now with a maximum of 20 for our main frontend. The median run time for our releases is around 25 minutes with deployments on several environments, end-2-end tests and so on.</p><p>Such improvements were either driven by some team with a need for it and then spread through the CoP or driven by the CoP itself. Workshops are especially effective for propagating best practices such as these.</p><p>However, the participation in these workshops, relative to the number of engineers working at Elli, has steadily decreased over the years.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*6YM9FztHm4gYwMsaAcIAzg.png" /></figure><p>To get a better understanding of why participation levels are declining, we administered a survey to identify the core causes of low participation rates. The main reasons for people not to join were:</p><ul><li>Feature pressure in the team</li><li>Sprint rituals and many other meetings collide with CoP meetings</li><li>Engineers like working on features better</li><li>Topics are not relevant or exciting</li></ul><p>The first two points are issues that came through our scale up. Engineering management needs to emphasize the priority and importance of the TOX budget repeatedly to our engineers to prevent it from falling by the wayside. Regarding scheduling, CoP meetings are visible in our company’s engineering department-wide calendar for everyone to see. Thus Product Owners and engineers should be wary of this and not schedule any conflicting meetings. Lack of prioritisation and commitment ultimately caused our CoP rituals to fall through.</p><p>But now to the really shocking part. Topics are not relevant and engineers rather choose more feature work. This is clearly on the CoPs and must be improved.</p><h4>Out of the rabbit hole and addressing the needs of engineers</h4><p>Every year, the CI/CD CoP, led by our senior infrastructure engineer –<a href="https://www.linkedin.com/in/madhusudan-ram-joshi-43750823/"><em>Madhusudan</em></a>, conducts a survey on the state of DevOps here at Elli. After the 2022 survey, they decided to address these shortcomings with a couple of initiatives:</p><ul><li><strong>Pipeline template repository</strong>: A dedicated repository with templates for the initial pipeline setup and continuous improvements.</li><li><strong>Upskilling workshop</strong>: An Elli Engineering Academy workshop on our general pipeline setup and fundamentals of Azure Pipelines.</li><li><strong>“Bring your own pipeline” (BYOP) workshops</strong>: Teams can look into and improve their pipelines together with experts from the CI/CD CoP.</li><li><strong>Data analysis</strong>: Collect data from Elli’s Azure Pipelines in Big Query for more in-depth analysis of our software delivery processes.</li></ul><p>Let’s examine some key insights from the 2022 survey and compare them to the new 2023 results:</p><ul><li><strong>Complexity of pipelines</strong>: In 2022, 35% of participants thought our pipelines are complex. This went down to 20% in 2023. The usage of the pipeline templates definitely helps here as well as the BYOP workshops.</li><li><strong>Lack of understanding of pipelines</strong>: In 2022, 45% of participants were not confident in modifying the pipelines. This decreased to 15% in 2023. The upskilling workshops surely did not hurt.</li><li><strong>Flaky pipelines</strong>: PR pipelines fail for the wrong reasons 60% of the time in 2022 and 2023.</li><li><strong>Slow pipelines</strong>: In 2022, 70% thought our pipelines are slow, in 2023 50% still have the need for speed. 60% of participants in 2022 actively waited for the pull request (PR) pipeline to complete. This dropped to 23% in 2023.</li></ul><p>The first two issues we address rather well with the measures taken. Regarding the workshop participation, we clearly see that these upskilling and BYOP workshops have way higher participation than other workshops (roughly double). However, especially Elli Engineering Academy workshops require a lot of time to prepare and facilitate. It is unrealistic to hold these lengthy workshops every week for each of the different engineering topics. So far this year we do not see an improvement in the overall participation.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*oYhxYE5Q81Hg79Gyis2iYw.png" /><figcaption>(illustration made by Lukas Hanke)</figcaption></figure><p>Flaky tests and slow pipelines however are more difficult. Let’s look into that in more detail and start with the drop of engineers waiting for the PR pipeline. We started collecting pipeline data only 3 months after the 2022 survey. For the time that we have data our median runtime of our PR pipelines is basically constant at about 15 minutes. So either, the drop of 60% to 23% is because of improvements before we collected data (I doubt it) or people adapted to the wait time and do other things in parallel, which is what I often hear when talking to other engineers.</p><p><strong>This is something which we actually want to avoid</strong>. Starting a new task, which will occupy you for the next 4 hours and increase the amount of work in the delivery system, all because of a 15 minute wait time, is not desirable. So this decrease in people waiting may look better than it actually is. However, sitting around for 15 minutes is not great as well. But getting the runtime down to 10–12 minutes does not help. You still wait. You still jump to another task. We need less than 5 minutes, a small bathroom break, and getting the pipeline this fast is hard.</p><p>Also, it is difficult to make the pipelines and tests more robust. We have a failure rate of roughly 20% on the PR pipeline, also quite constant over time. The survey hints that these 20% are made up of 12% unnecessary failures and only 8% meaningful failures. This shows that teams see this pain point but struggle to significantly improve on it with their regular TOX work. For these hard topics, the small CoP time is not enough. What we need in these cases is a full time effort. For example an existing team could pick up the topic for one of their TOX sprints. Another option would be to form a task force or even an enabling team to have full time capacity temporarily but not move the ownership permanently away from CoPs and stream-aligned teams.</p><h3>Conclusion</h3><p>We have seen early on at Elli that CoPs can drive great improvements across the engineering teams. Especially in our company’s earlier stages, our CoPs tackled realistically achievable and time-bound topics with high impact. But at some point you need to shift gears when an existing way of working does not achieve desired results. Our CoP approach struggled to generate the desired impact and lost support among engineers. And hence the need to challenge the status quo, be agile and look for better solutions.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*_onGd9WqdVvo8tL3IBctMQ.png" /><figcaption>(illustration made by Lukas Hanke)</figcaption></figure><p>Setting goals that are achievable with the CoP capacity, as well as focusing on upskilling and knowledge sharing across teams, shows promising participation levels and improvements. At the same time we should not buckle when continuous improvement gets tough. Instead, we can seek alternative data-driven solutions. Surveys and having honest conversations with our fellow engineers helped us see the need for a proper full-time effort to dig deep down into rabbit holes. We see different possibilities here to address the issue. In the end it is important that we push forward on these topics in collaboration with CoPs and stream-aligned teams and keep the end-to-end ownership in the teams, while supporting them on their improvement efforts.</p><h3>Learn More</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/267/1*M2kDcN84CmcbRKcHVpBHvw.png" /></figure><p>At Elli, we are always in search of improving the way we work together as a team. If you are interested in finding out more about how we work, please subscribe to the <a href="https://elli-engineering.medium.com/">Elli Medium blog</a> and visit our company’s website at <a href="http://elli.eco/">elli.eco</a>! See you next time!</p><p><strong>About the author</strong></p><p><a href="http://www.linkedin.com/in/matthias-foerth-60b68a173"><strong><em>Matthias Förth</em></strong></a><strong><em> </em></strong><em>is</em> <em>a Product Owner and former Software Engineer focused on backend development. His current interests are the streamlining of development and pushing data-driven decision making at Elli.</em></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=c7a24f563ab0" width="1" height="1" alt=""><hr><p><a href="https://medium.com/elli-engineering/on-communities-of-practice-c7a24f563ab0">Navigating the Evolution of Stream-Aligned Teams: Lessons from Our CI/CD Journey</a> was originally published in <a href="https://medium.com/elli-engineering">Elli Engineering</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[To TOX or not to TOX]]></title>
            <link>https://medium.com/elli-engineering/to-tox-or-not-to-tox-6a089d2e75a2?source=rss-664565f11d82------2</link>
            <guid isPermaLink="false">https://medium.com/p/6a089d2e75a2</guid>
            <category><![CDATA[engineering]]></category>
            <category><![CDATA[technical-debt]]></category>
            <dc:creator><![CDATA[Elli Engineering]]></dc:creator>
            <pubDate>Thu, 13 Apr 2023 14:59:40 GMT</pubDate>
            <atom:updated>2023-04-13T14:59:40.846Z</atom:updated>
            <content:encoded><![CDATA[<h4>How technical and operational excellence is achieved</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*cdossdaAqkjHBfN5VV6TyQ.png" /><figcaption>(illustration made by Lukas Hanke)</figcaption></figure><p>In our very first blog <a href="https://medium.com/elli-engineering/introduction-to-elli-engineering-our-guiding-principles-cb0c9a1b907b">post</a>, we introduced you to Elli’s guiding principles for achieving engineering excellence. In this article, we delve deeper into one of the practices our engineering teams use on a per-sprint basis to maintain a high level of code quality and to upskill technically. It’s a practice we call “TOX”, and it stands for “Technical and Operational Excellence”.</p><h3>What are TOX stories?</h3><p>TOX are driven from engineering teams, and can be seen as a parallel stream to feature development. Each team dedicates some proportion of the time to work on such tasks. The amount of the time spent for TOX depends on the team and their priorities for the current sprint.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*_TveTACRukkJsL1-M_rnGQ.png" /><figcaption>The time spent on TOX stories depends on the team itself. (illustration by Lukas Hanke)</figcaption></figure><p>Such topics usually come to the team from the Tech Lead role, however all engineers are encouraged to initiate a TOX. The key is to understand what is included in such a task.</p><h3>Boundaries</h3><p>TOX is not about repaying technical debt that has been introduced as a consequence of feature pressure or to fulfil the “Definition of Done” for business requirements tasks.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*FfHKahnqj--PcXS8aL4QaA.png" /><figcaption>There are guidelines about Do’s and Don’ts when creating TOX stories. (illustration by Lukas Hanke)</figcaption></figure><h4><strong>Examples of TOX topics</strong></h4><ul><li>Minimising the risk of technical debt</li><li>Improving developer’s experience</li><li>Knowledge sharing within Elli</li><li>Improvements in the CI/CD pipelines</li><li>Testing improvements (introducing black box testing for example)</li><li>Upgrading software package dependencies</li><li>Automating the repetitive</li></ul><h4><strong>Examples of non TOX topics</strong></h4><ul><li>Product improvements</li><li>Fixing bugs</li><li>Tech debt that has been deliberately accepted by feature pressure or tight deadlines</li><li>Operational tasks that are vital to the product</li><li>Supporting existing / implementing new internal business processes</li><li>Supporting customer care</li><li>Preparing presentations / demos for sprint reviews</li><li>ISO related topics</li></ul><h3>The gist of it</h3><blockquote>“Leave your code better than you found it” — The Boy Scout Rule.</blockquote><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*6n6qHmgmtFWe9LlO9C0Yfg.png" /><figcaption>(illustration by Lukas Hanke)</figcaption></figure><p>The main point of following this principle is to maintain the quality and sustainability of the codebase. Additionally, we initiate TOX stories for several other reasons:</p><h4><strong>Improving CI/CD experience</strong></h4><p>We want to continuously work on CI/CD automation and improve our pipelines. This helps us in identifying and fixing bugs early in the development process. It also supports us in speeding up the delivery of software updates and new features to our users. By automating such repetitive tasks, it results in cost savings for organisations and ensures that the software being delivered to users is of high quality.</p><h4><strong>Improving developer experience</strong></h4><p>As engineers, we also want to boost our productivity. Improving our experience enables us to work more efficiently and produce higher quality code in less time. Delivering faster and better reduces the likelihood of costly errors and downtime. Additionally, a positive developer experience enables us to collaborate better with other team members.</p><h4><strong>Codebase housekeeping</strong></h4><p>In order to improve the quality of code, we focus on refactoring to make it more readable, maintainable and scalable. In this way, we reduce technical debt and increase efficiency of future development.</p><h4>Knowledge sharing</h4><p>Last but not least, we highly embrace the power of knowledge sharing. TOX stories are also used to initiate lightning talks so that we learn more from each other about a new tool, technology or anything else that promotes clean architecture and code.</p><h3>To Sum Up</h3><p>The practice of “TOX” or “Technical and Operational Excellence” is used by engineering teams to maintain high code quality and to upskill technically. TOX stories are tasks that are not directly related to feature development, but rather focus on improving developer experience, automating tasks, minimising technical debt, improving CI/CD pipelines, and sharing knowledge. The article emphasises the importance of maintaining code quality and sustainability through practices like TOX and refactoring, and encourages engineers to continuously improve their skills and productivity.</p><h3>Learn More</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*Xv2keYMXmK2A59CG.png" /></figure><p>At Elli, we are always in search of improving the way we work together as a team. If you are interested in finding out more about how we work, please subscribe to the <a href="https://elli-engineering.medium.com/">Elli Medium blog</a> and visit our company’s website at <a href="http://elli.eco/">elli.eco</a>! See you next time!</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=6a089d2e75a2" width="1" height="1" alt=""><hr><p><a href="https://medium.com/elli-engineering/to-tox-or-not-to-tox-6a089d2e75a2">To TOX or not to TOX</a> was originally published in <a href="https://medium.com/elli-engineering">Elli Engineering</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Parlez-vous OCPP?]]></title>
            <link>https://medium.com/elli-engineering/parlez-vous-ocpp-27054b092d44?source=rss-664565f11d82------2</link>
            <guid isPermaLink="false">https://medium.com/p/27054b092d44</guid>
            <dc:creator><![CDATA[Elli Engineering]]></dc:creator>
            <pubDate>Wed, 01 Feb 2023 11:54:53 GMT</pubDate>
            <atom:updated>2023-04-11T11:58:36.495Z</atom:updated>
            <content:encoded><![CDATA[<h4>How charging stations communicate with their central backend system</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*EaPJxExQZlvgj9i7_uBvQg.png" /></figure><h3>OCPP, c’est quoi?</h3><p>If we observed a group of people with different native tongues trying to engage in discussion, we would likely observe them consolidating to the language that is understood by all people. They converse in a <strong>common language</strong>–one that is universally spoken, to make communication easier for all members of the group.</p><p>Similarly, for an Electric Vehicle (EV) charging station (hardware) to communicate with a central backend system (software) they must speak a common language. This common language or “communication protocol” is the <strong>Open Charge Point Protocol (OCPP)</strong>. OCPP allows these two systems to <strong>exchange data and orchestrate the overall charging process for an electric vehicle</strong>.</p><p>A charging station that can speak this language is called <em>OCPP-compliant</em> and thus can communicate with any kind of software that is also OCPP-compliant. Any charge point from <em>any vendor</em> should be able to work with any charging management software as long as both can speak the language of OCPP.</p><blockquote><strong><em>Charge Station : </em></strong><em>The charger for an EV; the physical device that charges the electrical vehicle.</em></blockquote><blockquote><strong><em>Charging Station Management System</em></strong><em> (CSMS): The central backend system that controls the charge points (from authorisation to charging session management).</em></blockquote><h3>OCPP in action</h3><p>Imagine an EV driver who parks in front of a charging station to charge their vehicle. OCPP is involved in almost every step of the EV charging experience from reserving a charging station to the final payment process:</p><ul><li>The EV driver reserves a charging station for use.</li><li>The EV driver is authorised to start a charging session.</li><li>The charging station reports the real-time status of its connectors (suspended, available, unavailable, charging, etc.) to the backend system.</li><li>Any information about the current charging session is relayed to the management system.</li><li>The EV driver remotely starts / stops sessions via a web or mobile console.</li><li>Load of charging stations is regulated based on the state of charging.</li><li>One can monitor or change the settings configured on a charging station.</li><li>The EV driver decides to stop charging and gets billed for the session.</li></ul><p>In every action above, <strong>requests and responses</strong> are relayed back and forth between charging stations (client) and the CSMS (server).</p><p>The charging stations are <strong>connected to Elli’s CSMS via a WebSocket (WS) connection</strong>. <strong>Messages (JSON) are exchanged back and forth through this WS channel.</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*XNOxOiOpxdrCOykR3fUYkQ.png" /><figcaption>The charging station and the CSMS communicate via a back-and-forth rally of OCPP messages!</figcaption></figure><h3>Basic Communication Pattern</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*TORQ5cKRdHVhjV3F" /><figcaption>Diagram from OCPP 1.6 Specification illustrating the start/stop of a transaction</figcaption></figure><p>The diagram above illustrates a typical communication pattern between the Charge Point and the Charging Station Management System.</p><p>The Charge Point and CSMS speak to each other via messages that:</p><ul><li><strong>Request</strong> the execution of an operation and</li><li><strong>Respond/confirm</strong> that the requested operation has been performed</li></ul><h3><strong>Smart Charging</strong></h3><p>Smart Charging, in e-mobility jargon, means <strong>leveraging information about the status of the energy grid and the state of EV charging</strong> to make<strong> informed decisions </strong>on how to optimise the charging process. OCPP enables smart charging by enabling the regulation of power at a charging station. For example, OCPP enables the CSMS to modify the charging power in real-time based on the charging status of the station or allows CSMS to limit the total load on a group of stations so that they do not draw more power than necessary from the grid.</p><p>Let’s say that an owner of a charging station wishes to reduce the maximum load (power) allocated to a single station. On the Elli Management Console, a user can send a request (via OCPP) to the station to reduce the station’s maximum power limit.</p><p>This is what happens behind the scenes…</p><ul><li>Elli’s CSMS, which connects to the station via a WebSocket connection, sends a request to the station to down-regulate the power limit. The parameters of the request are described in what is called a “<strong><em>charging profile</em></strong>”.</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*7TffJ6W0_4DnQupHsLMWbQ.png" /><figcaption>A charging profile is a set of parameters that define how charging should proceed at a given station.</figcaption></figure><ul><li>The station receives the request with the charging profile from CSMS:</li></ul><p><strong>SetChargingProfile.req:</strong></p><pre>{<br>  &quot;connectorId&quot;: 0,<br>  &quot;csChargingProfiles&quot;: {<br>   &quot;chargingProfileId&quot;: 1,<br>        &quot;stackLevel&quot;: 0,<br>   &quot;chargingProfilePurpose&quot;: &quot;ChargePointMaxProfile&quot;,<br>   &quot;chargingProfileKind&quot;: &quot;Absolute&quot;,<br>   &quot;chargingSchedule&quot;: {<br>    &quot;startSchedule&quot;: &quot;2022-07-06T02:04:00Z&quot;,<br>    &quot;chargingRateUnit&quot;: &quot;W&quot;,<br>    &quot;chargingSchedulePeriod&quot;: [{<br>     &quot;startPeriod&quot;: 0,<br>     &quot;limit&quot;: 9000<br>    }]<br>   }<br>  }<br>}</pre><ul><li>The station returns a response message acknowledging that the message was received and</li><li>Executes the request: It adjusts its power limit based on the parameters described in the charging profile (in this case to 9 kW)</li></ul><h3>Pourquoi OCPP?</h3><p>If a network of charging stations all manufactured by different vendors are all OCPP-compliant, they will all be able to communicate with the OCPP-compliant backend system to which they are connected.</p><p>Conversely, a single charging station that is OCPP-compliant will be able to communicate with various management systems built and maintained by separate companies.</p><p>This interoperability lends itself to the following benefits:</p><h4><strong>Open Source</strong></h4><ul><li>Anyone can access and contribute to this protocol for continuous improvement.</li></ul><h4><strong>No Vendor Lock-in</strong></h4><ul><li>You can use any charge point provider as long as the hardware is OCPP-compliant. You are not “stuck” with a single brand and can switch at any time (e.g. if the manufacturer goes out of business!)</li></ul><h4><strong>Vendor-Agnostic Hardware within Single Network:</strong></h4><ul><li>You can keep charge points from various vendors in the same network and manage them in one place.</li></ul><h4><strong>Competitive Prices</strong></h4><ul><li>Providers following the OCPP protocol are in competition to provide the most cost-effective and feature-rich products to the market.</li></ul><p>At Elli, we recognise the advantages of developing vendor-agnostic software for our customers and ensure that our charge point management system complies with the Open Charge Point Protocol to manage the stations in our network. To learn more about OCPP: check out the OCPP documentation provided by the Open Charge Alliance: <a href="https://www.openchargealliance.org/protocols/ocpp-201/">https://www.openchargealliance.org/protocols/ocpp-201/</a></p><p>In this article, we gave a basic overview of OCPP, how it works, and the benefits that come from OCPP-compliancy. To learn more about e-mobility topics, and how we implement the latest standards and best practices for building Elli’s e-mobility solutions, please subscribe to our Medium blog or visit our <a href="http://www.elli.eco/">company website</a>! Stay tuned for the next article!</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*RcDL6KjiDgF7i4-5Sia7wg.png" /><figcaption>See you next time :)</figcaption></figure><p><a href="https://www.linkedin.com/in/jaryungkim/"><strong><em>Jane Kim</em></strong></a><em> is a Software Engineer in the Design System team and a part-time writer/illustrator for the Elli blog</em></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=27054b092d44" width="1" height="1" alt=""><hr><p><a href="https://medium.com/elli-engineering/parlez-vous-ocpp-27054b092d44">Parlez-vous OCPP?</a> was originally published in <a href="https://medium.com/elli-engineering">Elli Engineering</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Catch Me If You Can — Memory Leaks]]></title>
            <link>https://medium.com/elli-engineering/catch-me-if-you-can-memory-leaks-eb2604c0c7ea?source=rss-664565f11d82------2</link>
            <guid isPermaLink="false">https://medium.com/p/eb2604c0c7ea</guid>
            <category><![CDATA[memory-leak]]></category>
            <category><![CDATA[google-cloud-platform]]></category>
            <category><![CDATA[software-development]]></category>
            <category><![CDATA[javascript]]></category>
            <category><![CDATA[nodejs]]></category>
            <dc:creator><![CDATA[Elli Engineering]]></dc:creator>
            <pubDate>Thu, 01 Dec 2022 10:13:11 GMT</pubDate>
            <atom:updated>2023-12-29T14:05:50.527Z</atom:updated>
            <content:encoded><![CDATA[<h3>Catch Me If You Can — Memory Leaks</h3><h4>A retrospective on a memory leak</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*rrPt_ay6Q52RJfubN1zEuQ.png" /><figcaption>Elli engineers vs. memory leak (illus. by Jane Kim)</figcaption></figure><h3>Introduction</h3><p>Memory leaks are one of those things that, when they happen, can really throw you in at the deep end. Diagnosing them seems like a challenging task at first. They require a deep dive into the tools and components your service relies on. This close-up examination not only deepens your understanding of your service landscape, but also gives an insight into how things run under the hood. Although daunting at first glance, memory leaks are essentially a blessing in disguise.</p><p>At Elli, we do our best to minimise technical debt to a bare minimum. However, incidents still happen, and our approach is to learn and share knowledge by resolving such issues.</p><p>So, this article aims to do just this. In this post, we walk you through our approach of identifying a memory leak and share our learnings along the way.</p><h3>Context</h3><p>Before we dive into repairing the memory leak, we need some context on Elli’s infrastructure and where the memory leak occurred in the first place.</p><p>Elli, among other things, is a Charging Point Operator. We are responsible for connecting charging stations (CSs) to our backend and controlling them via <a href="https://www.openchargealliance.org/">OCPP</a> protocol. Ergo, our customers can charge their EVs at private or public stations. The CSs are connected to our systems via WebSockets. When it comes to authentication, we support connections via TLS or <a href="https://www.cloudflare.com/en-gb/learning/access-management/what-is-mutual-tls/">mutual TLS</a> (mTLS). During TLS, a CS will verify our server certificate and assure that it connects to an Elli backend. With mTLS, we also verify that the CS has a client certificate issued by us.</p><p>On the connectivity side, a server written in Node.js, is responsible for taking care of the <a href="https://www.rfc-editor.org/rfc/rfc7230#section-6.7">UPGRADE</a> logic from HTTP to WebSockets and keeping the state of the connections. It is deployed in a Kubernetes cluster and managed by a <a href="https://kubernetes.io/docs/tasks/run-application/horizontal-pod-autoscale/">Horizontal Pod Autoscaler</a> (HPA). Ideally, the HPA follows the traffic load and scales the pods up or down accordingly.</p><p><strong>We maintain tens of thousands of persistent and long-living TCP connections from the charging stations concurrently</strong>. This introduces complexity and significantly differs from typical RESTful services. A proxy metric that tracks the load is memory utilisation since it reflects the number of established connections, and the application’s logic does not require many computations. Our pods are long-lived, and scaling via memory led us to the observation that the number of pods is slowly increasing for a constant number of connections. Long story short, we spotted a memory leak.</p><h4>Impact assessment</h4><p>When faced with any kind of production issue, the Elli engineering team immediately assesses the implications of this incident on our customers and the business. So upon discovering this memory leak, we made the following assessment:</p><p><strong><em>The application is leaking memory in a matter of days. This means that without receiving any additional traffic our infrastructure continues to grow.</em></strong></p><p>When a pod cannot handle additional traffic, thanks to the readiness probe of Kubernetes, it stops receiving additional traffic but keeps serving the established connections. A pod that would serve X connections could end up only serving a fraction of its capabilities due to the leak, without causing any disruption on the customer side. This means that <strong>we can readily absorb the impact by simply spinning up more pods.</strong></p><h3>The Investigation</h3><p>Now for the actual technical deep-dive into the memory leak.</p><p>Here we explain the tools and methods we used to uncover the source behind the memory leak, what we expected to see from our experiment, and what we actually observed. We included links to the resources that we used in our investigation for your reference.</p><h4>A quick primer to JS memory</h4><blockquote>Variables in JavaScript (and most other programming languages) are stored in two places: stack and heap. A stack is usually a continuous region of memory allocating local context for each executing function. Heap is a much larger region storing everything allocated dynamically. This separation is useful to make the execution safer against corruption (stack is more protected) and faster (no need for dynamic garbage collection of the stack frames, fast new frame allocation).</blockquote><blockquote>Only primitive types passed by value (Number, Boolean, references to objects) are stored on the stack. Everything else is allocated dynamically from the shared pool of memory called the heap. In JavaScript, you do not have to worry about deallocating objects inside the heap. The garbage collector frees them whenever no one is referencing them. Of course, creating a large number of objects takes a performance toll (someone needs to keep all the bookkeeping), plus causes memory fragmentation.</blockquote><p>Source: <a href="https://glebbahmutov.com/blog/javascript-stack-size/">https://glebbahmutov.com/blog/javascript-stack-size/</a></p><h3>Taking a heap snapshot from a production pod | Heap Snapshots &amp; Profiling</h3><h4><strong>Expectations</strong></h4><p>We collected regular heap snapshots of our application to see an accumulation of objects over time. Due to the nature of the application, mostly holding WebSocket connections, we expected the TLSSocket objects to match the number of connections in the application. We hypothesised that when a station got disconnected, the object was somehow still referenced. Garbage collection works by cleaning up unreachable objects, so in this case, the objects would be left intact.</p><h4>Results</h4><p>Getting a heap dump from a 90% utilised pod resulted in the range of 100MB. Each pod requests around 1.5GB of RAM, and the heap was less than 10% of the allocated memory. This looked suspicious…</p><p><strong><em>Where was the rest of the memory allocated?</em></strong> Nonetheless, we continued the analysis. Taking three snapshots in intervals and observing the change in memory over time didn’t reveal anything. We didn’t notice an accumulation of objects nor were there any issues with garbage collection. The heap dump looked rather healthy.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/759/0*W07OlvWb-nApORER" /><figcaption><em>Image 1: Taking heap snapshots via chrome dev tools from a production pod. The number of TLSSocket objects aligns with the current pod’s connection contrary to the expected results.</em></figcaption></figure><p>The TLSSocket objects were matching the state of the application. Going back to the first observation, the heapdump is an order of magnitude less than the memory utilisation. We thought: <strong>“<em>This can’t be right. We are looking in the wrong place. We need to take a step back.”</em></strong></p><p>In addition, we profiled the application via the <a href="https://cloud.google.com/profiler/docs">Cloud Profiler</a> offered by GCP. We were interested in seeing how objects are allocated with the passing of time and potentially identifying the memory leak.</p><p><strong><em>Getting a heap dump blocks the main thread and can potentially kill the application, opposite to this the profiler can be kept in production with little overhead.</em></strong></p><blockquote>Cloud Profiler is a continuous profiling tool that is designed for applications running on Google Cloud. It’s a statistical, or sampling profiler with low overhead and is suitable for production environments.</blockquote><p>Although the profiler contributed to our understanding of the tenants of the heap, it still didn’t give us any leads on the investigation. On the contrary, it pushed us away from going in the right direction.</p><p>Spoiler alert: the profiler, however, did provide us with quite valuable information during an incident in production where we identified and fixed an aggressive memory leak, but that’s a story for another time.</p><h3>Memory usage statistics</h3><p>We needed greater insights into memory usage. We created dashboards for all metrics that<strong><em> </em></strong><a href="https://nodejs.org/api/process.html#processmemoryusage"><strong><em>process.memoryUsage()</em></strong></a> had to offer.</p><blockquote>The<strong> heapTotal</strong> and <strong>heapUsed</strong> refer to V8’s memory usage.</blockquote><blockquote>The<strong> external</strong> refers to the memory usage of C++ objects bound to JavaScript objects managed by V8.</blockquote><blockquote>The<strong> rss, Resident Set Size</strong>, is the amount of space occupied in the main memory device (that is, a subset of the total allocated memory) for the process, including all C++ and JavaScript objects and code.</blockquote><blockquote>The <strong>arrayBuffers</strong> refers to memory allocated for <strong>ArrayBuffers</strong> and <strong>SharedArrayBuffers</strong>, including all Node.js <strong>Buffers</strong>. This is also included in the external value. When Node.js is used as an embedded library, this value may be 0 because allocations for <strong>ArrayBuffers</strong> may not be tracked in that case.</blockquote><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*1JqU8aiqRnpFUlLI" /><figcaption>Image 2: A visualisation of RSS content. There is no official up-to-date model of V8’s memory as it’s changing quite frequently. This is our best effort to depict what lives under the RSS so we can have a clearer picture of potential memory components that leak memory. If you like to learn more about the garbage collector, we would suggest <a href="https://v8.dev/blog/trash-talk">https://v8.dev/blog/trash-talk</a>. Thanks to <a href="https://twitter.com/mlippautz">@mlippautz</a> for the clarification.</figcaption></figure><p>As we saw earlier we were getting ~100MB heap snapshots from a container that had more than 1 GB of memory utilisation. <strong><em>Where is the rest of the memory allocated?</em></strong> Let’s have a look.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*uMiF1mVB0bqv-vl7" /><figcaption>Image 3: Memory utilisation per pod (95th percentile). It grows over time. Nothing new here, we are aware of the memory leak.</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*3IBj7O191xJKTnts" /><figcaption>Image 4: The number of connections per pod over time (95th percentile); pods are handling fewer and fewer connections.</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*jSVcQcIYwwDooarp" /><figcaption>Image 5: Heap used memory (95th percentile). Heap is aligned with the size of the snapshots we collected and is stable over time.</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*zJLu0htplCHUdAM6" /><figcaption>Image 6: External memory (95th percentile): small in size and stable.</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*BUcJ_081WRnGvm-q" /><figcaption>Image 7: Memory utilisation and Resident Set Size (RSS) (95th percentile). There is correlation — RSS is following the pattern.</figcaption></figure><p><strong><em>What do we know so far?</em></strong> RSS is growing, the <em>heap</em> and <em>external</em> are stable which leads us to the stack. This could mean a method that gets called and never exits, thus leading to a stack overflow. However, the stack can’t be hundreds of MBs. At this point, we already tested in a non-production environment with multiple thousands of stations but didn’t get any results.</p><h3>Memory allocator</h3><p>While brainstorming, we considered memory fragmentation: chunks of memory are allocated non-sequentially leading to small chunks of memory that can’t be used for new allocations. In our case, the application is long-running and does a lot of allocation and freeing. Memory fragmentation is a valid concern in these cases. Extensive googling led us to a GitHub <a href="https://github.com/nodejs/help/issues/1518">issue</a> where the folks faced the same situation as us. The same pattern of the memory leak was observed, and it aligned with our hypothesis.</p><p>We decided to put a different memory allocator to test, and we switched from <a href="https://musl.libc.org/">musl</a> to <a href="https://github.com/jemalloc/jemalloc">jemalloc</a>. We found no meaningful results. At this point, we knew we needed to take a break. We had to rethink the approach entirely.</p><h4>Could it be that the leak only appears on mTLS connections?</h4><p>During our first tests, we tried to reproduce the issue in a non-production environment but had no luck. We ran load tests with thousands of stations simulating different scenarios, connecting/disconnecting stations for days, but they produced no meaningful results. However, we started to have a growing suspicion that there was something we missed while running these tests.</p><p>We didn’t take into account that our stations can connect via TLS or mTLS. Our first test included TLS stations, but not mTLS, and the reason for that is simple: we couldn’t easily create mTLS stations and the respective client certificates. A recent incident motivated us to minimise the blast radius and split the application’s responsibilities so that each deployment would handle TLS and mTLS traffic separately. <strong>Eureka! The memory leak appears only on our mTLS pods, while on TLS the memory is stable.</strong></p><h4><strong>Where do we go from here?</strong></h4><p>We decided that there are two options: (1) Move on to our next suspects — a library that handles all the Public Key Infrastructure tasks as well as a potential recursion somewhere in that code path, (2) or live with it until we rework our service entirely.</p><p>During the memory leak investigation, many unforeseen topics were brought to our attention that were related to the affected service. Taking into account the memory leak and everything else we discovered, <strong>we decided to improve our service landscape and split the responsibilities of the service</strong>. The CS authentication and authorisation flow, among others, would be delegated to the new service and we would use the right tools for handling PKI tasks.</p><h3>Summary</h3><p>Improving our scaling revealed that we have a memory leak that could have been left unnoticed for an indefinite period of time. Prioritising the customer and assessing the impact of the leak was first and foremost. Only then were we able to set the pace of our investigation since we realised that there was no customer impact. We started with the most obvious place to look when diagnosing a memory leak — the heap. Analysing the heap, however, showed us that we were looking at the wrong place. Further clues were needed and the <a href="https://nodejs.org/api/process.html#processmemoryusage">process</a> API of <a href="https://nodejs.dev/en/learn/the-v8-javascript-engine">V8</a> gave us exactly that. In the first results we got, the memory leak appeared in RSS. Finally, analysing all the information gathered, we suspected memory fragmentation.</p><p>Changing the memory allocator didn’t improve the situation. Rather, changing our approach and splitting the workload between TLS and mTLS, helped us narrow down the code path affected.</p><h4><strong>What were the final outcomes of our investigation?</strong></h4><p>Our plans to improve scalability along with addressing the memory leak, made us decide to split the service and write a new one to take care of the CS connectivity flow separately from the other CS specifics.</p><h4><strong>Did we fix our memory leak?</strong></h4><p>Time will tell, but I would say the investigation was way more than that. The experience of probing into the leak helped us grow as developers and our service to adopt a more resilient and scalable architecture.</p><h4>Key points and learnings</h4><ul><li><em>Τough engineering challenges bring people together; we played ping-pong on ideas with engineers outside of our team.</em></li><li><em>Gave us the motivation to rethink the service, which led to a more scalable architecture.</em></li><li><em>If it aches, it requires your attention; don’t ignore it.</em></li></ul><h4>References</h4><ul><li><a href="https://nodejs.org/en/docs/guides/diagnostics/memory/using-heap-snapshot/">https://nodejs.org/en/docs/guides/diagnostics/memory/using-heap-snapshot/</a></li><li>Enable remote debugging to a pod via port forwarding: <a href="https://kubernetes.io/docs/tasks/access-application-cluster/port-forward-access-application-cluster/">https://kubernetes.io/docs/tasks/access-application-cluster/port-forward-access-application-cluster/</a></li><li><a href="https://developers.google.com/cast/docs/debugging/remote_debugger">https://developers.google.com/cast/docs/debugging/remote_debugger</a></li></ul><h4>Learn More</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*y_ruRCcUupI-vfbfPgqIYA.png" /><figcaption>Follow us on Medium! (illus. by Jane Kim)</figcaption></figure><p>If you are interested in finding out more about how we work, please subscribe to the <a href="https://elli-engineering.medium.com/">Elli Medium blog</a> and visit our company’s website at <a href="http://elli.eco/">elli.eco</a>! See you next time!</p><p><strong>About the author</strong></p><p><a href="https://www.linkedin.com/in/thanosam/"><strong><em>Thanos Amoutzias</em></strong></a><em> is a Software Engineer, he develops Elli’s Charging Station Management System and drives SRE topics. He is passionate about building reliable services and delivering impactful products. You can find him on </em><a href="https://www.linkedin.com/in/thanos-amoutzias/"><em>LinkedIn</em></a><em> and in the 🏔️.</em></p><p><em>Credits: Thanks to all my colleagues who reviewed and gave feedback on the article!</em></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=eb2604c0c7ea" width="1" height="1" alt=""><hr><p><a href="https://medium.com/elli-engineering/catch-me-if-you-can-memory-leaks-eb2604c0c7ea">Catch Me If You Can — Memory Leaks</a> was originally published in <a href="https://medium.com/elli-engineering">Elli Engineering</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Don’t Do Small Pull Requests]]></title>
            <link>https://medium.com/elli-engineering/dont-do-small-pull-requests-2b9f8793e4e8?source=rss-664565f11d82------2</link>
            <guid isPermaLink="false">https://medium.com/p/2b9f8793e4e8</guid>
            <category><![CDATA[pull-request]]></category>
            <category><![CDATA[software-engineer]]></category>
            <category><![CDATA[software-engineering]]></category>
            <category><![CDATA[software-development]]></category>
            <category><![CDATA[pair-programming]]></category>
            <dc:creator><![CDATA[Elli Engineering]]></dc:creator>
            <pubDate>Tue, 04 Oct 2022 10:57:40 GMT</pubDate>
            <atom:updated>2024-01-15T11:15:30.485Z</atom:updated>
            <content:encoded><![CDATA[<h4>How asynchronous reviews and wait times harm throughput and code quality</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*HdWOvNDrnxe4mmzyJMtkiw.png" /><figcaption>Smaller the pull request, the better? (illustration by Jane Kim)</figcaption></figure><h3>Introduction</h3><p>In this blog post, I share my learnings in the last couple of months regarding the delivery process of our engineering team at Elli. It was during this time that I came across <a href="https://dragan-stepanovic.github.io/about.html">Dragan Stepanovic</a>’s presentation at the <a href="https://www.meetup.com/de-DE/seattle-software-craftsmanship/">Seattle Software Crafters Meetup</a>. This post is in large part inspired by the work Dragan Stepanovic presented in his talk.</p><h3>My Hypothesis</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*IRrTTwrtSkpPt-CgBSIjUQ.png" /><figcaption>Does smaller PR size always mean higher velocity and code quality? (illustration by Jane Kim)</figcaption></figure><p>During our sprint retrospective, my team and I identified a core bottleneck in our delivery process. During our sprint, we worked on pull requests (PRs) that stayed open for an unusually long time, waiting for reviews, comments to be addressed, merge conflicts to be resolved, you know the deal. As a way to eliminate this bottleneck, I suggested: <strong><em>let’s simply slice our tasks and PRs to smaller parts</em></strong>. My hypothesis was that smaller PRs are easier to comprehend, thereby allowing more thorough and time-efficient reviews and, as a result, increase our team’s velocity and code quality. I was convinced this would be the solution to our bottleneck.</p><h3>The Method</h3><p>In order to test my hypothesis I did two things:</p><ol><li>First, I started to <strong>slice my PRs to really small parts</strong>.</li><li>Second, I decided to <strong>analyse</strong> <strong>some</strong> <strong>data</strong>.</li></ol><p>I analysed the following variables over the course of 9 months:</p><ul><li><strong>Size of the PR: </strong>the unit of measurement for PR size in this case would be the number of changes that this PR includes. Azure DevOps reports a “<em>change count</em>” metric for pull requests, which we will conveniently use for our data analysis.</li><li><strong>PR’s Time to Completion</strong>: The total time (in business hours) taken from the point at which the PR was opened to when it was merged.</li><li><strong>Reaction time</strong> (or wait time): The time taken for a reviewer to “react” with a comment or an approval to the PR.</li></ul><p><strong>Disclaimer: </strong>This analysis does not consider the corresponding tasks and therefore ignores the time spent before the PR was created.</p><h3>The Analysis</h3><h4><em>Do we increase our velocity when we slice asynchronous pull requests to smaller parts?</em></h4><p>To get a better understanding of the relation between PR size and time to completion, I plotted a line chart that shows the number of business hours spent until the PR merges. The graph clearly shows that smaller PRs complete faster, which we all probably assumed from personal experience. The drop-off to the far right in the graph is an interesting exception and we will get back to this in a later part of this post.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*U3cB4kh6qxS3-gP_s7mkug.png" /><figcaption>The relation between PR size and time to completion of a PR in business hours, proving the initial hypothesis that smaller PRs complete faster. (graph by Matthias Förth)</figcaption></figure><p>However, when I sliced my PRs really small I did not get the impression that I completed my work faster, rather it felt like the opposite. The next graph reveals that the time per change actually reaches its maximum when change count is really small (to the left) and reaches its minimum when change count gets exceptionally large (to the right).</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*1qCqjywy5n0BGgdEn1QZ-g.png" /><figcaption>The time required to merge a change in business hours (purple) and the ratio of PR with comments (green) depending on the PR size to illustrate the effect of the asynchronous discussion and approval process. (graph by Matthias Förth)</figcaption></figure><p>Ultimately our goal is not to get as many PRs closed as possible but to <strong>merge meaningful code</strong>. If we want to achieve that in the fastest way possible we should, in fact, make larger PRs, according to these numbers.</p><p>Yet, in the middle, there is an interesting minimum on the completion time per change at 5–10 changes, which suggests that slicing a 22 change-PR in half actually reduces its time for completion. Let’s look at this dip more closely, since this area is quite important (we merge 50% of our changes as PRs with a change count between 5 and 23).</p><h4>What is the effect of asynchronous comments and responses?</h4><p>The green line shows the percentage of PRs with one or more comments. Below 13 changes there are more PRs without comments. Above 13 changes there are more PRs with comments. This asynchronous discussion that arises as a result of comment submission leads to the increase in the time to completion. Below 13 changes, the wait time for a reviewer to react to a pull request makes up a large part of the time for a PR to complete. Above 13 changes, the initial reaction time (or the wait time taken for a reviewer to react) makes up less of the total completion time since the asynchronous discussion includes not just the initial wait for comments but also addressing them, and potentially further review loops.</p><p>How big is the impact of these reaction times actually? To gauge the size of this impact, I plotted the wait time until the first reaction (approval or comment) as a percentage of the total time taken for the PR to complete.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*0MzPFot7s96wlv2oRm-u4A.png" /><figcaption>Comments per change as an indicator for code quality (purple) and the ratio of the time to a first reaction compared to the total completion time of the PR (pink) to show the tradeoff between the impact of wait times in the asynchronous review process and code quality.</figcaption></figure><p>We see that especially for smaller PRs this initial reaction time makes up the majority of the time to actually complete the PR. Yet, even though the majority of small PRs get approved without any comment from the reviewer, the ratio of comments per change is still the highest in this area. We would conclude from this that we get the most in-depth reviews on these small PRs, but for many, the code can be merged right away. So what do we make out of this now?</p><h3>My Findings</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*i10cJxx0F7nH8C0ZwXUvMA.png" /><figcaption>In general, aim for medium-sized PRs! (illustration by Jane Kim)</figcaption></figure><ul><li>To get the best combination of merging changes quickly but still receiving an in-depth review we should aim for <strong>medium-sized PRs</strong>.</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*FDAXCUUnMb33FGEFsx_KiQ.png" /><figcaption>Pass the PR-baton as soon as possible! (illustration by Jane Kim)</figcaption></figure><ul><li>On these medium-sized PRs, the reaction time makes up a considerable portion of the total time to complete the PR. Therefore, there is quite some potential to speed up delivery by <strong>keeping any</strong> <strong>reaction time low</strong> (i.e. complete reviews, addressing comments or pipeline failures as efficiently as possible)</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*UNA5dc_5N2ItrINp0iWd6A.png" /><figcaption>Do the heavy-lifting of PRs together in pairs! (illustration by Jane Kim)</figcaption></figure><ul><li>The first chart showed a drop-off in the “time to completion” for very large PRs. This is due to the fact that many of these PRs have been completed in <strong>pairing mode</strong>. At Elli, <strong>we tend to do large refactoring’s or complex business logic implementations via</strong> <strong>pair programming</strong>. Instead of doing the review process asynchronously, where one person creates the PR and another reviews, we synchronously code and review in the pairing session. Therefore, there are no comments on the PRs and <strong>approval may be granted right away</strong>. This leads to the drop in the completion time for very large PRs, but this reduction should very well apply to medium-sized PRs as well.</li></ul><h3>Conclusion</h3><p>Going back to the initial problem of long-running PRs in our retro, I would now no longer simply recommend slicing the PRs to really small pieces as a blanket solution to faster delivery times. Instead, I would recommend slicing PRs into parts such that the <strong>“heavy lifting” is done via pair-programming</strong>. The more independently manageable PRs should have a <strong>medium size</strong> to justify the wait times of asynchronous reviews.</p><p>But ultimately, there is <strong><em>no magic number</em></strong> when it comes to pull request size. However, what I did discover from my data analysis were several important guidelines to keep in mind in order to optimise the pull request workflow of our product teams. I hope these findings are useful and extend to anyone who is interested in improving delivery times within their team or organization.</p><h3>Learn More</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*y_ruRCcUupI-vfbfPgqIYA.png" /></figure><p>At Elli, we are always in search of data-driven ways to improve the way we work together as a team. If you are interested in finding out more about how we work, please subscribe to the <a href="https://elli-engineering.medium.com/">Elli Medium blog</a> and visit our company’s website at <a href="http://elli.eco/">elli.eco</a>! See you next time!</p><p><strong>About the author</strong></p><p><a href="http://www.linkedin.com/in/matthias-foerth-60b68a173"><strong><em>Matthias Förth </em></strong></a><em>is</em> <em>a Product Owner and former Software Engineer focused on backend development. His current interests are the streamlining of development and pushing data-driven decision making at Elli.</em></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=2b9f8793e4e8" width="1" height="1" alt=""><hr><p><a href="https://medium.com/elli-engineering/dont-do-small-pull-requests-2b9f8793e4e8">Don’t Do Small Pull Requests</a> was originally published in <a href="https://medium.com/elli-engineering">Elli Engineering</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Electric Vehicle Charging for Newbies]]></title>
            <link>https://medium.com/elli-engineering/electric-vehicle-charging-for-newbies-717b6d9d6946?source=rss-664565f11d82------2</link>
            <guid isPermaLink="false">https://medium.com/p/717b6d9d6946</guid>
            <category><![CDATA[emobility]]></category>
            <category><![CDATA[ev-charging]]></category>
            <category><![CDATA[electric-vehicle-market]]></category>
            <category><![CDATA[electric-vehicle-chargers]]></category>
            <category><![CDATA[ev]]></category>
            <dc:creator><![CDATA[Elli Engineering]]></dc:creator>
            <pubDate>Mon, 01 Aug 2022 07:14:11 GMT</pubDate>
            <atom:updated>2022-11-25T12:06:38.345Z</atom:updated>
            <content:encoded><![CDATA[<h4>A quick read for all newbies to EV charging!</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*uvtphGHIo7ngwqZnivQf1Q.png" /></figure><p>Creating a sustainable future means changing the way we get around. Perhaps this means switching to “greener” modes of transportation like commuting by bike or public transit. It could also mean reducing the ecological footprint of existing modes of transportation. The proliferation of <strong>electric vehicles (EV)</strong> is a prime example of this. Electric cars are becoming easier to manufacture and thus more accessible among the public. In addition, the European Commission has approved <strong>a ban on the sale of cars with combustion engines by 2035</strong>, with the aim of achieving a 100% reduction in carbon dioxide emissions from new cars. In the near future, electric vehicles will outnumber the petrol and diesel-based cars on the road!</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*P8IYXXma2pg9kecqK6euXQ.png" /><figcaption>Electric vehicles for the win! (illustration by Jane Kim)</figcaption></figure><p>As electric vehicles are becoming increasingly mainstream, it is important to get a gist of the technology behind them. This article will deliver the key points on e-mobility and EV charging without diving too deep on the technical details. ;)</p><h3><strong>What is a battery electric vehicle (BEV)?</strong></h3><p>An electric vehicle drives exactly as the petrol or diesel-based car that most of us are familiar with. However, the key difference is that in a battery electric vehicle (BEV), the internal combustion engine, present in most of the vehicles on the road today, has been replaced by an <strong>electric motor and a battery</strong>. This motor <strong>converts electrical energy stored in the battery to mechanical energy </strong>while producing <strong>zero tailpipe emissions</strong>. The battery that supplies power to the electric motor is most commonly a <strong>lithium-ion battery </strong>and must be plugged in and charged for continuous use.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*1YKIWKgal1ZMrpofKaz3bw.png" /><figcaption>Electric motor converts electrical energy to mechanical energy (illustration by Jane Kim)</figcaption></figure><h3><strong>How do electric vehicle batteries work?</strong></h3><p>The battery inside the EV itself is comparable to the battery in the smartphones that we use day-to-day and function more or less in the same way.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*6jQKsLJJR5VJz0gBEZA2Vg.png" /><figcaption>Current flows from the battery to the consumer and back to the battery again (illustration by Jane Kim)</figcaption></figure><p>As we all know, batteries have a positive pole and a negative pole. Current flows in a single direction from the positive pole into the electrical consumer (e.g. an EV or smartphone) and back into the negative pole. This kind of electric current flowing in a single direction is known as <strong>direct current (DC)</strong> and <strong>batteries are always charged and discharged in the form of DC.</strong></p><p>Conversely, the electricity coming from the power grid is always <strong>alternating current (AC)</strong>. As its name suggests, alternating current means that the flow of electric charge <strong>periodically changes direction</strong>. Because the power grid uses AC, but batteries only store DC power, there needs to be a <strong>conversion</strong> that takes place in order to charge any kind of battery.</p><h4>What are the differences between direct current (DC) and alternating current (AC)?</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Xj6QzxoKwSQtuyRfsuYm3A.png" /><figcaption>Direct current vs. alternating current (illustration by Jane Kim)</figcaption></figure><h3><strong>How do you charge your EV?</strong></h3><p>A car-to-station <strong>cable</strong> connects an EV to a socket or a charging station and the power flows from the power source to the EV via this cable. How fast the EV charges depends on the type (i.e. mode) of this charging cable that connects the car and station. Some charging cables deliver power faster than others and even allow for communication between the EV and the station.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*-vwtRcnihe99qxKCozKjTQ.png" /><figcaption>A cable connects the EV to the charging station (illustration by Jane Kim)</figcaption></figure><h3>What are the different types of charging stations?</h3><p>There are two different types of charging stations (a.k.a. chargers) and their difference lies in where this conversion from AC to DC occurs.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*yiqz_90ivgSgk39r9b7B-A.png" /><figcaption>The differences between an AC charger and DC charger (illustration by Jane Kim)</figcaption></figure><p>In summary, an AC charging station relies on the charger <strong>onboard the EV</strong> to convert the AC power from the grid to DC. On the other hand, a DC charging <strong>station does the conversion</strong> itself and feeds this power directly to the EV’s battery.</p><h3>What is Three-Phase Current?</h3><p>Alternating Current can be further categorised on the number of “phases”:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*u5HgffIHQ7uuBDoWnVlFyQ.png" /><figcaption>Single-phase vs. three-phase alternating current (illustration by Jane Kim)</figcaption></figure><p>In three phase installations or grid connections, power is supplied to your home or charging site from the power grid via three conductors or wires (see image below).</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*ySXK304buICRRyBasSC0Vw@2x.png" /><figcaption>Type 2 Connector Female (left) and Male (right) Plug Pinouts (illustration by Jane Kim)</figcaption></figure><p>Since the current flowing through these conductors is <strong>staggered</strong>, a three-phase charging station delivers a <strong>higher flux of power per unit time </strong>than a single-phase charging station.</p><h3>Good bye for now!</h3><p>Here are some key principles of EV charging that will serve as a basic foundation for e-mobility related topics. For more articles on the intersection between e-mobility and software engineering, please check out the other articles on the Elli engineering blog!</p><p>To stay in the loop, follow us on Medium and be sure to check out our website at <a href="http://elli.eco/">elli.eco</a>!</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Jgzo1nMt2X7ypZofsxR_Zw.png" /><figcaption>See you next time in the next article of “for Newbies”! (illustration by Jane Kim)</figcaption></figure><p><a href="https://www.linkedin.com/in/jaryungkim/"><strong><em>Jane Kim</em></strong></a><em> is a Software Engineer in Team Cannelloni and a part-time illustrator for the Elli blog</em></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=717b6d9d6946" width="1" height="1" alt=""><hr><p><a href="https://medium.com/elli-engineering/electric-vehicle-charging-for-newbies-717b6d9d6946">Electric Vehicle Charging for Newbies</a> was originally published in <a href="https://medium.com/elli-engineering">Elli Engineering</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Introduction to Elli Engineering: Our Guiding Principles]]></title>
            <link>https://medium.com/elli-engineering/introduction-to-elli-engineering-our-guiding-principles-cb0c9a1b907b?source=rss-664565f11d82------2</link>
            <guid isPermaLink="false">https://medium.com/p/cb0c9a1b907b</guid>
            <category><![CDATA[ev-charging]]></category>
            <category><![CDATA[volkswagen]]></category>
            <category><![CDATA[engineering]]></category>
            <category><![CDATA[electric-vehicles]]></category>
            <category><![CDATA[ellis]]></category>
            <dc:creator><![CDATA[Elli Engineering]]></dc:creator>
            <pubDate>Wed, 01 Jun 2022 09:26:34 GMT</pubDate>
            <atom:updated>2022-11-25T12:38:13.561Z</atom:updated>
            <content:encoded><![CDATA[<h4>The six guiding principles for technical excellence</h4><p>Elli is a brand of Volkswagen Group providing energy and electric charging solutions. Software and hardware engineering are key to the business. We as engineers focus on creating and maintaining awesome products for the customer’s needs.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*1ZxjD-iBU6EUIJqS2__B9A.png" /></figure><p>We’ve created this engineering blog to share insights from our exciting journey. To start with, we’re introducing our guiding principles for technical &amp; operational excellence.</p><h3>The Six Guiding Principles</h3><h3>1. Best idea wins</h3><p>By empowering an open &amp; collaborative culture, we tend to give and provide feedback seamlessly. In order to find the best solution in product development, we listen to everyone by removing the hierarchy of ideas. As engineers we are involved from the concept to planning execution.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*dxGczMiWvfM2b1ojy_DT_Q.png" /><figcaption>The best idea for the win! (illustration by Jane Kim)</figcaption></figure><h3>2. Knowledge sharing</h3><p>Knowledge is power and power must be shared. Following this principle, we unite in workshops every sprint. Engineers from different teams come together to create communities of practice and organise such workshops.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*wSBC2xe0Of8_XDfiA85Q5w.png" /><figcaption>We participate in knowledge-sharing workshops called “Communities of Practice” (CoP) every sprint. (illustration by Jane Kim)</figcaption></figure><h3>3. You build it, you run it</h3><p>We strongly believe that a team is most creative and productive when working as independently as possible. To avoid conflicting requirements, each team is accountable for a clear purpose. Teams have full control and ownership of their source code. The team designs it, builds it and runs it.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*7lML61tyXeh-6txUKh4Z5Q.png" /><figcaption>Each team is an independent entity that owns their own product! (illustration by Jane Kim)</figcaption></figure><h3>4. Minimise feedback cycle time</h3><p>We build things fast and don’t over-engineer them at a cost to maintainability. We prefer real-world data to drive improvements. To engage with stakeholders, we announce and explain technical details in Architectural Decision Records, or ADRs. Other teams can then give direct feedback on decisions.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*_BeqIpPFYfkw6gM8JZTrbg.png" /><figcaption>Team Stelline publishes an ADR. The members of the pasta teams review the ADR. (illustration by Jane Kim)</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*wJ2KJHXC8uEnk5O-N-_rTg.png" /><figcaption>Even Beyonce says so. (illustration by Jane Kim)</figcaption></figure><h3>5. Technical excellence</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*O3gCaovfP9e6xhthb3vM9g.png" /><figcaption>Hmm… Technical excellent… (illustration by Jane Kim)</figcaption></figure><p>We constantly strive for technical and operational excellence (TOX). What we mean by that is we focus on continuously enhancing our apps until there is no more room for improvement. While in a real world scenario this ideal can never be fully achieved, we still aim for it. 🚀</p><h4>How is technical excellence achieved?</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*GMJiZCcf0gjaBRst4IQj7g.png" /></figure><p>TOX activities are driven completely by us as engineers. They are seen as a parallel stream to feature development, and within our teams we are dedicated to investing ~ 25% of the time in such tasks.</p><h3>6. Automate the repetitive</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*DOWR6m1jZN8cuuX8EqEMQA.png" /><figcaption>(illustration by Jane Kim)</figcaption></figure><p>We automate everything that is worth automating. Hence, our creative minds are freed, and we can push Elli’s products forward instead. Automation helps us to execute repeatable tasks reliably.</p><h3>To Sum Up</h3><p>At Elli, software engineering is a crucial part for the business. As dedicated engineers, our goal is to strive for excellence in our platforms. In order to achieve our target, we created six guiding principles that are followed by all teams. With these principles, we encourage strong collaboration, team independence, and continuous improvements.</p><p>We strongly believe in open culture, and we want to share our insights publicly via this engineering blog.</p><h3>Interested? Get in touch!</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*y_ruRCcUupI-vfbfPgqIYA.png" /></figure><p>Elli is here to make our life with electricity easier, more reliable, and totally free of hassle. We are always looking out for new colleagues who can bring new perspectives to our team!</p><p>Let’s keep in touch:</p><ul><li><a href="https://www.linkedin.com/company/elli-vw">LinkedIn</a></li><li><a href="http://elli.eco/">Elli Site</a></li></ul><h3>Stay tuned!</h3><p>This article is just the start. We still have so much more knowledge to share, amazing people to introduce, and awesome things to brag about ;). Coming soon!</p><p><strong>About the Blog Team</strong></p><p><a href="http://linkedin.com/in/besmirethaqi"><strong><em>Besmire Thaqi</em></strong></a><em> is a Software Engineer focused in backend development. Her current interests include pragmatic programming and bridging the gap between tech and business.</em></p><p><a href="https://www.linkedin.com/in/d-owens/"><strong><em>Daniel Owens</em></strong></a><em> is a Software Engineer and Tech Lead with a keen interest in architecting reliable distributed systems.</em></p><p><a href="https://www.linkedin.com/in/jaryungkim/"><strong><em>Jane Kim</em></strong></a><em> is a Software Engineer in Team Cannelloni and a part-time illustrator for the Elli blog.</em></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=cb0c9a1b907b" width="1" height="1" alt=""><hr><p><a href="https://medium.com/elli-engineering/introduction-to-elli-engineering-our-guiding-principles-cb0c9a1b907b">Introduction to Elli Engineering: Our Guiding Principles</a> was originally published in <a href="https://medium.com/elli-engineering">Elli Engineering</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
    </channel>
</rss>