<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Steve Bowerman on Medium]]></title>
        <description><![CDATA[Stories by Steve Bowerman on Medium]]></description>
        <link>https://medium.com/@steve-bowerman?source=rss-a6ab148b5db3------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/1*tyO10Tab6XqIhK9DtvALxA.jpeg</url>
            <title>Stories by Steve Bowerman on Medium</title>
            <link>https://medium.com/@steve-bowerman?source=rss-a6ab148b5db3------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Mon, 16 Nov 2026 23:36:36 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@steve-bowerman/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[Guilds and Communities of Practice: Uniting People, Sharing Knowledge]]></title>
            <link>https://medium.com/edf-data-and-tech/guilds-and-communities-of-practice-uniting-people-sharing-knowledge-34c774a99cd4?source=rss-a6ab148b5db3------2</link>
            <guid isPermaLink="false">https://medium.com/p/34c774a99cd4</guid>
            <category><![CDATA[community-of-practice]]></category>
            <category><![CDATA[people]]></category>
            <category><![CDATA[town-hall]]></category>
            <category><![CDATA[inclusion]]></category>
            <category><![CDATA[guild]]></category>
            <dc:creator><![CDATA[Steve Bowerman]]></dc:creator>
            <pubDate>Fri, 09 May 2025 09:33:36 GMT</pubDate>
            <atom:updated>2025-05-09T09:33:36.766Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*Xg7C1X52BzT1uEls" /><figcaption>Photo by <a href="https://unsplash.com/@helenalopesph?utm_source=medium&amp;utm_medium=referral">Helena Lopes</a> on <a href="https://unsplash.com/?utm_source=medium&amp;utm_medium=referral">Unsplash</a></figcaption></figure><p>Our <a href="https://medium.com/@edfdatatech">Data &amp; Tech</a> ( Software Engineering, Data and AI) culture at EDF is formed upon a strong contribution and collaboration mindset which encourages and rewards:</p><ul><li>Breaking out of siloed thinking that often product teams can be guilty of.</li><li>A sense of belonging to a bigger collective / community rather than just a product team.</li><li>Access to a wider knowledge pool.</li><li>Mentoring / coaching and leadership opportunities.</li><li>Internal mobility.</li><li>Continuous improvement of self, team and wider community.</li></ul><p>Growing our practice from zero to over 200 members in just five years presented a dichotomy:</p><p><strong><em>more people &amp; more contribution opportunities and common challenges.</em></strong></p><p><strong><em>vs</em></strong></p><p><strong><em>more people being harder to co-ordinate, and the risk of members just being a number.</em></strong></p><h3>Solutions</h3><p>We’ve experimented and adopted these three useful methods.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*ak8fa4Jwt2SMnPY_" /><figcaption>Photo by <a href="https://unsplash.com/@john_cameron?utm_source=medium&amp;utm_medium=referral">John Cameron</a> on <a href="https://unsplash.com/?utm_source=medium&amp;utm_medium=referral">Unsplash</a></figcaption></figure><h4>1. Town Halls</h4><p>We run monthly Software Engineering and Data guild town halls. A town hall is very much a broad cascade approach for broadcast content to a large group of people, typically around 60–80 or more.</p><p>These are typically run as a hybrid meeting, so that we can accommodate folks meeting and interacting in person — the ‘water cooler moments’ as we’ll as accommodating our geographically disparate teams.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*HazOdRkyinBURE3Y" /><figcaption>Photo by <a href="https://unsplash.com/@austindistel?utm_source=medium&amp;utm_medium=referral">Austin Distel</a> on <a href="https://unsplash.com/?utm_source=medium&amp;utm_medium=referral">Unsplash</a></figcaption></figure><p><em>Good for</em>:</p><ul><li>Strategy &amp; direction updates.</li><li>Showcases.</li><li>Visibility — ‘oh we are trying to do something similar’.</li></ul><p><em>Less useful for</em>:</p><ul><li>Solving problems.</li><li>Discussion &amp; Ideation.</li></ul><h4>2. Communities of Practice</h4><p>These are smaller, more focused groups (10–20 people) that collaborate on a common areas to seek improvements, typically with a 3x month roadmap of activities.</p><p>Examples include:</p><ul><li>Front-End CoP — That own and curate our storybook and UI component library, and promote learning and development around UI design and build skills.</li><li>DevOps CoP — That own and curate our Terraform modules, Github actions library, and promote strong DevOps &amp; SRE practise and maturity.</li><li>Data &amp; Tech Brand — That own and curate our Medium and LinkedIn social presence and promote writing and personal brand building skills.</li></ul><p>The CoP’s tend to meet weekly and cascade progress out our town hall. They also tend to be online Teams calls rather than in-person.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*jYovhCi5wBJjfhgO" /><figcaption>Photo by <a href="https://unsplash.com/@hannahbusing?utm_source=medium&amp;utm_medium=referral">Hannah Busing</a> on <a href="https://unsplash.com/?utm_source=medium&amp;utm_medium=referral">Unsplash</a></figcaption></figure><p><em>Good for:</em></p><ul><li>Problem solving, ideation and discussion.</li><li>Tracking delivery progress.</li><li>Learning from others.</li><li>‘Getting stuff done’.</li></ul><p>Less useful for:</p><ul><li>Broadcast (updates given at town hall) or lightning talks.</li></ul><h4>3. In Person Events</h4><p>Periodically we run in-person events, typically hackathons, or deep dive sessions where we’ll pick a focus area (eg resilience) and work through concepts and practical implementation steps to broaden teams understanding and create some actionable take-aways for teams to implement.</p><p>In person events are super important given our geographic disparity and remote working. It gives us a good mixture of collaboration and work focus time as well as great social interaction and relationship building.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*cIKR3tWdTSrdRrfZ" /><figcaption>Photo by <a href="https://unsplash.com/@heftiba?utm_source=medium&amp;utm_medium=referral">Toa Heftiba</a> on <a href="https://unsplash.com/?utm_source=medium&amp;utm_medium=referral">Unsplash</a></figcaption></figure><p><em>Good for:</em></p><ul><li>Socialising!</li><li>Relationship building (non-verbal language, stuff you can’t get from Teams calls).</li><li>Sharing and embedding concepts, methodologies, new technologies.</li><li>Inclusive for all levels to ask questions. Also serves as an opportunity for more senior members to coach and guide.</li></ul><p>Less useful for:</p><ul><li>Large numbers. Typically we see 30–40 as good size</li><li>Long running activities that run over multiple days or weeks.</li></ul><h3>Observations</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*dki2h7nCYeMARI1Q" /><figcaption>Photo by <a href="https://unsplash.com/@rudoy?utm_source=medium&amp;utm_medium=referral">Ivan Rudoy</a> on <a href="https://unsplash.com/?utm_source=medium&amp;utm_medium=referral">Unsplash</a></figcaption></figure><p>We’ve made a few observations along the way which have caused us to adapt and evolve our approaches:</p><h4>1. Inclusivity</h4><p>We have a broad range of skills and experience, so how do we make things accessible for juniors whilst interesting for seniors?</p><p>We try and avoid ‘beginner’ or ‘advanced’ sessions, and instead ensure that the context, concepts are understood before jumping into detail — bring everyone along together.</p><p>For CoP’s specifically, we’ve seen an onion skin approach where folks can start off by attending and listening, gradually building up skills and expertise before contributing and ultimately leading. Its all about creating a safe space to build confidence and learn.</p><h4>2. Time away from ‘the day job’</h4><p>Whilst there is an argument that CoP’s and these types of activities are ‘extra curricular’, they are essential to both learning and cohesion. As such, they are treated as learning and development and hence:</p><ul><li>Time allocated for learning and development (recommend about 0.5 day a week)</li><li>Contribution to the practise aligns to ‘community’ objectives that our engineers have.</li><li>Senior leadership already bought into the value of the community from the outset of our formation.</li></ul><h4>3. Momentum &amp; the burden of organisation</h4><p>Initially much of the ‘push’ has been from engineering leadership to enable and organise. Once we established some repeatability, the organisation tasks could be picked up by anyone.</p><p>Our graduate community have been strong advocates and provide the energy and enthusiasm (with the reward of networking and recognition) driving things forward.</p><h3>Top Tops</h3><ul><li>Find your advocates and evangelists</li><li>Make it easy for content and topic ideas to be shaped — provide a service to help get an idea into fruition.</li><li>Less is more — a small number of higher value events and CoP’s are more valuable for CoP’s for too many topics.</li></ul><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=34c774a99cd4" width="1" height="1" alt=""><hr><p><a href="https://medium.com/edf-data-and-tech/guilds-and-communities-of-practice-uniting-people-sharing-knowledge-34c774a99cd4">Guilds and Communities of Practice: Uniting People, Sharing Knowledge</a> was originally published in <a href="https://medium.com/edf-data-and-tech">EDF Data and Tech</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[How EDF Built a DevOps Practice on AWS]]></title>
            <link>https://medium.com/edf-data-and-tech/how-edf-built-a-devops-practice-on-aws-84e7ab3d0777?source=rss-a6ab148b5db3------2</link>
            <guid isPermaLink="false">https://medium.com/p/84e7ab3d0777</guid>
            <category><![CDATA[technology]]></category>
            <category><![CDATA[talent-retention]]></category>
            <category><![CDATA[culture-change]]></category>
            <category><![CDATA[devops-practice]]></category>
            <category><![CDATA[communities-of-practice]]></category>
            <dc:creator><![CDATA[Steve Bowerman]]></dc:creator>
            <pubDate>Mon, 03 Jun 2024 08:14:01 GMT</pubDate>
            <atom:updated>2024-06-03T08:14:01.825Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*jwydgvm80ayBueQL" /><figcaption>Photo by <a href="https://unsplash.com/@growtika?utm_source=medium&amp;utm_medium=referral">Growtika</a> on <a href="https://unsplash.com/?utm_source=medium&amp;utm_medium=referral">Unsplash</a></figcaption></figure><p>Adopting DevOps is widely considered as the ‘right’ way of working when building and supporting modern cloud based solutions.</p><p>Whilst the DevOps enabling technologies are mature and plentiful, and there are an abundance of engineers that work (and want to work) in a DevOps way, there are many barriers to entry, particularly for larger organisations with heritage processes and technologies.</p><p>I’ll step you through EDF’s journey over the last 3–4 years.</p><h3>Why DevOps?</h3><p>Recognising the need to be more <strong>agile</strong>, to <strong>innovate, </strong>and to<strong> do-more-with-less </strong>in a highly competitive market<strong>, </strong>we set-out with a hypothesis that we’d attain this by:</p><ul><li>Forming ourselves around product teams and agile methodologies to provide greater end-to-end ownership and accountability.</li><li>Adopting DevOps as a delivery and operate model that builds on the product ownership mindset, that removes hand-offs between build vs run, and that promotes a single team — single goals culture.</li><li>Building an internal capability of engineers to own and adapt the IP and enable better use of partners (capacity and skills gaps) rather than traditional outsourcing.</li><li>Leveraging our existing AWS usage, but shifting towards higher levels of automation and use of serverless architectures.</li></ul><h3>Stepping Stones</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*9uE4wRA66WwYr-LRlkEmBA.png" /><figcaption>High level journey</figcaption></figure><p>Rome wasn’t built in day, and nor can your DevOps transformation. Whilst our journey was (and is) a continuous evolution over time, we’ve chunked it up into 3x discrete phases of maturity:</p><h4>1. Learn by Doing, and prove that we can do this!</h4><p>Building upon the need to act and to work out how we can operate in a DevOps mode, we ran an initial incubation project.</p><p>The goal was to form a product team, deliver customer facing product and to run that product as a DevOps / product that acted as the North Star for how we wanted to work:</p><ul><li>Product team</li><li>Embedded engineers</li><li>Agile roles like Product Owner rather than traditional PM</li><li>AWS serverless architecture that used high levels of automation</li><li>Full ownership of product lifecycle including design, build, deploy, operate.</li></ul><p>As we lacked a lot of the core skills (but have plenty of delivery and AWS experience) we hired in some DevOps experts to work with the team. It should also be noted that the product team was formed of individuals wanting to help make this work — and hence good messengers and evangelists for success.</p><p><strong>Key outcomes</strong></p><ul><li>We succeeded in getting a product to market in &lt; 6 months and learned a huge amount in the process. Most importantly we learned that this was achievable and that the DevOps way of working contributed to the team cohesion and fast delivery</li><li>New AWS account topology pattern (known internally a Landing Zone — not to be confused with the AWS product) ensured that new AWS accounts were vended for this team to operate in — one set of accounts per team. This gave is isolation and accountability</li><li>The use of serverless first architecture (rather than E2C, K8 etc) served as a blueprint for our serverless first approach that we have since adopted. We’d done some serverless previously, but not with IaC (Serverless Framework and Terraform)</li><li>High levels of automation; CI/CD based on github, CodeBuild and IaC with Terraform and Serverless framework. Also the shift-left of testing into the pipelines.</li></ul><h4>2. Walking the Walk — Doing it ourselves!</h4><p>Having proved the value of DevOps, we had a mandate to proceed and form another 4–5 product teams to deliver new products.</p><p>At this point we had a very basic blueprint which we needed to make repeatable and to start building up skills so that our DevOps partner could exit — this was a key success criteria for them.</p><p>We can break down this stage into 3 areas:</p><ul><li><strong>Lighthouse Projects — </strong>Selecting suitable green field products to deliver and to form new product teams around so that the outcome was a product in market and an enduring product team that owned that product and used DevOps/product engineering to do so.</li><li><strong>Talent Pipeline </strong>— Starting to build a talent pipeline; forming job definitions, landing the culture we wanted and starting the treadmill of recruitment. Later in this phase we introduced our Data &amp; Tech graduate scheme.</li><li><strong>Account Vending Machine </strong>— Taking the Landing Zone account topology that was incubated previously and turning it into a self service vending machine (Terraform + gitops) to help teams onboard quicker.</li></ul><p><strong>Key outcomes:</strong></p><ul><li>4x product teams formed and products live in market — mixture of internal staff facing and external customer facing services</li><li><a href="https://www.edfenergy.com/careers/graduates/data-tech">Data &amp; Tech Graduate</a> scheme started.</li><li>The AWS Account vending machine that enables accounts to be created / bootstrapped in minutes</li><li><a href="https://www.edfenergy.com/careers/software-engineer">Recruitment</a> of 8 internal software engineers and up-skilling of 5 existing partner engineers.</li><li>Formation of a central Cloud Services team that owns the AWS Account vending machine.</li><li>DevOps as a way of working is normalised.</li></ul><h4>3. Scale Up</h4><p>This stage happened around 2022 where it had been just over 2 years since we’d started the incubation project. We’d seen product teams grow to 10 and the DevOps way of working becoming the default approach.</p><p>Whilst we’d scaled-up from zero to around 50 engineers / 10 teams, it was not quite enough to enable us fully for the typical levels of change and delivery that we encounter, plus we were aware of a number of larger programmes of change that DevOps would need to support.</p><p>This phase focused on:</p><ul><li><strong>Recruitment Scale Up </strong>— We learned alot recruiting. Its a treadmill, its hard and costly in terms of time and effort. We needed to find a more enduring / scalable approach.</li><li><strong>Partner Evolution</strong> — We always realised from day 1 that partners play a key role in our talent sourcing, but shifting from a traditional ‘outsource the work’ mindset, to a ‘bring in talent when needed’ — typically to support burst capacity and skills gaps. All the time ensuring that the focus was on the enduring product team size and capability.</li><li><strong>SRE &amp; Operations</strong> — By now, we’d have products that have been live and in the wild for around 1–2 years, and had gained alot of operational insight from using out of the box AWS tooling.</li></ul><p><strong>Key outcomes:</strong></p><ul><li>Creation of Data &amp; Tech brand and in-house headhunting / recruitment approach that uses paid campaigns rather than expensive agency placement fees. We found this not only more cost effective, but a more human centric approach as applicants where talking to EDF from the outset, rather than via a proxy. Ultimately, people want to work for people. By over-indexing on this process made recruitment more effective and reduced the friction. This approach enabled us to double our engineering workforce in a 18 months.</li><li>Partners now aligned to our Competency Framework as a common language for roles, skill levels and rates. Regular ‘resourcing’ (dont really like that term!) conversations to ensure we maintain the right balance of internal skills vs partner.</li><li>SRE Community of Practise formed, and the SRE culture / mindset adopted into our competencies and ways of working. Essentially ‘design for operation’. We also went to market to select a more centralized observability platform — Dynatrace which enables better dependancy visibiity between teams and components, and faster triage / fault isolation.</li></ul><h3>Building Blocks</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*BknJvrdvXxKpUCbZ" /><figcaption>Photo by <a href="https://unsplash.com/@mourimoto?utm_source=medium&amp;utm_medium=referral">Mourizal Zativa</a> on <a href="https://unsplash.com/?utm_source=medium&amp;utm_medium=referral">Unsplash</a></figcaption></figure><p>Reflecting on this journey, we observed the following key building blocks that we’d created along the way. That might help you on your journey:</p><ul><li><strong>Account Vending</strong> — If you are using a cloud platform like AWS, leverage the value of separate accounts for isolation and ownership. Utilise a vending machine (Landing Zone or Control Tower, or roll your own — as we did, as these didnt exist at the time). Adopting that vending machine / self service / gitops mindset for regular change is essential to reducing friction and delay.</li><li><strong>Governance as Code</strong> — Typically the larger the organisation, the larger the governance overhead is. We found that baking in standard, guardrails and governance into pipelines, template etc was more effective that traditional eye-balling. Lots of good tools our there to do this like Checkov, AWS Guard Duty etc. Plus leveraging security vulnerability tools like Dependabot, AWS Inspector gives that blended coverage of static vs dynamic code analysis.</li><li><strong>Recruitment as a Service</strong> — We needed to balance centralizing recruitment for consistency and brand purposes vs federating for practical purposes — ie a team should be recruiting its people, not someone else! — our learning from recruitment has essentially formed this ‘service’, its iterated through feedback, owned and directed centrally by the Principal Software Engineer.</li><li><strong>Communities</strong> — Leveraging communities of practice and interest groups to decompose and drive forward a range of engineering enablers like; SRE, Security, Front-End frameworks, Integration Patterns is a great engagement tool to empower engineers to contribute to the bigger picture and have greater ownership on the technology direction.</li></ul><h3>Lessons Learned</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*DRSRqiPD2GGTG0we" /><figcaption>Photo by <a href="https://unsplash.com/@timmossholder?utm_source=medium&amp;utm_medium=referral">Tim Mossholder</a> on <a href="https://unsplash.com/?utm_source=medium&amp;utm_medium=referral">Unsplash</a></figcaption></figure><h4>1. Vision rather than Plan</h4><p>Form a vision of what DevOps looks like and anchor that as your goal, rather than making static plans. The vision will evolve over time, but core tenets will remain consistent — these are your ‘why are we doing this’ answers.</p><p>The AWS ‘working backwards’ is often a great way to approach this.</p><h4>2. Start Small and validate/iterate</h4><ul><li>You will get stuff wrong, but equally you won’t know its wrong until you try!</li><li>Don’t try to move too fast. Steady, demonstrable value and growth builds reputation and greater buy-in from detractors</li><li>Iteration and continuous improvement is a fundamental behaviour, set that into the DNA and ways of working from day 1.</li></ul><h4>3. Culture is key</h4><p>Defining what culture you want to cultivate is essential to success. Ours is formed on DevOps normals like:</p><ul><li>Contribution &amp; Collaboration — Break down silos and increase engagement by active Communities of Practice, Game Days and a sense of belonging to a collective that is more powerful than the individual teams</li><li>Recognise and award exemplar behaviours around</li><li>Ensure that culture is embedded in the recruitment, metrics, job definitions, competency frameworks etc — anything that you use to select and retain the right talent.</li><li>Leverage the value of evangelists and messengers to adopt, promote and extol the virtues of DevOps. Top down mandates are less inclusive and instantly provoke reaction. However, seeing demonstrable value and benefit promotes more ‘we should be doing that’ rather than ‘you must do this’.</li></ul><h4>4. Building and retaining talent is hard</h4><ul><li>Treat recruitment as regular exercise rather than a fad diet. Invest in making it efficient, transparent and fair so that its as low effort as possible to keep switched on.</li><li>People want to work with people (mainly), so your culture and how you operate needs to be visible and showcased. Reputation breeds reputation</li></ul><h3>Further Reading</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*R1Z7_rYhD7KE8sH0" /><figcaption>Photo by <a href="https://unsplash.com/@gaellemarcel?utm_source=medium&amp;utm_medium=referral">Gaelle Marcel</a> on <a href="https://unsplash.com/?utm_source=medium&amp;utm_medium=referral">Unsplash</a></figcaption></figure><ul><li>AWS London Summit 2024 presentation: <a href="https://www.youtube.com/watch?v=RdArvf_5xG4">https://www.youtube.com/watch?v=RdArvf_5xG4</a></li><li>Case Studies and Collateral <a href="https://github.com/edfenergy-stevebowerman/aws-summit-2024">https://github.com/edfenergy-stevebowerman/aws-summit-2024</a></li></ul><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=84e7ab3d0777" width="1" height="1" alt=""><hr><p><a href="https://medium.com/edf-data-and-tech/how-edf-built-a-devops-practice-on-aws-84e7ab3d0777">How EDF Built a DevOps Practice on AWS</a> was originally published in <a href="https://medium.com/edf-data-and-tech">EDF Data and Tech</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Our Journey into Site Reliability Engineering, Part 2]]></title>
            <link>https://medium.com/edf-data-and-tech/our-journey-into-site-reliability-engineering-part-2-192d9584948c?source=rss-a6ab148b5db3------2</link>
            <guid isPermaLink="false">https://medium.com/p/192d9584948c</guid>
            <category><![CDATA[technology]]></category>
            <category><![CDATA[site-reliability]]></category>
            <category><![CDATA[community-of-practice]]></category>
            <category><![CDATA[self-service]]></category>
            <category><![CDATA[dynatrace]]></category>
            <dc:creator><![CDATA[Steve Bowerman]]></dc:creator>
            <pubDate>Thu, 13 Jul 2023 11:22:03 GMT</pubDate>
            <atom:updated>2023-07-13T11:22:03.742Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*LMaCHUPRxd9NN6Sy" /><figcaption>Photo by <a href="https://unsplash.com/@nxvision?utm_source=medium&amp;utm_medium=referral">Nigel Tadyanehondo</a> on <a href="https://unsplash.com/?utm_source=medium&amp;utm_medium=referral">Unsplash</a></figcaption></figure><p>In <a href="https://medium.com/edf-data-and-tech/our-journey-into-site-reliability-engineering-a59cc2f43902">part 1</a>, I outlined our evolution from DevOps towards SRE and a culture of observability. In this second part, I’ll outline how we got started.</p><h3>Approach</h3><p><strong>How</strong> — Start small, prove out, and demonstrate. We’ve used this approach many times in our Software Engineering Practice journey, so it’s well adopted, proven and mostly expected by our stakeholders.</p><p><strong>What</strong> — We started with our Central Engineering Services team — the same team driving the SRE initiative — a classic case of ‘eating our own dog food’, which enabled us to go through the learning experience and guide others from a position of experienced authority rather than an Ivory Tower.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/536/1*O_vPOwI83R-hjEHTVUA8Qw.png" /></figure><h4>Phase 1 : Basic Tooling</h4><p>Run with basic tools and build experience &amp; knowledge in SRE Concepts.</p><p>In terms of tools, solutions and scope; we took small set of our AWS Serverless applications (which represent the same architecture as 99% of our Product Teams). Initially using AWS native resources like Cloudwatch logs, metrics, dashboards &amp; alarms to get familiar with the concepts.</p><p>In parallel we started immersing ourselves in the Google SRE Handbook — <a href="https://sre.google/sre-book/table-of-contents/">https://sre.google/sre-book/table-of-contents/</a> and putting the concepts into practice.</p><p>Our initial focus, out of the 7 principles was:</p><ul><li><strong>Monitoring Distributed Systems</strong> — Our top priority as we didn’t want to inherit the traditional ‘IT Services’ approach — ie a support team. We wanted to ensure that our end-to-end design-&gt; ship -&gt; run process understood how to instrument, how to alert (real-time) and how to track over time (dashboard / helicopter view / trend analysis), and to encourage the team into the cycle of actively sensing, reacting and enhancing production service health.</li><li><strong>Eliminating Toil — </strong>The phrase ‘is it going to hurt us’ — ie will we waste time investigating, fixing repetitive and preventable issues that will sap energy and time away from feature build and value to our customers / consumers. Generally across our teams we are already strong on managing technical debt, but with toil there is the added pressure and balance around feature vs fix, and therefore how reactive vs proactive each team chooses to be. Accepting that this balance will continually flex.</li><li><strong>Embracing Risk</strong> — Getting our team into a mindset of thinking about ‘build to run’ — instrumentation and quantifying systemic and business impact as part of our design process. Focusing less on establishing every conceivable boundary event, and instead about core health, and how to architect to improve and iterate. This required us to embrace failure and quantifying toil and technical debt impact to ensure sprint and backlog refinement conversations were rich in detail, justification, pragmatism. Eliminating Toil and Embracing Risk are counter-points, along with feature delivery that teams need to learn to understand, balance and moderate.</li></ul><p>Since we already have strong CI pipelines and adopt AWS well architected principles, we felt that these were good enough initially, and would circle back round these and the other 4 principles once we’d moved everything else along.</p><h4>Phase 2: Revised Tooling</h4><p>The previous phase provided us with great insight into ‘how to do SRE’ — albeit in a very skeleton fashion. It identified the areas what we needed to prioritise and invest more in.</p><p>From an observability tooling point of view, we found limitations (2021) with managing observability across multiple AWS accounts (we have around 20 as a team, and the wider practice operates across around 200+!). We also found with native AWS tooling that whilst the possibilities are endless, we did spend significant time engineering &amp; building gaps and automation — this is common with AWS; amazing components but often need a lot of glue.</p><p>Our traditional IT Services teams used New Relic, which we explored too as part of a POC and ultimately resulted in a tool selection process where we landed on <a href="https://www.dynatrace.com/news/press-release/edf-uk-choisit-dynatrace-pour-assurer-la-bonne-distribution-du-courant-electrique-a-des-millions-de-clients/">Dynatrace</a>.</p><p>Adopting a COTS/SaaS observability platform allowed us to focus more time and value in getting the metrics / log data into the system and then leveraging inbuilt dash-boarding and alerting frameworks (Slack, Service Now etc) . This also resulted in a series of ongoing design patterns, collateral and re-usable Terraform modules that we knew we wanted to build up to support wider roll-outs.</p><h4>Phase 3: Iterate</h4><p>Having now shifted away from the initial AWS observability toolset to <a href="http://www.dynatrace.com/">Dynatrace</a>, and achieved a high level of repeatability around onboarding additional products and services, we expanded to the remaining products and services that our Central Engineering Team owns and maintains.</p><p>From this stage, we embarked upon a heavy refinement period where our goal was to create a Git-ops style self service approach to allow for more efficient onboarding for other teams, and a set of enablers / accelerators to get them up and running in hours rather than days that it initially took us. All the while, bottling the good learning into a rich set of Confluence pages and example code / modules etc.</p><h4>Phase 4: Scale</h4><p>Establish a Community of Practice and Scale out.</p><p>With around 2–3 months elapsed since the previous phase, we drew up a plan to start scaling out the service to other teams in order to enable other product teams with SRE concepts and tooling, and to further iterate our SRE framework towards a wider practice enabler.</p><p>We have been fortunate enough to leverage an existing larger business transformation programme that has been modernising our residential and SME landscape, where there was a significant catalyst to shift towards the formation of new product engineering teams — plenty of new software and services being built, and a rich environment to challenge and evolve.</p><p>This phase helped us accelerate the pace of scaling out, and importantly — demonstrate the value of SRE — as this new way of working has been adopted from the outset so teams were very supportive as it gave them with an instant set of accelerators, and a significant role to contribute and shape SRE.</p><p>Along side the scaling out, we established an SRE Community of Practice to support and engage in the SRE adoption and evolution, rather like a user group on steroids!</p><p>As of writing this article we have around 10+ (and growing) teams using this framework, and it has formed a core foundation of our new team bootstrapping process.</p><h3>Key Learnings</h3><h4>1. Decentralisation of Skills &amp; Community of Practice</h4><p>Our observation around the existing operating model was that having central ownership for the tool is still the right thing to do — vendor relationship, roadmaps, being evangelistic for the use and deployment of the tool.</p><p>However, centralisation had extended to implementation and management, with teams requesting setup and management, and being hands off , resulting in rich tool understanding and best practice knowledge being retained in the central team rather than being socialised out to the community so that ‘everyone is better’.</p><p>Our approach of creating a CoP and providing enablers and accelerators to teams for them to self service ensures that SRE skills and knowledge is being developed, shared and retained across the product engineering teams.</p><h4>2. Self Service</h4><p>As an engineering community we have been used to a culture of GitOps and self service to vend and onboard products provided by our platform teams. We saw significant value in utilising that approach in our SRE tooling — Dynatrace onboarding. Using config files, pipeline automation and Terraform to allow teams to quickly add themselves, users and services and easily effect change via regular YML files and Github pull requests. With PR’s being the lightweight governance we wanted to apply.</p><p>This learning alone as reduced the onboarding process from days to hours and also ensured a higher level of consistency and ‘getting it right first time’.</p><h4>3. Start Small, Learn, Adapt and Scale</h4><p>It might be obvious, but this approach really works well. It builds credibility and buy-in gradually and consistency, whilst helping the service and learning evolve.</p><h3>Next Steps</h3><p>In part 3 I’ll cover of how we are circling back around the Google SRE Handbook and implementing:</p><ul><li>Duty Engineer as an approach to manage on-call and interruptions</li><li>SLO’s and DORA metrics to help quantify the effectiveness of teams</li></ul><p>Plus plenty of others!</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=192d9584948c" width="1" height="1" alt=""><hr><p><a href="https://medium.com/edf-data-and-tech/our-journey-into-site-reliability-engineering-part-2-192d9584948c">Our Journey into Site Reliability Engineering, Part 2</a> was originally published in <a href="https://medium.com/edf-data-and-tech">EDF Data and Tech</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Our Journey into Site Reliability Engineering]]></title>
            <link>https://medium.com/edf-data-and-tech/our-journey-into-site-reliability-engineering-a59cc2f43902?source=rss-a6ab148b5db3------2</link>
            <guid isPermaLink="false">https://medium.com/p/a59cc2f43902</guid>
            <category><![CDATA[observability]]></category>
            <category><![CDATA[service-level-objectives]]></category>
            <category><![CDATA[site-reliability]]></category>
            <category><![CDATA[technology]]></category>
            <category><![CDATA[evolution]]></category>
            <dc:creator><![CDATA[Steve Bowerman]]></dc:creator>
            <pubDate>Tue, 02 May 2023 13:53:51 GMT</pubDate>
            <atom:updated>2023-05-02T13:53:51.501Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*Z7vjVf6JxjiFwEYP" /><figcaption>Photo by <a href="https://unsplash.com/@foxxmd?utm_source=medium&amp;utm_medium=referral">Matt Duncan</a> on <a href="https://unsplash.com/?utm_source=medium&amp;utm_medium=referral">Unsplash</a></figcaption></figure><h3>Context</h3><p>Steve is EDF’s Principal Software Engineer, accountable for defining, shaping, and executing EDF’s Software Engineer strategy. Steve has lead our engineering practice through our Central Engineering Services Team, which specialises in the tools &amp; services that enable and empower our product and engineering teams, helping them deliver great products and services to our customers.</p><h3>Part 1 — DevOps isn’t enough.</h3><p>As part of EDF’s ongoing digital transformation and optimisation activities, we have been building and optimising our software engineering practices for the last couple of years.</p><p>Whilst our default mode of operation was DevOps, we felt that DevOps needed a bit of added sauce to avoid being just a series of teams that “just” build and run services. We wanted to avoid that hamster wheel approach as our industry is constantly evolving, price sensitive and demanding of efficiencies. So there is need for a relentless forward advance.</p><p>Furthermore, most operational issues with software stem from poor design and/or a lack of operational empathy.</p><p>We started our journey by inheriting a legacy organisational approach &amp; mindset to running IT services — a culture focused more on counting CPU cycles, HDD, memory usage, and binary “service up or down” status, rather than business value and impact.</p><p>We observed (pun intended) the following key impacts:</p><ol><li>Difficult to link our software products and services with the value our business and customers expect, which in turn makes it hard for engineers to equate their work and impact on business success.</li><li>Little incentive to manage technical debt and invest in refactoring or other more proactive approaches.</li><li>Acceptance of toil, rather than investment in automation.</li><li>A confused and watered-down accountability in the product teams who saw running software as an “IT thing” yet wanted to be agile and move at pace.</li></ol><p>To evolve our product teams and empower them to build great products that our customers love, it took a complete rethink. We like rethinking and resetting occasionally as it’s easy to subconsciously carry cruft and outdated thinking when you are evolving and iterating quickly.</p><p>Our mission was quite bold, but achievable:</p><blockquote><strong><em>To foster a Site Reliability Engineering (SRE) culture in our product engineering teams and to support and empower them with a better operational framework and tooling.</em></strong></blockquote><p>This mission can be decomposed into two areas: <strong>Shifting from DevOps to SRE</strong> and <strong>Shifting from Monitoring to Observability</strong></p><p><strong>Shifting from DevOps to SRE</strong></p><p>We want to evolve from DevOps to SRE so that the “operational” thinking is present at the point of design. Werner Vogels states that “<em>everything fails all the time”</em>, therefore expecting failure, designing around it, and ensuring that failure is measurable and actionable was a core part of this shift.</p><p>This meant:</p><ul><li><strong>Understanding/predicting failure</strong> points — defensive architecture, leveraging existing best practice like retires, loose coupling, and strongly typed interfaces.</li><li>Mapping failure points to automated recovery (back-off/retry) or, at a very minimum, ensuring the failures are <strong>observable</strong> and <strong>recoverable</strong>, leading to strong <strong>correlation</strong> between failures and quality run-books.</li><li><strong>Continuous cycle </strong>of measuring, reviewing, and understanding failure occurrences and automating or re-designing to resolve the root cause of issues.</li></ul><p>Accepting that sometimes the operational inputs — load, demand can be less visible or apparent during design and that only when live can you discover these.</p><p><strong>Shift from Monitoring to Observability</strong></p><p>The traditional monitoring “bridge” is not effective in modern organisations (aside from core infrastructure like networking and comms) as engineering tends to be federated or devolved into the business where it should be — close to the action and impact.</p><p>With this ongoing organisational shift, it meant also that the traditional ownership and accountability shifts towards business native product teams, and hence a greater emphasis is needed on empowering these teams to be self-sufficient and accountable. As such, they need to be the ones detecting and reacting to service quality, not a centralised monitoring team.</p><p>Whilst there is additional burden on the teams for this activity, it removes a large amount of admin and cruft that typically get in the way between software issues/incidents and resolutions, and the multitude of hand-off points in the more traditional landscape. This embodies the “you build it, you own it, you run it” culture that we want to foster in our product engineering teams which cultivates better software/product design through greater product intimacy, context, and ownership amongst engineers in the products they are building.</p><p>We saw monitoring as a slightly purgative term like an afterthought, or marking someone’s homework, rather than being a core part of what makes a modern product team effective.</p><p>We can summarise this shift as:</p><ul><li>Observability is an <strong>intentional</strong> design activity.</li><li>Use of SLO’s for objectively defining <strong>service qualities</strong> that are measurable.</li><li><strong>Alignment</strong> between SLO’s and service quality metrics with real business impact — churn, CSAT, complaint levels, channel shift.</li><li><strong>Use of metrics</strong> to drive priority of work and to accurately quantify action vs inaction.</li></ul><p>In my next post, I’ll continue our journey from “how did we start”.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=a59cc2f43902" width="1" height="1" alt=""><hr><p><a href="https://medium.com/edf-data-and-tech/our-journey-into-site-reliability-engineering-a59cc2f43902">Our Journey into Site Reliability Engineering</a> was originally published in <a href="https://medium.com/edf-data-and-tech">EDF Data and Tech</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Mastering Serverless Framework and Lambda Versions]]></title>
            <link>https://steve-bowerman.medium.com/mastering-serverless-framework-and-lambda-versions-d07d47bff6d3?source=rss-a6ab148b5db3------2</link>
            <guid isPermaLink="false">https://medium.com/p/d07d47bff6d3</guid>
            <category><![CDATA[serverless]]></category>
            <category><![CDATA[aws-lambda]]></category>
            <dc:creator><![CDATA[Steve Bowerman]]></dc:creator>
            <pubDate>Mon, 19 Sep 2022 16:25:47 GMT</pubDate>
            <atom:updated>2022-09-19T16:25:47.910Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*yZP7zJztzUnpDXpys9Jdkw.jpeg" /></figure><p>I’ve been tidying up a few personal projects lately, many of which have been incubation and proof of concepts. Its one of the many unglamorous but necessary activities that good devs need to keep on-top of; removing / undeploying POC code, bumping npm versions, node versions, and generally keeping up with the pace of deprecation.</p><p>One topic that jumped out that I’ve seen many devs fall foul of (including myself when I was learning AWS &amp; Serverless) is how Lambda versions work, and more importantly how to control and influence this, and not run out of storage space in your AWS accounts!</p><h3>Lambda function versions</h3><p>A quick primer on Lambda function versions — they allow you to run multiple versions of your code, typically to support the rollout of breaking changes, deprecation of features, blue/green deployment etc.</p><p>Generally speaking I’d suggest not using versioning in the first instance unless you are clear on the need for it, as it can add further complexities and administration. Often its simply not required if you have broader control over your dependancies. I dare say this comment will spark debate, but I stand in the camp of ‘build what you need now’.</p><h3>Serverless Framework</h3><p>I’m a big user and advocate of<a href="https://www.serverless.com/"> Serverless Framework</a> for personal and professional projects. In both instances I’m in the ‘serverless first’ camp as means more time writing functionality and features than battling infrastructure.</p><p>Serverless Framework is a very elegant framework for neatly bundling up your code and infra. While there are other frameworks like SAM and CDK (<a href="https://aws.amazon.com/serverless/sam/">https://aws.amazon.com/serverless/sam/</a>), I like the simplicity of Serverless Framework when I’m building Lambda centric components.</p><h3>Serverless and Lambda — whats the problem?</h3><p>The default behaviour for Serverless Framework is to enable versioning, regardless if you need it or not. It should be noted that AWS provides a maximum of 75Gb storage per account for holding your Lambda code base, and while that sounds loads, it can be easily eaten up if you have many Lambda functions in your account and relatively active change cadence.</p><p>Some crude calculations to colour this:</p><blockquote>2x changes per week</blockquote><blockquote>50Mb per Lambda package</blockquote><blockquote>100 Lambdas in an AWS account</blockquote><blockquote>= 10Gb of code per week</blockquote><h3>How do I tame this?</h3><p>Thankfully this is simple!, first decide if you need versioning or not, and if you are not sure, then you probably don’t need it.</p><p>Second, explicitly set versionFunctions parameter to false<strong> —</strong> it defaults to true.</p><p>Like the rest of the serverless framework, the property can be set globally for all functions in the provider section or at a function level if you need a mixture — I’d recommend setting it globally, then overriding if needed at function level</p><pre>provider:</pre><pre>  name: aws</pre><pre>  runtime: nodejs16.x</pre><pre>  versionFunctions: false</pre><pre>  region: eu-west-1</pre><pre><br>...<br></pre><pre>function:<br>  foo:<br>    versionFunctions: false</pre><h3>Oh, didn’t know that, and I’ve been running like this for a while!</h3><p>Thankfully its relatively easy to prune and clean out old versions manually or automatically. There are plugins for this — serverless-prune-plugin, (though I’ve not used them).</p><p>I’ve found the easiest thing is a shell script that uses the AWS CLI. Something like this which can be run periodically or included in your pipelines. Obviously where need to use versioning, ensure you exclude functions from this list!</p><pre>declare -a lambdas=($(aws lambda list-functions --query &#39;Functions[*].FunctionName&#39; --output text))</pre><pre>for lambda in  &quot;${lambdas[@]}&quot;</pre><pre>do</pre><pre>  declare -a versions=($(aws lambda list-versions-by-function --function-name ${lambda} --query &#39;Versions[*].Version&#39; --output text))</pre><pre>  for version in &quot;${versions[@]}&quot;</pre><pre>  do</pre><pre>    echo &quot;Checking: ${lambda}&quot;</pre><pre>    if [ &quot;${version}&quot; != &quot;\$LATEST&quot; ]; then</pre><pre>      echo &quot;\tDeleting: ${lambda}:${version}&quot;</pre><pre>      aws lambda delete-function --function-name ${lambda}:${version}</pre><pre>    fi</pre><pre>  done</pre><pre>done</pre><h3>Further reading</h3><ul><li>Lambda versions — AWS documentation <a href="https://docs.aws.amazon.com/lambda/latest/dg/configuration-versions.html">https://docs.aws.amazon.com/lambda/latest/dg/configuration-versions.html</a></li><li>Serverless Framework reference — <a href="https://www.serverless.com/framework/docs/providers/aws/guide/functions/#versioning-deployed-functions">https://www.serverless.com/framework/docs/providers/aws/guide/functions/#versioning-deployed-functions</a></li></ul><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=d07d47bff6d3" width="1" height="1" alt="">]]></content:encoded>
        </item>
    </channel>
</rss>