<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Alex Potter Dixon on Medium]]></title>
        <description><![CDATA[Stories by Alex Potter Dixon on Medium]]></description>
        <link>https://medium.com/@alex.potter.dixon?source=rss-cd7b1b8adb8e------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/1*o4ZaFIeH15RM0De27zfsZQ.jpeg</url>
            <title>Stories by Alex Potter Dixon on Medium</title>
            <link>https://medium.com/@alex.potter.dixon?source=rss-cd7b1b8adb8e------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Sun, 22 Nov 2026 01:34:47 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@alex.potter.dixon/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[Failure Friday: Simulated Chaos]]></title>
            <link>https://medium.com/@alex.potter.dixon/failure-friday-simulated-chaos-624602f46861?source=rss-cd7b1b8adb8e------2</link>
            <guid isPermaLink="false">https://medium.com/p/624602f46861</guid>
            <category><![CDATA[growth]]></category>
            <category><![CDATA[it]]></category>
            <category><![CDATA[development]]></category>
            <category><![CDATA[learning]]></category>
            <category><![CDATA[operations]]></category>
            <dc:creator><![CDATA[Alex Potter Dixon]]></dc:creator>
            <pubDate>Sat, 18 Apr 2020 14:56:33 GMT</pubDate>
            <atom:updated>2020-04-18T14:56:33.308Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*iV943Q-Tr0P4aukQ" /><figcaption>Photo by <a href="https://unsplash.com/@brenomachado?utm_source=medium&amp;utm_medium=referral">Breno Machado</a> on <a href="https://unsplash.com/?utm_source=medium&amp;utm_medium=referral">Unsplash</a></figcaption></figure><p>In any operations team, you will always try and plan for every system failure. You may do this by writing playbooks, reading documentation, watching webinars but in the end, until you’ve experienced your system failing in real life you truly never know. This is where Failure Fridays come into play.</p><h3>What is Failure Friday?</h3><p>We initially got the inspiration for Failure Friday from PagerDuty’s ops team. They described how they would intentionally break their live system and see how their operations team would handle it. For us, this means every Friday causing a simulated failure on the systems we look after in ways the team has never handled before.</p><p>Now you ask why on earth do you want to intentionally break things? Is it not your job to make sure things don’t break? Well, there are a few reasons we wanted to do this:</p><ol><li>Strengthen and stretch our team&#39;s incident management skill set.</li><li>Identify resilience issues within the platforms we manage.</li><li>Identify any missing processes or procedures.</li><li>Improve our communication to stakeholders and customers during system downtime.</li><li>Train new SRE team members during business hours rather than 1 in the morning on a Sunday.</li></ol><h3>How do we do it?</h3><p>Our first Failure Friday required a few things to get in place:</p><h4>Roles</h4><p>In any simulation, you need clearly defined roles. In Glasswall’s SRE team we have 3 key roles with clear responsibilities:</p><ol><li>DM (Disaster Master): The DM will cause the incident. They will delete the node, increase the traffic or create the latency.</li><li>Primary on-call: The Primary will be the main person to handle the incident.</li><li>Secondary on-call: The Secondary is the backup who can be summoned by the Primary to assist in the incident or act as an incident communicator.</li></ol><h4>Scenario</h4><p>Every simulation needs a scenario. With our scenarios, we try to follow some simple guidelines:</p><ul><li>Severe enough so the team is stretched sufficiently for growth and learning.</li><li>Something that’s not been experienced before. We want the team to be out of their comfort zone.</li><li>A scenario that scares you. Those water cooler conversations of: “Imagine if the databases got deleted, we’d be screwed”. Those are perfect scenarios! You want to be pro-active and experienced in those situations so if it happens you know exactly what to do.</li></ul><h4>Process</h4><p>Last but not least is a process. You want to make sure that a process is defined to make sure:</p><ul><li>It is as real as possible. The simulation needs to be a surprise as it would be in real life. No hints or no planning to the Primary or Secondary on what it will be.</li><li>Communication with stakeholders should be as if there was an incident. We are reviewing the whole process end to end and communication to stakeholders and users is an absolutely key part of any incident.</li><li>A postmortem is performed after. As a team we need to review how it went, what went well, what didn’t go so well and most importantly what actions need to take place so we can learn, grow and ultimately improve as an individual and as a team.</li></ul><h3>What have we done so far?</h3><p>We’ve only completed one Failure Friday so far and it was a great success for us from a performance and learning perspective.</p><h4>We deleted SQL tables</h4><p>Our first Failure Friday involved the DM deleting a table in one of the main databases that manage our users for our FileTrust email system. This meant that users were not able to login to the portal. Users were still able to send mail to the system though and this was thanks to the hight availability Redis cache we use to cache tenant data incase of database issues.</p><h4>How did we recover?</h4><p>Initially, the team started to perform a database recovery via Azure SQL but realised that they could re-run the database creation pipeline. This pipeline restored the tables within the database but not the data. To restore the data we were able to run a separate pipeline that re-populated the table with the missing users.</p><h4>What went well?</h4><ul><li>We were able to restore the data to exactly before the deletion via two methods: Pipelines and our live backups.</li><li>The system was able to keep mail flowing even with major data loss.</li><li>The team worked well in communicating the incident, co-ordinating with other SRE team members and keeping the incident up to date in PagerDuty.</li></ul><h4>What did we learn?</h4><ul><li>We need a clear playbook on restoring SQL data. It was clear that the team had the capability to restore data loss but that was from past experience and not a well-defined playbook. The team shouldn’t be crawling through documentation at 2 in the morning. They should be using a step by step playbook.</li><li>We need to create a public-facing status page like statuspage.io to make communication and system statuses to our customer&#39;s much clearer and easier. Currently, it requires a chain of people to get comms out which can cause delays.</li><li>We need clearer SQL auditing for deletion. Currently, we only notice when error logs appear or systems go down. We need those alerts combined with audit alerts of major changes to the database to make sure we know about the problem is instantly and in turn, improve mean time to resolution.</li></ul><p>Failure Fridays are a great way of stretching any operations team and system they look after. We learned a lot from our first one and looking forward to future ones to see how much we can learn and grow.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=624602f46861" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Want to be an SRE? Join Glasswall!]]></title>
            <link>https://medium.com/@alex.potter.dixon/want-to-be-an-sre-join-glasswall-a4b10ab94220?source=rss-cd7b1b8adb8e------2</link>
            <guid isPermaLink="false">https://medium.com/p/a4b10ab94220</guid>
            <category><![CDATA[hiring]]></category>
            <category><![CDATA[devops]]></category>
            <category><![CDATA[job-hunting]]></category>
            <category><![CDATA[jobs]]></category>
            <category><![CDATA[tech]]></category>
            <dc:creator><![CDATA[Alex Potter Dixon]]></dc:creator>
            <pubDate>Mon, 03 Feb 2020 13:20:12 GMT</pubDate>
            <atom:updated>2020-02-12T09:59:58.204Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/800/1*2calJfKF1jjqz3BM9qNFIQ.png" /></figure><h3>We’re hiring!</h3><p>The SRE team at Glasswall is looking for a new Junior member to help make all our production systems scalable, reliable and thrive!</p><p>If you get a kick from engineering resilient solutions, automation and helping others build amazing services for our customers. We want to hear from you!</p><h3>What is an SRE at Glasswall?</h3><p>At Glasswall we follow the company that found SRE: Google and their implementation of SRE. Why? What’s the first website you check to see if your internet is up? Google.</p><p>Our main aim is to <strong>“Make our production services as reliable as possible while keeping the pace of innovation high”.</strong></p><p>We do this by implementing SLOs and SLIs, defining best practices, automating all repetitive manual work, keeping changes to production frequent but in small manageable chunks, building disaster recovery tools and using industry-standard incident management.</p><p>An important part of being an SRE is being on call. At Glasswall you will be on call for one week and then off for the next 3 weeks. Being on call you are the interrupt person, you handle any incidents during and out of office hours for that week including weekends. This is the operations part of being an SRE and we make sure your workload is split <strong>50/50 </strong>operations and software engineering. Being on call also means you will be doing the deployments to production and answering any external requests. We have this so the rest of the team can have a deep focus on other engineering work.</p><p>During the time you are not on call, you will be mainly focused on software engineering automation tools, providing support to other teams on resilience and scalability, building infrastructure using true IaC and building disaster recovery tools. We strongly believe this is the most important part of the role and will help us be successful in our goal of <strong>keeping our services as reliable as possible</strong>.</p><h3>What are we looking for?</h3><p>Not all of these skills are needed but should give you a good idea of the sort of person we are looking for. For us, the soft skills are more important, you can train technical, soft is more of who you are as a person.</p><p><strong>Soft skills</strong></p><ul><li>You are passionate about the world of cloud engineering and programming.</li><li>You love learning new skills and are always pushing yourself to learn more.</li><li>You can deal with high-pressure situations, step back and get the full picture before jumping in.</li><li>Complex polymorphic problem solving is your bread and butter. You love the solution and want more!</li><li>You grasp new concepts quickly and to a high standard.</li><li>You thrive in a strong teamwork environment and put the team first in all of your actions.</li></ul><p><strong>Technical skills</strong></p><ul><li>Strong software engineering practices that include clean architecture, clean code, high but quality test code coverage and creating documentation the whole team and community as a whole can use.</li><li>Programming languages such as Python or Go.</li><li>Key DevOps practices such as CI/CD and source control with Git.</li><li>Experience with cloud technologies such as Kubernetes and Serverless.</li></ul><h3><strong>What are we offering?</strong></h3><ul><li>High autonomy. You’re the engineer and craftsman we are here to enable you to make your masterpiece your way.</li><li>Space to learn. 10% of your time is reserved training days. Access to online training tools including Linux Academy and Pluralsight.</li><li>Career progression and mentorship. We want you to be the best version of yourself. Clear goals and development plan to get you where you want with space and tools to achieve those goals. You will be stretched daily which creates real rock star engineers.</li><li>Bottom-up change. The team has strong control over its destiny with the decision on new technologies and project work is decided by the team as a whole, not management.</li><li>Interesting work. We are open sourcing our tools, building chat ops bots, using true infrastructure as code and building beautiful monitoring platforms.</li><li>Glasswall company benefits: Private health care and share options.</li><li>Once passed probation can work from home up to 2 days a week.</li><li>Total Package from<strong> £35,000</strong> based on ability.</li></ul><h3><strong>How to apply</strong></h3><p><strong>This role does include shift on-call work. One week on 3 weeks off.</strong></p><p><strong>Must be able to commute to our offices in Chelmsford, Essex.</strong></p><p>Email us at <strong>careers@glasswallsolutions.com </strong>with your CV and an answer to this question:</p><p>If you had a simple HTTP web server and you had an agreed SLA of 99.9% with all your users what tools, cloud infrastructure, and best practices would you use?</p><p>We don’t need an essay but feel free to express your ideas in a Word document, diagram, markdown file, PowerPoint or YouTube video. Use whatever tools work best for you!</p><p><strong>This position is open until we find a suitable candidate.</strong></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=a4b10ab94220" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Postmortem: Production gets some new IPs!]]></title>
            <link>https://medium.com/@alex.potter.dixon/postmortem-production-gets-some-new-ips-a000146c63c6?source=rss-cd7b1b8adb8e------2</link>
            <guid isPermaLink="false">https://medium.com/p/a000146c63c6</guid>
            <category><![CDATA[incident-response]]></category>
            <category><![CDATA[learning]]></category>
            <category><![CDATA[cloud-computing]]></category>
            <category><![CDATA[devops]]></category>
            <category><![CDATA[engineering]]></category>
            <dc:creator><![CDATA[Alex Potter Dixon]]></dc:creator>
            <pubDate>Wed, 29 Jan 2020 08:25:28 GMT</pubDate>
            <atom:updated>2020-01-29T08:25:28.147Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*7M7SyZl0Rf_GMt0Y" /><figcaption>Photo by <a href="https://unsplash.com/@rojekilian?utm_source=medium&amp;utm_medium=referral">Sarah Kilian</a> on <a href="https://unsplash.com/?utm_source=medium&amp;utm_medium=referral">Unsplash</a></figcaption></figure><p>Postmortems are a massive part of any incident management process. They allow us to break down what happened, reflect and most importantly learn.</p><h3><strong>Background</strong></h3><p>Before I deep dive into the postmortem, we need some context on what we have running under the hood.</p><p>Currently, all of our FileTrust email services run in Microsoft’s cloud services: Azure. In Azure, we use a key piece of technology called Kubernetes. Kubernetes basically takes all of our different programs like receiving mail, using the Glasswall SDK, our web portal and runs them for us in one big virtual environment. What runs the Kubernetes in the cloud is an Azure service called VM scale sets, these are a group of computers that are scalable so more can be added or removed depending on the load. Remember these VM scale sets as they are key to what happened.</p><p>Each Kubernetes cluster also has a set of static Public Internet Protocol addresses, IPs so that our customers can open their firewalls to us so we can send their emails to them.</p><h3>What happened?</h3><p>At around 2:23 am on Friday 24th Azure Container Services ran a “create or update” event on our VM Scale Sets, the same one that runs our FileTrust Email services for all of the UK. This update was big enough to cause all the VMs to be updated causing some key services such as a certain public load balancer to be recreated.</p><p>When the new public load balancer was created it needed a public Ip but instead of picking our previous public Ip that all of our customers have whitelisted it generated a completely new one. What impact does this have? Well, as you would expect all of our customer&#39;s mail servers rejected the connections of our FileTrust email service so no mail could be delivered.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/601/1*DJgDAEyH5nQ2Nbfv1hOA3Q.png" /><figcaption>Customers firewalls rejecting us</figcaption></figure><p>This wasn’t the only problem we experienced from the VM Scale Set being updated. We also have an internal Redis cache that is used by a lot of programs to prevent unnecessary repetitive calls to our databases. The programs inside the Kubernetes cluster were unable to contact this Redis cache. Now, this wouldn’t normally be a problem as the programs still have the information from the last time the Redis cache was available but this information goes stale and eventually the programs will shut down to prevent that stale information from becoming false information. This meant email wasn’t able to flow through the service.</p><h3>What was the impact?</h3><p>As with any incident its important to understand what was the impact on our customers.</p><p>All of our customers would have experienced a maximum delay of up to 3 hours for the delivery of their email. This was during the early hours of Friday morning so most customers would not have noticed.</p><p>One customer did experience a delay of up to 6 hours with their mail as it got “stuck” in an earlier part of the FileTrust Email so required a bit more work to get sent out.</p><p>No mail was lost.</p><h3>How did we fix it?</h3><p>To fix the first problem of the false public IP we had to rerun our infrastructure pipeline. That associated the correct IP with our UK clusters. This meant our customer&#39;s mail servers would now accept our connection requests and in turn their mail.</p><p>The second problem with Redis&#39;s issues was to redeploy Redis and the services that used Redis so they could all connect and work in harmony.</p><h3>What went well</h3><p>As we were experiencing issues with contacting our customer&#39;s mail servers we had to bring in our customer support team who answered the call straight away at 4 in the morning. This was the first time we worked together in an on-call situation and they were a great help and provided support throughout the incident.</p><p>We also had to bring the on-call second SRE team member into the incident. They were a pair of fresh eyes and were absolutely key to diagnosing the problem!</p><p>The FileTrust Email service behaved exactly as expected and stored the emails it couldn’t process in queues so as soon as we fixed the issue we could open up the queues and send the mail through. Great work from the cloud development team!</p><h3>Action items</h3><p>The final part of any postmortem is the action items. These are a list of actions we need to take to make sure this event doesn’t happen again. The SRE team takes these action items very seriously and they are completed before any new work is started.</p><ul><li>Bind the IP to the creation of the Kubernetes cluster using the <strong>load-balancer-outbound-ips </strong>flag. This means that anytime an update happens a new IP isn’t associated preventing the above incident.</li><li>Move our Redis out of the Kubernetes cluster and use a high availability Redis in Azure. A high resilient Kubernetes cluster should not store state and moving the Redis out into a separate service will drastically improve the resilience of our cluster.</li><li>Deploy our Kubernetes clusters to other availability zones in Azure. This means the VMs we use for our cluster will be in different data centers within an Azure region. Preventing a single point of failure and improving the resilience of our cluster.</li><li>Speak to Microsoft support and find out why this “create or update” event happened to our cluster. We should have been able to handle this but this is the first time it has happened since we’ve been live.</li><li>Refactor our current infrastructure code to use Pulumi. This will allow the above changes to be quickly enacted, tested and reviewed.</li></ul><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=a000146c63c6" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Manage Things, Lead People]]></title>
            <link>https://medium.com/@alex.potter.dixon/manage-things-lead-people-26a4ad9b2ea?source=rss-cd7b1b8adb8e------2</link>
            <guid isPermaLink="false">https://medium.com/p/26a4ad9b2ea</guid>
            <category><![CDATA[management]]></category>
            <category><![CDATA[leadership]]></category>
            <category><![CDATA[inspiration]]></category>
            <category><![CDATA[saas]]></category>
            <category><![CDATA[tech]]></category>
            <dc:creator><![CDATA[Alex Potter Dixon]]></dc:creator>
            <pubDate>Sat, 07 Dec 2019 15:28:45 GMT</pubDate>
            <atom:updated>2019-12-07T15:28:45.992Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/960/1*NbeabtW3GlWRQmLiTd4m6A.jpeg" /><figcaption>Leading others</figcaption></figure><p>Having recently been given the amazing opportunity to lead the SRE team at Glasswall I’ve had time to reflect on how best to lead the team. I’ve had experience of management before but never in such a specialised and creative environment. Here are some of the ideas I feel strongly about.</p><h3>Trust</h3><p>I fully trust my team and 100% believe in them and their work. I believe that if something goes sideways that they will have my back and I have theirs. This is incredibly important, we run production systems that handle the security around companies email and files and if we all aren’t there for each other then things can easily fall apart.</p><p>This allows everyone to be open with each other when mistakes happen or concerns are raised. If I’ve made a poor decision I want my team to hold me accountable without me pulling rank so we can all learn together.</p><h4>Embrace Failure</h4><p>With high levels of trust, it allows us to embrace failure and be open with everyone about it. We are all human and mistakes happen to the best of us.</p><p>When something goes wrong the questions to ask are what happened and how can we fix it? Some people can fixate on who the blame lies with and this can be very toxic. At the end of the day, we want to learn how we got to that mistake and as a team how can we prevent it from happening again. People need to learn from their mistakes and be enabled to grow as a person, that’s why we run blameless post-mortems at the core of our incident management.</p><h3>Lead by example</h3><p>This has always been a big thing for me. Being a leader doesn’t provide me with any special privileges, my only privilege is leading a group of incredibly talented and intelligent individuals.</p><p>Leading by example stretches from workflow and to in house benefits. If the team agrees we must be following a particular coding standard then I must follow that, if I have flexi-time so I can miss traffic then everyone from Junior to Senior will also have those benefits.</p><p>If you think you are above anyone in your team and deserve special treatment then you shouldn’t be leading them.</p><h3>Vision, purpose and core values</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*4fET-tfBnawgeCm4" /><figcaption>Photo by <a href="https://unsplash.com/@heytowner?utm_source=medium&amp;utm_medium=referral">JOHN TOWNER</a> on <a href="https://unsplash.com/?utm_source=medium&amp;utm_medium=referral">Unsplash</a></figcaption></figure><h4>Vision</h4><p>As a leader of any team, you need to be able to share your vision and the direction the team will be going in. Within SRE, I’ve taken on what the team personally want to work on and what’s best for our customers and forged a 6-month road map. This is fluid as tech changes daily but it allows the team to plan for work, know what the priorities are and know what to expect.</p><h4>Purpose</h4><p>What’s my purpose? It’s a life long question a lot of us have. If your team members have it about their work then somethings gone wrong. At Glasswall our SRE teams purpose is:</p><p><strong>Keep the services we own up and thriving!</strong></p><p>This will help everyone in the team know what their key purpose is within the company.</p><h4>Core Values</h4><p>Core values are very important in creating a healthy culture within any team. They set out how we are to work and to interact with each other. This is why we have 3 simple values:</p><ul><li>First-class engineering.</li><li>Continuous learning.</li><li>Ownership.</li></ul><h3>Growth and learning</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*nOCmIWU3kQKUppWj" /><figcaption>Photo by <a href="https://unsplash.com/@adelau1?utm_source=medium&amp;utm_medium=referral">Austin D</a> on <a href="https://unsplash.com/?utm_source=medium&amp;utm_medium=referral">Unsplash</a></figcaption></figure><p>Within any tech-based job you always need to be learning and that’s no different within SRE. Learning is one of the core values and we instil it in everything we do.</p><p>We have 10% innovation time which is a closed-off day every two weeks where an SRE team member can complete training, try new tools or read books completely uninterrupted so they can maintain focus.</p><p>We also regularly explore new technologies and see if they can enhance our workflows. I make sure we never keep still and are always pushing the envelope.</p><h3>My role</h3><p>Within the team, I feel that my role is defined as a set of simple points:</p><ul><li>Make sure everyone is performing at their best.</li><li>Be there for everyone in the good and bad times.</li><li>Provide strong leadership with vision and decision making.</li></ul><p>I’m there to enable the team to perform and provide the tools and resources to allow that to happen. I need to be able to provide reassurance and guidance during rough times and to share in our celebration.</p><p>As a leader knowing that my team are happy because they are using cutting edge technologies aren’t feeling burnt out because they&#39;re not expected to work late, are able to work from home or have flexi-time and can spend time with family or friends while not worrying about work.</p><p>For me, that’s the greatest reward.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=26a4ad9b2ea" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Are you ready? New starters journey to production.]]></title>
            <link>https://medium.com/@alex.potter.dixon/are-you-ready-new-starters-journey-to-production-bcf0f7e30ac2?source=rss-cd7b1b8adb8e------2</link>
            <guid isPermaLink="false">https://medium.com/p/bcf0f7e30ac2</guid>
            <category><![CDATA[hr]]></category>
            <category><![CDATA[training]]></category>
            <category><![CDATA[startup]]></category>
            <category><![CDATA[development]]></category>
            <category><![CDATA[devops]]></category>
            <dc:creator><![CDATA[Alex Potter Dixon]]></dc:creator>
            <pubDate>Sun, 01 Dec 2019 12:43:25 GMT</pubDate>
            <atom:updated>2019-12-01T12:43:25.086Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*qbgZQ7N1gKTkft-G" /><figcaption>Photo by <a href="https://unsplash.com/@marco_blackboulevard?utm_source=medium&amp;utm_medium=referral">Marc-Olivier Paquin</a> on <a href="https://unsplash.com/?utm_source=medium&amp;utm_medium=referral">Unsplash</a></figcaption></figure><p>One of the main and if not the most important part of being an SRE is running production system. We have to protect our customer&#39;s data, make sure production is healthy and perform regular updates and maintenance. It&#39;s incredibly important to make sure that anyone that has the keys to the castle fully knows what they are doing otherwise things can go wrong very quickly.</p><p>So how do you make sure they are <strong>ready?</strong></p><h3>Training</h3><p>As with any new starter in any job, it is incredibly important that you provide adequate training but in the SRE world this especially apparent.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1006/1*5uEm5ueJgP-54Ar0N3MA8A.png" /><figcaption>Glasswall’s SRE training plan.</figcaption></figure><p>At Glasswall all of our new SRE team members have to go through a specialised and personally curated training book. It’s broken up into several key areas:</p><ul><li>SAAS, Our Email FileTrust service.</li><li>Kubernetes</li><li>Azure DevOps</li><li>Prometheus/Monitoring</li><li>Incident Management</li></ul><p>Each area is broken up into specific learning areas with references to training resources such as Linux Academy, training videos or talks given by other SRE members. On top of the learning resources, we have questions and activities that the new starter must complete such as building a new Kubernetes cluster or writing up a set of Prometheus queries. Finally, after each task or question has been completed it must be signed off by another member of the SRE team.</p><p>This training is a shared responsibility within the team this is because we all run the production system together and we all need to feel confident that they will have our back!</p><h3>Shadowing</h3><p>On top of the training, we need to make sure the new starter knows how to apply the new skills they have learned to the 50% operational work that is part of the SRE role.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/794/1*yDsE8hqegRlxGIeKw0o7Hg.png" /><figcaption>SRE training timeline.</figcaption></figure><p>As you can see they are shadowing an SRE team member from day one. This allows the new team member to get their feet wet as quickly as possible. The job of the on-call person is to show the new team member how they perform BAU (Business as Usual), explaining all the steps in detail and showing them how to handle any incidents that may happen. It is expected that the new team member to ask any questions and ask the on-call team member to repeat anything they aren’t sure about.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*odxVRS03lJOZBPsg" /><figcaption>Photo by <a href="https://unsplash.com/@jdent?utm_source=medium&amp;utm_medium=referral">Jason Dent</a> on <a href="https://unsplash.com/?utm_source=medium&amp;utm_medium=referral">Unsplash</a></figcaption></figure><h3>Sign off</h3><p>Once the new starter has completed the training booklet, shadowed for a month and everyone feels the new starter is ready its time to sign them off!</p><p>We want to make sure this is celebrated. They put the work in and are ready to be a fully fledge SRE!</p><h3>Final thoughts</h3><p>It’s important for us that the new starter succeeds and can handle their new role as an SRE. Their success is our success and ultimately it is the team&#39;s responsibility for making this happen. We have a feedback session once the new SRE has been initiated to make sure we can fill in any gaps and improve it for the next member, as one of our core values is <strong>Always Learning.</strong></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=bcf0f7e30ac2" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Terraform?]]></title>
            <link>https://medium.com/@alex.potter.dixon/terraform-5ce52a5865c6?source=rss-cd7b1b8adb8e------2</link>
            <guid isPermaLink="false">https://medium.com/p/5ce52a5865c6</guid>
            <category><![CDATA[aws]]></category>
            <category><![CDATA[azure]]></category>
            <category><![CDATA[terraform]]></category>
            <category><![CDATA[iac]]></category>
            <dc:creator><![CDATA[Alex Potter Dixon]]></dc:creator>
            <pubDate>Sun, 24 Nov 2019 13:02:28 GMT</pubDate>
            <atom:updated>2019-11-24T13:02:28.104Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/225/1*LUfMu8eWL7yuHNE9shnAZw.png" /><figcaption>Terraform by HashiCorp</figcaption></figure><p>At Glasswall our SRE (Site Reliability Engineering) team currently manages our IaC (Infrastructure As Code) deployments via Azure CLI. This is great for having direct access to the latest Azure resources however we are coming up against some challenges we’d like to solve:</p><ul><li>Our IaC is tightly coupled to Azure as its Azure CLI. If we wanted to move to AWS we would have to learn a new toolset.</li><li>It can be brittle. Microsoft regularly changes their CLI which can generate a lot of maintenance work and frustration.</li><li>Testing can be difficult. If you want to test the IaC you need to be able to quickly build, test and then tear down. This isn’t built into Azure CLI.</li><li>It’s difficult to visualise what is going to be built or changed. If you have a large set of Azure CLI commands unless you have a full understanding of what is currently deployed its difficult to know the impact of your changes.</li></ul><p>So how can Terraform solve these challenges for us?</p><h3>What is Terraform?</h3><p>Before understanding how Terraform can help us, what actually is Terraform?</p><p>As the name suggests it’s for building and changing the environments it is interacting with. It allows you to use a high-level configuration language. For example, creating a Redis cache in Azure is simple as:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/334/1*1f59BTxfUzWfO0KCONwbxw.png" /><figcaption>Azure Redis Cache definition.</figcaption></figure><p>This simplicity and simple to use configuration language have helped make Terraform one of the most popular IaC tools within the industry.</p><h3>Cloud agnostic, sort of.</h3><p>In an ideal world, anyone that maintains IaC would love to be able to deploy into Azure, AWS or DigitalOcean with a single code base but as they all offer services in different formats this can be difficult. With Terraform we can hit a middle ground: a common configuration language.</p><p>With Terraform we can use a common configuration language to interact with different cloud providers.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/868/1*-r2lU4V0-4yKIEnIC5tvwQ.png" /><figcaption>Single technology, multiple clouds.</figcaption></figure><p>This common configuration language allows teams to deep dive into one particular language and build specialties. This, of course, doesn’t remove the learning curve with each new cloud provider we support but it takes the initial edge off.</p><h4>Abstracting the brittle.</h4><p>Another benefit of using a single configuration language is it abstracts away the use of the cloud SDK directly. The challenge of the constantly changing CLI and strange nuances are all managed by Terraform. We no longer have to manage the changes, HashiCorp does.</p><h3>Testing</h3><p>As with standard Software Engineering, SRE engineers have to make sure what they are deploying or managing has been tested. How do we know what we are going to deploy is correct and what we designed? Gruntworks has us covered.</p><p>Gruntworks has an amazing opensource Go testing library for testing Terraform code called Terratest.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/907/1*jKpm9Ts0z6I3w7MLToh0nA.png" /><figcaption><a href="https://github.com/gruntwork-io/terratest">https://github.com/gruntwork-io/terratest</a> Usage example.</figcaption></figure><p>From this simple example we can answer the basic questions:</p><ul><li>Is the configuration valid syntax?</li><li>Are there any mistakes in the use of Terraform?</li><li>Will it create what we asked for?</li><li>Are the services up and running?</li></ul><p>Gruntworks is able to do this amazing work because of the power of Terraform&#39;s quick provisioning and down tearing. In any testing framework, you need to be able to quickly set up the environment and then, in turn, tear it down quickly to run the next test.</p><h3>Visualise the actions</h3><p>As with any action we perform on our infrastructure we want to know what our code change will actually change.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/999/1*Mo-tC2fVa2ZJMwDwOfh63w.png" /><figcaption><a href="https://github.com/coinbase/terraform-landscape">https://github.com/coinbase/terraform-landscape</a> AWS Terraform Plan example</figcaption></figure><p>The above example shows what is being added, green, what’s being modified, in yellow and anything that is being deleted is in red. When developing or modifying infrastructure visualisations like this are invaluable to engineers in helping them understand what impact their change is going to have.</p><p>The way Terraform does this is by storing the state of the infrastructure in a remote store and in this case, it is Azure Blob Storage. Terraform can then know what is currently there and how it will be affected by the code change. Some see this as a downside by having to store a separate remote store but for us, this is invaluable in planning infrastructure changes.</p><h3>Not all perfect</h3><p>As with any technology, there is no “silver bullet” to solve all of our challenges and one of Terraforms strengths is also its weakness. The abstracted configuration language. So what are the downsides of this?</p><ul><li>If any new infrastructure from a provider comes out we have to wait for HashiCorp to update their libraries for us to use it.</li><li>Some of our networking infrastructure is not fully implementable with Terraform.</li></ul><p>To solve this we will have to use a hybrid approach of Azure/Cloud Platform CLI/SDK and Terraform to fill the gap until these challenges are solved.</p><p>In the end, the benefits for us completely outweigh the downsides of Terraform. Keep an eye on GlasswallEngineering as we transition over to Terraform.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=5ce52a5865c6" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Balance of Reliability and New Features]]></title>
            <link>https://medium.com/@alex.potter.dixon/balance-of-reliability-and-new-features-7435e8bc788a?source=rss-cd7b1b8adb8e------2</link>
            <guid isPermaLink="false">https://medium.com/p/7435e8bc788a</guid>
            <category><![CDATA[cloud-services]]></category>
            <category><![CDATA[software-development]]></category>
            <category><![CDATA[sre]]></category>
            <dc:creator><![CDATA[Alex Potter Dixon]]></dc:creator>
            <pubDate>Sun, 17 Nov 2019 10:59:16 GMT</pubDate>
            <atom:updated>2019-11-17T11:05:59.455Z</atom:updated>
            <content:encoded><![CDATA[<h3>Balance of Reliability and Innovation</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/840/1*ee4ZixqRag541Je7SL0iFQ.png" /><figcaption>SRE Balance of Reliability and New Features</figcaption></figure><h3><strong>The Balance</strong></h3><p>At Glasswall one of our new products is our SAAS FileTrust email solution and two questions regularly come up:</p><ol><li>Is our service stable enough so that our users are having a good experience?</li><li>When can we get this new feature out?</li></ol><p>We want our users to feel like the service is mature, stable and safe to use otherwise why would they use it? We also want our service to be seen as exciting and engaging by evolving it with new features on a regular basis.</p><p>The problem is the more new features and innovations you bring to your online service the more instability it can bring!</p><p>So how does Glasswall solve this problem? <strong>SRE</strong></p><h3><strong>SRE</strong></h3><p>Glasswall has recently established an in house SRE team which stands for Site Reliability Engineers. Our main purpose is:</p><p>“<strong>Make the site as reliable as possible while keeping the pace of innovation high.”</strong></p><p>Our goal is to hold the balance and allow developers to create all the new exciting features for our customers but also making sure the customer doesn’t experience a degradation of service.</p><p>There are 3 simple tools that we use to make sure this happens: <strong>SLO, SLI and Error budget.</strong></p><h4><strong>SLO</strong></h4><p>What is an SLO? It is a Service Level Objective. A bit like an SLA, it&#39;s a metric that SRE negotiates with the business on what is an acceptable level of service.</p><p>An example of an SLO might be:</p><p><strong>99% of requests to our site are successful.</strong></p><p>This simply means out of 100 user requests to the site 99 must be successful and 1 can be unsuccessful such as a 500 error.</p><h4><strong>SLI</strong></h4><p>We need to be able to measure an SLO and this is where SLIs come into play. Service Level Indicators are the ways we measure our SLO.</p><p>Following on from the SLO example our SLI might be:</p><p><strong>Successful requests (2**,3**) / Total number of requests.</strong></p><p>This will generate a percentage that signifies what our SLO is.</p><h4><strong>Error Budget</strong></h4><p>Error Budget is the final piece of the puzzle. It is simply the gap between 100% SLO and your defined SLO. The Error Budget gives you the breathing room to deploy changes to your site, it’s the control mechanism for the balance we’ve been seeking.</p><p>Following on from the SLO example our Error budget is:</p><p><strong>100–99 = 1%</strong></p><p>This means that we have a 1% Error Budget for our requests to be unsuccessful.</p><p>Error Budgets are also designed to fit over a specific time period and within Glasswall we currently run our Error Budgets on a rolling 4 week period.</p><h3><strong>How does this all fit together?</strong></h3><p>Now we have all the tools we need, how do they fit together to answer our two previous questions:</p><ol><li>Is our service stable enough so our users are having a good experience?</li><li>When can we get this new feature out?</li></ol><p>Let&#39;s use the examples we’ve been using but with the added detail of 4 weeks traffic:</p><p><strong>4 weeks traffic: </strong>10,000</p><p><strong>SLO: </strong>99% of requests to our site are successful.</p><p><strong>SLI: </strong>Successful requests (2**,3**) / Total number of requests.</p><p><strong>Error Budget: </strong>100–99 = 1% or 100 unsuccessful requests</p><p>Now we know we have a 100 unsuccessful requests Error Budget we know what is an acceptable level of service.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*9dR9mh9bUJf_lrClPBmntg.png" /><figcaption>In budget or out of budget</figcaption></figure><p>If the service is in budget we keep developing new features and deploying them in new releases. If we are out of budget then we need to <strong>STOP.</strong></p><p>If the Error Budget is blown at Glasswall we will halt all changes and releases other than P1 issues, security fixes and bug fixes to our production systems until the following has been completed:</p><ul><li>Postmortem has been completed.</li><li>Bugs from the postmortem have been fixed in production.</li><li>Other actions from the postmortem such as process change have been enacted.</li><li>Service is back within its SLO.</li></ul><p>Using this mechanism we can control the flow of new features going to the platform while making sure the customer is experiencing the level of service they expect.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=7435e8bc788a" width="1" height="1" alt="">]]></content:encoded>
        </item>
    </channel>
</rss>