<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by feidhlim o&#39;neill on Medium]]></title>
        <description><![CDATA[Stories by feidhlim o&#39;neill on Medium]]></description>
        <link>https://medium.com/@feidhlimoneill?source=rss-39b1ce642c79------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/0*PqLlNrLzEWaVhFKW.JPG</url>
            <title>Stories by feidhlim o&amp;#39;neill on Medium</title>
            <link>https://medium.com/@feidhlimoneill?source=rss-39b1ce642c79------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Thu, 19 Nov 2026 09:48:15 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@feidhlimoneill/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[Scaling our infrastructure at TransferWise: how we make it work]]></title>
            <link>https://medium.com/wise-engineering/scaling-our-infrastructure-at-transferwise-how-we-make-it-work-a4c7a1bd5b8e?source=rss-39b1ce642c79------2</link>
            <guid isPermaLink="false">https://medium.com/p/a4c7a1bd5b8e</guid>
            <category><![CDATA[platform]]></category>
            <category><![CDATA[infrastructure]]></category>
            <category><![CDATA[specialization]]></category>
            <category><![CDATA[engineering-culture]]></category>
            <category><![CDATA[transferwise]]></category>
            <dc:creator><![CDATA[feidhlim o'neill]]></dc:creator>
            <pubDate>Wed, 16 Sep 2020 08:32:30 GMT</pubDate>
            <atom:updated>2021-06-08T10:19:09.149Z</atom:updated>
            <content:encoded><![CDATA[<h3><strong>Scaling our infrastructure at Wise: how we make it work</strong></h3><p><em>At Wise, we’re proud of our autonomous team culture and </em><a href="https://www.transferwise.jobs/2019/12/05/how-we-work-a-practical-guide/"><em>the way we work</em></a><em>. Over the past few years we’ve been growing rapidly. We now have over 60 independent, autonomous teams building and improving our product for our 8 million customers. Scaling our product has resulted in our infrastructure and the team behind it to grow and evolve accordingly. This has meant moving from a focus on physical infrastructure to introducing specialised teams to accommodate for new requirements of the business.</em></p><p><em>This blog is about the rise of specialisation at Wise and how we make it work. We’ll deep-dive into the upsides and challenges of specialisation, as seen by our Infrastructure team, and how our ways of working, customer focus and </em><a href="https://transferwise.com/gb/blog/the-transferwise-mission"><em>mission</em></a><em> help us balance the trade-offs.</em></p><p>The last few years have been quite a ride for Wise. We’ve grown into a global team of 2200+ employees with over 400 Engineers and now have over 300 microservices running in production. Our autonomous team structure has allowed us to scale our teams, product and infrastructure quickly. But this structure also requires constant support and decision-making, while balancing between reflecting back to our foundations and thinking strategically about the future. We need to think through how and what can we scale, what’s culturally important, what behaviours should be shared and what shouldn’t and, of course, always keep in mind how our decisions benefit (or hurt) our customers.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*lZ66Nf66o12PFc_N" /></figure><h3>Scaling fast = more complexity</h3><p><strong>As we grow, the way our autonomous teams work together and ship new features is becoming more complex.</strong></p><p>We work in a heavily regulated environment so many requirements aren’t optional. We also support a lot more customers, who have different types of needs from our product. We’re operating and expanding into a lot more countries, while building lots of new features. It’s just more complex!</p><p><strong>Our technical platform is a key part of how we operate and scale. It helps us ship products quickly, while reducing the burden of managing complex requirements.</strong></p><p>This technical platform covers a broad number of shared requirements like messaging, service layer and observability. To design and operate these shared technologies, we’ve built teams around these domains. The teams work out how to provide good enough basic capabilities and where we can invest in to give us advantage. They aim to abstract some of the complexity away from our Product teams so teams can work efficiently.</p><h3>So, how do we evolve our platform?</h3><p>Our Product teams own the full life-cycle and decision-making for their areas. This comes with lots of questions and decisions to make about how a feature is designed, built and operated. They often come with new requirements for our Platform teams.</p><p>For example:</p><ul><li>Does our Database team onboard a new technology to support a new Wise product?</li><li>Can the Product team self-support?</li><li>Or do we even have to compromise on their technology choice and use existing tech?</li></ul><p>The answer is of course, that it depends.<strong> </strong>But decisions like this are always guided by how worthwhile the tradeoffs are and our long-term customer benefit. Teams make these decisions in collaboration.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*WGfdWgMeBzWizh9bVzrclQ.jpeg" /></figure><p>Looking back, the genesis of these specialist platform teams that support these shared requirements, is in the small ‘infra’ focused team. Back in the day our Infrastructure team was very much focused on physical infrastructure when we ran in our own data centre. From this beginning we’ve built out our foundation technical platform (and migrated to AWS) that our products run on. What used to be a single Infra team, has grown into over 12 Platform component teams with over 60 Engineers.</p><p>To understand what works for us and where teams can accept a shared approach <strong>we’ve developed our opinionated platform that provides a standard pattern</strong>. In short, an opinionated platform is where we have coded a preferred way of working that makes it easy to adopt that preferred way but is flexible enough to allow teams to deviate from that standard when needed.</p><p><strong>Creating teams around these shared components means we can manage more scale and business complexity</strong> and provide more polished solutions. Specialisation is essential. An example of this would be observability, monitoring and alerting. We’ve developed libraries that integrate into our observability stack that provide standard ways of exposing metrics as well as monitoring and alerting capabilities that build on these metrics. This means the ‘out of the box’ experience is good for teams for all the basic things they need to get their new product live. But, if the Product team has a requirement for monitoring or tooling, we collaboratively discuss how to accommodate and may decide the best approach is for them to build their own capabilities.</p><p><strong>Fun fact: we collect over 5billion metric reading an hour from our platform!</strong></p><p>If you want to know more about our stack read Yurii’s excellent post <a href="https://medium.com/transferwise-engineering/the-transferwise-stack-2020-edition-68f70267501b">here</a>.</p><h3>How we overcome some of the challenges of specialisation</h3><p>While our collaborative way of working helps us scale fast, there are some risks to it. Platform teams are specialists and have the challenge of siloed decision-making. With the complexity in their domain, it can make it more difficult for people outside their team to engage in decisions they make. This is problematic as we know it’s exactly that external perspective and collaboration that’s needed to ensure the team is on the right track.</p><p><strong>So, we need specialist teams, but want to avoid silos. How do we do it? Here’s a few ways we make it work:</strong></p><h4><strong>Staying aligned and starting with the why</strong></h4><p>First off, we make sure our Platform teams build systems the same way other product-focused Engineering teams do. We can avoid some of these pitfalls by focusing on <em>why this work benefits all our customers</em>. Understanding the why behind what we’re building, before thinking about the how of doing it, we stay on the same track and collaborate on decisions.</p><p>We ensure transparency and gain engagement by sharing plans and sharing approaches as soon as possible. We do this through open communication channels and quarterly planning sessions and this allows us to be transparent about work being planned and alternatives we can and could use.</p><h4><strong>Ensuring feedback between Platform and Product teams</strong></h4><p>Sharing openly about our plans and work brings me to feedback. Looking for feedback has always been part of how Engineering teams work at Wise. Evolving our forums to make it easy to discuss and contribute across Engineering is part of this.</p><p>As an example, we host open ‘show and tell’ type sessions where teams present ideas and proposed solutions. The emphasis on these forums is to share ideas early, get feedback from others and discuss solutions together. Platform teams also share their quarterly plans to the entire company and get lots of feedback on how to improve their impact.</p><p>Of course, this is a two way process — Platform Engineers are expected to give feedback on Product teams’ plans too. This way, solutions naturally evolve from early stage feedback.</p><h4><strong>Finding a balance between enabling teams to work and building up new capabilities.</strong></h4><p>With our opinionated platform we’re aiming to avoid a restrictive environment and provide just enough guard-rails so that teams can move fast while maintaining high non-functional quality. When thinking about work we can do, <strong>we always refer back to the impact on our customers</strong> rather than doing something because it’s the easiest thing to do.</p><p>In the future we expect this model to mature. It’s all about balancing between good enough capabilities that teams just need to operate and things that help us achieve our mission quicker. On a company level, we use our mission to guide our work, whether it’s creating new teams, making product decisions or prioritising new features. It helps us to stay aligned, share learnings and prioritise what’s best for our customers.</p><p>Specifically in the Platform team, a key focus area for us is to better understand those use cases and teams that our platform doesn’t support well today. This means understanding when to actively unblock teams so they can work efficiently while not getting involved with teams who don’t need support at the moment. Our future plans are to iterate on the current model, consciously incorporating new requirements in support of our team’s shared mission and better quantifiably measure our impact.</p><p>I hope this blog gives you a good overview of how we look at specialisation at Wise along with insights into how we make it work for our Platform team. If you have any questions, I’m happy to answer them in the comments section.</p><p>P.S. Interested to join us? We’re hiring. Check out our <a href="https://www.wise.jobs/search/?t=engineering&amp;sort=oldest&amp;utm_source=medium&amp;utm_medium=referral&amp;utm_campaign=engineering_blog">open Engineering roles</a>.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=a4c7a1bd5b8e" width="1" height="1" alt=""><hr><p><a href="https://medium.com/wise-engineering/scaling-our-infrastructure-at-transferwise-how-we-make-it-work-a4c7a1bd5b8e">Scaling our infrastructure at TransferWise: how we make it work</a> was originally published in <a href="https://medium.com/wise-engineering">Wise Engineering</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Microservices Hygiene Day]]></title>
            <link>https://medium.com/@feidhlimoneill/microservices-hygiene-day-42100038b3af?source=rss-39b1ce642c79------2</link>
            <guid isPermaLink="false">https://medium.com/p/42100038b3af</guid>
            <category><![CDATA[microservices]]></category>
            <category><![CDATA[software-development]]></category>
            <category><![CDATA[devops]]></category>
            <dc:creator><![CDATA[feidhlim o'neill]]></dc:creator>
            <pubDate>Tue, 15 Sep 2015 09:11:15 GMT</pubDate>
            <atom:updated>2015-09-22T19:29:41.312Z</atom:updated>
            <content:encoded><![CDATA[<p>How Hailo ensures its microservices platform is kept up to date.</p><p>Hailo launched its golang microservices platform over 12 months ago and so far it’s delivered on all that it promised in terms of agility (in all senses), availability and scalability. Our attention to tooling on provisioning, debugging and monitoring have, for the most part, made the platform a dream to develop on and support. With over 200 services running and increasing every day we always knew that at some point we would need to think about service lifecycle and decommissioning microservices. Last week we held our inaugural ‘Hygiene’ day to begin to address this. What follows are some details and observations about what we did.</p><p>Someone who knows about these things (aka <a href="http://martinfowler.com/">Martin Fowler</a>) said “You can usually only judge architectural decisions after a system has matured and you’ve learned what it’s like to work with years after development began. We don’t have many anecdotes yet about long-lived microservice architectures”. We have not quite reached that milestone but we had begun to realise that some microservice on our platform were not being refactored, rebuilt or redeployed as much as we expected. Of course our vision is to automatically build and deploy everything regularly but this still remains our medium term goal so what to do? Having these old stable microservices is good in one way (it just works) but it meant our services were not taking advantage of new capabilities or bug fixes in our own or 3rd parties libraries. There is nothing worse that having an incident and it turning out that that particular edgecase was solved 12 months ago but this service was never redeployed!</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/900/0*GHIX2zDHCVjVTPsw." /></figure><p>While we wait for our CI/CD nirvana we initiated a ‘Platform Hygiene Day’ when all of engineering takes a day out of their sprints and focused on non-functional tasks. Merging fixes, resetting error levels, minor service refactoring form a part of this but for our inaugural session we decided to build using latest and greatest everything and redeploy to live as much of the platform as possible. Our CI/CD pipe is very capable of doing hundreds of deployments in a day but we had not done so in live in quite a while so there were a degree of apprehension. I certainly expected some low level impact during the day!</p><p>So here is what we did. We divided our engineers into 3 separate teams of engineers that normally do not work closely together — we assigned a team leader for the day for each group. Secondly we divided the entire list of microservices running in live into 3 lists ordered by build age and assigned a list to each team. We removed some of the kernel services from the lists because for this first pass we didn’t want to risk platform wide issues and kernel services would have that impact. Some teams did some prep — some didn’t. Nothing was mandated.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/291/0*SdXKQDcvm70lObP7." /></figure><p>On the day we met up first thing (10am is first thing right) and called out to ground rules. Firstly, pizza was to be provided for lunch. We assigned an engineer to role of controller for the day to be point for conflicts, advice and cross team coordination on the code side. We assigned two operations focused engineers as guardians of the live environment to manage the flow of change to live and stop the flow if instabilities were detected — you needed a positive acknowledgement from one of them before deploying. Each of the three teams was given a room for the day to collocate and 10 minutes later off they went. During the day the team came up with lightweight solutions to issues as they came up and QA did quick regression tests on services of concern.</p><p>And the results…</p><p>We build and shipped 104 services on the day with over 5000 bug fixes included in there. We retired a handful of services on the day with more identified for retiring. Just as importantly we had zero incidents which is as much a testament to our platform as to the quality of our engineers. For the future, besides getting back to minor refactoring etc as mentioned before, we want to improve our tooling around mass deployments in the short-term to make these days less onerous ahead of getting our deployment pipe fully automated. Of course non functional work needs to be part of teams normal backlogs but having a day a month to focus on these tactical tasks is very beneficial to the team and helps guide those backlog items. Finally, we came out of the day encouraged by how well the platform and team performed and confident we had avoided incidents those unfixed bugs were going to cause. Overall a great day.</p><p>Try the Hailo platform for yourself — head over to <a href="https://github.com/hailocab/h2">https://github.com/hailocab/h2</a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=42100038b3af" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[TechOps, DevOps, NoOps oh my!]]></title>
            <link>https://medium.com/@feidhlimoneill/techops-devops-noops-oh-my-3e4827d56a9a?source=rss-39b1ce642c79------2</link>
            <guid isPermaLink="false">https://medium.com/p/3e4827d56a9a</guid>
            <category><![CDATA[culture]]></category>
            <category><![CDATA[operations]]></category>
            <category><![CDATA[devops]]></category>
            <dc:creator><![CDATA[feidhlim o'neill]]></dc:creator>
            <pubDate>Sat, 04 Jul 2015 12:25:00 GMT</pubDate>
            <atom:updated>2018-09-10T13:27:36.095Z</atom:updated>
            <content:encoded><![CDATA[<p>Everyone needs to code but to run systems at scale that is not enough!</p><p>Coding is now a skill all IT professionals need. From the most junior operations engineer to senior Infrastructure architects, they all need to be proficient in coding and the development process. Cloud and visualization, in all its guises, and the rise of the API has driven this.</p><p>With Amazon Web Services and other public and private clouds technologies we have seen the rise of “Infrastructure as Code” where infrastructure is defined programmatically and invoked across an API. Infrastructure as Code has removed the need to manage physical hardware and has deskilled (maybe re-skilled) the provisioning process to a degree that product developers can eliminate the need for SysAdmins or other operational specialists to develop, test, deploy and launch their product. This model is called NoOps and while NoOps absolutely can work, as your product scales and you bring on real customers you inevitably start layering back in other disciplines to ensure you can operate as scale, reliably and cost effectively.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/316/1*83J-BDrfDpIj7c5xxpPhtw.jpeg" /><figcaption>multiple disciplines</figcaption></figure><p>So for some terminology… TechOps is the traditional sysadmin discipline focused on running systems. TechOps are experts in running services but not generally skilled in building them. The focus on incident, capacity management, monitoring systems, support tasks, infrastructure planning etc. In recent years there has been a tendency to relabel TechOps as DevOps but DevOps is not a role, it is the collaboration between TechOps and product development. TechOps can be also referred to as Service Reliability Engineering but what really distinguishes the roles is the % of time spent coding with TechOps &gt; 40% and the focus on running systems not building them.</p><p>As mentioned DevOps is not a role but the rise of the developer sysadmin is real. A better term for these developer sysadmins is ‘automation engineering’. Although still vague automation engineering reflects the skills and focus. Automation engineering is in the weeds of CI/CD, environment provisioning and resource scheduling, configuration management and the likes. From a development process they focus on non-functional requirements (NFRs) needed to make the product run efficiently, reliably and cost effectively both from the standpoint of running and developing products. Automation engineers spend &lt; 60% of their time coding.</p><p>NoOps is by definition when product developers do it all. Two reasons why NoOps exists, firstly with public cloud it can and secondly, if no operational teams to throw stuff over the wall at engineers should produce better software (from a NFR perspective). At small scale NoOps does work but it does not work at scale.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/100/1*NC8dYJSyBbe_Y-GbzCM7dA.jpeg" /><figcaption>noops</figcaption></figure><p>You can have an excelled product engineer who is excellent in an incidents, and DevOps encourages this crossover, but in general for engineers to excel they need a core focus and that focus needs to be the type of work that satisfies and interests them. You can have sysadmins who dabble in automation, or product engineers that dabble in capacity management and load testing all of which should be nurtured but in general these discipline are happiest in their chosen focus area and you cannot depend on dabbling to run an effective service.</p><p>To run a product reliably at scale you need people who are focused on TechOps, automation and of course product development with clear definition of responsibilities between them. You need someone to think of capacity management, someone to manage incidents when Amazon is having issues (which it does), someone to create tools to allow seamless migration of code to production, someone to create tools to create development environments that match productions configuration and without all these things your going to really struggle maintaining a service.</p><p>There are obvious overlaps and shared responsibilities — like on-call — but these are shared responsibilities and not areas of focus. On the other side, you may have deep pockets of technical or process skill that are so specific they are islands of knowledge. These subject matter experts like DBA or security need a singular focus to be effective but while creating roles with deep specializations may be necessary in some cased it is not to be encouraged as it tend to create silios and reduce knowledge sharing. So areas of focus but not areas of exclusive focus.</p><p>So, to sum up. NoOps can get you going but does not scale in the long-term. Traditional TechOps disciplines are still essential but need to evolve with the rise of IaaS and the API and automation engineering complements Techops but does not replace it.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=3e4827d56a9a" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Microservices — a DevOps architecture.]]></title>
            <link>https://medium.com/@feidhlimoneill/microservices-a-devops-architecture-12caceda4a71?source=rss-39b1ce642c79------2</link>
            <guid isPermaLink="false">https://medium.com/p/12caceda4a71</guid>
            <category><![CDATA[microservices]]></category>
            <category><![CDATA[devops]]></category>
            <category><![CDATA[change-management]]></category>
            <dc:creator><![CDATA[feidhlim o'neill]]></dc:creator>
            <pubDate>Tue, 12 May 2015 15:16:00 GMT</pubDate>
            <atom:updated>2015-05-12T15:16:00.854Z</atom:updated>
            <content:encoded><![CDATA[<p>Microservices — a DevOps architecture.</p><p>DevOps is all about breaking down the barriers between development and operational teams and thereby unlocking a wide range of benefits in how we design, build, deploy and support software. DevOps has been the key driver behind the adoption of a whole raft of technologies, architectures, processes and skills and one of the best of these is continuous delivery — delivering business value in a continuous stream without creating chaos as you do so. There are a number of aspects to continuous delivery but a key point, certainly from an Ops perspective, is that new features or functionality get deployed into production as part of a continuous process and isn’t just “thrown over the wall” to be installed (and supported) by a separate team.</p><p>Complementary to continuous deployment has been the rise of the microservices architecture. Microservices are, as the name suggests, breaking down your business logic into 10s if not 100s of small independent services and ideally suited to the low risk changes that continuous deployment promises. To see why lets look back when code was deployed in large single application with separate operations teams.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/337/1*sc4HSSJWYf_Ad4w4_9vXHQ.png" /><figcaption>Big Bang!</figcaption></figure><p>In the past, besides handing the code off to a separate Ops team, we had to go through a change process to assess the risks associated with the change as well as things like compliance. Production had to be protected from developers who considered their job done once the change ticket was created. In larger companies there was a function specifically created to oversee this process. It was a slow process and because getting approval for a deployments was onerous it was natural to get as much into each release as possible. Therefore, ironically, the change process designed to manage risk was in some cases actually increasing it. Not good.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/373/1*mISFkQVWd1pa4kUM640DeQ.png" /><figcaption>Continuous Delivery</figcaption></figure><p>So along came DevOPS. Continuous delivery is about deploying small changes continually so the risk on each change is lower. This certainly helped lower the risk of each deployment but when dealing with monolithic systems the business impact, of a bad change, could still be large and difficult to diagnose and the the number of releases needed to achieve the same results as a bigger release meant there was more scope for human error. In response to this DevOps engineers used their development skills to create tools to make deployment easier.</p><p>Automation tools allow us to better manage the change process and de-skill making the change itself so engineers could focus on the impact and do more releases. With better tools to both orchestrate the change and monitor more widely in real-time the impact of the change we could react quicker to issues. The automation even allowed you to automate the release process formalities too if needed — to the satisfaction of compliance.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/510/1*dgwGWsWsEdxl5iQC1sHgJA.png" /><figcaption>Microservices</figcaption></figure><p>Now I, finally, get to the point of micro-services and change management. Microservices decomposes the business logic into smaller, independent chunks of code that are loosely coupled and ideally suited to continuous deployment. Of course a misbehaving microservice can still cause significant business issues but with correct tooling should be easily identified and reverted and most importantly can be done without slowing down other independent changes — so it scales.</p><p>Because microservices decompose the business logic, having 100s of Microservices is to be expected so good tooling and automation is essential for both release and debugging issues. If you invest upfront in good monitoring both at the macro (metrics, graph) and the micro (traces, correlation) level as well as orchestration tools then the actual process of releasing is low risk with limited scope of change. Related services in the architecture can help identify issues and in more mature platforms, automated rollback can be trigged if issues detected via the extensive monitoring.</p><p>Of course these microservices will have dependencies on infrastructure services and platforms that need updating and may not lend themselves to decomposition in the same way but they also need to change less and may indeed, in the next round of architectural enhancement, follow a similar path.</p><p>In short, microservices combined with comprehensive monitoring and orchestration eliminates the need for manual change management processes even at scale.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=12caceda4a71" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[The Post Mortem and Continuous Improvement]]></title>
            <link>https://medium.com/@feidhlimoneill/the-post-mortem-and-continuous-improvement-38040a8096c?source=rss-39b1ce642c79------2</link>
            <guid isPermaLink="false">https://medium.com/p/38040a8096c</guid>
            <category><![CDATA[risk-management]]></category>
            <category><![CDATA[postmortem]]></category>
            <category><![CDATA[continuous-improvement]]></category>
            <dc:creator><![CDATA[feidhlim o'neill]]></dc:creator>
            <pubDate>Fri, 17 Apr 2015 12:34:11 GMT</pubDate>
            <atom:updated>2015-04-17T12:46:28.607Z</atom:updated>
            <content:encoded><![CDATA[<p>This post is about people and technology delivery and how to manage risk. Its not a complex post but an important message for anyone interested in technology delivery.</p><p>Risk, for our purposes, leads to outages or data loss or being hacked or more subtle things like service degradation.The likeness of something bad happening. Risk can be reduced but never eliminated and the cost of reducing risk increases as you get closer to zero risk. So you can only ever manage risk.</p><p>So, you’re responsible for an online service your company depends on. You need to deploy it on a redundant architecture, architect out risks, you need to have a change process or, if you’re able to, you build a deployment pipeline to manage your risk.</p><p>First thing is to work out your appetite for risk. Everything you do that reduces risk beyond what you absolutely need is a waste. A waste of money, of time, of effort. So you need to find the balance. You need Business understandable ways of expressing your risk like Monthly uptime %, Time to Recover, Time between Failure or Cost of Outages and then determine with your business the gap between the appetite for the the risk and your reality.</p><p>Now I’ll mention ITIL. I’m not a huge fan in general because of its emphasis on the process rather than the people. Having said that when taken in context of when and why it was written there are some good basic principles in there. #1 of these is ‘Continuous Improvement’ where you create a feedback loop between the observed behaviour and the future work being planned to improve the observed behaviour. The Plan — Do — Check — Act loop.</p><p>In the context of technology delivery we are applying continuous improvement to reduce risk (of bad things happening) to the negotiated levels. In practice technology delivery is very interrupt driven so we need a simple and compelling way to make sure this happens and make prioritizing your precious time on the areas with the biggest return and the postmortem is an ideal tool.</p><p>Most are familiar with the idea of post-mortems or incident analysis — where you review an incident to ensure the impact, the cause and the remediations of an issue are understood but many organizations see this as a box ticking exercise and miss the real value. To get the most from post-mortems you need to make sure they are blameless, fact based and focused on two questions.</p><ol><li>How could we have recovered quicker?</li><li>How could we have avoided the issue in the first place?</li></ol><p>And most importantly we need to make sure we do not forget the answers to these questions as soon as we walk out of the meeting.</p><p>So you track the remediations to answer those two questions. Put them on a whiteboard or even better open a ticket and you get to work on the remediations with the biggest impact. If you do this for every issue soon patterns emerge. Nothing formal (like ITILs Problem Management) just go after the low hanging fruit. You ring-fence some resources and put them to work. You have just started a Continuous Improvement program!</p><p>So now as you get your metrics (risk profile) to the negotiated level you can look to refine expectations or redirect focus onto other areas such as cost management, development productivity or some functional development. The beauty of Post-Mortem remediations is they are incident driven so you are guaranteed to be fixing something that did cause an issue and you are making things better.</p><p>In the event you run out of incidents you need a more proactive approach but thats for another day…</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=38040a8096c" width="1" height="1" alt="">]]></content:encoded>
        </item>
    </channel>
</rss>