<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Ryan Dawson on Medium]]></title>
        <description><![CDATA[Stories by Ryan Dawson on Medium]]></description>
        <link>https://medium.com/@ryandawsonuk?source=rss-5e3e8868caf------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/1*wZOqXUDA8VwvP0fjeEZFpw.jpeg</url>
            <title>Stories by Ryan Dawson on Medium</title>
            <link>https://medium.com/@ryandawsonuk?source=rss-5e3e8868caf------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Wed, 18 Nov 2026 22:22:06 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@ryandawsonuk/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[How to evaluate MLOps Platforms]]></title>
            <link>https://medium.com/mlops-community/how-to-evaluate-mlops-platforms-c98cf7874cca?source=rss-5e3e8868caf------2</link>
            <guid isPermaLink="false">https://medium.com/p/c98cf7874cca</guid>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[mlops]]></category>
            <category><![CDATA[artificial-intelligence]]></category>
            <dc:creator><![CDATA[Ryan Dawson]]></dc:creator>
            <pubDate>Wed, 22 Dec 2021 16:12:52 GMT</pubDate>
            <atom:updated>2022-01-19T12:07:33.762Z</atom:updated>
            <content:encoded><![CDATA[<h3>How do you evaluate MLOps platforms?</h3><p>Companies that pioneered application of AI at scale did so using in-house ML platforms (facebook, uber, LinkedIn etc.). These capabilities are now available in off-the-shelf products. The rush to MLOps has led to <strong>too much choice</strong>. There are <a href="https://huyenchip.com/2020/12/30/mlops-v2.html">hundreds of tools</a> and at least 40 platforms available:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*k2E8GGcrK_9rIml8" /><figcaption>Image by Thoughtworks, from <a href="https://www.thoughtworks.com/what-we-do/data-and-ai/cd4ml/guide-to-evaluating-mlops-platforms">Guide to Evaluating MLOps Platforms</a></figcaption></figure><p>This is a very difficult landscape to navigate. Let’s understand the big challenges and then we’ll introduce some new free material that aims to address these problems.</p><h3>Challenge #1: Overwhelming Choice</h3><p>Chip Huyen gathered data on the <a href="https://huyenchip.com/2020/12/30/mlops-v2.html">ML tool scene in 2020</a>. <strong>Chip found 284 tools and the number keeps growing.</strong> Evaluation don’t have to consider all of these tools in detail as long it’s possible to narrow them down to the ones that are most relevant to start with. That’s not easy though because to do that we would want clear categories and MLOps categories are not clear…</p><h3>Challenge #2: Blurred Categories</h3><p>Typically we get a picture of what software does what by putting products in categories. There are attempts to do this with the <a href="https://landscape.lfai.foundation/">LFAI Landscape diagram</a>, the <a href="http://46eybw2v1nh52oe80d3bi91u-wpengine.netdna-ssl.com/wp-content/uploads/2021/10/2021-MAD-Landscape-V2-Small.jpg">MAD landscape</a> and <a href="https://github.com/EthicalML/awesome-production-machine-learning">github lists</a>. But often ML software does more than one thing and could fit into any of several categories. Platforms are especially hard to categorise as they explicitly aim to do more than one thing.</p><p>Because software that does several things is hard to categorize, ML platforms all tend to end up in a single category of ‘platform’. This <strong>obscures what emphasis each platform has</strong> and also <strong>loses all of the nuance</strong> of how different platforms do similar things in different ways.</p><h3>Challenge #3: Shifting Landscape</h3><p>ML categories are hard to keep track of in part because <strong>new categories keep appearing</strong>. Feature stores, for example, haven’t been around very long but are now a significant category. This affects platforms too as platforms are introducing big new features and changing their emphasis (in part in response to what new tools appear).</p><h3>Challenge #4: Complex Problems</h3><p>ML is complex. It’s also a big field. <strong>Platforms do a wide range of things</strong> so that means understanding regression, classification, nlp, image processing, reinforcement learning, explainability, AutoML and a lot more.</p><h3>Challenge #5: Range of Roles and Stakeholders</h3><p>Not only are the problems various and complex, there’s also a <strong>range of roles involved in the ML lifecycle</strong>. Data Scientists, ML Engineers, Data Engineers, SRE, Product Managers, Architects, Application Developers, Platform Developers, End Users etc. Different roles have different points of interaction with the software and different needs from it.</p><h3>Challenge #6: Build vs Buy and other controversies</h3><p>There’s a lot of discussion of build vs buy trade-offs in the ML space. How much control do organisations need over their stack? How much does that control cost? Build vs buy is often presented as an either-or but it is more of a spectrum. This is just one of the <strong>confusing controversies in the ML space</strong> (consider how controversial AutoML is).</p><h3>How do we get on top of all this?</h3><p>Let’s looks separately at specialist tools vs platforms.</p><p>For specialist tools the mlops.community has some great profiles that you can drill into <a href="https://mlops.community/learn/">on the website</a>. Tools are broken down by category and the website offers feature lists, introduction videos and a feature to compare tools side-by-side.</p><p>For understanding the platforms landscape and dealing with the trade-offs Thoughtworks has launched a Guide to Evaluating MLOps Platforms:</p><p><a href="https://www.thoughtworks.com/what-we-do/data-and-ai/cd4ml/guide-to-evaluating-mlops-platforms">https://www.thoughtworks.com/what-we-do/data-and-ai/cd4ml/guide-to-evaluating-mlops-platforms</a></p><p>This is available for free without any sign-up. It addresses <strong>how to distinguish MLOps platforms. </strong>The guide looks at MLOps platforms through different lenses and offer categories to break them down. For example, here’s a picture from the guide showing the distinction between MLOps platforms and tools:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*O5befU4kqKjPECuSliV4fA.png" /><figcaption>Image by Thoughtworks, from <a href="https://www.thoughtworks.com/what-we-do/data-and-ai/cd4ml/guide-to-evaluating-mlops-platforms">Guide to Evaluating MLOps Platforms</a></figcaption></figure><p>It’s a spectrum with some all-in-one platforms trying to address the whole ML lifecycle, specialist tools address individual parts of the lifecycle and in between there’s many focused platforms that address multiple elements of the lifecycle but not the whole lifecyle. Elsewhere in the guide, we look at other questions such as the strategies for low-code and AutoML platforms<strong>, </strong>the roles and personas involved in MLOps and <strong>how to structure an evaluation to suit the needs of your organisation</strong>.</p><p>Understanding the high-level platforms landscape is just part of an evaluation. We also need to apply this knowledge and see how to compare platforms against each other. For this we’ve released an open source comparison matrix:</p><p><a href="https://github.com/thoughtworks/mlops-platforms">https://github.com/thoughtworks/mlops-platforms</a></p><p>The matrix is structured to highlight how vendors do things in their own ways and also point to more detail in the product documentation. We’ve also included in the repository a series of profiles that describe the product directions of popular platforms concisely and in a marketing-free way.</p><p>We hope you find this material helpful and welcome contributions in github. Feel free to ask any questions on the <a href="https://mlops.community/">mlops.community slack</a> (and tag Ryan Dawson on there so that I see it).</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=c98cf7874cca" width="1" height="1" alt=""><hr><p><a href="https://medium.com/mlops-community/how-to-evaluate-mlops-platforms-c98cf7874cca">How to evaluate MLOps Platforms</a> was originally published in <a href="https://medium.com/mlops-community">MLOps.community</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Too Much Product Roadmap May Kill You — Here’s a Better Strategy]]></title>
            <description><![CDATA[<div class="medium-feed-item"><p class="medium-feed-image"><a href="https://medium.com/better-programming/too-much-product-roadmap-may-kill-you-heres-a-better-strategy-a383823d53f5?source=rss-5e3e8868caf------2"><img src="https://cdn-images-1.medium.com/max/1280/1*xyFCtBwjy7JGeajy5aD1jw.jpeg" width="1280"></a></p><p class="medium-feed-snippet">Don&#x2019;t be a feature factory. Get empowered instead</p><p class="medium-feed-link"><a href="https://medium.com/better-programming/too-much-product-roadmap-may-kill-you-heres-a-better-strategy-a383823d53f5?source=rss-5e3e8868caf------2">Continue reading on Better Programming »</a></p></div>]]></description>
            <link>https://medium.com/better-programming/too-much-product-roadmap-may-kill-you-heres-a-better-strategy-a383823d53f5?source=rss-5e3e8868caf------2</link>
            <guid isPermaLink="false">https://medium.com/p/a383823d53f5</guid>
            <category><![CDATA[software-development]]></category>
            <category><![CDATA[programming]]></category>
            <category><![CDATA[software-engineering]]></category>
            <category><![CDATA[agile]]></category>
            <category><![CDATA[product-management]]></category>
            <dc:creator><![CDATA[Ryan Dawson]]></dc:creator>
            <pubDate>Wed, 23 Jun 2021 16:02:02 GMT</pubDate>
            <atom:updated>2021-06-23T16:02:02.190Z</atom:updated>
        </item>
        <item>
            <title><![CDATA[Your Team Structures Ain’t Working. Let’s Apply Team Topologies]]></title>
            <description><![CDATA[<div class="medium-feed-item"><p class="medium-feed-image"><a href="https://medium.com/better-programming/your-team-structures-aint-working-let-s-apply-team-topologies-470e8d4f7fe5?source=rss-5e3e8868caf------2"><img src="https://cdn-images-1.medium.com/max/1969/1*-AU7zzF_jri1rpxakB-gFA.png" width="1969"></a></p><p class="medium-feed-snippet">Structuring teams is always tricky. Let&#x2019;s do it right</p><p class="medium-feed-link"><a href="https://medium.com/better-programming/your-team-structures-aint-working-let-s-apply-team-topologies-470e8d4f7fe5?source=rss-5e3e8868caf------2">Continue reading on Better Programming »</a></p></div>]]></description>
            <link>https://medium.com/better-programming/your-team-structures-aint-working-let-s-apply-team-topologies-470e8d4f7fe5?source=rss-5e3e8868caf------2</link>
            <guid isPermaLink="false">https://medium.com/p/470e8d4f7fe5</guid>
            <category><![CDATA[software-engineering]]></category>
            <category><![CDATA[software-development]]></category>
            <category><![CDATA[software-architecture]]></category>
            <category><![CDATA[agile]]></category>
            <category><![CDATA[programming]]></category>
            <dc:creator><![CDATA[Ryan Dawson]]></dc:creator>
            <pubDate>Mon, 08 Feb 2021 15:02:36 GMT</pubDate>
            <atom:updated>2021-02-08T15:02:36.175Z</atom:updated>
        </item>
        <item>
            <title><![CDATA[5 Myths Holding Back Agile]]></title>
            <link>https://medium.com/better-programming/5-myths-holding-back-agile-de922ec17045?source=rss-5e3e8868caf------2</link>
            <guid isPermaLink="false">https://medium.com/p/de922ec17045</guid>
            <category><![CDATA[programming]]></category>
            <category><![CDATA[scrum]]></category>
            <category><![CDATA[agile]]></category>
            <category><![CDATA[software-development]]></category>
            <category><![CDATA[startup]]></category>
            <dc:creator><![CDATA[Ryan Dawson]]></dc:creator>
            <pubDate>Fri, 08 Jan 2021 16:18:56 GMT</pubDate>
            <atom:updated>2021-07-06T11:09:29.634Z</atom:updated>
            <content:encoded><![CDATA[<h4>Rethinking productivity and culture</h4><figure><img alt="Elephant display in front of store" src="https://cdn-images-1.medium.com/max/1024/1*DcG6i3J75z3e5wn_XmP4ow.jpeg" /><figcaption>Photo by <a href="https://pixabay.com/users/barroa_artworks-1783849/?utm_source=link-attribution&amp;utm_medium=referral&amp;utm_campaign=image&amp;utm_content=1191266">Barroa_Artworks</a> on <a href="https://pixabay.com/?utm_source=link-attribution&amp;utm_medium=referral&amp;utm_campaign=image&amp;utm_content=1191266">Pixabay</a>.</figcaption></figure><p>According to <a href="https://www.scrumalliance.org/ScrumRedesignDEVSite/media/Forbes-Media/ScrumAlliance_REPORT_FINAL-WEB.pdf">Forbes Insights and ScrumAlliance</a>, only 47% of executives believe they will see sustainable value from their Agile transformations. This is both worrying for the organizations and painful for the developers struggling through these programs.</p><p><a href="https://www.qagile.pl/wp-content/uploads/2020/06/14th-annual-state-of-agile-report.pdf">State of Agile surveys</a> repeatedly tell us that culture change and management participation are crucial to Agile transformations. So what is holding back the cultural change? Let’s look at key myths that prevent us from fully grasping the necessary shift in mindset.</p><p>The problems come out most clearly in Agile at scale, but the myths are wide and deep in the software industry.</p><h3>1. Productivity Is About Developers</h3><p>I’ve worked on some great small Agile projects where I felt super productive. I was able to look back at a month’s worth of coding work and be really pleased with the features I’d put in.</p><p>Then I worked on a big Agile project doing Scrum of Scrums. I would look back at a month of work and wonder where it had gone. I knew other highly capable developers who felt the same.</p><p>It was not just that the project was complex or involved talking to more people. The key reason was that the project had a defensive and bureaucratic culture that made everything difficult to achieve.</p><p>People would refuse meetings unless you escalated to someone senior. Any communication beyond my most immediate colleagues was difficult. People had given up on discussing concerns and had become more worried about avoiding blame.</p><p>I’m not talking about bureaucracy in the old waterfall sense of requiring lots of documents. It was everyone refusing to communicate unless you had an item on their queue. We don’t realise how much goodwill smooths everything out until it disappears.</p><h3>2. Culture Is Fluffy Stuff</h3><p>Google did a big survey of 180+ teams over two years. This was their <a href="https://rework.withgoogle.com/blog/five-keys-to-a-successful-google-team/">key finding</a>:</p><blockquote>“Who is on a team matters less than how the team members interact, structure their work, and view their contributions.”</blockquote><p>The big thing marking out the best teams was simply that they answered “yes” to this question:</p><blockquote>“Can we take risks on this team without feeling insecure or embarrassed?”</blockquote><p>Trust trumped everything.</p><p>Google has also connected this with academic work on how <a href="https://cloud.google.com/solutions/devops/devops-culture-westrum-organizational-culture">bureaucratic cultures are low-trust cultures</a>. They state the top pitfall of organizational culture as:</p><blockquote>“Ignoring the importance of culture altogether, and treating technical work as a technology and process challenge only.”</blockquote><p>Of course, we’d all like to work on projects with good cultures. But many regard it as a nice, optional extra. We don’t fully appreciate how a bureaucratic culture slows everything down.</p><p>Perhaps we choose to overlook the importance of culture because we don’t see how to do anything about it.</p><h3>3. Culture Just Evolves</h3><blockquote>“We create our organizational culture by the actions we take; not the other way around.” <a href="https://hbr.org/2016/10/leaders-can-shape-company-culture-through-their-behaviors">James Whitehurst</a>, President of IBM, via the Harvard Business Review</blockquote><p>There’s some great work in the management space on how cultures are influenced. <a href="https://www.amazon.co.uk/Silos-Politics-Turf-Wars-Competitors/dp/0787976385">Patrick Lencioni</a> has looked at how organizations get mired in politics and become dysfunctional. He argues that much of it comes down to a lack of clear direction.</p><p>When the intended direction for a program is not clear, people form their own interpretations — typically interpretations favourable to their particular sub-function. Then you get factions. And each faction will be more interested in advancing its status than how to best move the program ahead.</p><p>A toxic political culture like that is a big part of how defensiveness can override trust.</p><p>Setting direction is not as simple as just outlining a clear top-level direction. There has to be a structure for how it breaks down into parts and how each team fits in. And it can’t simply be top-down. It has to be consultative in order to ensure it’s workable and that everyone can <a href="https://dzone.com/articles/agile-teamwork">voice concerns and buy in</a>.</p><p>Setting a direction has to come from leadership. Factions will only come together when somebody with authority makes it clear what is expected. Leaders both need to make the direction clear and model behaviours that enable people to debate openly in order to buy in.</p><p>Clear, sensitive leadership is delicate. The most senior leaders have the most influence, but anyone with influence has a big part to play.</p><h3>4. Agile Means No Central Planning</h3><p>It’s tempting to think that central planning is not the Agile way. We think of small self-organizing teams. Just ensure the backlog is prioritised, then the team will work out what to do. This seems to be what the <a href="https://agilemanifesto.org/principles.html">Agile Manifesto envisaged</a>.</p><p>On small projects, it is often true that there’s not much need for centralized planning. If there’s just one team and everyone can speak directly to each other, then things can be worked out as you go along.</p><p>When you’ve got multiple teams working together, you need some central planning. Work has to be divided between the different teams. It has to be clear who is responsible for what. It’s not practical for 50 people to organically work that out between themselves.</p><p>Planning need not be done in a top-down fashion. There can be initial proposals, consultations, and refinements.</p><p>Having some central planning does not mean a return to command-and-control leadership or the end of developer empowerment.</p><p>People feel empowered so long as there’s a vision they can get behind and they <a href="https://hackernoon.com/agile-is-not-enough-empowered-on-innovation-he9m35s1">have a voice in the direction</a>.</p><h3>5. Agile Means No Technical Leadership</h3><p>The Architect role has come under some <a href="https://www.fromhereon.com/blog-list/agile-architecture-an-oxymoron">criticism in recent years</a>. A stereotype had emerged of an “Ivory Tower Architect” who draws diagrams and has little interest in the realities of implementation. That top-down approach does not sit well with Agile.</p><p>The current <a href="https://www.qagile.pl/wp-content/uploads/2020/06/14th-annual-state-of-agile-report.pdf">most popular choice</a> for a scaled Agile framework (SAFe) envisages <a href="https://www.scaledagileframework.com/agile-architecture/">Architects</a> as facilitators as well as leaders. The role does involve setting a direction but also adapting the direction to what’s happening on the ground and enabling teams to realise the direction.</p><p>We need to recognise that some technical elements are cross-team, that there has to be some planning of how <a href="https://www.scaledagileframework.com/architectural-runway/">near-term work</a> will be spread across teams — and that’s in part a design question.</p><p>Even the question of team structures is in part a technical question. If we want teams to be able to work mostly independently, we have to design for that. We have to think about how parts of the system will interact and how the APIs will be built.</p><p>There are also other team-spanning technical considerations (release conventions, documentation and code conventions, and integration testing).</p><p>Technical leadership has a key role to play. Vision and direction can’t only be the responsibility of business leadership.</p><h3>Changing Our Mindset</h3><p>Most senior managers <a href="https://assets.kpmg/content/dam/kpmg/be/pdf/2019/11/agile-transformation.pdf">believe</a> Agile is a project management methodology with no implications for them. We need to realise that big Agile requires shaping a healthy culture. A healthy culture requires a clear direction. That direction needs to be set with collaborative and flexible leadership.</p><p>We think of software projects too much in terms of coding and Gantt charts. We all need to think more in terms of vision, collaboration, and culture.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=de922ec17045" width="1" height="1" alt=""><hr><p><a href="https://medium.com/better-programming/5-myths-holding-back-agile-de922ec17045">5 Myths Holding Back Agile</a> was originally published in <a href="https://betterprogramming.pub/">Better Programming</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[How To Scale Agile: It’s Delicate but Not Alchemy]]></title>
            <link>https://itnext.io/how-to-scale-agile-its-delicate-but-not-alchemy-6ac24963ffbe?source=rss-5e3e8868caf------2</link>
            <guid isPermaLink="false">https://medium.com/p/6ac24963ffbe</guid>
            <category><![CDATA[project-management]]></category>
            <category><![CDATA[scaled-agile-framework]]></category>
            <category><![CDATA[agile-methodology]]></category>
            <category><![CDATA[agile]]></category>
            <category><![CDATA[scrum]]></category>
            <dc:creator><![CDATA[Ryan Dawson]]></dc:creator>
            <pubDate>Mon, 04 Jan 2021 09:45:35 GMT</pubDate>
            <atom:updated>2021-07-06T11:06:14.515Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*-uS_xyYR7iF-_37qwPPcEQ.png" /><figcaption>Image by author using designs from OpenClipart-Vectors and StarGladeVintage on Pixabay.</figcaption></figure><p>Agile has conquered the world. Iterative feedback for software development makes sense.</p><p>But talk about Agile on big projects and you get war stories. “We tried to be Agile but…” You hear that Agile did not produce a harmonious project.</p><p>It can seem as though getting Agile to work at scale requires a kind of alchemy. A secret formula that hasn’t been made accessible to mere mortals who haven’t done all of the many framework certifications.</p><p>Scaling Agile is not alchemy. But it is delicate. Let’s demystify this.</p><p>First we need to remind ourselves of the core of Agile. Then we’ll see why it’s difficult to scale and understand how the obstacles can be overcome. We’ll get a good grasp of the core concepts before looking at approaches for scaling Agile.</p><h3>Why We Love Agile</h3><p>In the beginning there was bureaucracy. There were people with problems. And there were problem solvers. But the two groups could only communicate via endless paperwork.</p><p>Sometimes it was confusing paperwork. But you couldn’t change it. Once you placed your order that was it.</p><p>Then there was Agile. The Agile Manifesto was handed down to us from a Ski Lodge in Utah in 2001.</p><blockquote><strong>Individuals and interactions</strong> over processes and tools</blockquote><blockquote><strong>Working software </strong>over comprehensive documentation</blockquote><blockquote><strong>Customer collaboration</strong> over contract negotiation</blockquote><blockquote><strong>Responding to change</strong> over following a plan</blockquote><p>The magic of Agile is that people with problems get to talk to people who can solve problems.</p><blockquote>Business people and developers must work together daily throughout the project.</blockquote><p>And they get to fix any confusions as they go along.</p><blockquote>Welcome changing requirements, even late in development.</blockquote><p>Turns out this works. If you keep making small adjustments you better get in sync with your users and their evolving needs. You can also help your users get clearer about those needs.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*y4NKR4bg4jZm98XoBDYheQ.png" /><figcaption>Feedback with Agile. Image by author.</figcaption></figure><p>Things got better. Projects started <a href="https://www.infoq.com/articles/standish-chaos-2015/">succeeding a bit more</a>. People got excited.</p><h3>The Agile Hangover</h3><p>People maybe got a bit too excited.</p><p>Most big corporations adopted Agile. Consultancies started teaching Agile frameworks.</p><p>At first everyone wanted to work on an Agile project. But when they did then they found paperwork. And developers being told to follow the paperwork.</p><p>This <a href="https://www.slideshare.net/jaxlondon2012/curing-agile-hangover-sandro-mancuso-and-mashooq-badar">beast</a> was given multiple names. <a href="https://www.thoughtworks.com/insights/blog/good-and-bad-wagile">Wagile</a>. <a href="https://ronjeffries.com/articles/016-09ff/defense/">Dark Scrum</a>. <a href="https://martinfowler.com/articles/agile-aus-2018.html">Fake Agile</a>.</p><p>It’s not pretty. Rigid relationships. Poor collaboration. Conflict and dysfunction.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*2f2V46DD_sAA_zbCUvocwA.png" /><figcaption>Stick men by<a href="https://thenounproject.com/leremy/"> Khoon Lay Ghan</a>. Arrangement by author.</figcaption></figure><p>How do these big Agile projects get so broken? How do we fix this?</p><p>We need to appreciate why scaling Agile is delicate.</p><h3>Why Scaling is Hard</h3><p>On smaller projects everyone can talk to everyone. People with problems really can talk to the problem solvers.</p><p>Everyone can do a bit of everything. Developers know the whole codebase. They do all their own devops. They speak directly to the users. Everyone can meet in the same room.</p><p>Now imagine the system gets bigger. Now people don’t fit in the same room. The developers can’t know the whole codebase anymore.</p><p>As the system grows then everyone does something more specialised. Everyone has to narrow their focus.</p><p>So now we’ll have teams devoted to different areas. Maybe to different sets of pages within a website. It’s more efficient.</p><p>Structuring into teams changes the dynamics of collaboration.</p><p>Once things get specialised then talking to people who do very different stuff takes effort. You don’t understand what they do. Their aims don’t immediately make sense to you.</p><p>We can try to structure the project to give teams independence. But some things inevitably span teams.</p><blockquote>“ <em>Specialization allows us to handle ever-growing complexity, but the benefits of specialization can only be fully realized if the silos that it creates can be connected effectively.</em>” Mik Kersten, <em>Project to Product</em></blockquote><p>Teams need to work together sometimes. And there needs to be planning of how the system fits together. And prioritisation.</p><p>You need management. Central planning and prioritisation.</p><p>Not quite what the Agile Manifesto seemed to be getting at:</p><blockquote>Build projects around motivated individuals. Give them the environment and support they need, and trust them to get the job done… The best architectures, requirements, and designs emerge from self-organizing teams.</blockquote><p>Scaling requires formal structure and process. The Agile manifesto favours empowering individuals to collaborate freely. There is real tension.</p><h3>Why Does Fake Agile Happen?</h3><p>Fake Agile is not simply about process. Having process and structure in itself is not a bad thing. People are happy to follow processes that work for everyone.</p><p>The problem is when people become inflexible.</p><p>Imagine that everyone takes a narrow interpretation of their role. Imagine a story requires building a new integration to another system. You ask the product owner about this and they respond it’s a technical question and not in their remit. You find an architect and they say they don’t deal in that area. Who do you even talk to for moving forward?</p><p>Now imagine if the product owner had said “I know so-and-so from that team, I’ll help you to set up some meetings.” This makes a huge difference to a developer who may not have the authority to get time from another team.</p><p>Imagine that everyone on the project interprets their role inflexibly. Infrastructure becomes hard to get. Defects get assigned back and forth. Everything that falls between roles or <a href="https://dzone.com/articles/agile-teamwork">requires cross-team collaboration</a> gets difficult.</p><blockquote>“Bureaucracy is the art of making the possible impossible.” Credited to Javier Pascual Salcedo</blockquote><p>We can take this inflexible behaviour as the core symptom of Fake Agile. It amounts to a <a href="https://cloud.google.com/solutions/devops/devops-culture-westrum-organizational-culture">bureaucratic culture</a>.</p><p>There might be other negative characteristics of Fake Agile but it helps to consider bureaucratic culture as key. This is because a culture of collaboration is so important. This was the <a href="https://rework.withgoogle.com/blog/five-keys-to-a-successful-google-team/">key finding</a> of a big survey Google performed of 180+ teams over two years:</p><blockquote>“Who is on a team matters less than how the team members interact, structure their work, and view their contributions.” Julia Rozovsky, Google</blockquote><p>So poor collaboration both dampens morale and hurts performance.</p><p>Poor culture also comes up around the top of <a href="https://www.qagile.pl/wp-content/uploads/2020/06/14th-annual-state-of-agile-report.pdf">surveys</a> on blockers for Agile transformations and for <a href="https://services.google.com/fh/files/misc/state-of-devops-2019.pdf">effective software delivery in general</a>. But why does it happen?</p><p>When inflexible behaviour goes wide and deep in a project then it’s likely a <a href="https://www.jstor.org/stable/3791022?seq=1">defence mechanism</a>. People become inflexible as a way of protecting themselves from being overwhelmed or exposed to blame.</p><p>So what could lead to such a defensive culture emerging? And how can we encourage a healthy culture?</p><p>It is often suggested that central planning allows command-and-control management in by the back door. Then people become inflexible because they’ve been instructed what to do. This is likely the most common (though not only) path to a defensive culture. It is avoidable.</p><p>There is a style of leadership that can facilitate empowerment and promote a healthy culture.</p><h3>Empowerment Subtle But Achievable</h3><p>Empowerment is a key motivator and a theme of the Agile Manifesto. It tends to happen naturally on small projects where everyone automatically has a significant influence. It is not so automatic on big projects. But it is possible to run big projects where developers are sufficiently empowered.</p><p>Structure does have to sacrifice some equality of influence. For central planning there has to be some centralised leadership. But this does not have to mean a big sacrifice of empowerment.</p><p>It is possible for leaders to speak for others. Leadership can be consultative. Servant leadership.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*cSH5fIqiBFhZw26wuz915g.png" /><figcaption>Image by author. <a href="https://thenounproject.com/search/?q=commander&amp;i=1587086">Commander</a>, <a href="https://thenounproject.com/search/?q=person&amp;i=23359">person</a> and <a href="https://thenounproject.com/search/?q=weightlifter&amp;i=2838688">weightlifter</a> images from thenounproject. Inspiration from <a href="https://www.5dvision.com/post/the-scrum-master-is-a-servant-leader/">5dvision</a>.</figcaption></figure><p>It is possible for large groups to be united by a shared vision. People feel empowered so long as they can contribute to a shared vision.</p><h3>Avoiding Fake Agile</h3><p>A defensive culture happens when everyone is overwhelmed and fearing failure.</p><p>This means anything that can derail the project risks leading to a defensive culture if it isn’t managed well. And lots of things can go wrong on a software project.</p><p>There are common patterns to problems that derail software projects. And it is possible to weather challenges successfully. Consultative leadership makes for a healthy culture that is better able to manage risks. It also makes serious issues less likely. Let’s see why.</p><h3>The Delicate Challenges of Scale</h3><p>Let’s think about some of the <a href="https://spectrum.ieee.org/computing/software/why-software-fails">key problems</a> we see on big projects:</p><ul><li>Work gets underestimated and overruns.</li><li>System architecture becomes complex and difficult to extend.</li><li>Teams get stuck waiting on other teams.</li><li>Teams have to rework pieces due to differing interpretations of goals.</li><li>Low morale and high staff turnover.</li></ul><p>Now consider these remedies:</p><ul><li>Collaborative planning with consultations and spikes before committing to budgets.</li><li>Flexible architecture that is sensitive to problems emerging on the ground.</li><li>Logical team structures that fit to and evolve with the architecture.</li><li>Clear project vision and architectural direction.</li><li>Inspiring project goals and clarity on how everyone contributes.</li></ul><p>Much of this comes down to having a <a href="https://www.infoq.com/articles/silos-politics-software-products/">clear vision</a> of what the product aims to be and flexible, consultative planning for how we aim to achieve it. Let’s go through these key areas in more detail.</p><h4>Vision</h4><p>The biggest thing we can do to set a project on a good path is to define a clear vision. The high level vision does not need to be detailed but it needs to be ready to be fleshed out into detail. If it contains conflicting priorities or muddles then these need to be worked out.</p><p>Working the vision out into details enables internal disputes to be resolved. Many projects fall into unresolved disagreements because teams <a href="https://www.infoq.com/articles/silos-politics-software-products/">interpret the aims differently</a>. Clarity of purpose is the only solution to this.</p><h4>Planning and Design</h4><p>The design provides a structure for which teams will work on what. It can evolve as the project evolves. But foresight and balance is important. Big change to the design will likely require changes to team structures.</p><p>The key problems of project realisation are problems of vision, design, prioritisation and planning. We need to be making clear decisions about these things to ensure direction is clear. We also need to be adapting these decisions in the light of new feedback. This is a sensitive balancing act.</p><p>Doing budgeting and estimation well is also delicate. How difficult it is stems from how clear the vision is and how well we can see the path ahead.</p><h4>Sponsorship</h4><p>It’s important that the vision and plan be sufficiently backed within the organization. Otherwise budget requests might not be forthcoming. The backing also needs to be visible. Otherwise other projects or departments might not make necessary resources available.</p><h3>Leadership and Scaling Agile</h3><p>We tend to underestimate the importance of software delivery leadership activities. We see problems emerging in the development process and think the problem starts there. That is like trying to fix a washing machine when the power has gone out.</p><blockquote>“Eighty-five percent of the reasons for failure are deficiencies in the systems and process rather than the employee. The role of management is to change the process rather than badgering individuals to do better.” W. Edwards Deming</blockquote><p>We’ve seen we have these key elements in keeping a project healthy and how to scale agile:</p><ul><li>Vision</li><li>Sponsorship</li><li>Prioritisation</li><li>Design</li><li>Planning and budgeting</li></ul><p>How can we achieve these in a big Agile context? And achieve them with the right balance and tone that enables everyone to see how to contribute their best?</p><h3>Scaling Agile Well</h3><p>There is no magic formula. There are a set of different elements that all have to come together.</p><p>There are frameworks for managing Agile at scale. SAFe, LeSS, Nexus and others have guides that spell out structures, steps and practices. These are important and valuable tools.</p><p>We should be wary of treating a framework like a magic formula. How we apply the guidance to our particular case is crucial.</p><p>Whether or not you choose to use a scaling framework, they can provide a valuable reference to mine for ideas and what to emphasise. <em>SAFe 5.0 Distilled </em>starts with the need for clear leadership and then works into more detail from there.</p><blockquote>“Leaders set the example through coaching, empowering, and engaging individuals and teams to reach their highest potential.” SAFe 5.0 Distilled</blockquote><p>Much of the <em>SAFe </em>literature is structured as a flexible knowledge base. It invites you to think of your case. But that’s not to say you won’t get as much value from other frameworks. It’s your application of the framework that matters. That said, the frameworks themselves are not neutral and it’s worth considering <a href="https://hackernoon.com/agile-is-not-enough-empowered-on-innovation-he9m35s1">what criticisms they’ve faced too (which I’ve written about elsewhere)</a>.</p><h3>Scaling Agile Is Not Alchemy</h3><p>It can look as though Agile frameworks are trying to codify the uncodifiable. As though they were trying to create a mechanical formula for the true essence of leadership.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*aXKnlSQ2ZOW4UPOXlg_zEA.png" /><figcaption>Image by author. Based on books from <a href="https://www.mycutegraphics.com/graphics/book/black-white-bunch-of-books.html">mycutegraphics</a> and hand symbols from fluf on <a href="https://pixabay.com/illustrations/hand-symbol-magic-sign-triangle-5278129/">pixabay</a>.</figcaption></figure><p>The air of mystery that can attach to frameworks owes to us. Of course frameworks can’t tell us precisely what to do, no matter how much we want them to. They are just guides.</p><p>There’s a feedback loop going on between practice and frameworks. Frameworks will carry on improving and adapting in response to how we use them. They will carry on changing as our needs change.</p><p>We don’t have a system for doing software development with the predictability of Newton’s Laws of Motion. We don’t have precise formulas. We do have a body of knowledge. It takes skill, experience and judgement to apply that knowledge.</p><p>Software delivery lives on the boundary between business and engineering. Software initiatives are very much business initiatives. And they inherit all of the messiness and risk of business initiatives.</p><p>If there were a general formula to turn business initiatives into great software then that formula really would be alchemy.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=6ac24963ffbe" width="1" height="1" alt=""><hr><p><a href="https://itnext.io/how-to-scale-agile-its-delicate-but-not-alchemy-6ac24963ffbe">How To Scale Agile: It’s Delicate but Not Alchemy</a> was originally published in <a href="https://itnext.io/">ITNEXT</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Applying the MLOps Lifecycle]]></title>
            <link>https://medium.com/data-science/applying-the-mlops-lifecycle-3b60033b7cbf?source=rss-5e3e8868caf------2</link>
            <guid isPermaLink="false">https://medium.com/p/3b60033b7cbf</guid>
            <category><![CDATA[mlops]]></category>
            <category><![CDATA[software-life-cycle]]></category>
            <category><![CDATA[devops]]></category>
            <category><![CDATA[machine-learning]]></category>
            <dc:creator><![CDATA[Ryan Dawson]]></dc:creator>
            <pubDate>Tue, 01 Dec 2020 14:41:24 GMT</pubDate>
            <atom:updated>2020-12-01T14:41:24.657Z</atom:updated>
            <content:encoded><![CDATA[<h4>Understand MLOps needs and how they arise through the MLOps Lifecycle. Apply this to better scope and tackle MLOps projects.</h4><p>MLOps can be difficult for teams to get a grasp of. It is a new field and most teams tasked with MLOps projects are currently coming at it from a different background.</p><p>It is tempting to copy an approach from another project. But the needs of MLOps projects can vary greatly. What is needed is to understand the specific needs of each MLOps project. This requires understanding the types of MLOps needs and how they arise.</p><h3>A Simplified MLOps Project</h3><p>At the simpler end of the spectrum, an MLOps setup can closely resemble a mainstream DevOps lifecycle.</p><p>Traditional DevOps empowers the build-deploy-monitor lifecycle for software applications. An executable is built from source code, deployed to an environment and then monitored for any issues.</p><p>Training of a model can be thought of like building an executable. We’ll see later that it can also get more complex. But if the training process is simple then it can be handled as a traditional continuous integration build.</p><p>There are different <a href="https://towardsdatascience.com/navigating-ml-deployment-34e35a18d514">deployment patterns for MLOps </a>but let’s assume we have a real-time serving use case. Then we want to take the trained model and use it to make predictions via HTTP.</p><p>Trained models may be specific to the framework that they were trained on. A common approach to deployment is to use an off-the-shelf serving solution that can work for the framework. One open source serving solution that supports a range of frameworks is Seldon Core, a project I work on.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*CCHSVwpcDDuDqby8uIoJaA.jpeg" /><figcaption>Image by Seldon. Shared under<a href="https://creativecommons.org/licenses/by-sa/4.0/"> CC BY-SA 4.0</a></figcaption></figure><p>With mainstream DevOps the monitoring phase centres on looking for issues such as error codes or poor response times. For MLOps we also want to monitor for prediction quality.</p><p>A model is trained on a certain chosen dataset. That data may not match the live data perfectly. To ensure the model keeps performing well, we can monitor for key business or model metrics.</p><p>The above flow can be workable <em>if the data is well-structured and predictable and there aren’t any additional governance needs</em>. Otherwise we get into more complex MLOps needs.</p><h3>An Advanced MLOps Lifecycle</h3><p>Machine Learning is <a href="https://hackernoon.com/why-is-devops-for-machine-learning-so-different-384z32f1">exploratory and data-driven</a>. In essence it is about extracting patterns from data and reapplying those patterns to new data. If all goes well you get good predictions. The exploratory part is finding the right patterns for the data you intend to make predictions on.</p><p>When the data is not well-structured or predictable then the MLOps lifecycle can look very different to mainstream DevOps. Then we see a range of approaches come in that are specific to MLOps.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*lw1xmotLneAd6kzAf8da5A.jpeg" /><figcaption>Image by Seldon. Shared under<a href="https://creativecommons.org/licenses/by-sa/4.0/"> CC BY-SA 4.0</a></figcaption></figure><p>Let’s go through each of the phases in turn and the approaches that come into play. We’ll see how each MLOps need arises.</p><h4>Training</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*z1CCsWhtJoeuSD7v5KJ_yg.jpeg" /><figcaption>Image by Seldon. Shared under<a href="https://creativecommons.org/licenses/by-sa/4.0/"> CC BY-SA 4.0</a></figcaption></figure><p>Training is about finding the best patterns to extract from training data. We encapsulate these patterns in models. Training runs have parameters that can be tweaked to result in different models. Encapsulating the best patterns in models is exploratory.</p><p>To explore parameters for the best models it makes sense to run multiple jobs in parallel. This is done in a hosted training environment running a specialist training platform. The best model for deployment then needs to be selected and packaged.</p><p>Data is a big part of why these platforms are run hosted rather than on a Data Scientist’s laptop. The volumes can be large. And data rarely starts ready for training models. That means a lot of preparation needs to be performed on it. This can take a lot of time and hardware resources. For governance and reproducibility reasons the preparation operations might also all need to be tracked.</p><h4>Deployment</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*XZmmxFpx16yKdXpEYjK2lA.jpeg" /><figcaption>Image by Seldon. Shared under<a href="https://creativecommons.org/licenses/by-sa/4.0/"> CC BY-SA 4.0</a></figcaption></figure><p>When we’ve selected a new model then we need to work out how to get it running. That means determining whether it’s really better than the version already running. It may have performed better on the training data but the live data could be different.</p><p>MLOps rollout strategies tend to be cautious. Traffic may be split between the new model and the old model and monitored for a while (using an A/B test or canary). Or the traffic may be duplicated so that the new model can receive requests but just have its responses tracked rather than used (a shadow deployment). Then the new model is only promoted when it has been shown to perform well.</p><p>We need to know a model is performing safely in order to promote it. This means deployment needs support from monitoring. We can also find that deployment may need to support a feedback mechanism for optimum monitoring. Sometimes a model makes predictions that turn out to be right or wrong e.g. whether a customer chose a recommendation. To make use of that we’d need a feedback mechanism.</p><p>An advanced case of splitting traffic for optimization is use of multi-armed bandits. With a bandit the traffic is split in a continuously-adjusting way. The model performing best gets most of the traffic and the others continue to get a small portion of traffic. This is handled by an algorithmic router in an inference graph. If the data changes later then a lesser-performing model may shift to becoming the dominant model.</p><p>Deployment can be intimately tied to monitoring. Deployment tools such Seldon therefore not only support deployment-phase features but also have integrations for the MLOps needs of the monitoring phase.</p><h4>Monitoring</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*yYPVX0Xa31rPdBcshMvGPQ.jpeg" /><figcaption>Image by Seldon. Shared under<a href="https://creativecommons.org/licenses/by-sa/4.0/"> CC BY-SA 4.0</a></figcaption></figure><p>Monitoring for model accuracy is possible only if you have feedback. This is a good example of a monitoring feature needing a deployment-stage feature. In some cases live accuracy may be the key metric and in other cases a custom business metric may be more important. But these are only part of the monitoring picture.</p><p>The other side of ML monitoring is seeing <em>why</em> a model performs well or badly. That requires an insight into the data.</p><p>A top reason why model performance may drop is a change in the live data. If the data distribution shifts away from the training data then performance can drop. This is called<em> data drift</em> or <em>concept drift</em>.</p><p>Even if the overall distribution remains in line with the training data, some predictions may still be dramatically wrong. This could happen if some individual data points are outside of the distribution. These outliers can be damaging in cases where predictions need to be reliable across the board.</p><p>Fully understanding why a model made a certain prediction can require looking at how a model makes predictions and not just the input data. Explanation techniques can reveal the key patterns that a model is relying on for its predictions and show us which patterns applied to a particular case. Achieving this level of insight is a data science challenge in itself.</p><p>There are different approaches to achieving advanced monitoring. At Seldon we make heavy use of asynchronous logging of requests. Logged requests can then be fed into detector components to monitor for drift or outliers. Requests can also be stored for later analysis e.g. explanations.</p><h3>Assessing MLOps Needs</h3><p>It’s difficult to offer hard rules about what MLOps projects need. Parts of the lifecycle might be considered a nice-to-have for some projects (e.g. request logging). But for other projects that same element might be essential (e.g. compliance-focused projects). This suggests an approach that centres on asking the right questions.</p><h4>MLOps Scoping Questions</h4><p>Here is a sample set of questions to use to assess an MLOps project:</p><ul><li>Do we have the data and is it clean?</li><li>How and how often will we get new iterations of the data and will it be clean?</li><li>Do we need a training platform and does it need to be linked to CI?</li><li>What ML framework is being used and what’s the easiest way to serve models for it?</li><li>Does it look like a real-time serving use case or an <a href="https://towardsdatascience.com/navigating-ml-deployment-34e35a18d514">offline/batch use case</a>?</li><li>Will the format of prediction requests be similar to the data format the model is trained on (check the raw form for reference)?</li><li>Do we think the data could vary/shift?</li><li>Do we expect to do online learning or to train offline?</li><li>Do we expect to need to retrain the model due to data change and if so how quickly?</li><li>Would outliers cause a problem?</li><li>Do we need explainability?</li></ul><h4>Platforms vs Projects</h4><p>Many organizations are not only looking at one-off MLOps projects. Many are looking to build a platform capability to cover a range of projects. Given how much MLOps projects can vary, this calls for a different set of questions.</p><ul><li>What are the range of use cases?</li><li>Who will be responsible for deployment and what skills do they have?</li><li>What tools are the team familiar with and what would they be comfortable using?</li><li>What audit, compliance and reporting requirements do we have? Are there any high-compliance/high-risk use cases?</li><li>How many teams and models will it have to scale to?</li><li>Is there a product or set of products that could be a fit?</li><li>What’s the budget, resources and skills available internally?</li></ul><h4>Advanced Platform Supporting Tools</h4><p>Some platform use cases employ specialist MLOps tools that are not currently part of the mainstream for single project use-cases.</p><h4>Model Registries</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*cPYK1kLtg7n9bpH4f2QLOQ.jpeg" /><figcaption>Image by Seldon. Shared under<a href="https://creativecommons.org/licenses/by-sa/4.0/"> CC BY-SA 4.0</a></figcaption></figure><p>If you’re running lots of deployments of models then it becomes important to record which versions were deployed and when. This is needed to be able to go back to specific versions. Model registries help with this problem by providing ways to store and version models.</p><h4>Feature Stores</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*iVguIOCb9XCEhg4a_11_6A.jpeg" /><figcaption>Image by Seldon. Shared under<a href="https://creativecommons.org/licenses/by-sa/4.0/"> CC BY-SA 4.0</a></figcaption></figure><p>In larger organizations multiple teams may be using the same raw data in different ways. <a href="https://medium.com/data-for-ai/the-importance-of-having-a-feature-store-c91419b08dcf">Feature stores</a> allow teams to drill into the raw data and see what preparations other teams have already performed on it. This can help reduce the need for teams to reinvent the wheel on data preparation.</p><h4>Metadata Stores</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*OB1pfyLkv98WYL9TgG1gFg.jpeg" /><figcaption>Image by Seldon. Shared under<a href="https://creativecommons.org/licenses/by-sa/4.0/"> CC BY-SA 4.0</a></figcaption></figure><p>For platform-level tools to offer insights about data going through models, it is important to know what type of data is in play. Text, image and tabular data are all very different for instance. Tracking this is the role of ML metadata. It’s early days for this space and Seldon is<a href="https://blog.kubeflow.org/release/official/2020/11/18/kubeflow-1.2-blog-post.html"> contributing the Kubeflow Model Management SIG</a> to help advance metadata tooling.</p><h3>Approaching MLOps Projects with the MLOps Lifecycle</h3><p>There’s much to approaching MLOps projects that we’ve not covered here. We’ve not talked about estimation, scheduling or team composition. We’ve not even got very far into the <a href="https://github.com/EthicalML/awesome-production-machine-learning">tool landscape</a>. Hopefully what we have achieved is to find our feet.</p><p>We’ve learnt to understand the MLOps lifecycle in terms of a set of needs. We’ve seen that ML is about taking patterns from data and reapplying those patterns. Data can be unpredictable and that can mean we have to be cautious about rollouts and that we have to monitor at the level of data and not just errors.</p><p>Once we understand the needs behind MLOps then we’re better placed to see how to leverage MLOps tools. We’re also better placed to ask the right questions in order to scope and approach MLOps projects.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=3b60033b7cbf" width="1" height="1" alt=""><hr><p><a href="https://medium.com/data-science/applying-the-mlops-lifecycle-3b60033b7cbf">Applying the MLOps Lifecycle</a> was originally published in <a href="https://medium.com/data-science">TDS Archive</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Navigating ML Deployment]]></title>
            <link>https://medium.com/data-science/navigating-ml-deployment-34e35a18d514?source=rss-5e3e8868caf------2</link>
            <guid isPermaLink="false">https://medium.com/p/34e35a18d514</guid>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[deployment]]></category>
            <category><![CDATA[model-serving]]></category>
            <category><![CDATA[editors-pick]]></category>
            <category><![CDATA[deployment-pipelines]]></category>
            <dc:creator><![CDATA[Ryan Dawson]]></dc:creator>
            <pubDate>Sun, 15 Nov 2020 14:31:38 GMT</pubDate>
            <atom:updated>2020-11-15T15:16:29.026Z</atom:updated>
            <content:encoded><![CDATA[<p>We often think of ‘deployment’ as packaging software into an artifact and moving it to an environment to run on. For Machine Learning it can be better to think of deployment as “the action of bringing resources into effective action” (one of Oxford’s definitions of ‘deployment’).</p><p>There are a range of patterns for using ML models to make business decisions. Deploying machine learning models can mean different things, depending on the context. Understanding the key prediction-making patterns can help to decide on which tools apply to your use case.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1015/0*rC56xhHJvhbXlXAO" /><figcaption>Image based on Clip Art by Vector Toons, <a href="https://creativecommons.org/licenses/by-sa/4.0">CC BY-SA 4.0</a>, via Wikimedia Commons</figcaption></figure><p>Different situations lend themselves to different deployment patterns. But sometimes the same model might be deployed in more than one way. The use cases for the patterns are distinguished more by how a model is to be consumed than the type or meaning of the predictions.</p><p>Many projects currently assume they have to build their own ML deployment infrastructure. Armed with a knowledge of the deployment patterns, we’ll see that there are tools available to help.</p><h3>Offline Prediction</h3><p>Offline prediction is predictions made on an ad-hoc or one-off basis. This could be done on a local machine, depending on the scale of predictions and compute resources required.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*yqcyWySAJaNf9skA" /><figcaption>Image by author. Business people clip art from <a href="https://clipartdoc.com/category/business_clipart/">clipartdoc.com</a>, used in accordance with their <a href="https://clipartdoc.com/terms/">terms</a>.</figcaption></figure><p>Offline predictions might be generated directly from python code (e.g. a jupyter notebook) and given to the intended audience (e.g. emailed as a CSV). This applies most to where the predictions are for a single event and a new model would be required for a new event. Example cases could be predicting the outcome of an election or a sporting event.</p><h3>Batch Prediction</h3><p>Batch prediction is when a set of predictions need to be made from a file or store, typically on a regular time cycle (e.g. weekly). Imagine a model being used to predict income for the next quarter. Or to predict how much water a crop will need next week.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*u3lWgsQPw2soYzD9" /><figcaption>Image by author. Business people clip art from <a href="https://clipartdoc.com/category/business_clipart/">clipartdoc.com</a>, used in accordance with their <a href="https://clipartdoc.com/terms/">terms</a>.</figcaption></figure><p>With batch use-cases there’s a regular predict cycle and it might be that the same training data or same model is used each cycle. Or it might be that new training data becomes available for each cycle and a new model needs to be produced each time. Either way, the use case lends itself to trigger-activated processing jobs that predict by feeding their way through the store of input data.</p><h3>Real-time Serving</h3><p>Real-time serving is making predictions on-demand, usually via an HTTP call. These online use-cases include (but are not limited to) e-commerce applications for recommending content such as products or advertising.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*5kvuj4nDNeQfZXuS" /><figcaption>Image by author. Web server icon public domain from <a href="https://freesvg.org/world-wide-web-server-icon-vector-drawing">freesvg.org</a></figcaption></figure><p>Commonly a model is trained on a training environment and packaged for real-time serving. The packaged model goes through a promotion process to the live serving environment. The train-package-promote process starts again if new training data becomes available.</p><p>There are other less common serving use cases such as online learning. With online learning training happens continuously and every new prediction request is also a new piece of training data. Most serving tools centre on separate training and serving, not online learning.</p><p>Real-time serving can be intimately tied to roll-out and monitoring. The served model becomes a microservice and new versions of it need to be rolled out safely and their performance monitored.</p><h3>Streaming</h3><p>Streaming use cases are marked out by throughput of predictions required, which can be high and variable. Streaming can be similar to batch or real-time but with the addition of a messaging system to queue up predictions to be handled at the rate of processing rather than the rate of arrival.</p><p>What distinguishes streaming from other real-time cases is the use of a queue to smooth out the processing of a variable workload of predictions.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*rntLsY803ubVJ5Uu" /><figcaption>Image by Author. Message queue icon from <a href="https://www.pngwing.com/en/free-png-pshyn">pngwing</a> -non-commercial use, DMCA.</figcaption></figure><p>The live predictions when generated might go to a file, to a database or to an HTTP service.</p><p>An example of streaming might be online fraud detection, where perhaps transactions are allowed to go through and are then asynchronously queued up for post-transaction verification by the fraud detection model.</p><h3>Leveraging ML Deployment Tools</h3><p>Machine Learning Deployment becomes easier to navigate if we break it down into categories. Here we’ve suggested the spaces of <strong>Offline Prediction, Batch Prediction, Real-time Serving</strong> and <strong>Streaming</strong>. There are open source tools in each of these spaces. Established tools are designed from the beginning to handle scaling and operating prediction systems in production.</p><p>A great place to find out about tools is the <a href="https://landscape.lfai.foundation/">Linux Foundation for AI’s (LFAI) Landscape Diagram</a>. Here’s part of the LFAI diagram:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*PB4foZoEtnDKbsKtksFoUw.png" /><figcaption>Seldon, Cortex and Ray are used for Serving. Kubeflow and mlflow are end-to-end platforms including serving capabilities. Spark is popular for batch processing. Snippet of <a href="https://landscape.lfai.foundation/">LFAI Landscape Diagram</a>, captured 2020–11–14 23:28:19Z. Used under <a href="https://github.com/lfai/lfai-landscape/blob/master/LICENSE">Apache License 2.0</a>.</figcaption></figure><p>Featured in the middle of the diagram is Seldon, where I work. Seldon Core is an open source real-time serving solution that has integrations for <a href="https://docs.seldon.io/projects/seldon-core/en/latest/examples/argo_workflows_batch.html">batch</a> or <a href="https://docs.seldon.io/projects/seldon-core/en/latest/examples/knative_eventing_streaming.html">streaming</a> use cases. Seldon has integrations for <a href="https://docs.seldon.io/projects/seldon-core/en/latest/examples/istio_canary.html">progressive rollouts</a>, <a href="https://docs.seldon.io/projects/seldon-core/en/latest/analytics/analytics.html">metrics</a>, <a href="https://docs.seldon.io/projects/seldon-core/en/latest/examples/payload_logging.html">audit logging</a>, <a href="https://docs.seldon.io/projects/seldon-core/en/latest/examples/explainer_examples.html?highlight=explanations">explanations</a> and <a href="https://docs.seldon.io/projects/seldon-core/en/latest/examples/drift_cifar10.html">data drift detection</a>. Seldon supports a range of different frameworks and allows different types of models to be seamlessly swapped or mixed (for example, with a <a href="https://docs.seldon.io/projects/seldon-core/en/latest/analytics/routers.html">multi-armed bandit router</a>).</p><p>Our philosophy at Seldon is to focus on key use cases and work with the open source community, as well as enterprise customers, to form flexible and scalable solutions. This has led to certain technology choices. Kubernetes for target platform. Knative, with its pluggable options of message broker (including a <a href="https://knative.dev/docs/eventing/broker/kafka-broker/">Kafka option</a>), for streaming architecture. An option for either of Argo workflows or kubeflow pipelines for batch.</p><p>To delve deeper into the range of tools available for each area, I’d suggest the ‘<a href="https://github.com/EthicalML/awesome-production-machine-learning">Awesome Production Machine Learning</a>’ GitHub repo. It has 7k starts at the time of writing and has dedicated sections on <a href="https://github.com/EthicalML/awesome-production-machine-learning#model-serving-and-monitoring">Model Serving (Real-time)</a>, <a href="https://github.com/EthicalML/awesome-production-machine-learning#data-pipeline-etl-frameworks">Data Pipelines (Batch)</a> and <a href="https://github.com/EthicalML/awesome-production-machine-learning#data-stream-processing">Data Stream Processing (Streaming)</a>.</p><h3>Summary: Navigating ML Deployment</h3><p>Deciding how best to get predictions from ML models can be confusing. We can approach decisions with more clarity if we keep in mind the key deployment patterns — <strong>Offline, Real-time, Batch</strong> and <strong>Streaming</strong>.</p><p>Many projects currently assume they have to build their own ML Deployment infrastructure. But there are technologies out there for each of the four deployment patterns. Knowledge of the patterns also brings reassurance that you are not alone.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=34e35a18d514" width="1" height="1" alt=""><hr><p><a href="https://medium.com/data-science/navigating-ml-deployment-34e35a18d514">Navigating ML Deployment</a> was originally published in <a href="https://medium.com/data-science">TDS Archive</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Tips for Getting Started with Helm]]></title>
            <link>https://medium.com/hackernoon/tips-for-getting-started-with-helm-99b01efdab90?source=rss-5e3e8868caf------2</link>
            <guid isPermaLink="false">https://medium.com/p/99b01efdab90</guid>
            <category><![CDATA[devops]]></category>
            <category><![CDATA[docker]]></category>
            <category><![CDATA[helm]]></category>
            <category><![CDATA[software-development]]></category>
            <category><![CDATA[kubernetes]]></category>
            <dc:creator><![CDATA[Ryan Dawson]]></dc:creator>
            <pubDate>Mon, 15 Oct 2018 08:52:23 GMT</pubDate>
            <atom:updated>2018-10-16T17:27:46.425Z</atom:updated>
            <cc:license>http://creativecommons.org/licenses/by/4.0/</cc:license>
            <content:encoded><![CDATA[<p>You’ve got an application that you want to deploy with Helm. Probably you’ve seen the <a href="https://docs.helm.sh/">official docs</a> and some examples. But you’re not sure how to apply them to your app. Here are some tips.</p><h3>1 Know Your Docker Image</h3><p>Have a docker image/s for your app first and be sure you can work with it. Deploy first with plain Kubernetes descriptors or start with docker-compose.</p><p>When you have everything working with plain k8s descriptors then you’ll want to think about how to make deployment more flexible. You may want to pass in deployment switches for different configurations or for different environments. That’s a great time to introduce helm.</p><h3>2 Understand the Role of the values File</h3><p>To make sure this is clear, let’s remind ourselves of the <a href="https://www.nclouds.com/blog/simplify-kubernetes-deployments-helm-part-2/">structure</a> of a helm chart:</p><pre>├── Chart.yaml<br>├── README.md<br>├── templates<br>│   ├── NOTES.txt<br>│   ├── _helpers.tpl<br>│   ├── <strong>deployment.yaml</strong><br>│   ├── <strong>secrets.yaml</strong><br>│   └── <strong>...more yaml...</strong><br>└── <strong>values.yaml</strong></pre><p>When the user runs helm install then the entries from the values.yaml in the chart and the helm release information (such as unique release name) get injected into the templated yaml resource descriptors as the template is evaluated and rendered into pure Kubernetes deployment descriptors. When the user runs helm install with parameters or a values file, then the parameters or values file will be overlaid on the one in the chart. Effectively the chart’s values file sets defaults that can be overridden.</p><p>If this isn’t clear, try it out. Create a file called mysql-values.yaml:</p><pre>imageTag: “5.7.10”</pre><p>Then run helm install stable/mysql --values=mysqlvalues.yaml. Helm generates a unique release name for us (`ignorant-camel`) and mysql is deployed in the cluster. The output of kubectl describe pod ignorant-camel-mysql-5dc6b947b-lf6p8 tells us that our chosen imageTag has been applied.</p><h3>3 Try Something Out</h3><p>When you’re done with intalling and overriding values, try creating a basic chart. You can use helm create to get a starter chart that you can modify — there’s an example on how to do this <a href="https://docs.helm.sh/chart_template_guide/">in the official docs</a>. Try it.</p><p>Or find a guide online that deploys an app built in the same language you’re using. If you can find an example of a chart for an app that is similar enough to yours then you can copy-paste and use that as a starting point. There are even some tools out there that can do give you starter examples — Jenkins-X for examples has these, which it calls <a href="https://github.com/jenkins-x/draft-packs">draft packs</a>.</p><h3>4 Start Simple</h3><p>Most charts are designed for a single docker image. If you’ve got a microservices app with several interacting parts, you can take a small part of your app that you think you can deploy on its own and try writing a helm chart for that.</p><h3>5 Start with Environment Variables and `if` conditions</h3><p>Most charts involve injecting environment variables into docker containers in a Pod. So you’ll see something like this in a deployment.yaml:</p><pre>- name<strong>: </strong>ENV_VAR1<br>  value: {{ .Values.var1 }}<br>- name<strong>: </strong>ENV_VAR2<br>  value: {{ .Values.var2 }</pre><p>With this, users of the chart can override these values in their values.yaml or with —set var1=foo .</p><p>Often there are also options that control whether to enable a configuration. So you might find something like:</p><pre>- name<strong>: </strong>ENV_VAR1<br>  value: {{ .Values.var1 }}<br>{{- if .Values.var2 }}<br>- name<strong>: </strong>ENV_VAR2<br>  value: {{ .Values.var2 }<br>{{- end }}</pre><p>Then ENV_VAR2 will only be set if a value is set for var2. Or you could do</p><pre>- name<strong>: </strong>ENV_VAR1<br>  value: {{ .Values.var1 }}<br>{{- if .Values.var2enabled }}<br>- name<strong>: </strong>ENV_VAR2<br>  value: {{ .Values.var2 }<br>{{- end }}</pre><p>Then ENV_VAR2 will only be set if var2enabled is true.</p><p>The if directive is an easy Helm <a href="https://github.com/helm/helm/blob/master/docs/chart_template_guide/control_structures.md#ifelse">control structure</a> to understand and allows you to do quite a lot.</p><h3>6 You Can Use Different values Files</h3><p>We’ve already seen that we can create a custom mysqlvalues.yaml file and pass it into the official mysql chart. We can do that with our own charts too. This can be very handy if we’ve got different configurations we want to apply for different environments e.g. staging and prod.</p><h3>7 Expect to Debug</h3><p>Helm install with the --dry-run --debug option is immensely useful. It doesn’t deploy your chart for real but it does run the checks involved in a deploy so you’ll know if you’ve done something fundamentally wrong. It also shows the Kubernetes descriptors that are generated. This let’s you check that your chart is doing what you want it to do.</p><h3>8 Don’t Worry About Releasing Yet</h3><p>Eventually you will probably want to <a href="https://github.com/helm/helm/blob/master/docs/helm/helm_package.md">package </a>your charts and release them in a <a href="https://github.com/helm/helm/blob/master/docs/chart_repository.md">chart repository</a>. But that’s only needed when you want others to be able to consume your charts and build on top of them. To begin with you can work from just source code as it makes things easier. If they’re in source control then others can check them out of source control in the same way you do until you get a charts repository set up.</p><h3>9 Understand Release Names</h3><p>We saw before that when we installed the mysql chart Helm generated the release name ‘ignorant-camel’ for the installation we created. This release name is injected by helm into the kubernetes deployment descriptors that helm generates from our chart. This ensures that every resource that is deployed has a unique name (<a href="https://kubernetes.io/docs/concepts/overview/working-with-objects/names/">and kubernetes needs it to</a>), even if we install the chart multiple times. This happens because the templates contain content like:</p><pre>apiVersion: v1<br>kind: Service<br>metadata:<br>  name:  {{ .Release.Name }}-myservice</pre><p>So {{ .Release.Name }} is a placeholder into which Helm will inject the release name. For many charts you’ll more likely see this as:</p><pre>apiVersion: v1<br>kind: Service<br>metadata:<br>  name: {{ template &quot;fullname&quot; . }}</pre><p>This amounts to the same thing as the template fullname function is defined in _helpers.tpl in the chart and uses release name and chart name.</p><p>You need to be aware of this when you start deploying multiple resource in your chart which need to reference one another. If the resources are defined inside your chart then you need to be careful to use the same names that are used in the definitions by using the same functions. So if you wanted to inject an environment variable into a Pod containing the name of the above Service then you could do:</p><pre>- name: SERVICE_NAME<br>  value: {{ template &quot;fullname&quot; . }}</pre><h3>10 Interaction is Tricky</h3><p>Let’s say you have a bunch of microservices that work together as an app. This might be one big chart if the services are only ever used together and don’t need to be individually deployable. But most likely you do need to be able to separate. So the tricky bit is how to build individual charts for each part and also get them to talk to each other.</p><p>So you’re going to want to use <a href="https://github.com/helm/helm/blob/master/docs/chart_template_guide/subcharts_and_globals.md">subcharts</a>. This lets us include other charts within our chart to create a composite/umbrella chart. We do this by referencing the chart we want to include in the requirements.yaml and including it in the charts directory of our chart. If we’re including a chart we’ve defined ourselves (and haven’t published yet) called mysubchart, we can put this in the requirements.yaml:</p><pre>- name: mysubchart<br>  version: 0.1.0</pre><p>And put the whole source code of mysubchart (its whole directory) in the charts directory of our umbrella chart. If we were including an official/published chart then we’d include its published location in the requirements.yaml:</p><pre>- name: postgresql                         <br>  repository: <a href="https://kubernetes-charts.storage.googleapis.com/">https://kubernetes-charts.storage.googleapis.com</a><br>  version: 0.11.0</pre><p>Then we would do a helm dep build before deploying the chart.</p><p>When charts have resources that need to references in other subcharts then things get a bit trickier. You need to know what the names are of the resources you’re looking for. The names are typically generated dynamically to include a release name.</p><p>Hopefully the chart will make this clear in its documentation but you might have to go to its source code. As an example, the <a href="https://github.com/helm/charts/tree/master/stable/postgresql">official postgresql chart</a> can create a secret containing the password we set for the postgresql instance. If we’re using postgresql in our app (adding it as a subchart) then we’ll want to reference this and inject it into an environment variable.</p><p>We can look at how that <a href="https://github.com/helm/charts/blob/master/stable/postgresql/templates/secrets.yaml">secret is defined</a> or consult <a href="https://github.com/helm/charts/blob/f7cfc026e69a5b5109a5ba97caf9b4115659d57b/stable/xray/templates/xray-persist-deployment.yaml#L54">examples </a>of how it is used to work out we need to set our environment variable using:</p><pre>— name: POSTGRESS_PASSWORD<br>  valueFrom:<br>    secretKeyRef:<br>      name: {{ .Release.Name }}-postgresql<br>      key: postgres-password</pre><p>This works since the postgres chart is included in ours. Its resources are part of the same release so the same release name is used.</p><p>The complexity of these references can grow depending on how the charts need to interact and how they will be used. For example, what if you need to support either creating a postgres along with your chart or pointing at an existing one? Or provide the option to switch to mysql? This sort of thing can get complex. The official charts <a href="https://hackernoon.com/the-art-of-the-helm-chart-patterns-from-the-official-kubernetes-charts-8a7cafa86d12">show patterns for handling these kinds of cases</a> but you’re best to start by limiting options so that initially you just have to handle the simpler case/s.</p><h3>11 Duplication Across Charts Can Be OK</h3><p>Each chart is for deploying a specific docker image or images. (Different tags or versions of the same image are normally handled by the same chart.) But what if you’ve got several images that are quite similar? Can we re-use sections across different charts?</p><p>The offical helm documentation gives <a href="https://github.com/helm/helm/blob/master/docs/chart_template_guide/subcharts_and_globals.md#overriding-values-from-a-parent-chart">an example</a> of how you can ‘inherit’ features from a parent chart. It defines a chart called mychart that includes a subchart called mysubchart in its charts directory. The values.yaml for mysubchart contains:</p><pre>dessert: cake</pre><p>This value goes into a ConfigMap resource created by mysubchart. The values.yaml of mychart can then override this with:</p><pre>mysubchart:<br>  dessert: ice cream</pre><p>So mychart can define the same ConfigMap defined by mysubchart because it includes mysubchart but it can give it different content by overriding the values.</p><p>You might therefore use a base chart to define common resources and have other charts include that chart and override them in the way that they need to. For example, you might imagine having a base chart for java applications. However, you don’t currently see this pattern used very much in that way. Actually, many of the official helm charts include sections that are pretty much the same for many charts. Lots handle ingress in the same way. Rather than try to re-use a common parent, they duplicate. (Actually they try to broadly <a href="https://github.com/helm/charts/blob/master/REVIEW_GUIDELINES.md#ingress">follow a common guideline</a>.) Why?</p><p>One reason is that helm has no notion of an abstract parent. Each chart, including the parent, is meant to be individually deployable. If it isn’t then it gets a bit tricky to test it.</p><p>Subcharts are more like overrideable members of a chart than parents in OO languages. Each has to be addressed with a prefix (in our example mysubchart). These names would grow quite awkward if you had multiple layers. You can’t modify resources in a subchart unless they are exposed through values. This means having to go and change a subchart in order to change the chart consuming it, which gets more awkward when charts are released as then each change is a version change and requires a rebuild and release.</p><p>Further, if there were a common parent for defining Ingress, it would need to refer to the Service being exposed. That means having to inject the service name into the Ingress resource. Since the parent would really be an included 3rd-party chart, that would bring in all the complexity of interactions we saw in the last section.</p><p>This is not to say that we shouldn’t try to avoid duplication. If we’ve got several charts that are deployed together and all need the same set of environment variables then using <a href="https://github.com/helm/helm/blob/master/docs/chart_template_guide/subcharts_and_globals.md#global-chart-values">globals</a> or the option to point at a <a href="https://dzone.com/articles/configuring-java-apps-with-kubernetes-configmaps-a">common ConfigMap</a> could make a lot of sense. It’s also possible to reduce duplication in the content of templates with <a href="https://github.com/helm/helm/blob/master/docs/chart_template_guide/named_templates.md">named templates</a> but that’s a bit more complex (although example <a href="https://stackoverflow.com/questions/43923749/kubernetes-how-to-avoid-duplicating-env-in-multiple-deployments">here</a>).</p><p>The point is that you can save yourself some headaches initially by duplicating content and looking to reduce duplication later if you find the duplication to be a problem. (Hopefully some of this will <a href="https://github.com/helm/helm/issues/3920">get smoother in Helm3</a> as the area is <a href="https://github.com/helm/helm/issues/4535">under discussion</a>.)</p><h3>12 You Don’t Have to Automate Your Helm Testing Right Away</h3><p>Probably you’re testing your app after deploying it anyway. If so, don’t feel you have to go full TDD your chart deployment process. Helm does have options for that and it’ll be great to use them but it’s more to learn.</p><h3>13 It’s OK to Treat Helm Like a Box of Tricks</h3><p>Many programmers are used to there being established design patterns to follow. From this perspective it’s surprising how quickly you end up looking at the helm <a href="https://github.com/helm/helm/blob/master/docs/charts_tips_and_tricks.md">‘tips and tricks’ pages</a> or documentation of <a href="https://github.com/helm/helm/blob/master/docs/chart_template_guide/functions_and_pipelines.md#template-functions-and-pipelines">template functions</a>. Embrace this as OK. There’s<a href="https://medium.com/@ryandawsonuk/the-art-of-the-helm-chart-patterns-from-the-official-kubernetes-charts-8a7cafa86d12"> lots of variation in the official Kubernetes charts.</a> It’s ok to do things in the way that works for you.</p><h3><strong>14 Help Yourself Out with the Official Charts</strong></h3><p>Official charts deal with things you won’t have thought of and they are maintained. So if you need a mysql database or anything else you can find in the official charts, look to use it instead of creating your own chart for that.</p><p>Note that the <a href="https://hub.kubeapps.com/">official charts </a>are quite complex because they have to be usable in so many ways. Don’t feel you need to delve into their code and follow their example right away, <a href="https://hackernoon.com/the-art-of-the-helm-chart-patterns-from-the-official-kubernetes-charts-8a7cafa86d12">although you might later benefit from that</a>. Initially you can just use them as it suits you. Each should have an explanation of how to use it.</p><h3>15 Use the Helm Community</h3><p>If you join the<a href="http://slack.k8s.io/"> official kubernetes slack</a>, there are channels there for helm and for helm charts. You can get great advice there. Also don’t be afraid to ask on <a href="https://stackoverflow.com/">stackoverflow</a>. You are not alone.</p><h3>Takeaways</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*bQTLNJuF4FItLXYUZ4mPcw.png" /><figcaption><a href="https://vimeo.com/127298945">Getting the hang of Helm now</a></figcaption></figure><p>There’s a lot to learn with Helm and it’s quite different from the approaches that many programmers are used to. Break it down and take it one step at a time. You’ll be making waves faster than you think.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=99b01efdab90" width="1" height="1" alt=""><hr><p><a href="https://medium.com/hackernoon/tips-for-getting-started-with-helm-99b01efdab90">Tips for Getting Started with Helm</a> was originally published in <a href="https://medium.com/hackernoon">HackerNoon.com</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[The Art of the Helm Chart: Patterns from the Official Kubernetes Charts]]></title>
            <link>https://medium.com/hackernoon/the-art-of-the-helm-chart-patterns-from-the-official-kubernetes-charts-8a7cafa86d12?source=rss-5e3e8868caf------2</link>
            <guid isPermaLink="false">https://medium.com/p/8a7cafa86d12</guid>
            <category><![CDATA[kubernetes]]></category>
            <category><![CDATA[devops]]></category>
            <category><![CDATA[docker]]></category>
            <category><![CDATA[helm]]></category>
            <category><![CDATA[development]]></category>
            <dc:creator><![CDATA[Ryan Dawson]]></dc:creator>
            <pubDate>Thu, 20 Sep 2018 18:26:02 GMT</pubDate>
            <atom:updated>2019-03-21T11:54:46.740Z</atom:updated>
            <cc:license>http://creativecommons.org/licenses/by/4.0/</cc:license>
            <content:encoded><![CDATA[<p>Helm charts package up applications for installation on Kubernetes Clusters. Installing a helm chart is a bit like running an install wizard. So helm chart developers face some of the same challenges faced by developers producing installers:</p><ul><li>What assumptions can be made about the environment that the install is running into?</li><li>Is the application meant to be able to interact with other applications?</li><li>What configurations need to be made available to the user and how should they be offered to the user?</li></ul><p>But these questions come with helm-specific twists. To see why, let’s start with a picture of what happens when a user runs a`helm install`. Then we can move on to see how some of the <a href="https://github.com/helm/charts">official Kubernetes charts</a> deal with these questions.</p><h3>A Picture of a Helm Install</h3><p>I want to install mysql in my cluster. But I don’t want the mysql version that the stable/mysql chart sets <a href="https://github.com/helm/charts/blob/fce283ced152e4eed93da8574022a5bcaff9b98d/stable/mysql/values.yaml#L5">in its values.yaml file in the official charts repo</a>. So I create my own values.yaml file called mysql-values.yaml with just one line:</p><pre>imageTag: “5.7.10”</pre><p>And then I run helm install stable/mysql --values=mysqlvalues.yaml. Helm generates a unique release name for me (`ignorant-camel`) and mysql is deployed in my cluster. The output of kubectl describe pod ignorant-camel-mysql-5dc6b947b-lf6p8 tells me that my chosen imageTag has been applied.</p><p>Actually I didn’t need to install anything in my cluster to see that my selection for imageTag would be applied. I could have run helm install stabe/mysql --values=mysqlvalues.yaml --dry-run --debug and helm would have simply shown me the content of kubernetes deployment descriptor yaml it generated without installing anything.</p><p>The process of getting to the generated kubernetes deployment descriptors can be better understood by thinking of the <a href="https://www.nclouds.com/blog/simplify-kubernetes-deployments-helm-part-2/">structure</a> of a helm chart:</p><pre>├── Chart.yaml<br>├── README.md<br>├── templates<br>│   ├── NOTES.txt<br>│   ├── _helpers.tpl<br>│   ├── <strong>deployment.yaml</strong><br>│   ├── <strong>secrets.yaml</strong><br>│   └── <strong>...more yaml...</strong><br>└── <strong>values.yaml</strong></pre><p>When the user runs helm install stable/mysql then the entries from the values.yaml in the chart and the helm release information (such as unique release name) get injected into the templated yaml resource descriptors as the template is evaluated and rendered into pure Kubernetes deployment descriptors. When the user runs helm install stable/mysql with parameters or a values file, then the parameters or values file will be overlaid on the one in the chart. Effectively the in-chart values file sets defaults that can be overridden.</p><p>So the values.yaml is the main interface between us as chart developers and our users. What we expose in the values.yaml determines what users can and can’t do with our charts.</p><p>Our charts and our values.yaml files also need to satisfy another kind of scenario. Another chart developer might like our app and decide they want to include it inside a package they want to release. So they add our chart to the requirements.yaml of a new chart they create, along with a bunch of other charts. They override some of our defaults by setting <em>their</em> values.yaml. And then they distribute it to users who do helm install with that chart.</p><p>So our chart goes into other charts and they feed onto users in a charts food chain. Seeing how to satisfy our consumers in this food chain is one of the key challenges of writing helm charts.</p><h3>The Challenges of Writing Helm Charts</h3><p>Applications often have lots of configuration options and Kubernetes Clusters can be configured in lots of different ways. So when writing a helm chart we naturally face questions like:</p><ol><li>What if I forget to expose a configuration parameter in the values.yaml? What if my application parses configuration dynamically so that I can’t say upfront what all the possible parameter names will be?</li><li>How can I allow users to use resources defined not directly in my chart but in a chart in which my chart is used as a subchart?</li><li>What if another chart developer is using my chart in their own and they need to add significant sections inside a resource that I defined (e.g. add an extra container to a pod or a custom initialization script)?</li><li>How can I allow my users to expose the app externally in the various ways that they might want?</li></ol><p>We can learn a lot on how to deal with these kinds of problems from the charts in the <a href="https://github.com/helm/charts">official helm charts repository</a>. Let’s take a look at some of the patterns that those charts use, so that we can understand how to use the same patterns in our own charts.</p><p>Before diving in though it’s worth keeping in mind that the public charts are rather advanced. If you’re worrying about some of the above questions when developing your first helm charts, try to park the more difficult worries to begin with. Start by familiarising yourself with <a href="https://docs.helm.sh/chart_template_guide/#getting-started-with-a-chart-template">the helm documentation</a> and try to build a chart that works for just the simplest cases first. You can then add <a href="https://github.com/helm/helm/blob/master/docs/charts_tips_and_tricks.md">advanced options</a> later.</p><p>So let’s try to get a snapshot of patterns from the official charts. This won’t be comprehensive and I expect new options to become available and new patterns to emerge when <a href="https://sweetcode.io/a-first-look-at-the-helm-3-plan/">helm 3 </a>adds scripting with Lua. But it will give us look at the current state of the art.</p><h3>1 Patterns for Exposing Configuration Parameters</h3><p>Let’s say we’ve defined a Deployment resource in our template and we’ve configured the env section to allow some environment variables to be set from the values.yaml:</p><pre>- name<strong>: </strong>ENV_VAR1<br>  value: {{ .Values.var1 }}<br>- name<strong>: </strong>ENV_VAR2<br>  value: {{ .Values.var2 }}</pre><p>Now users of our chart can override these values in their values.yaml or with —set var1=foo . But what will our users be able to do if we’ve overlooked one? Or even worse, what if our application can dynamically parse configuration options (e.g. it might read ENV_VAR1 and create an internal variable called var1 )? Then there isn’t even a finite set of configuration option names to expose. So how do we let users set the variable names as well as the values?</p><p>As the <a href="https://docs.helm.sh/chart_template_guide/#variables">helm charts developer guide</a> says, we could create a configmap with a range function. A nice example of this is in the stable/unbound chart. It contains a <a href="https://github.com/helm/charts/blob/cbd5e811a44c7bac6226b019f1d1810ef5ee45fa/stable/unbound/templates/configmap.yaml">configmap</a> that defines its unbound.conf file. It <a href="https://github.com/helm/charts/blob/cbd5e811a44c7bac6226b019f1d1810ef5ee45fa/stable/unbound/templates/deployment.yaml#L71">mounts</a> this file into Pods created by its Deployment. Inside the configmap it has entries <a href="https://github.com/helm/charts/blob/cbd5e811a44c7bac6226b019f1d1810ef5ee45fa/stable/unbound/templates/configmap.yaml#L35">such as</a>:</p><pre>{{- range .Values.localRecords }}<br>local-data: &quot;{{ .name }} A {{ .ip }}&quot;<br>local-data-ptr: &quot;{{ .ip }} {{ .name }}&quot;<br>{{- end }}</pre><p>And its <a href="https://github.com/helm/charts/blob/cbd5e811a44c7bac6226b019f1d1810ef5ee45fa/stable/unbound/values.yaml#L54">values.yaml</a> lets the entries in localRecords be set as a list e.g.:</p><pre>localRecords:<br>- name: &quot;fake3.host.net&quot;<br>  ip: &quot;10.12.10.10&quot;<br>- name: &quot;fake4.host.net&quot;<br>  ip: &quot;10.13.10.10&quot;</pre><p>The <a href="https://github.com/helm/charts/blob/0445fad8d8308f089f5fb6756a5570bc6d6f0bf5/stable/sonarqube/templates/deployment.yaml#L80">sonarqube chart</a> applies a similar approach directly to environment variables, defining some explicitly and allowing further variables to be set using an extraEnv collection:</p><pre>{{- range $key, $value := .Values.extraEnv }}<br> — name: {{ $key }}<br>   value: {{ $value }}<br>{{- end }}</pre><p>So that variables would be set in a values.yaml like:</p><pre>extraEnv:<br>  ENV_VAR1: var1<br>  ENV_VAR2: var2</pre><p><a href="https://github.com/helm/charts/search?q=extraEnv&amp;unscoped_q=extraEnv">Many</a> of the official charts define an extraEnv but in slightly different ways. The <a href="https://github.com/helm/charts/blob/cbd5e811a44c7bac6226b019f1d1810ef5ee45fa/stable/buildkite/templates/deployment.yaml#L43">buildkite chart</a> defines it differently, without using range. Instead it takes the values.yaml section and simply puts it in the template:</p><pre>{{- if .Values.extraEnv }}<br>{{ toYaml .Values.extraEnv | indent 12 }}<br>{{- end }}</pre><p>So this means that instead of setting extraEnv entries in the values.yaml as simple pairs we would also need to name each of the keys (name) and values (value) in the pairs like:</p><pre>extraEnv:<br> — name: ENV_VAR1<br>   value: &quot;var1&quot;<br> — name: ENV_VAR2<br>   value: &quot;var2&quot;</pre><p>The <a href="https://github.com/helm/charts/blob/0445fad8d8308f089f5fb6756a5570bc6d6f0bf5/stable/keycloak/templates/statefulset.yaml#L67">keycloak chart</a> does this differently again:</p><pre>{{- with .Values.keycloak.extraEnv }}<br>{{ tpl . $ | indent 12 }}<br>{{- end }}</pre><p>So it treats extraEnv as a string, sends it through the tpl function so that it is evaluated as part of the template and ensures that the correct indenting is applied. This is interesting as it allows values to be set like:</p><pre>extraEnv: |<br> — name: KEYCLOAK_LOGLEVEL<br>   value: DEBUG<br> — name: HOSTNAME<br>   value: {{ .Release.Name }}-keycloak</pre><p>Normally it wouldn’t be possible to use a template directives like {{ .Release.Name }} inside the values.yaml but here we can because the content will go through tpl. This could be a big advantage in cases where we’re including the chart inside a parent chart that we’re developing and we need to refer to a service that is part of the same parent chart. (More on this in the next section.) The disadvantages are that it’s a bit more abstract and that the content in the values.yaml is treated as a string there so formatting issues aren’t necessarily found by your editor.</p><h3>2 Referencing Mutually-deployed Resources</h3><p>Typically resources deployed through helm are prefixed with a release name, so that multiple releases can be installed from the same chart (and in the same namespace) without naming conflicts between the installed resources. This means that template directives need to be used to prefix the resource names when one resource with a chart needs to refer to another, or to resources within the same parent chart.</p><p>A common case where a mutually-deployed resource needs to be referred to is a database secret. The <a href="https://github.com/helm/charts/blob/cbd5e811a44c7bac6226b019f1d1810ef5ee45fa/incubator/xray/templates/xray-indexer-deployment.yaml#L56">xray chart</a>, for example, includes within it the option to deploy a postgres database. It sets up its indexer component’s Deployment with credentials by pointing to the postgres secret (which itself is part of the chart as a subchart and therefore also prefixed with the same release name):</p><pre>{{- if .Values.postgresql.enabled }}<br> — name: POSTGRES_USER<br>   value: {{ .Values.postgresql.postgresUser }}<br> — name: POSTGRESS_PASSWORD<br>   valueFrom:<br>     secretKeyRef:<br>       name: {{ .Release.Name }}-postgresql<br>       key: postgres-password<br> — name: POSTGRESS_DB<br>   value: {{ .Values.postgresql.postgresDatabase }}<br> {{- else }}<br>...</pre><p>Here the xray chart knows which database it needs to connect to since it is itself including the postgres chart. But what if we were developing a chart for an application where the user can choose to supply a database? Then it might be necessary to allow the user to include our chart inside a parent chart alongside a database of their choosing. How would we allow the user to set database configuration then?</p><p>One option would be the extraEnv approach we saw already with the keycloak chart. Then the user could then set our chart’s extraEnv to point to postgres even if we’ve not defined it explicitly in our original chart. The values.yaml would then have an entry like:</p><pre>extraEnv: |<br>- name: POSTGRES_USER<br>  value: {{ .Values.postgresql.postgresUser }}<br>— name: POSTGRESS_PASSWORD<br>  valueFrom:<br>    secretKeyRef:<br>      name: {{ .Release.Name }}-postgresql<br>      key: postgres-password<br>— name: POSTGRESS_DB<br>  value: {{ .Values.postgresql.postgresDatabase }}</pre><p>The | is there because the section is treated as a string to be passed explicitly to the tpl function.</p><p>A similar question emerges if a release name prefix needs to be used within an entry in a file that is loaded into a configmap. A typical way of loading a whole file into a configmap is to use .Files.Get. However, like the values.yaml, the contents of files loaded into this way are then not part of the template and so template directives can’t be used in them. What does allow template directives to be used inside a file is to load the <a href="https://stackoverflow.com/questions/47595295/how-do-i-load-multiple-templated-config-files-into-a-helm-chart/52009992#52009992">file content with .Files.Get and pass that into </a><a href="https://stackoverflow.com/questions/47595295/how-do-i-load-multiple-templated-config-files-into-a-helm-chart/52009992#52009992">tpl</a>. Then the data section of the configmap would have entries like:</p><pre>conf_file1: {{ tpl (.Files.Get &quot;files/conf_file1&quot;) . | quote }}</pre><p>Or to load into a Secret we need to encode the content in base64:</p><pre>conf_file1: {{ tpl (.Files.Get &quot;files/conf_file1&quot;) . | b64enc | quote }}</pre><p>Instead of just one file can with load a <em>set </em>of files into a ConfigMap <a href="https://www.nclouds.com/blog/simplify-kubernetes-deployments-helm-part-3-creating-configmaps-secrets/">using </a>.Files.Glob :</p><pre>{{ (tpl (.Files.Glob &quot;files/*&quot;).AsConfig . ) | indent 2 }}</pre><p>Though we can’t quite apply the same approach with AsSecret as then the content would be encoded before going through tpl . To load a set of files into a secret we could use <a href="https://github.com/helm/helm/blob/master/docs/chart_template_guide/accessing_files.md#glob-patterns">Glob to find the files and </a><a href="https://github.com/helm/helm/blob/master/docs/chart_template_guide/accessing_files.md#glob-patterns">Get to load them</a>:</p><pre>{{ range $path, $bytes := .Files.Glob &quot;files/*&quot; }}<br>{{ base $path }}: &#39;{{ tpl ($root.Files.Get $path) . | b64enc }}&#39;<br>{{ end }}</pre><h3>3 Empowering Our Fellow Chart Developers</h3><p>The use of extraEnv in the keycloak chart suggests a possible pattern for dealing with other kinds of definitions that might need to be injected into a chart. For example, the keycloak chart needs to allow a user to package the keycloak chart inside a parent chart that also supplies a user-defined a json file that the user needs to be able to mount into keycloak Pods. The chart supports this by <a href="https://github.com/helm/charts/blob/0445fad8d8308f089f5fb6756a5570bc6d6f0bf5/stable/keycloak/templates/statefulset.yaml#L115">exposing</a> an extraVolumes:</p><pre>{{- with .Values.keycloak.extraVolumes }}<br>{{ tpl . $ | indent 8 }}<br>{{- end }}</pre><p><a href="https://github.com/helm/charts/blob/0445fad8d8308f089f5fb6756a5570bc6d6f0bf5/stable/keycloak/templates/statefulset.yaml#L73">And</a> extraVoumeMounts:</p><pre>          volumeMounts:<br>            - name: scripts<br>              mountPath: /scripts<br>{{- with .Values.keycloak.extraVolumeMounts }}<br>{{ tpl . $ | indent 12 }}<br>{{- end }}</pre><p>The user <a href="https://github.com/helm/charts/tree/cbd5e811a44c7bac6226b019f1d1810ef5ee45fa/stable/keycloak#setting-a-custom-realm">can then</a> point to their secret (which contains the json file) and mount it through the values.yaml:</p><pre>extraVolumes: |<br> — name: custom-secret<br>   secret:<br>     secretName: custom-secret<br>extraVolumeMounts: |<br> - name: custom-secret<br>   mountPath: &quot;/realm/&quot;<br>   readOnly: true</pre><p>Effectively the configuration of volumes and volumeMounts is externalized to the values.yaml. The chart also applies this pattern to allow users to inject further resources into the template such as initContainers and even adding entire additional containers (or ‘sidecars’). This pattern gives chart users a lot of power and flexibility, especially for other chart developers using our chart.</p><p>The pattern can be especially powerful when combined with the keycloak chart’s exposing of other parameters, such as a <a href="https://github.com/helm/charts/blob/0445fad8d8308f089f5fb6756a5570bc6d6f0bf5/stable/keycloak/templates/configmap.yaml#L19">preStartScript</a> variable that is used within the chart’s initialization script:</p><pre>{{- with .Values.keycloak.preStartScript }}                           echo &#39;Running custom pre-start script...&#39;                       <br>{{ . | indent 4 }}                         <br>{{- end }}</pre><p>The user can inject whatever shell script content they like into .Values.keycloak.preStartScript in their values.yaml. So this approach makes it possible for users to not just set their own configuration paramters but also load their own files and run their own custom scripts using those files.</p><h3>4 Exposing Externally in Different Ways</h3><p>The starter helm chart generated by helm create includes a Service specification but not an Ingress. Many of the public charts do define an Ingress resource. This needs to be configurable as users might not want to use ingress. So the <a href="https://github.com/helm/charts/blob/cbd5e811a44c7bac6226b019f1d1810ef5ee45fa/stable/rabbitmq/templates/ingress.yaml#L1">rabbitmq chart</a>, like many others, wraps its whole Ingress resource definition with:</p><pre>{{- if .Values.ingress.enabled }}<br>...<br>{{-end}</pre><p>This is in line with the <a href="https://github.com/helm/charts/blob/master/REVIEW_GUIDELINES.md#ingress">review guidelines</a> for the official charts repo, which recommends that ingress should be disabled by default.</p><p>The rabbitmq chart, for example, needs to provide the option whether to expose it by setting a <a href="https://github.com/helm/charts/blob/cbd5e811a44c7bac6226b019f1d1810ef5ee45fa/stable/rabbitmq/templates/ingress.yaml#L19">host-based</a> ingress rule:</p><pre>rules:<br>  {{- if .Values.ingress.hostName }}<br>  - host: {{ .Values.ingress.hostName }}<br>    http:<br>  {{- else }}<br>  - http:<br>  {{- end }}</pre><p>(Actually the user might want multiple hosts to route to the service so the <a href="https://github.com/helm/charts/blob/master/REVIEW_GUIDELINES.md#ingress">review guidelines</a> suggest using a range function for hosts. The rabbitmq chart just offers a single host.)</p><p>The rabbitmq chart also allows for the host to be not set (the else condition above). In that case it’s likely that the user will instead override the path (so that rabbitmq is exposed on a unique route, distinct from other exposed services):</p><pre>- path: {{ default &quot;/&quot; .path }}<br>  backend:<br>    serviceName: {{ template &quot;rabbitmq.fullname&quot; . }}<br>    servicePort: {{ .Values.rabbitmq.managerPort }}</pre><p>It’s also especially important that the user has flexibility to set the annotations on an ingress resource, since these are used to control routing configuration options. The <a href="https://github.com/helm/charts/blob/master/REVIEW_GUIDELINES.md#ingress">review guidelines</a> suggest supporting this with toYaml:</p><pre>{{- with .Values.ingress.annotations }}<br> annotations:<br>{{ toYaml . | indent 4 }}<br>{{- end }}</pre><p>This allows the user of the chart to set annotations in the values.yaml such as:</p><pre>annotations:<br>  kubernetes.io/ingress.class: nginx<br>  nginx.ingress.kubernetes.io/rewrite-target: /</pre><p>So the user can set whatever annotations they need to on the ingress through the values.yaml without the chart needing to be aware of what those annotations might be. Just exposing these annotations can amount to giving the power to apply script-like configuration on the ingress routing. For example, with an nginx ingress a user can apply rules to set headers in custom ways through configuration snippets:</p><pre>annotations:<br>  kubernetes.io/ingress.class: nginx<br>  nginx.ingress.kubernetes.io/rewrite-target: /<br>  nginx.ingress.kubernetes.io/configuration-snippet: |<br>     more_set_headers &#39;Access-Control-Allow-Origin: $http_origin&#39;;                            </pre><h3>What We’ve Learned About the Art of the Helm Chart</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/622/0*oOXN1R9rF3Lblnzn" /><figcaption>This <a href="https://www.pexels.com/photo/grayscale-photo-of-man-wearing-black-dress-shirt-with-hat-sitting-on-folding-armchair-beside-family-sketch-1046123/">guy</a> is just chillin because his charts are so good</figcaption></figure><p>A good helm chart needs to anticipate what level of flexibility is needed by its users. More flexibility tends to mean either more complexity or greater abstraction or both. This can hamper chart readability and put more burden on chart users. The challenge is to choose the tools that best fit with what users need for that particular chart.</p><p>There are other concerns to balance too that we’ve not touched on, such as testing and security. This has just been a look at a particular slice of the official charts. I’ve tried to focus on patterns that seem particularly helpful to me in making sure that users can do what they need to do with your charts. The official Kubernetes charts have been extremely helpful to me in my experience of working on <a href="https://github.com/Activiti/activiti-cloud-charts/">the helm charts</a> for the <a href="https://github.com/Activiti">Activiti</a> project. Hopefully the explanation in this post can help encourage others to dive into the official repo and take inspiration from its charts.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=8a7cafa86d12" width="1" height="1" alt=""><hr><p><a href="https://medium.com/hackernoon/the-art-of-the-helm-chart-patterns-from-the-official-kubernetes-charts-8a7cafa86d12">The Art of the Helm Chart: Patterns from the Official Kubernetes Charts</a> was originally published in <a href="https://medium.com/hackernoon">HackerNoon.com</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
    </channel>
</rss>