<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Alejandro Saucedo on Medium]]></title>
        <description><![CDATA[Stories by Alejandro Saucedo on Medium]]></description>
        <link>https://medium.com/@AxSaucedo?source=rss-32de426f7278------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/1*BbRpUh3j5fjz5b1hylRNPw.jpeg</url>
            <title>Stories by Alejandro Saucedo on Medium</title>
            <link>https://medium.com/@AxSaucedo?source=rss-32de426f7278------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Wed, 18 Nov 2026 22:23:13 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@AxSaucedo/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[Claude Code: It’s not replacing devs. It’s moving them to a higher altitude.]]></title>
            <link>https://medium.com/@AxSaucedo/claude-code-its-not-replacing-devs-it-s-moving-them-to-a-higher-altitude-8fbac096f579?source=rss-32de426f7278------2</link>
            <guid isPermaLink="false">https://medium.com/p/8fbac096f579</guid>
            <category><![CDATA[genai]]></category>
            <category><![CDATA[agentic-ai]]></category>
            <category><![CDATA[agentic-development]]></category>
            <category><![CDATA[github-copilot]]></category>
            <category><![CDATA[claude]]></category>
            <dc:creator><![CDATA[Alejandro Saucedo]]></dc:creator>
            <pubDate>Mon, 26 Jan 2026 19:47:58 GMT</pubDate>
            <atom:updated>2026-01-28T16:07:26.537Z</atom:updated>
            <content:encoded><![CDATA[<p>Join me in this post as I explore new paradigms brought by agentic development tools, lessons learned from building systems with AI tooling, and what that could mean for the software engineering practice (not “just coding”) within the next few years.</p><p><em>“AI is making developers 10x” [</em><a href="https://github.blog/news-insights/research/research-quantifying-github-copilots-impact-on-developer-productivity-and-happiness/"><em>link</em></a><em>],</em> <em>“AI is making developers less productive” [</em><a href="https://metr.org/blog/2025-07-10-early-2025-ai-experienced-os-dev-study/"><em>link</em></a><em>],</em> <em>“AI is coming for our jobs” [</em><a href="https://www.windowscentral.com/artificial-intelligence/engineers-dont-write-code-anymore-anthropic-ceo-says-ai-is-about-to-eat-the-entire-profession"><em>link</em></a><em>], “AI enables coders” [</em><a href="https://www.atlassian.com/blog/artificial-intelligence/how-ai-turns-software-engineers-into-product-engineers"><em>link</em></a><em>].</em></p><p>Well, which one is it?</p><p>Let’s start this exploration with this meme:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*GZ5vTJC4TAxXhKx9" /></figure><p>I’ve been thinking about this meme for a while. The punchline is feels like a great take to the question on how agentic development is impacting the software engineering practice.</p><p>I’ve been able to further process this thought particularly as throughout the winter break I got a Copilot Pro subscription and went on a building spree; initially building one project per day, but then getting stuck on a longer project which provided me with a more clear view of what the future may look like for the tech industry.</p><h3>Building a team-sized projects in days instead of weeks</h3><p>If someone was to tell me they single-handedly built a project that spans across custom Kubernetes operators, backend, frontend and SDKs, with robust CI &amp; release process within a couple of days, I would have struggled to believe them.</p><p>That is about how shocked I was when I took on the challenge, and successfully built <a href="https://github.com/axsaucedo/kaos"><strong>K.A.O.S.</strong></a><strong> </strong>in just a couple of days<strong>, a Kubernetes Agent Orchestration System</strong> that deploys and manges multi-agent systems, which consists of a non-trivial set of components:</p><ul><li>Golang control-plane that manages the kubernetes native resources</li><li>Python data-plane that implements Agents, MCPs and ModelAPI client/servers</li><li>React based user interface with interactive CRUD, Agent chat, MCP debugging, etc</li><li>CLI for managing resources and easily performing workflows</li><li>CI/CD with KIND based e2e testing and automated release management</li></ul><p>Traditionally building a project like KAOS would’ve taken weeks, and would’ve required a handful of skilled individuals with different skill-sets (eg frontend, backend, infra, etc).</p><p>But oh how things have changed.</p><p>I was blazing through development milestones and the code was produced at real time speed. It got to the point where iteration cycles were faster than my capability to review changes and design further extensions. This brought two particular themes that really surprised me.</p><h3>Surprise #1: Time Constant — Watching tech debt form in real time</h3><p>The first surprise was the <strong>time constant</strong>.</p><p>We all know tech debt is a real problem; in practice it creeps in slowly &amp; silently, and we only realise it when it’s too late. <a href="https://martinfowler.com/bliki/TechnicalDebt.html">The interest payments eventually show up</a>: small changes require larger investments, developer productivity stalls, interesting new bugs arise from everywhere, code areas become fragile, etc.</p><p>What I was particularly surprised about was not about tech debt itself, but about <strong>the time constant</strong> in which was being created<strong>.</strong> If at some point I came across an implementation that was slightly off (eg. a quick interface hack, a decision in the wrong layer, etc) and didn’t address it right there and then, within hours I could feel the codebase start bending around it.</p><p>I have to say it was fascinating seeing tech debt form in a few hours in real time, when normally this spans across weeks, or even months.</p><p>This also coupled with the realisation that now I had the luxury to backtrack on particular implementation directions when seeing something was off. This was only possible due to low cost/effort for development.</p><p>This enabled for an <strong>exploration-exploitation branching-like </strong>development approach, as opposed to a purely linear one where otherwise retracting may be too expensive compared to accepting the short term debt.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*_aejdELxeEydy7ze" /></figure><h3>Surprise #2: Cognitive Level — Operating at a higher level of abstraction whilst staying tactical</h3><p>The second surprise was at the <strong>cognitive level</strong>.</p><p>Historically when coding the traditional cycle is along the lines of: pick up a task, dive into a module, load that context into your brain’s RAM, and descend through levels of abstractions into the internals to fix what you can as further rabbit-holes arise (aka “being in the zone”). Basically this was the status quo, where we had accepted that our brains (+ typing speed) are the bottleneck.</p><p>With agentic workflows, I noticed a different mode:</p><ul><li>Load the entire picture at once and reason about at the <strong>system level </strong>(backend, infrastructure, frontend, SDK, CI, etc) without losing the tactical thread.</li><li>Delegate chunks of implementation whilst being able to steer the direction in real time whilst having the big picture in mind.</li><li>Review the resulting artifacts as discrete, comprehensible deltas noting down extensions and improvements.</li><li>Match against overarching direction otherwise backtrack and iterate; often with various threads in parallel.</li></ul><p>This didn’t feel like I was “no longer coding”. It just felt like I was <strong>coding at a different altitude</strong>.</p><p>The way I like to think about this is the following analogy: how a tech lead or staff engineer thinks when they’re guiding the team(s) and shaping the system across months of development cycles, instead of just editing files.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*iT2uZ6Ia_FZc26yx" /></figure><h3>The real unlock is abstraction: specs become complex systems</h3><p>Now bringing these two “surprises” together is what makes me think: we have been over-emphasising the <strong>improved time constant</strong> and under-discussing the <strong>higher cognitive level</strong> that is being unlocked.</p><p>If we do now focus on the higher cognitive level unlocked, a more complex question arises:</p><p><strong>If we have unlocked this higher level of abstraction, what could this enable in practice?</strong></p><p>In organisations we may have a functional setups (teams of backend vs teams of frontend, etc) or cross functional setups (teams with mixes of frontend, backend, etc). Each of these team members would have a manager; teams of managers would report to a head of department; heads would report to a director; directors would report to… you get the point.</p><p>As of today, we build these layers because <strong>cognition</strong> and <strong>coordination</strong> don’t scale linearly. At some point, you need a layer whose job is <strong>not</strong> writing the code, but <strong>steering</strong> the system; i.e. aligning architecture, interfaces, guardrails, and operational quality. Further layers then arise to steer strategy, business priorities, etc — which may or may not be aligned with the execution (and vice-versa).</p><p>Why are we talking about organisational setups? Here is the twist: with this higher level of abstraction, a single developer may be able to move upwards to that next layer of cognitive abstraction <em>without giving up the ability to ship tactical changes</em>. The metric is no longer “how fast do I code”; it becomes:</p><p><strong>How many layers of complexity can one person hold and steer effectively?</strong></p><p>And once you look at it through that lens, the most interesting question becomes not “how many tickets can I close”, but:</p><h3>What is an example of what we could build at this higher level of abstraction?</h3><p>The first potential answer to this question came up as I was exploring it through various conversations. Namely on the topic of a common organizational reality: departmental systems optimizing in silos.</p><p>In a typical e-commerce setup you might have separate systems for:</p><ul><li><strong>Pricing</strong> — optimising discounting, demand shaping, stock clearance</li><li><strong>Replenishment</strong> — optimising inventory coverage to enable re-stocking</li><li><strong>Marketing</strong> — optimising campaigns, conversion, and demand spikes</li></ul><p>Each system can be locally <em>smart</em>, but globally <strong>dumb</strong>.</p><p>Pricing may discount inventory down because it sees a stock issue. Replenishment may ramps stock up because it sees demand. Marketing may run a campaign that shifts demand curves. If these systems aren’t aligned, you get cannibalisation: adversarial optimisation that makes the overall customer lifetime value outcome worse.</p><p>Today, organizations try to solve this with process and coordination: roadmaps, planning, architecture governance, quarterly alignment, and a lot of “let’s align”. It works… sometimes. But it’s expensive and slow.</p><p>If individuals can operate one abstraction layer higher, it enforces that more of this coordination gets embodied in <em>systems</em>, not meetings:</p><ul><li>shared constraints,</li><li>shared feedback loops,</li><li>shared intent,</li><li>and shared operational reality.</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/744/0*lG130wSi0bNLl7gb" /></figure><p>However, this still doesn’t answer my question; as it only provides the “means”, and not the “end”.</p><p>Namely it doesn’t answer the question: <strong>What is an *<em>actual*</em> example of what could be built at this level of abstraction?</strong></p><p>The next attempt was explored through the concept of programming languages:</p><h3>From objects and functions to architectural primitives (K8s CRDs as a mental model)</h3><p>When thinking of this, the mental model I kept returning to was Kubernetes: specifically the <a href="https://kubernetes.io/docs/concepts/extend-kubernetes/operator/">Operator Pattern — also known as Custom Resource Definitions (CRD)</a>.</p><p>A CRD is not just a schema. It’s a way to define a higher-level concept as architectural components that can be instantiated, reconciled, and operationalized into sophisticated and complex applications.</p><p>In the example of KAOS we abstract the concept of an <a href="https://axsaucedo.github.io/kaos/v0.1.3/operator/overview.html">Agent, or an MCP or a ModelAPI into an architectural component with a set of attributes</a>, that reconciles into a broader set of system and application components that interact across a complex system.</p><p>So the thought experiment then becomes:</p><p>What if “programming language constructs” were CRD-level like architectural components; namely if we didn’t operate in objects but instead with dynamic architectural components via a <em>data plane</em> of concepts which reconcile into services, policies, interactions, dependencies and so on.</p><p>This is where another project I built (bit more half baked this one) gave me a different angle.</p><p>Basically, I wondered:</p><h3>What would a programming language look like if it natively operated at the next level of abstraction?</h3><p>And before someone comes here to just shout “erlang!”, I want to explore a bit of blue sky thinking with a slightly more naive exploration. Starting with the idea of a programming language where an agent can be a first-class architectural component in a specification; then what happens if we design it where:</p><ul><li>modules are agents,</li><li>control flow is agentic,</li><li>and even the interpreter loop is an agent?</li></ul><p>Not an interpreter in the classic sense (“parse -&gt; execute”), but something closer to a runtime that plans, delegates, reconciles, validates, and iterates.</p><p>That line of thinking is what led me to experiment with <a href="https://github.com/axsaucedo/agenticscript/"><strong>AgenticScript</strong></a><strong>:</strong> another weekend project — this time more half-baked — but that may help answer the bigger question.</p><p>AgenticScript was an idea to a distributed agent programming language explicitly designed around coordination: agent spawning, inter-agent communication (ask / tell), tool management, a message bus, and rich debugging via a REPL that surfaces agent status, message bus performance, and communication flows.</p><p>It was really fun (and impressive) to work through a lexer, parser and interpreter with a code copilot (+ learned there’s a whole sub-field <a href="https://en.wikipedia.org/wiki/Agent-oriented_programming">of Agent-Oriented Programming</a>). However although it wasn’t a success in regards to pursuing the project end-to-end, it was a success in a different sense as it helped progress this question.</p><p>And the answer seemed to lead to the potential of a more uncomfortable conclusion:</p><h3>A language at this next abstraction layer might not look like a programming language at all</h3><p>If we actually want a layer above today’s languages, it may not be a rigid syntax like we’re used to.</p><p>This line of thinking is closer to Andrej Karpathy’s famous 2023 phrase of <a href="https://x.com/karpathy/status/1617979122625712128">“the hottest new programming language is English”</a>. Not sure if it’s right to call this a “superset of human language”, but analogous to semi-structured Technical Design Documents (TDDs) that describe concepts, constraints, and components in ways that can compile into living artifacts that interact with broader systems.</p><p>Unsurprisingly, this is where the industry is already at the moment; large changes already require TDDs to be written before a solution is implemented.</p><p>The difference is that here “spec-driven development” would be framed as treating the specification as the source of truth and regenerating derived artifacts when it changes. And at the more radical end, there’s an explicit argument that code becomes a byproduct between requirements and system outcomes.</p><p>If you zoom out, you can see this as the latest iteration of a long lineage of code generation that has been exploring “spec/models -&gt; generated artifacts” for years. The difference now is the “runtime” that would be required to execute such a new paradigm: the interpreter isn’t just executing instructions, <strong>it’s continuously reconciling intent against reality</strong>.</p><p>Which implies something important:</p><p>If we really want that future, the execution layer will likely be <em>more complex</em> than most of the software we build today. Namely because it needs to absorb the semantic and operational burden that currently lives in teams, process, and organizational structure.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/742/0*I49qfcDeTeo2zbmD" /></figure><h3>A pragmatic cop-out (that might actually be the answer)</h3><p>I can use this as a cop-out and say: maybe this already answers the question of what could be an example of what could be built at this higher level of abstraction.</p><p>One plausible answer is:</p><ul><li>an ecosystem that itself exists at a higher abstraction layer (primitives, protocols, declarative intent),</li><li>enabling individuals (and teams) to build at that same layer,</li><li>with higher leverage in speed, integration, and reliability — iff the specs and guardrails are treated as first-class.</li></ul><p>Which brings us right back to the meme.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/856/0*d7v8IjmDVn9Ac9U4" /></figure><p>A “comprehensive and precise spec” is called code.</p><p>The shift is that we’re increasingly moving the definition of “code” upward: from functions and classes toward <strong>specifications, architectural primitives, and executable intent</strong>.</p><h3>Three final reflections from this philosophical rant</h3><h3>1) We’re heading toward an “agentic SDLC” standardization moment</h3><p>I am getting flashbacks of the early 2010s, where many of us may remember those were the days where Scrum coaches were being hired everywhere to teach us “the way” of agile (and move away from the dark side of waterfall).</p><p>Whether or not you love Scrum, the pattern is the interesting part: <strong>practice standardizes after fragmentation</strong>.</p><p>I expect something similar here:</p><ol><li>A small minority becomes visibly faster.</li><li>Organizations try to replicate the gains.</li><li>We get a proliferation of incompatible “agentic workflows.”</li><li>Then we converge into a more standardized agentic SDLC — maybe even with something like “agentic coaches” (a phrase that sounds ridiculous until it doesn’t).</li></ol><p>So it may not be too far until we start seeing agentic coaches as part of teams, pushing for day-long cycles, renaming spikes to something else (agentic runs?), and inventing another set of agentic poker cards (oh my).</p><h3>2) Developer profiles are diverging (fast)</h3><p>And we’re already seeing the developer profiles shift. At least I have noticed three broad categories:</p><ul><li>High performers becoming disproportionately more productive.</li><li>Low-discipline workflows becoming disproportionately more dangerous (generating code, not reading it, opening PRs before running it, creating more work downstream).</li><li>The middle majority watching from the sidelines, waiting for “the way we do it here” to crystallize, instead of venturing into the unknown.</li></ul><p>This last group is important. Most industries don’t shift because early adopters are excited. They shift when norms, expectations, and playbooks form.</p><h3>3) The bottleneck moves upstream: from “writing code” to judgment, constraints, and verification</h3><p>The new 10x isn’t typing faster; it’s steering better. When you can generate and integrate <em>at </em>a higher level, the differentiator becomes (a) what you choose to build, (b) how precisely you specify it, and © how aggressively you verify it.</p><p>This is also where the tech debt surprise lands me: accelerated iteration doesn’t just accelerate output, it accelerates <strong>misalignment</strong>. People have started naming the new type failure <a href="https://failingfast.io/ai-epistemic-debt/?utm_source=chatgpt.com"><em>“epistemic debt”</em></a>: shipping systems you can’t explain, defend, or reliably change (even if tests pass today).</p><p>So the agentic SDLC endgame isn’t “everyone becomes 10x”. It is: <strong>speed becomes commoditised; rigour becomes the moat.</strong> Teams that turn specs, guardrails, evals, and operational feedback loops into first-class artifacts get compounding leverage. Teams that don’t… compound confusion at machine speed.</p><p>It is now our responsibility as practitioners and leaders figure out this shift upwards in the cognitive stack; individuals can now do what teams could; teams can do what departments; and what follows should be able to invent the future. This analogy may go beyond the field of software engineering.</p><p>As they say: the future is here, it’s just not evenly distributed.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=8fbac096f579" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Production Machine Learning Monitoring: Outliers, Drift, Explainers & Statistical Performance]]></title>
            <link>https://medium.com/data-science/production-machine-learning-monitoring-outliers-drift-explainers-statistical-performance-d9b1d02ac158?source=rss-32de426f7278------2</link>
            <guid isPermaLink="false">https://medium.com/p/d9b1d02ac158</guid>
            <category><![CDATA[monitoring]]></category>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[notes-from-industry]]></category>
            <category><![CDATA[alibi]]></category>
            <category><![CDATA[mlops]]></category>
            <dc:creator><![CDATA[Alejandro Saucedo]]></dc:creator>
            <pubDate>Sun, 13 Dec 2020 15:52:48 GMT</pubDate>
            <atom:updated>2021-03-18T00:28:43.774Z</atom:updated>
            <content:encoded><![CDATA[<h4><a href="https://towardsdatascience.com/tagged/notes-from-industry">Notes from Industry</a></h4><h4>A practical deep dive on production monitoring architectures for machine learning at scale using real-time metrics, outlier detectors, drift detectors, metrics servers and explainers.</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*sUnPNsdXbzjfqKdKA3vxdQ.png" /><figcaption>Image by Author</figcaption></figure><blockquote>“The lifecycle of a machine learning model only begins once it’s in production”</blockquote><p>In this article we present an end-to-end example showcasing best practices, principles, patterns and techniques around monitoring of machine learning models in production. We will show how to adapt standard microservice monitoring techniques towards deployed machine learning models, as well as more advanced paradigms including concept drift, outlier detection and AI explainability.</p><p>We will train an image classification machine learning model from scratch, deploy it as a microservice in Kubernetes, and introduce a broad range of advanced monitoring components. The monitoring components will include outlier detectors, drift detectors, AI explainers and metrics servers — we will cover the underlying architectural patterns used for each, which are developed with scale in mind, and designed to work efficiently across hundreds or thousands of heterogeneous machine learning models.</p><p>You can also view this blog post in video form, which was presented as the keynote at the PyCon Hong Kong 2020 — the main delta is that the talk uses an Iris Sklearn model for the e2e example instead of the CIFAR10 Tensorflow model.</p><iframe src="https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Fwww.youtube.com%2Fembed%2FQcevzK9ZuDg%3Ffeature%3Doembed&amp;display_name=YouTube&amp;url=https%3A%2F%2Fwww.youtube.com%2Fwatch%3Fv%3DQcevzK9ZuDg&amp;image=https%3A%2F%2Fi.ytimg.com%2Fvi%2FQcevzK9ZuDg%2Fhqdefault.jpg&amp;key=a19fcc184b9711e1b4764040d3dc5c07&amp;type=text%2Fhtml&amp;schema=youtube" width="854" height="480" frameborder="0" scrolling="no"><a href="https://medium.com/media/41896880d5fca52904204d70b1c96031/href">https://medium.com/media/41896880d5fca52904204d70b1c96031/href</a></iframe><h3>End to end machine learning monitoring example</h3><p>In this article we present an end-to-end hands on example covering each of the high level concepts outlined in the sections below.</p><ol><li>Introduction to Monitoring Complex ML Systems</li><li>CIFAR10 Tensorflow Renset32 model training</li><li>Model Packaging &amp; Deployment</li><li>Performance monitoring</li><li>Eventing Infrastructure for Monitoring</li><li>Statistical monitoring</li><li>Outlier detection monitoring</li><li>Concept drift monitoring</li><li>Explainability monitoring</li></ol><p>We will be using the following open source frameworks in this tutorial:</p><ul><li><a href="https://github.com/tensorflow/tensorflow"><strong>Tensorflow </strong></a>— Widely used machine learning framework.</li><li><a href="https://github.com/SeldonIO/alibi"><strong>Alibi Explain</strong></a> — White-box and black-box ML model explanation library.</li><li><a href="https://github.com/SeldonIO/alibi-detect"><strong>Albi Detect </strong></a>— Advanced machine learning monitoring algorithms for concept drift, outlier detection and adversarial detection.</li><li><a href="https://github.com/SeldonIO/seldon-core/"><strong>Seldon Core</strong></a> — Machine learning deployment and orchestration of the models and monitoring components.</li></ul><p>You can find the full code for this article in the <a href="https://github.com/axsaucedo/seldon_experiments/blob/master/monitoring-talk/cifar10_example.ipynb">jupyter notebook provided</a> which will allow you to run all relevant steps throughout the model monitoring lifecycle.</p><p>Let’s get started.</p><h3>1. Introduction to Monitoring Complex ML Systems</h3><p>Monitoring of production machine learning is hard, and it becomes exponentially more complex once the number of models and advanced monitoring components grows. This is partly due to how different production machine learning systems are compared to traditional software microservice-based systems — some of these key differences are outlined below.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*P6oOUY-sNfNDH1rGMGDN5Q.png" /><figcaption>Image by Author</figcaption></figure><ul><li><strong>Specialized hardware</strong> —Optimized implementation of machine learning algorithms often require access to GPUs, larger amounts of RAM, specialised TPUs/FPGAs, and other even dynamically changing requirements. This results in the need for specific configuration to ensure this specialised hardware can produce accurate usage metrics, and more importantly that these can be linked to the respective underlying algorithms.</li><li><strong>Complex Dependency Graphs </strong>— The tools and underlying data involves complex dependencies that can span across complex graph structures. This means that the processing of a single datapoint may require stateful metric assessment across multiple hops, potentially introducing additional layers of domain-specific abstraction which may have to be taken into consideration for reliable interpretation of monitoring state.</li><li><strong>Compliance Requirements</strong> — Production systems, especially in highly regulated environments may involve complex policies around auditing, data requirements, as well as collection of resources and artifacts at each stage of execution. Sometimes the metrics that are being displayed and analysed will have to be limited to the relevant individuals, based on the specified policies, which can vary in complexity across use-cases.</li><li><strong>Reproducibility </strong>— On top of these complex technical requirements, there is a critical requirement around reproducibility of components, ensuring that the components that are run can be executed at another point with the same results. When it comes to monitoring, it is important that the systems are built with this in mind such that it’s possible to re-run particular machine learning executions to reproduce particular metrics, whether for monitoring or for auditing purposes.</li></ul><p>The anatomy of production machine learning involves a broad range of complexities that range across the multiple stages of the model’s lifecycle. This includes experimentation, scoring, hyperparameter tuning, serving, offline batch, streaming and beyond. Each of these stages involve potentially different systems with a broad range of heterogeneous tools. This is why it is key to ensure we not only learn how we are able to introduce model-specific metrics to monitor, but that we identify the higher level architectural patterns that can be used to enable the deployed models to be monitored effectively at scale. This is what we will cover in each of the sections below.</p><h3>2. CIFAR10 Tensorflow Renset32 model training</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/507/0*JT6j5KK9zAZJocDD.png" /><figcaption>Image from the open source <a href="https://www.cs.toronto.edu/~kriz/cifar.html">CIFAR10 dataset</a></figcaption></figure><p>We will be using the intuitive <a href="https://www.cs.toronto.edu/~kriz/cifar.html"><strong>CIFAR10 dataset</strong></a>. This dataset consists of images that can be classified across one of 10 classes. The model will take as an input an array of shape 32x32x3 and as output an array with 10 probabilities for which of the classes it belongs to.</p><p>We are able to load the data from the Tensorflow datasets — namely:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/f7e9d698397accda35b1e341b4f43428/href">https://medium.com/media/f7e9d698397accda35b1e341b4f43428/href</a></iframe><p>The 10 classes include: cifar_classes = [“airplane”, “automobile”, “bird”, “cat”, “deer”, “dog”, “frog”, “horse”, “ship”, “truck”].</p><p>In order for us to train and deploy our machine learning model, we will follow the traditional machine learning workflow outlined in the diagram below. We will be training a model which we will then be able to export and deploy.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Y1vB1vG70Uu39zw1mx_TMA.png" /></figure><p>We will be using Tensorflow to train this model, leveraging the <a href="https://towardsdatascience.com/an-overview-of-resnet-and-its-variants-5281e2f56035">Residual Network</a> which is arguably one of the most groundbreaking architectures as it makes it possible to train up to hundreds or even thousands of layers with good performance. For this tutorial we will be using the Resnet32 implementation, which fortunately we’ll be able to use through the utilities provided by the Alibi Detect Package.</p><p>Using my GPU this model took about 5 hours to train, luckily we will be able to use a pre-trained model which can be retrieved using the Alibi Detect fetch_tf_model utils.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/d27fcbee196c26e633bd1200d23afef2/href">https://medium.com/media/d27fcbee196c26e633bd1200d23afef2/href</a></iframe><p>If you want to still train the CIFAR10 resnet32 tensorflow model, you can use the helper utilities provided by the Alibi Detect package as outlined below, or even just import the raw Network and train it yourself.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/e2677855b9bb825dcdd8fe7b03fd2913/href">https://medium.com/media/e2677855b9bb825dcdd8fe7b03fd2913/href</a></iframe><p>We can now test the trained model on “unseen data”. We can test it using a CIFAR10 datapoint that would be classified as a truck. We can have a look at the datapoint by plotting it using Matplotlib.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/3dcfe03107831ea176b8876c04693fa4/href">https://medium.com/media/3dcfe03107831ea176b8876c04693fa4/href</a></iframe><figure><img alt="" src="https://cdn-images-1.medium.com/max/251/1*gU3zhU5xqo-NiDZ2OpUAPQ.png" /></figure><p>We can now process that datapoint through the model, which as you can imagine should be predicted as a “truck”.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/524763e72ca7715989393c6deb215898/href">https://medium.com/media/524763e72ca7715989393c6deb215898/href</a></iframe><p>We can find the class predicted by finding the index with the highest probability, which in this case it is index 9with a high probability of 99%. From the names of the classes (e.g. cifar_classes[ np.argmax( X_curr_pred )]) we can see that class 9 is “truck”.</p><h3>3. Package &amp; deploy Model</h3><p>We will be using Seldon Core for the deployment of our model into Kubernetes, which provides multiple options to convert our model into a fully fledged microservice exposing REST, GRPC and Kafka interfaces.</p><p>The options we have to deploy models with Seldon Core include 1) the <a href="https://docs.seldon.io/projects/seldon-core/en/latest/wrappers/language_wrappers.html">Language Wrappers</a> to deploy our Python, Java, R, etc code classes, or 2) the <a href="https://docs.seldon.io/projects/seldon-core/en/latest/servers/overview.html">Prepackaged Model Servers</a> to deploy model artifacts directly. In this tutorial we will be using the <a href="https://docs.seldon.io/projects/seldon-core/en/latest/servers/tensorflow.html">Tensorflow Prepackaged Model</a> server to deploy the Resnet32 model we were using earlier.</p><p>This approach will allow us to take advantage of the cloud native architecture of Kubernetes which powers large scale microservice systems through horizontally scalable infrastructure. We will be able to learn about and leverage cloud native and microservice patterns adopted to machine learning throughout this tutorial.</p><p>The diagram below summarises the options available to deploy model artifacts or the code itself, together with the abilities we have to deploy either a single model, or build complex inference graphs.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*IDs8rEaWDOq19h1jrdGT4A.png" /></figure><blockquote>As a side note, you can get set up you on Kubernetes using a development environment like <a href="https://github.com/kubernetes-sigs/kind">KIND (Kubernetes in Docker)</a> or <a href="https://github.com/kubernetes/minikube">Minikube,</a> and then following the instructions in the <a href="https://github.com/axsaucedo/seldon_experiments/blob/master/monitoring-talk/cifar10_example.ipynb">Notebook for this example</a>, or in the <a href="https://docs.seldon.io/projects/seldon-core/en/latest/workflow/install.html">Seldon documentation</a>. You will need to make sure you install Seldon with a respective ingress provider like Istio or Ambassador so you can send the REST requests.</blockquote><p>To simplify the tutorial, we have already uploaded the trained Tensorflow Resnet32 model, which can be found this public Google bucket: gs://seldon-models/tfserving/cifar10/resnet32. If you have trained your model, you are able to upload it to the bucket of your choice, which can be Google Bucket, Azure, S3 or local Minio. Specifically for Google you can do it using the gsutil command line with the command below:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/47756761685ab933676530bad3e4ad92/href">https://medium.com/media/47756761685ab933676530bad3e4ad92/href</a></iframe><p>We can deploy our model with Seldon using the custom resource definition configuration file. Below is the script that converts the model artifact into a fully fledged microservice.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/14675939f6d7524d47d97f073cc26f64/href">https://medium.com/media/14675939f6d7524d47d97f073cc26f64/href</a></iframe><p>We can now see that the model has been deployed and is currently running.</p><pre>$ kubectl get pods | grep cifar</pre><pre>cifar10-default-0-resnet32-6dc5f5777-sq765   2/2     Running   0          4m50s</pre><p>We can now test our deployed model by sending the same image of the truck, and see if we still have the same prediction.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/251/1*gU3zhU5xqo-NiDZ2OpUAPQ.png" /><figcaption>Datapoint displayed with plt.imshow(X_curr[0])</figcaption></figure><p>We will be able to do this by sending a REST request as outlined below, and then print the results.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/e2b50dee90097a650198794075cb1ec5/href">https://medium.com/media/e2b50dee90097a650198794075cb1ec5/href</a></iframe><p>The output of the code above is the JSON response of the POST request to the url that Seldon Core provides us through <a href="https://docs.seldon.io/projects/seldon-core/en/latest/ingress/istio.html">the ingress</a>. We can see that the prediction is correctly resulting in the “truck” class.</p><pre>{&#39;predictions&#39;: [[1.26448288e-06, 4.88144e-09, 1.51532642e-09, 8.49054249e-09, 5.51306611e-10, 1.16171261e-09, 5.77286274e-10, 2.88394716e-07, 0.00061489339, 0.999383569]]}<br><br>Prediction: truck</pre><h3>4. Performance Monitoring</h3><p>The first monitoring pillar we will be covering is the good old performance monitoring, which is the traditional and standard monitoring features that you would find in the microservices and infrastructure world. Of course in our case we will be adopting it towards deployed machine learning models.</p><p>Some high level principles of machine learning monitoring include:</p><ul><li><strong>Monitoring the performance of the running ML service</strong></li><li><strong>Identifying potential bottlenecks or runtime red flags</strong></li><li><strong>Debugging and diagnosing unexpected performance of ML services</strong></li></ul><p>For this we will be able to introduce the first two core frameworks which are commonly used across production systems:</p><ul><li>Elasticsearch for logs — A document key-value store that is commonly used to store the logs from containers, which can then be used to diagnose errors through stack traces or information logs. In the case of machine learning we don’t only use it to store logs but also to store pre-processed inputs and outputs of machine learning models for further processing.</li><li>Prometheus for metrics —A time-series store that is commonly used to store real-time metrics data, which can then be visusalised leveraging tools like Grafana.</li></ul><p>Seldon Core provides integration with Prometheus and Elasticsearch out of the box for any model deployed. During this tutorial we will be referencing Elasticsearch but to simplify the intuitive grasp of several advanced monitoring concepts we will be using mainly Prometheus for metrics and Grafana for the visualisations.</p><p>In the diagram below you can visualise how the exported microservice enables any containerised model to export both metrics and logs. The metrics are scraped by prometheus, and the logs are forwarded by the model into elasticsearch (which happens through the eventing infrastructure we cover in the next section. For explicitness it is worth mentioning that Seldon Core also supports Open Tracing metrics using Jaeger, which shows the latency throughout all microservice hops in the Seldon Core model graph.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*9fmkJUH87r3afi-LHzI3EQ.png" /><figcaption>Image by Author</figcaption></figure><p>Some examples of performance monitoring metrics that are exposed by Seldon Core models, and that can be also added through further integrations include:</p><ul><li><strong>Requests per second</strong></li><li><strong>Latency per request</strong></li><li><strong>CPU/memory/data utilisation</strong></li><li><strong>Custom application metrics</strong></li></ul><p>For this tutorial you can set up Prometheus and Grafana by using the <a href="https://docs.seldon.io/projects/seldon-core/en/latest/examples/metrics.html#Install-Seldon-Analytics">Seldon Core Analytics package</a> that sets everything up for metrics to be collected in real time, and then visualised on the dashboards.</p><p>We can now visualise the utilization metrics of the deployed models relative to their specific infrastructure. When deploying a model with Seldon you will have multiple attributes that you will want to take into consideration to ensure optimal processing of your models. This includes the allocated CPU, Memory and Filesystem store reserved for the application, but also the respective configuration for running processes and threads relative to the allocated resources and expected requests.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*S9Pu4INKasOgOo4pmIxnug.jpeg" /><figcaption>Image by Author</figcaption></figure><p>Similarly, we are also able to monitor the usage of the model itself — every seldon model exposes model usage metrics such as requests-per-second, latency per request, success/error codes for models, etc. These are important as they are able to map into the mode advanced/specialised concepts of the underlying machine learning model. Large latency spikes could be diagnosed and explained based on the underlying requirements of the model. Similarly errors that the model displays are abstracted into simple HTTP error codes, which allows for standardisation of advanced ML components into microservice patterns that then can be managed more easily at scale by DevOps / IT managers.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*95vtHPvYiVFmMqbiT5pZUg.jpeg" /><figcaption>Image by Author</figcaption></figure><h3>5. Eventing Infrastructure for Monitoring</h3><p>In order for us to be able to leverage the more advanced monitoring techniques, we will first introduce briefly the eventing infrastructure that allows Seldon to use advanced ML algorithms for monitoring of data asynchronously and in a scalable architecture.</p><p>Seldon Core leverages <a href="https://knative.dev/docs/eventing/">KNative Eventing</a> to enable Machine Learning models to forward the inputs and outputs of the model into the more advanced machine learning monitoring components like outlier detectors, concept drift detectors, etc.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*XC5rrUDvO9tAD3y7HmfvfA.png" /></figure><p>We will not be going into too much detail on the eventing infrastructure that KNative introduces, but if you are curious there are multiple hands on examples in the Seldon Core documentation in regards to how it leverages the KNative Eventing infrastructure to <a href="https://docs.seldon.io/projects/seldon-core/en/latest/analytics/log_level.html">forward payloads to further components</a> such as Elasticsearch, as well as how Seldon models can also be connected to <a href="https://docs.seldon.io/projects/seldon-core/en/latest/streaming/knative_eventing.html">process events</a>.</p><p>For this tutorial, we need to enable our model to forward all the payload inputs and ouputs processed by the model into the KNative Eventing broker, which will enable all other advanced monitoring components to subscribe to these events.</p><p>The code below adds a “logger” attribute to the deployment configuration which specifies the broker location.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/e89481e951f7f62d6575aa99eb19fb72/href">https://medium.com/media/e89481e951f7f62d6575aa99eb19fb72/href</a></iframe><h3>6. Statistical monitoring</h3><p>Performance metrics are useful for general monitoring of microservices, however for the specialised world of machine learning, there are widely-known and widely-used metrics that are critical throughout the lifecycle of the model beyond the training phase. More common metrics can include accuracy, precision, recall, but also metrics like <a href="https://en.wikipedia.org/wiki/Root-mean-square_deviation">RMSE</a>, <a href="https://en.wikipedia.org/wiki/Relative_entropy">KL Divergence</a>, between many many more.</p><p>The core theme of this article is not just to specify how these metrics can be calculated, as it’s not an arduous task to enable an individual microservice to expose this through some Flask-wrapper magic. The key here is to identify scalable architectural patterns that we can introduce across hundreds or thousands of models. This means that we require a level of standardisation on the interfaces and patterns that are required to map the models into their relevant infrastructure.</p><p>Some high level principles that revolve around more specialised machine learning metrics are the following:</p><ul><li><strong>Monitoring specific to statistical ML performance</strong></li><li><strong>Benchmarking multiple different models or different versions</strong></li><li><strong>Specialised for the data-type and the input/output format</strong></li><li><strong>Stateful asynchronous provisioning of “feedback” on previous requests (such as “annotations” or “corrections”)</strong></li></ul><p>Given that we have these requirements, Seldon Core introduces a set of architectural patterns that allow us to introduce the concept of “Extensible Metrics Servers”. These metric servers contain out-of-the-box ways to process data that the model processes by subscribing to the respective eventing topics, to ultimately expose metrics such as:</p><ul><li><strong>Raw metrics: True-Positives, True-Negatives, False-Positives, False-Negatives</strong></li><li><strong>Basic metrics: Accuracy, precision, recall, specificity</strong></li><li><strong>Specialised metrics: KL Divergence, RMSE, etc</strong></li><li><strong>Breakdowns per class, features, and other metadata</strong></li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*27KFq_dka2Vpz6o7H90pkw.png" /><figcaption>Image by Author</figcaption></figure><p>From an architectural perspective, this can be visualised more intuitively in the diagram above. This showcases how a single datapoint can be submitted through the model, and then processed by any respective Metric Servers. The metric servers can also process the “correct/annotated” labels once they are provided, which can be linked with the unique prediction ID that Seldon Core adds on every request. The specialised metrics are calculated and exposed by fetching the relevant data from the Elasticsearch store.</p><p>Currently, Seldon Core provides the following set of out-of-the-box Metrics Servers:</p><ul><li>BinaryClassification — Processes data in the form of binary classifications (e.g. 0 or 1) to expose raw metrics to show basic statistical metrics (accuracy, precision, recall and specificity).</li><li>MultiClassOneHot — Processes data in the form of one hot predictions for classification tasks (e.g. [0, 0, 1] or [0, 0.2, 0.8]), which can then expose raw metrics to show basic statistical metrics.</li><li>MultiClassNumeric — Processes data in the form of numeric datapoints for classification tasks (e.g. 1, or [1]), which can then expose raw metrics to show basic statistical metrics.</li></ul><p>For this example we will be able to deploy a Metric Server of the type “MulticlassOneHot” — you can see the parameters used in the summarised code below, but you can find the full YAML in the jupyter notebook.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/34cead550363c3a98b1e4de59bd6ebbb/href">https://medium.com/media/34cead550363c3a98b1e4de59bd6ebbb/href</a></iframe><p>Once we deploy our metrics server, we can now just send requests and feedback to our CIFAR10 model, through the same microservice endpoint. To simplify the workflow, we will not send asynchronous feedback (which would perform the comparisons with the elasticsearch data), but instead we’ll send “self-contained” feedback requests, which contain the inference “response” and the inference “truth”.</p><p>The following function provides us with a way to send a bunch of feedback requests to achieve an approximate accuracy percent (number of correct vs incorrect predictions) for our usecase.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/e001c8bf3d30810a081f750fc9374b22/href">https://medium.com/media/e001c8bf3d30810a081f750fc9374b22/href</a></iframe><p>Now we can first send feedback to get 90% accuracy, and then to make sure our graphs look pretty, we can send another batch request that would result in 40% accuracy.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/11daa7cc4ebfc870be048830033c7dfb/href">https://medium.com/media/11daa7cc4ebfc870be048830033c7dfb/href</a></iframe><p>This now basically gives us the ability to visualise the metrics that the MetricsServer calculates in real time.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*lBfSek4AxiFxypmH-t2pjQ.jpeg" /><figcaption>Image by Author</figcaption></figure><p>From the dashboard above we can get a high level intuition of the type of metrics that we are able to get through this architectural pattern. The stateful statistical metrics above in particular require extra metadata to be provided asynchronously, however even though the metrics themselves may have very different ways of being calculated, we can see that the infrastructural and architectural requirements can be abstracted and standardised in order for these to be approached in a more scalable way.</p><p>We will continue seeing this pattern as we delve further into the more advanced statistical monitoring techniques.</p><h3>7. Outlier detection monitoring</h3><p>For a more advanced monitoring technique, we will be leveraging the Alibi Detect library, particularly around some of the advanced outlier detector algorithms it provides. Seldon Core provides us with a way to perform the deployment of outlier detectors as an architectural pattern, but also provides us with a prepackaged server that is optimized to serve Alibi Detect outlier detector models.</p><p>Some of the key principles for outlier detection include:</p><ul><li>Detecting anomalies in data instances</li><li>Flagging / alerting when outliers take place</li><li>Identifying potential metadata that could help diagnose outliers</li><li>Enable drill down of outliers that are identified</li><li>Enabling for continuous / automated retraining of detectors</li></ul><p>In the case of outlier detectors it is especially important to allow for the calculations to be performed separate to the model, as these tend to be much heaver and may require more specialised components. An outlier detector that is deployed, may come with similar complexities to the ones from a machine learning model, so it’s important that the same concepts of compliance, governance and lineage are covered with these advanced components.</p><p>The diagram below shows how the requests are forwarded by the model using the eventing infrastructure. The outlier detector then processes the datapoint to calculate whether it’s an outlier. The component is then able to store the outlier data in the respective request entry asynchronously, or alternatively it is able to expose the metrics to prometheus, which is what we will visualise in this section.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*0RS9UfFOEgNCljY7MEkmmg.png" /><figcaption>Image by Author</figcaption></figure><p>For this example we will be using the <a href="https://docs.seldon.io/projects/alibi-detect/en/stable/examples/od_vae_cifar10.html">Alibi Detect Variational Auto Encoder</a> outlier detector technique. The outlier detector is trained on a batch of unlabeled but normal (inlier) data. The VAE detector tries to reconstruct the input it receives, if the input data cannot be reconstructed well, then it is flagged as an outlier.</p><p>Alibi Detect provides us with utilities that allow us to export the outlier detector from scratch. We can fetch it using the fetch_detector function.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/792f509ec7d82f11e5ca2c1adf7cab84/href">https://medium.com/media/792f509ec7d82f11e5ca2c1adf7cab84/href</a></iframe><p>If you want to train the outlier, you can do so by simply leveraging the OutlierVAE class together with the respective encoder and decoders.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/05fce6b3f866bbb0249df3b32b4d15b3/href">https://medium.com/media/05fce6b3f866bbb0249df3b32b4d15b3/href</a></iframe><p>To test the outlier detector we can take the same picture of the truck and see how the outlier detector behaves if noise is added to the image increasingly. We will also be able to plot it using the Alibi Detect visualisation function plot_feature_outlier_image.</p><p>We can create a set of modified images and run it through the outlier detector using the code below.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/bab05890208d780703393a587a1ca472/href">https://medium.com/media/bab05890208d780703393a587a1ca472/href</a></iframe><p>We now have an array of modified samples in the variable all_X_mask , each with an increasing amount of noise. We can now run all these 10 through the outlier detector.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/55c15089ed7ae8c410d64db868b7a355/href">https://medium.com/media/55c15089ed7ae8c410d64db868b7a355/href</a></iframe><p>When looking at the results, we can see that the first 3 were not marked as outliers, whereas the rest were marked as outliers — we can see it by printing the value print(od_preds[“data”][“is_outlier”]). Which displays the array below, where 0 is non-outliers and 1 is outliers.</p><pre>array([0, 0, 0, 1, 1, 1, 1, 1, 1, 1])</pre><p>We can now visualise how the outlier instance level score maps against the threshold, which reflects the results in the array above.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/44f99082808ae0351976d9bde4f7d3cf/href">https://medium.com/media/44f99082808ae0351976d9bde4f7d3cf/href</a></iframe><figure><img alt="" src="https://cdn-images-1.medium.com/max/392/1*nGewv7rR3dpjQk5iGIjXeQ.png" /></figure><p>Similarly we can dive deeper into the intuition of what the outlier detector score channels look like, as well as the reconstructed images, which should provide a clear picture of how its internals operate.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/0af0672fb4d248758153e0788f15c51b/href">https://medium.com/media/0af0672fb4d248758153e0788f15c51b/href</a></iframe><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*GMCd6VogQ757lx4RH5p-cQ.png" /></figure><p>We will now be able to productionise our outlier detector. We will be leveraging a similar architectural pattern to the one from the metric servers. Namely the Alibi Detect Seldon Core server, which will be listening to the inference input/output of the data. For every data point that goes through the model, the respective outlier detector will be able to process it.</p><p>The main step required will be to first ensure the outlier detector we trained above is uploaded to an object store like Google bucket. We have already uploaded it to gs://seldon-models/alibi-detect/od/OutlierVAE/cifar10, but if you wish you can upload it and use your own model.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/be25a066cede460370aff27078b8a18b/href">https://medium.com/media/be25a066cede460370aff27078b8a18b/href</a></iframe><p>Once we deploy our outlier detector, we will be able to send a bunch of requests, many which will be outliers and others that won’t be.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/6e498c645db2504657985d48f3ea55f4/href">https://medium.com/media/6e498c645db2504657985d48f3ea55f4/href</a></iframe><p>We can now visualise some outliers in the dashboard — for every data point there will be a new entry point and will include whether it would be an outlier or not.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*xH2BW1UizoZDKsX9_cfCxg.jpeg" /><figcaption>Image by Author</figcaption></figure><h3>8. Drift monitoring</h3><p>As time passes, data in real life production environments can change. Whilst this change is not drastic, it can be identified through drifts in the distribution of the data itself particularly in respect to the predicted outputs of the model.</p><p>Key principles in drift detection include:</p><ul><li>Identifying drift in data distribution, as well as drifts in the relationship between input and output data from a model</li><li>Flagging drift that is found together with the relevant datapoints where it was identified</li><li>Allowing for the ability to drill down into the data that was used to calculate the drift</li></ul><p>In the concept of drift detection we deal with further complexities when compared to the outlier detection usecase. The main one being the requirement to run each drift prediction on a batch input as opposed to a single datapoint. The diagram below shows a similar workflow to the one outlied in the outlier detector pattern, the main difference is that it keeps a tumbling or sliding window of data to perform the processing against.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*f8QKiGpZVEc9i8VwSceIsQ.png" /><figcaption>Image by Author</figcaption></figure><p>For this example we will once again be using the Alibi Detect library, which provides us with the <a href="https://docs.seldon.io/projects/alibi-detect/en/stable/examples/cd_ks_cifar10.html">Kolmogorov-Smirnov data drift detector on CIFAR-10</a>.</p><p>For this technique will be able to use the KSDrift class to create and train the drift detector, which also requires a preprocessing step which uses an “Untrained Autoencoder (UAE)”.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/4c92da2d2db2f8a1b344db456a407aec/href">https://medium.com/media/4c92da2d2db2f8a1b344db456a407aec/href</a></iframe><p>In order for us to test the outlier detector we will generate a set of detectors with corrupted data. Alibi Detect provides a great set of utilities that we can use to generate corruption/noise into images in an increasing way. In this case we will be using the following noise: [‘gaussian_noise’, ‘motion_blur’, ‘brightness’, ‘pixelate’]. These will be generated with the code below.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/71f0c17bab5d446fb28eebc10982a13d/href">https://medium.com/media/71f0c17bab5d446fb28eebc10982a13d/href</a></iframe><p>Below is one datapoint from the created corrupted dataset, which contains images with an increasing amount of corruption of the different types outlined above.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/231/1*8Ih2Yg7gvDhqZXQzzad-BQ.png" /></figure><p>We can now attempt to run a couple of datapoints to compute whether drift is detected or not. The initial batch will consist of datapoints from the original dataset.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/c3cd95d1f9663b0958f442007383d832/href">https://medium.com/media/c3cd95d1f9663b0958f442007383d832/href</a></iframe><p>This as expected outputs: Drift? No!</p><p>Similarly we can run it against the corrupted dataset.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/951c086fb075ff11effbc5487bf119c0/href">https://medium.com/media/951c086fb075ff11effbc5487bf119c0/href</a></iframe><p>And we can see that all of them are marked as drift as expected:</p><pre>Corruption type: gaussian_noise<br>Drift? Yes!<br><br>Corruption type: motion_blur<br>Drift? Yes!<br><br>Corruption type: brightness<br>Drift? Yes!<br><br>Corruption type: pixelate<br>Drift? Yes!</pre><h4>Deploy Drift Detector</h4><p>Now we can move towards deploying our drift detector following the architectural pattern provided above. Similar to the outlier detector we first have to make sure that the drift detector we trained above can be uploaded to an object store. We currently will be able to use the Google bucket that we have prepared under gs://seldon-models/alibi-detect/cd/ks/drift to perform the deployment.</p><p>This will have a similar structure, the main difference is that we will also specify the desired batch size to use for the Alibi Detect server to keep as a buffer before running against the model. In this case we select a batch size of 1000.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/dd6b5ac84aa177451a25b96edb2273db/href">https://medium.com/media/dd6b5ac84aa177451a25b96edb2273db/href</a></iframe><p>Now that we have deployerd our outlier detector, we first try sending 1000 requests from the normal dataset.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/e58d126339bc26f79a53605f16d13e11/href">https://medium.com/media/e58d126339bc26f79a53605f16d13e11/href</a></iframe><p>Next we can send the corrupted data, which would result in drift detected after sending the 10k datapoints.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/4c5a9ae541f00ddd99b209fb739fc9f2/href">https://medium.com/media/4c5a9ae541f00ddd99b209fb739fc9f2/href</a></iframe><p>We are able to visualise each of the different drift points detected in the Grafana dashboard.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*xEWWrPC3rJX1QcNOs5i2tg.jpeg" /><figcaption>Image by Author</figcaption></figure><h3>9. Explainability monitoring</h3><p>AI Explainability techniques are key to understanding the behaviour of complex black box machine learning models. There is a broad range of content that explores the different algorithmic techniques that can be used in different contexts. Our current focus in this context is to provide an intuition and a practical example of the architectural patterns that can allow for explainer components to be deployed at scale. Some key principles of model explainability include:</p><ul><li><strong>Human-interpretable insights for model behaviour</strong></li><li><strong>Introducing use-case-specific explainability capabilities</strong></li><li><strong>Identifying key metrics such as trust scores or statistical perf. thresholds</strong></li><li><strong>Enabling for use of some more complex ML techniques</strong></li></ul><p>There are a broad range of different techniques available around explainability, but it’s important to understand the high level themes around the different types of Explainers. These include:</p><ul><li><strong>Scope (local vs global)</strong></li><li><strong>Model type (black vs white box)</strong></li><li><strong>Task (classification, regression, etc)</strong></li><li><strong>Data type (tabular, images, text, etc)</strong></li><li><strong>Insight (feature attributions, counterfactuals, influential training instances, etc)</strong></li></ul><p>For explainers as interfaces, these have similarities in the data flow patterns. Namely many of them require interacting with the data that the model processes, as well as the ability to interact with the model itself — for black box techniques it includes the inputs/outputs whereas for white-box techniques it includes the internals of the models themselves.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*xxgx-byArF5id6wwxJeL7w.png" /><figcaption>Image by Author</figcaption></figure><p>From an architectural perspective, this involves primarily a separate microservice which instead of just receiving an inference request, it would be able to interact with the respective model and “reverse engineer” the model by sending the relevant data. This is shown in the diagram above, but it will become more intuitive once we dive into the example.</p><p>For the example, we will be using the Alibi Explain framework, and we will use the <a href="https://docs.seldon.io/projects/alibi/en/latest/examples/anchor_image_imagenet.html">Anchor Explanation</a> technique. This local explanation technique tells us what are the features in a particular data point with the highest predictive power.</p><p>We can simply create our Anchor explainer by specifying the structure of our dataset, together with a lambda that allows the explainer to interact with the model’s predict function.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/78b42181be1d2734311a4db0608e17fa/href">https://medium.com/media/78b42181be1d2734311a4db0608e17fa/href</a></iframe><p>We are able to identify what are the anchors in our model that would predict in this case the image of the truck.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/251/1*gU3zhU5xqo-NiDZ2OpUAPQ.png" /></figure><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/3b7e9e6aa6ed96b8b73497a8895d48e6/href">https://medium.com/media/3b7e9e6aa6ed96b8b73497a8895d48e6/href</a></iframe><p>We can visualise the anchors by displaying the output anchor of the explanation itself.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/2182b42b73a6ab1f57bf204e3764df7e/href">https://medium.com/media/2182b42b73a6ab1f57bf204e3764df7e/href</a></iframe><p>We can see that the anchors of the image include the windshield and the wheels of the truck.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/251/1*Gc7r2vw-aZZMB8NuVbossw.png" /></figure><p>Here you can see that the explainer interacts with our deployed model. When deploying the explainer we will be following the same principle but instead of using a lambda that runs the model locally, this will be a function that will call the remote model microservice.</p><p>We will follow a similar approach where we’ll just need to upload the image above to an object store bucket. Similar to the previous example, we have provided a bucket under gs://seldon-models/tfserving/cifar10/explainer-py36–0.5.2. We will now be able to deploy an explainer, which can be deployed as part of the CRD of the Seldon Deployment.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/5cfb88006e0ac7e18d4cc49955066663/href">https://medium.com/media/5cfb88006e0ac7e18d4cc49955066663/href</a></iframe><p>We can check that the explainer is running with kubectl get pods | grep cifar , which should output both running pods:</p><pre>cifar10-default-0-resnet32-6dc5f5777-sq765   2/2     Running   0          18m<br>cifar10-default-explainer-56cd6c76cd-mwjcp   1/1     Running   0          5m3s</pre><p>Similar to how we send a request to the model, we are able to send a request to the explainer path. This is where the explainer will interact with the model itself and print the reverse engineered explanation.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/f44adba6b07166f8779e28149d3ff4d0/href">https://medium.com/media/f44adba6b07166f8779e28149d3ff4d0/href</a></iframe><p>We can see that the output of the explanation is the same as the one we saw above.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/251/1*Gc7r2vw-aZZMB8NuVbossw.png" /></figure><p>Finally we can also see some of the metric related components that come out of the explainer themselves, which can then be visualised through dashboards.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/e0474d68905ba1223509f5beeb5b518c/href">https://medium.com/media/e0474d68905ba1223509f5beeb5b518c/href</a></iframe><pre>Coverage: 0.2475<br>Precision: 1.0</pre><p>Similar to the other microservice based machine learning components deployed, the explainers also can expose these and other more specialised metrics for performance or advanced monitoring.</p><h3>Closing Thoughts</h3><p>Before wrapping up, one thing to outline is the importance of abstracting these advanced machine learning concepts into standardised architectural patterns. The reason why this is crucial is primarily to enable machine learning systems for scale, but also to allow for advanced integration of components across the stack.</p><p>All the advanced architectures covered above not only are applicable across each of the advanced components, but it is also possible to enable for what we can refer to as “ensemble patterns” — that is, connecting advanced components on the outputs of other advanced components.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*7OcRdq1eHA0fQuVFm6dw_g.png" /><figcaption>Image by Author</figcaption></figure><p>It is also important to ensure there are structured and standardised architectural patterns also enable developers to provide the monitoring components, which are also advanced machine learning models, have the same level of governance, compliance and lineage required in order to manage the risk efficiently.</p><p>These patterns are continuously being refined and evolved through the Seldon Core project, and advanced state of the art algorithms on outlier detection, concept drift, explainability, etc are improving continuously — if you are interested on furthering the discussion, please feel free to reach out. All the examples in this tutorial are open source, so suggestions are greatly appreciated.</p><p>If you are interested in further hands on examples of scalable deployment strategies of machine learning models, you can check out:</p><ul><li><a href="https://docs.seldon.io/projects/seldon-core/en/latest/examples/argo_workflows_batch.html">Batch Processing with Argo Workflows</a></li><li><a href="https://docs.seldon.io/projects/seldon-core/en/latest/streaming/knative_eventing.html">Serverless eventing with Knative</a></li><li><a href="https://docs.seldon.io/projects/seldon-core/en/latest/analytics/explainers.html">AI Explainability Patterns with Alibi</a></li><li><a href="https://docs.seldon.io/projects/seldon-core/en/latest/examples/sklearn_spacy_text_classifier_example.html">Seldon Model Containerisation Notebook</a></li><li><a href="https://github.com/SeldonIO/seldon-core/blob/master/examples/kafka/sklearn_spacy/README.ipynb">Kafka Seldon Core Stream Processing Deployment Notebook</a></li></ul><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=d9b1d02ac158" width="1" height="1" alt=""><hr><p><a href="https://medium.com/data-science/production-machine-learning-monitoring-outliers-drift-explainers-statistical-performance-d9b1d02ac158">Production Machine Learning Monitoring: Outliers, Drift, Explainers &amp; Statistical Performance</a> was originally published in <a href="https://medium.com/data-science">TDS Archive</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Beyond CUDA: GPU Accelerated Python for Machine Learning on Cross-Vendor Graphics Cards Made Simple]]></title>
            <link>https://medium.com/data-science/beyond-cuda-gpu-accelerated-python-for-machine-learning-in-cross-vendor-graphics-cards-made-simple-6cc828a45cc3?source=rss-32de426f7278------2</link>
            <guid isPermaLink="false">https://medium.com/p/6cc828a45cc3</guid>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[deep-learning]]></category>
            <category><![CDATA[gpu]]></category>
            <category><![CDATA[python]]></category>
            <category><![CDATA[kompute]]></category>
            <dc:creator><![CDATA[Alejandro Saucedo]]></dc:creator>
            <pubDate>Fri, 13 Nov 2020 00:05:14 GMT</pubDate>
            <atom:updated>2021-08-21T12:53:13.599Z</atom:updated>
            <content:encoded><![CDATA[<h4>A practical deep dive into GPU Accelerated Python on cross-vendor graphics cards (AMD, Qualcomm, NVIDIA &amp; friends) building machine learning algorithms using the Kompute Python Framework and the Vulkan SDK</h4><iframe src="https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Fwww.youtube.com%2Fembed%2FAJRyZ09IUdg%3Fstart%3D662%26feature%3Doembed%26start%3D662&amp;display_name=YouTube&amp;url=https%3A%2F%2Fwww.youtube.com%2Fwatch%3Fv%3DAJRyZ09IUdg&amp;image=https%3A%2F%2Fi.ytimg.com%2Fvi%2FAJRyZ09IUdg%2Fhqdefault.jpg&amp;key=a19fcc184b9711e1b4764040d3dc5c07&amp;type=text%2Fhtml&amp;schema=youtube" width="854" height="480" frameborder="0" scrolling="no"><a href="https://medium.com/media/debc0e0f7db38f064bd3e1c9cf51f59e/href">https://medium.com/media/debc0e0f7db38f064bd3e1c9cf51f59e/href</a></iframe><p>Machine learning algorithms — together with many other advanced data processing paradigms — fit incredibly well to the parallel-architecture that GPU computing offers. This has driven massive growth in the advancement and adoption of graphics cards for accelerated computing in recent years. This has also driven exciting research around techniques that optimize towards concurrency, such as <a href="https://mxnet.apache.org/versions/1.7/api/faq/model_parallel_lstm.html">model parallelism</a> and <a href="https://en.wikipedia.org/wiki/Data_parallelism">data parallelism</a>.</p><p>In this article you’ll learn how to write your own GPU accelerated algorithms in Python, which you will be able to run on virtually any GPU hardware — including non-NVIDIA GPUs. We’ll introduce core concepts and show how you can get started with the <a href="https://github.com/axsaucedo/vulkan-kompute#vulkan-kompute"><strong>Kompute Python framework</strong></a><strong> with </strong>only a handful of lines of code.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*EZ27EB0hMtEdgm0d" /><figcaption><strong>Ben-Nun, Tal, and Torsten Hoefler. “Demystifying parallel and distributed deep learning: An in-depth concurrency analysis.” <em>ACM Computing Surveys (CSUR)</em> 52.4 (2019): 1–43.</strong></figcaption></figure><p>First we will be building a simple GPU Accelerated Python script that will multiply two arrays in parallel which this will introduce the fundamentals of GPU processing. We will then write a Logistic Regression algorithm from scratch on the GPU. Below are the core topics that we will cover, together with the respective resource links:</p><ol><li><a href="https://pypi.org/project/kp/">Kompute python package</a> Installation</li><li>GPU Accelerated <a href="https://github.com/EthicalML/vulkan-kompute/blob/master/python/test/test_array_multiplication.py">Array Multiplication Example</a> in Python</li><li>GPU Accelerated <a href="https://github.com/EthicalML/vulkan-kompute/blob/master/python/test/test_logistic_regression.py">Logistic Regression Example</a> in Python</li></ol><p>Only <strong>basic programming experience</strong> is required for anyone reading this article, no knowledge of GPU computing is required. You can find the full code in the <a href="https://github.com/EthicalML/vulkan-kompute"><strong>main repository</strong>,</a> and we also created an online Google Colab Notebook where you’ll be able to run the example with a GPU for free — you can find it in <a href="https://github.com/EthicalML/vulkan-kompute/tree/master/examples/python"><strong>this link</strong></a>.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*LBLVNwHYnCmZ6IDZ.jpg" /><figcaption>Google Colab Notebook with Examples for GPU</figcaption></figure><h3>Enter Kompute &amp; the Vulkan SDK</h3><p>There are two parts to the Python framework we will be using today, both which are in the name itself —the <strong>Vulkan SDK</strong>, and <strong>Kompute</strong>.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/512/0*mzEbYNdCJ129eSz3" /><figcaption>Playing “where’s waldo” with Khronos Membership (Image by Vincent Hindriksen via <a href="https://streamhpc.com/blog/2017-05-04/what-is-khronos-as-of-today/">StreamHPC</a>)</figcaption></figure><p>The <strong>Vulkan </strong>SDK is an Open Source project led by the <a href="https://www.khronos.org/">Khronos Group</a>, a consortium consisting of numerous tech companies that have come together to work towards defining and advancing the open standards for mobile and desktop media (and compute) technologies.</p><p>A large number of high profile (and new) machine learning frameworks such as Google’s <a href="https://github.com/tensorflow/tensorflow">Tensorflow</a>, Facebook’s <a href="https://github.com/pytorch/pytorch">Pytorch</a>, Tencent’s <a href="https://github.com/Tencent/ncnn">NCNN</a>, Alibaba’s <a href="https://github.com/alibaba/MNN">MNN </a>—between others — have been adopting Vulkan as their core cross-vendor GPU computing SDK. This is primarily to enable the frameworks for cross platform and cross vendor graphics card support.</p><p>As you can imagine, the Vulkan SDK provides very low-level C / C++ access to GPUs, which allows for very specialized optimizations. This is a great asset for GPU computing— the main disadvantage is the verbosity involved, requiring 500–2000+ lines of C++ code to only get the base boilerplate required to even start writing the application logic. This can result in expensive developer cycles and errors that can lead to larger problems. This was one of the main motivations for us to start the <strong>Kompute </strong>project.</p><p><a href="https://github.com/EthicalML/vulkan-kompute#vulkan-kompute"><strong>The Kompute Python package</strong></a> is built on top of the Vulkan SDK through optimized C++ bindings, which exposes Vulkan’s core computing capabilities. Kompute is the Python <a href="https://en.wikipedia.org/wiki/General-purpose_computing_on_graphics_processing_units">GPGPU framework</a> that we will be using in this tutorial to build the GPU Accelerated machine learning algorithms.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*hkFiBPfbdHQqhHKA.png" /><figcaption>Kompute <a href="https://ethicalml.github.io/vulkan-kompute/">Documentation</a> (Image by Author)</figcaption></figure><h3>Installing the Python Kompute Package</h3><p>In order for us to start using the Kompute Python Package we will need to install its required dependencies. The package is available in Pypi, which means we can install it with pip install. You will however require the following key components installed on your machine before being able to use it:</p><ul><li>CMAKE v3.41+ (install in <a href="https://tulip.labri.fr/TulipDrupal/?q=node/1081">Windows</a>, <a href="https://vitux.com/how-to-install-cmake-on-ubuntu-18-04/">Linux (Ubuntu)</a>, <a href="https://stackoverflow.com/a/59825656/1889253">Mac</a>)</li><li>Vulkan SDK installed via <a href="https://vulkan.lunarg.com/sdk/home">official website</a></li><li>C++ compiler (eg. gcc for linux / mac, MSVC for Windows)</li></ul><p>Once you have these dependencies installed, you can simply run:</p><ul><li>pip install kp==0.5.1</li></ul><p>You should now see a success message confirming that the Kompute Python package has been installed. You can try it out yourself in the Google Colab Notebook <a href="https://github.com/EthicalML/vulkan-kompute/tree/master/examples/python">provided in the repository</a>, which you can set up with a GPU.</p><h3>Writing your first Kompute: GPU Multiplication</h3><p>To build our first simple array-multiplication GPU computing application using Kompute, we will write a simple python program that will do the following:</p><ol><li>Create a Kompute Manager (selects device 0 by default)</li><li>Create Kompute Tensors to hold data (two input one output)</li><li>Initialise the Kompute Tensors in the GPU</li><li>Define the code to run on the GPU</li><li>Dispatch GPU shader execution against Kompute Tensors</li><li>Use Kompute Operation to map GPU output data into local Tensors</li><li>Print your results</li></ol><p>The full Python code required is quite minimal, so we are able to show the full script below. We’ll break down each of the sections in more detail.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/ab5a6d276be3659d9009e2c5bfed5240/href">https://medium.com/media/ab5a6d276be3659d9009e2c5bfed5240/href</a></iframe><h4>1. Create a Kompute Manager (selects device 0 by default)</h4><p>First, we’ll create our Kompute Manager, which is in charge of creating and managing all the underlying Vulkan resources.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/7d05f8d7e3a78fe300e939c01e09088f/href">https://medium.com/media/7d05f8d7e3a78fe300e939c01e09088f/href</a></iframe><p>As you can see, here we are initializing our Kompute Manager, which by default creates all the base Vulkan resources on Device 0 (in my case it’s an NVIDIA card, and Device 1 is my integrated graphics card). For more advanced use-cases it’s also possible to provide the underlying GPU queues that you’d like to load — in <a href="https://towardsdatascience.com/parallelizing-heavy-gpu-workloads-via-multi-queue-operations-50a38b15a1dc">this other tutorial</a> we show how this can lead to significant speedups, but this is outside of scope of this article.</p><h4>2. Create Kompute Tensors to hold data (two input one output)</h4><p>We will now create the Kompute Tensors that will be used for input and output. These will hold the data required which will be mapped into the GPU to perform this simple multiplication.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/472f93cf2151644f512147d890c7f888/href">https://medium.com/media/472f93cf2151644f512147d890c7f888/href</a></iframe><p>When the tensors are created, the data is only initialized in the local CPU memory (aka RAM), but in order to use it in the GPU we’ll have to map the data into the GPU memory.</p><h4>3. Initialise the Kompute Tensors in the GPU</h4><p>Now that we have our Tensors created with local data, we will map the data into the GPU. For this we will use the eval_tensor_create_def, which will initialize the underlying Vulkan buffer and GPU memory, and perform the respective mapping into the GPU.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/7015cf29150a53f0075b3b4ff4dafd0f/href">https://medium.com/media/7015cf29150a53f0075b3b4ff4dafd0f/href</a></iframe><h4>4. Define the code to run on the GPU</h4><p>Now that we’ve initialized the necessary Kompute Tensor components and they are mapped in GPU memory, we can add the Kompute Algorithm that will be executed in the GPU. This is referred to as the “shader” code, which we build using the pyshader library. You can see the full shader code below, and we’ll break down each of the section below.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/42d4c325f9bb903b5e5d35208e6be029/href">https://medium.com/media/42d4c325f9bb903b5e5d35208e6be029/href</a></iframe><p>The GPU shader code can be defined as a Python function with the decorator @ps.python2shader , and the parameters in this case include the variables that we’ll be using. This includes the Tensor inputs and outputs that we’ll be processing — the parameter format is the following:</p><ul><li>&lt;param≥=(“&lt;memory&gt;”, &lt;binding&gt;, &lt;type&gt;, ...)</li></ul><p>In this case we are using Tensors with float values, which inherently would be equivalent to the ps.Array value, with ps.f32 float values as elements.</p><p>The first parameter index is of type GlobalInvocationId , and provides the shader with the current index location in the execution GPU dispatch structure. This is what allows us to know what index in the parallel execution loop we are currently running, which is what we extract from the component i = index.x — the reason why here we select x is because the execution index can be defined as a vec3 component, where there would be execution indices for inedx.x , index.y and index.z .</p><p>The final component is the actual equation used, which in this case is a simple multiplication of the first and second parameter, and stored in the output (third) parameter.</p><h4>5. Dispatch GPU shader execution against Kompute Tensors</h4><p>In order to run the shader above we will use the eval_algo_data_def function. The parameters required for this Kompute Operation includes the Tensors to bind into the GPU instructions, as well as the GPU shader code that we defined in the Python function above.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/734c69d5177716d245069ccfd171aac7/href">https://medium.com/media/734c69d5177716d245069ccfd171aac7/href</a></iframe><p>It’s worth mentioning that Kompute allows the user to also pass the shader as a raw glsl string, or alternatively a file path to a SPIR-V binary or raw glsl/hlsl file. For context, <a href="https://www.khronos.org/opengl/wiki/SPIR-V">SPIR-V is the intermediate representation</a> that GPUs can use to process relevant operations.</p><h4>6. Use Kompute Operation to map GPU output data into local Tensors</h4><p>Once the algorithm runs successfully, the result data will now be we held in the GPU memory of our output tensor. We can now use the function eval_tensor_sync_local_def to sync the Tensor GPU memory into the local tensor.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/e250290d25e5c92338d158f5ea247cc5/href">https://medium.com/media/e250290d25e5c92338d158f5ea247cc5/href</a></iframe><h4>7. Print your results</h4><p>Finally, we can print the output data of our tensor.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/b038f121cc43b6c1e7426d28b0faaa0d/href">https://medium.com/media/b038f121cc43b6c1e7426d28b0faaa0d/href</a></iframe><p>When you run this, you will see the values of your output tensor printed. That’s it, you’ve written your first Kompute!</p><p>Although it may not seem obvious, the above introduced some intuition around core concepts and design thinking in GPU computing, whilst still abstracting a couple of the more in-depth concepts. In the following sections we will be providing more concrete terminology and at the end we’ll also outline a set of articles to dive into if you’re interested to learn more.</p><h3>Diving into the Machine Learning intuition</h3><p>Now we’ll look into the more advanced GPU compute use-case, specifically implementing the “hello world of machine learning”: <strong>logistic regression</strong>. Before we cover the implementation we will provide some intuition on the concepts and the terminology that we’ll be using throughout the following sections.</p><p>In machine learning we always have two stages, training and inference. In the diagram below you can see the two simplified flows. At the top is the training flow, where you identify some data, extract some features, and train a model until you are happy with the accuracy. Once you have a trained model, you persist the model “weights” and deploy the model into the second workflow, where the model would perform inference on unseen data.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/791/0*qjzd2Q5vinPtO5c6" /><figcaption>Data Science Process (Image by Author)</figcaption></figure><p>In this case we will have an input dataset X , where each element is a pair xi and xj . Our input data will be the following:</p><ul><li>xi = { 0, 1, 1, 1, 1 }</li><li>xj = { 0, 0, 0, 1, 1 }</li></ul><p>With this input data, the expected target value Y to be predicted will be the following:</p><ul><li>Y = {0, 0, 0, 1, 1}</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/485/1*C9Sl7wor6-pydyD7NXcC5Q.png" /><figcaption>Logistic Regression Example from <a href="https://www.datasciencecentral.com/profiles/blogs/why-logistic-regression-should-be-the-last-thing-you-learn-when-b">DS Central</a></figcaption></figure><p>Our core objective in machine learning is to learn using this training data to find the function (and parameters) that will allow us to predict values Y from new “previously unseen” inputs.</p><p>It’s worth noting that the predicted values will be defined as ŷ , which are specifically the values computed with our “prediction” function, distinct to the “true” or “actual” values of Y that we defined above.</p><p>The functions that we will be using for logistic regression will be the following:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/102/0*4FrI-KB2_9n7y0Go.png" /></figure><p>Let’s break down this function:</p><ul><li>z — is our linear mapping function</li><li>ŷ —is the resulting predicted outputs</li><li>Xᵀ —Transpose of the matrix of vectors we’ll represent as x_i and x_j</li><li>σ — The sigmoid function which is covered in more detail below</li></ul><p>And the parameters that we’ll be looking to learn with our machine learning algorithm are:</p><ul><li>w— The weights that will be applied to the inputs</li><li>b — The bias that will be added</li></ul><p>There is also the surrounding function σ which is the sigmoid function. This function forces our input to be closer to 0 or 1, which could be intuitively seen as the probability of our prediction to be “true” or “false”, and is defined as following:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/117/0*B4vJKaqNBrPgeAG_.png" /></figure><p>This is now the prediction/inference function that will allow us to process predictions from new data points. If we say for example that we have a new unseen set of inputs X = { (0, 1) }, and we assume that the learned parameters were W = (1, 1), b = 0 after running our machine learning algorithm through our training data (which we’ll do later on), then we’ll be able to run this through our prediction function by substituting the values as follows:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/463/0*vPB55aWa874FYfwa.png" /></figure><p>In this case the prediction is 0.73..., which would be a positive prediction. This of course is just to demonstrate what our inference function will look like once we learn the parameters W and b.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/368/1*e88JKNWAFok3vpjeuPfHig.gif" /><figcaption>Gradient descent visualized from <a href="https://mi-academy.com/2018/10/04/the-history-of-gradient-descent/">ML Academy</a></figcaption></figure><p>The way that we will be learning the parameters is by performing a prediction, calculating the error, and then re-adjusting the weights accordingly. The method used to “re-adjust” the weights based on the “prediction error” will be done by leveraging gradient descent. This will be repeated multiple times to find more accurate parameters.</p><p>For this we will need to use the derivatives of each of the formulas. The first one, which is the derivative of our linear mapping function z is using the partial derivatives of the variables w, z and b. First, the partial derivative ∂z:</p><ul><li>∂z = z(X) — y</li></ul><p>Where the variables are defined as follows:</p><ul><li>∂z — The partial derivative of the linear mapping function z(x)</li><li>z(X) — the result of the linear mapping function applied to input x</li><li>y — the actual value label expected for that input x</li></ul><p>Similarly the derivatives for w and b respectively are the following:</p><ul><li>∂w = (X — ∂z)/m</li><li>∂b = ∂z/m</li></ul><p>In this case m is the total number of input elements.</p><p>We will now be able to re-adjust the parameters using the above as follows:</p><ul><li>w = w — θ · ∂w</li><li>b = b — θ · ∂b</li></ul><p>In this case θ is the learning rate, which as the name suggests controls the ratio by which the parameters will be modified on each iteration. Intuitively, the smaller, the more iterations it will be required for the algorithm to converge, however if the learning rate is too big, it will overshoot, leading to never being able to converge (from the image above you can imagine it will keep bouncing from side to side never reaching the bottom).</p><p>In order for us to calculate loss, we will be using the log loss function, known also as cross-entropy loss function. This function is defined as follows:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/207/0*TgKVkmzh9rpbnlaD.png" /><figcaption>Log loss (cross entropy loss) function</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*SQh95-Fn58JTDgzX.png" /><figcaption>Intuitive diagram to visualize cost function <a href="https://machinelearningmastery.com/how-to-score-probability-predictions-in-python/">from ML Mastery</a></figcaption></figure><p>The function itself is set up such that the larger the difference between the predicted class and the expected class, the larger the error (you can see how much it punishes if the predicted class is on the complete different label).</p><p>The loss function will provide us an idea of the improvement of our algorithm across iterations.</p><p>Finally, one of the most important points here will be the intuition behind how we can leverage the parallel architecture of the GPU to optimize computation. In this case, we’ll be able to do it by processing multiple input parameters at the same time, referred to as a micro-batch, and then re-adjusting the parameters in batch. This is known as data-parallelization, and is one of many techniques available. In the next section we will see how this is implemented, namely passing a mini-batch of inputs, storing the weights, and then re-adjusting them before the next iteration.</p><blockquote>Note: In this post we won’t delve into much detail, nor best practices on machine learning, however at the end of the article we will be listing a broad range of sources for people interested to take their machine learning (or GPU compute) knowledge to the next level.</blockquote><p>Now that we have covered some of the core concepts, we will be able to learn about the implementation.</p><h3>Machine Learning GPU Shader Implementation</h3><p>First we will start with the GPU compute shader, which is the code that will be executed in the GPU. The full shader is outlined below, and we’ll be breaking down each section in detail to explain what each part is doing.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/e9c2398539fb8d6f0518f4d7570cd408/href">https://medium.com/media/e9c2398539fb8d6f0518f4d7570cd408/href</a></iframe><h4>1. Define input and output parameters</h4><p>First we define all input parameters that are analogous to the input and output components we mentioned in the previous sections.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/8122fea00f08a6a322c8c491c49bad31/href">https://medium.com/media/8122fea00f08a6a322c8c491c49bad31/href</a></iframe><p>If you remember, at the end of the last section we mentioned how we will be leveraging the concept of micro-batches in order to use the parallel architecture of GPU processing. What this means in practice, is that we will be passing multiple instances of X to the GPU to process at a time, instead of expecting the GPU to process it one by one. This is why we see that above we have an array for xi, xj, y, wOuti, wOutj, andbOut respectively.</p><p>In more detail:</p><ul><li>The input X as arrays x_i and x_j will hold the micro-batch of inputs</li><li>The array y will hold all the expected labels for micro-batch inputs</li><li>The two input weight parameters w_in_i and w_out_j will be used for calculating predictions</li><li>The input parameter b which will be used for calculating the predictions</li><li>The output weights w_out_i and w_out_j contains weights and will store the derivative of W for all micro-batches that should be subtracted</li><li>Similarly the output bias array contains the derivatives of b for all micro-batches that should be subtracted in batch</li><li>Finally l_out contains the output array where losses will be returned</li></ul><h4>2. Define the size of the input buffers as M</h4><p>We also receive the constant M, which will be the total number of elements — if you remember this parameter will be used for the calculation of the derivatives. We will also see how these parameters are actually passed into the shader from the Python Kompute side.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/2458e850b1d5719c28ccc48cc0e47f18/href">https://medium.com/media/2458e850b1d5719c28ccc48cc0e47f18/href</a></iframe><p>Now that we have all the input and output parameters defined, we can start defining the core logic, which will contain the implementation of our machine learning training algorithm.</p><h4>3. Keep track of the execution index</h4><p>We will need to keep track of the current index of the global invocation. Since the GPU executes in parallel, each of these runs will be running directly in parallel, so this allows the current execution to consistently keep track of what iteration index is currently being executed.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/83f602c94f09cf72a325279a8d873811/href">https://medium.com/media/83f602c94f09cf72a325279a8d873811/href</a></iframe><h4>4. Define the variables from the input parameters</h4><p>We now can start preparing all the variables that we’ll be using throughout the algorithms. All our inputs are buffer arrays, so we’ll want to store them in vec2 and float32 variables.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/343840b8afcdedb1877e8aae3360c665/href">https://medium.com/media/343840b8afcdedb1877e8aae3360c665/href</a></iframe><p>In this case we’re basically making explicit the variables that are being used for the current “thread run”. The GPU architecture consists of slightly more nuanced execution structures that involve thread blocks, memory access limitations, etc — however we won’t be covering these in this article.</p><p>Now we get into the more fun part — implementing the inference / predict logic. Below we will implement the inference logic to calculate ŷ, which involves both the linear mapping function, as well as the sigmoid function which we defined above.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/93ee49715feac4578ee1754f9b734609/href">https://medium.com/media/93ee49715feac4578ee1754f9b734609/href</a></iframe><h4>5. Calculate derivatives to “re-adjust” parameters</h4><p>Now that we have y_hat, we can now use it to calculate the derivatives (∂z, ∂w and ∂b), which in this case are the derivative of the currently-executed index input element.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/dfb0e9e252ff624687c33725a5195e30/href">https://medium.com/media/dfb0e9e252ff624687c33725a5195e30/href</a></iframe><h4>6. Calculate the loss from the current iteration</h4><p>Using the expected prediction output and the calculated prediction output we are now able to compute the loss for the current iteration. As covered above, we are using the log loss (cross entropy) function to calculate the loss.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/b2bc9edef3673de54f392b22d1e60588/href">https://medium.com/media/b2bc9edef3673de54f392b22d1e60588/href</a></iframe><h4>7. Store the data on the output parameters</h4><p>Finally we are able to pass all respective calculated metrics to our output buffers. This will allow us to re-adjust for the next iteration.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/5cc70de57c315519821d51cf824aa50c/href">https://medium.com/media/5cc70de57c315519821d51cf824aa50c/href</a></iframe><p>We’ve now finished the shader that will enable us to train a Logistic Regression algorithm in the GPU —we will now cover the rest of the logic that will call this shader and orchestrate the machine learning training and inference. The full script is outlined below, and you can also try it in the Google Colab notebook with a GPU.</p><h3>Machine Learning Orchestration from Kompute</h3><p>We will be using a few more advanced components from Kompute, which can be more intuitively visualised in the diagram below.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/462/0*Env2QAL1QVnE-XeR.jpg" /><figcaption>Kompute <a href="https://ethicalml.github.io/vulkan-kompute/overview/reference.html">Architecture Design</a> (Image by Author)</figcaption></figure><p>At the core of Kompute are Kompute “Sequences” and “Operations”, which are used for GPU actions. A Kompute Section can record and execute a batch of Kompute Operations for more efficient processing. In this example we will be leveraging Sequences to manage more efficient execution of the machine learning processing.</p><p>Similar to the example above, we will be will setting up the following steps:</p><ol><li>Create Kompute Manager with the device explicitly defined</li><li>Create all the Kompute Tensors required</li><li>Execute the Kompute Tensor GPU initialization via Kompute Manager</li><li>Create Kompute Sequence and record operations for execution</li><li>Iterate 100 times: Run micro-batch execution &amp; update weights</li><li>Print resulting parameters to use for future inference</li></ol><p>As you can see this is more involved than the simpler example we used above. In this case we will use the Kompute Sequence instead of the Kompute Manager directly, as we want to have deeper control on the commands that can be recorded to send in batch to the GPU. We will discuss this in more detail as we cover each of the steps. Let’s get started.</p><h4>1. Create Kompute Manager with the device explicitly defined</h4><p>We will be creating the Kompute Manager with the device 0 explicitly defined — you can define another device as required.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/7043a94edaa1323eba8128169d565b28/href">https://medium.com/media/7043a94edaa1323eba8128169d565b28/href</a></iframe><h4>2. Create all the Kompute Tensors required</h4><p>Now we’ll be creating all the tensors required. In this sub-section you will notice that we will be referencing all the buffers/arrays that are being used in the shader. We’ll also cover how the order in the parameters passed relates to the way data is bound into the shaders so it’s accessible.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/db4c3579aaa0c3c48f0f0357d5fa5966/href">https://medium.com/media/db4c3579aaa0c3c48f0f0357d5fa5966/href</a></iframe><p>We also store them in a list params for easier access:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/ae0ff74fb7de4a2faba7ff5eade32cda/href">https://medium.com/media/ae0ff74fb7de4a2faba7ff5eade32cda/href</a></iframe><h4>3. Execute the Kompute Tensor GPU initialization via Kompute Manager</h4><p>The Kompute Tensor initialisation is quite standard so we’ll be able to do this step directly through the manager as we did in the simple array multiplication example previously.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/eadeab4d3aa422f1cadae8e1e5c4c431/href">https://medium.com/media/eadeab4d3aa422f1cadae8e1e5c4c431/href</a></iframe><h4>4. Create Kompute Sequence and record operations for execution</h4><p>In this section we will want to clear the previous recordings of the Kompute Sequence and begin recording a set of sequences. You will notice that unlike the previous section, in this case we won’t be running the eval() straight away as we’ll have to first record the operations.</p><p>You will also notice that we will be recording three types of Kompute Operations through separate functions:</p><ul><li>record_tensor_sync_device(...) — This operation ensures that the Tensors are synchronized with their GPU memory by mapping their local data into the GPU data. In this case, these Tensors use Device-only memory for processing efficiency, so the mapping is performed with a staging Tensor inside the operation (which is re-used throughout the operations for efficiency). Here we’re only wanting to sync the input weights, as these will be updated locally with the respective derivatives.</li><li>record_algo_base_data(...) — This is the Kompute Operation that binds the shader that we wrote above with all the local CPU/host resources. This includes making available the Tensors. It’s worth mentioning that the index of the tensors provided as parameters is the order in which they are mapped in the shaders via their respective bindings.</li><li>record_tensor_sync_local(...)— This Kompute Operation performs a similar set of instructions as the sync operation above, but instead of copying the data to the GPU memory, it does the converse. This Kompute Operation maps the data in the GPU memory into the local Tensor vector so it’s accessible from the GPU/host. As you can see we’re only running this operation in the output tensors.</li></ul><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/5123f3c7cb552c62e91d63ebe9fe071c/href">https://medium.com/media/5123f3c7cb552c62e91d63ebe9fe071c/href</a></iframe><h4>5. Iterate 100 times: Run micro-batch execution &amp; update weights</h4><p>Now that we have the command recorded, we can start running executions of these pre-loaded commands. In this case, we will be running the execution of a micro-batch iteration, followed by updating the parameters locally, so they are used in the following iteration.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/ed11dfee59309b27af815b21df0681ee/href">https://medium.com/media/ed11dfee59309b27af815b21df0681ee/href</a></iframe><h4>7. Print resulting parameters to use for future inference</h4><p>We now have a trained logistic regression model, or at least we’ve been able to optimize its respective function to identify suitable parameters. We are now able to print these parameters and use the parameters for inference in unseen datasets.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/77c3c611d9ecc9aa2565742074207c51/href">https://medium.com/media/77c3c611d9ecc9aa2565742074207c51/href</a></iframe><p>And we’re done!</p><p>You are able to find this entire example in the example repository, which you’ll be able to run and extend.</p><h3>What next?</h3><p>Congratulations, you’ve made it all the way to the end! Although there was a broad range of topics covered in this post, there is a massive amount of concepts that were skimmed through. These include the underlying Vulkan concepts, GPU computing fundamentals, machine learning best practices, and more advanced Kompute concepts. Luckily, there are a broad range of resources online to expand your knowledge on each of these. Some links I recommend as further reading include the following:</p><ul><li><a href="https://kompute.cc/">Kompute Documentation</a> for more details and further examples</li><li><a href="https://ethical.institute/mle.html">The Machine Learning Engineer Newsletter</a> if you want to keep updated on articles around Machine Learning</li><li><a href="https://github.com/EthicalML/awesome-production-machine-learning/">Awesome Production Machine Learning</a> list for open source tools to deploy, monitor, version and scale your machine learning</li><li><a href="https://www.fast.ai/2018/09/26/ml-launch/">Introduction to ML for Coders course</a> by FastAI to learn further machine learning concepts</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/800/1*OeK5rt6Taw51IDAFNT2I-Q.gif" /><figcaption>Image by Author</figcaption></figure><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=6cc828a45cc3" width="1" height="1" alt=""><hr><p><a href="https://medium.com/data-science/beyond-cuda-gpu-accelerated-python-for-machine-learning-in-cross-vendor-graphics-cards-made-simple-6cc828a45cc3">Beyond CUDA: GPU Accelerated Python for Machine Learning on Cross-Vendor Graphics Cards Made Simple</a> was originally published in <a href="https://medium.com/data-science">TDS Archive</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Parallelizing GPU-intensive Workloads via Multi-Queue Operations]]></title>
            <link>https://medium.com/data-science/parallelizing-heavy-gpu-workloads-via-multi-queue-operations-50a38b15a1dc?source=rss-32de426f7278------2</link>
            <guid isPermaLink="false">https://medium.com/p/50a38b15a1dc</guid>
            <category><![CDATA[parallel-computing]]></category>
            <category><![CDATA[deep-learning]]></category>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[vulkan]]></category>
            <category><![CDATA[gpu]]></category>
            <dc:creator><![CDATA[Alejandro Saucedo]]></dc:creator>
            <pubDate>Sun, 18 Oct 2020 18:01:07 GMT</pubDate>
            <atom:updated>2021-08-21T12:54:45.334Z</atom:updated>
            <content:encoded><![CDATA[<h3>Parallelizing GPU-intensive Workloads via Multi-Queue Operations using Kompute &amp; Vulkan</h3><h4>Achieving 2x+ performance improvements on GPU-intensive workloads by leveraging multi-queue operation parallelism using Kompute and the Vulkan SDK</h4><iframe src="https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Fwww.youtube.com%2Fembed%2FXz4fiQNmGSA%3Fstart%3D813%26feature%3Doembed%26start%3D813&amp;display_name=YouTube&amp;url=https%3A%2F%2Fwww.youtube.com%2Fwatch%3Fv%3DXz4fiQNmGSA&amp;image=https%3A%2F%2Fi.ytimg.com%2Fvi%2FXz4fiQNmGSA%2Fhqdefault.jpg&amp;key=a19fcc184b9711e1b4764040d3dc5c07&amp;type=text%2Fhtml&amp;schema=youtube" width="854" height="480" frameborder="0" scrolling="no"><a href="https://medium.com/media/d55650e6507e75aef82b6c877b9582be/href">https://medium.com/media/d55650e6507e75aef82b6c877b9582be/href</a></iframe><p>GPUs have proven <a href="https://arxiv.org/abs/1802.09941">extremely useful for highly parallelizable data processing use-cases</a>. The computational paradigms found in machine learning &amp; deep learning for example fit extremely well to <a href="http://cs149.stanford.edu/fall19/lecture/gpuarch/">the processing architecture graphics cards provide</a>.</p><p>However, when it comes to multiple GPU workloads, one would assume that these would be processed concurrently, but this is not the case. Whilst a single GPU compute workload is parallelized across the numerous GPU cores, multiple workloads are run one by one sequentially. That is of course <strong>until recent improvements in graphics card architectures </strong>which are now enabling for hardware parallelization across multiple workloads. This can be achieved by submitting the workloads to different underlying physical GPU “queue families” that support concurrency. Practical tecniques in machine learning that would benefit from this include <a href="https://mxnet.apache.org/versions/1.7/api/faq/model_parallel_lstm">model parallelism</a> and <a href="https://en.wikipedia.org/wiki/Data_parallelism">data parallelism</a>.</p><p>In this example we will show how we can achieve <strong>a 2x performance improvement</strong> on a synchronous example by simply submitting multiple workloads across two queue families, resulting in these workloads running in parallel.</p><p>This is an important optimization technique, as recent announcements outlined in NVIDIA’s Ampere GA10x architecture specifications <a href="https://www.nvidia.com/content/dam/en-zz/Solutions/geforce/ampere/pdf/NVIDIA-ampere-GA102-GPU-Architecture-Whitepaper-V1.pdf">page 19 in this document</a> will enable for <strong>3x performance improvements </strong>(i.e. concurrency across one graphics queue and two compute queues)<strong>,</strong> making it clear that this trend will only continue to bring further optimization improvements in this area.</p><p>We will be implementing this using Vulkan and the <a href="https://github.com/EthicalML/vulkan-kompute">Kompute framework</a>. More specifically we will cover:</p><ul><li>Disambiguation of “asynchronous” and “parallel” in GPU processing</li><li>A base synchronous example that we will build upon</li><li>Steps to extend the example for asynchronous workload submission</li><li>Steps to extend the example for parallel multi-queue GPU processing</li></ul><p>You can find the <a href="https://github.com/EthicalML/vulkan-kompute/blob/0b221c9ebd3c8d7c8a81ef2ce80627c9460ec9c2/test/TestAsyncOperations.cpp#L10"><strong>full code in this file</strong></a> — instructions on how to run the full suite using CMAKE can be found in the <a href="https://github.com/EthicalML/vulkan-kompute#build-overview">main Kompute repository build section</a>.</p><h3>About Vulkan and Kompute</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/512/0*hkZp9ffOun6EHRlD" /><figcaption>Khronos Membership (Image by Vincent Hindriksen via <a href="https://streamhpc.com/blog/2017-05-04/what-is-khronos-as-of-today/">StreamHPC</a>)</figcaption></figure><p><a href="https://en.wikipedia.org/wiki/Vulkan_(API)"><strong>The Vulkan SDK</strong></a><strong> </strong>is an Open Source project led by the <a href="https://www.khronos.org/"><strong>Khronos Group</strong></a>, which enables for highly optimized cross-vendor/cross-platform GPU processing.</p><p><a href="https://github.com/axsaucedo/vulkan-kompute#vulkan-kompute"><strong>Kompute</strong></a> is a framework built on top of the Vulkan SDK which abstracts the thousands of lines of boilerplate code required, introducing best practices that expose Vulkan’s core computing capabilities. Kompute is the <a href="https://en.wikipedia.org/wiki/General-purpose_computing_on_graphics_processing_units">GPGPU computing framework</a> that we will be using in this tutorial to build the core asynchronous and parallel code implementation.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/800/0*ASHn5x_l8sB6OXhm.gif" /><figcaption>The “Komputer” from <a href="https://github.com/EthicalML/vulkan-kompute">Kompute Repo</a> (Image by Author)</figcaption></figure><h3>Asynchronous vs Parallel Processing</h3><p>Before diving into the code, it is important to disambiguate two concepts — <strong>asynchronous workload submission</strong> and <strong>parallel workload processing</strong>.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/568/1*gEBXV6hJGPMfRZWvF3kENQ.png" /><figcaption>Simplified Vulkan Architecture (Image by Author)</figcaption></figure><p>The way parallel workloads are submitted for processing when using the <strong>Vulkan SDK </strong>through <strong>GPU Queues</strong>. This can be visualised in the simplified Vulkan Architecture diagram (pipeline and descriptor components were left out for simplicity).</p><h4>Asynchronous Workload Submission</h4><p>Asynchronous processing encompasses the ability for the CPU host side to be able to do other work whilst the GPU is processing the workload. “Other work” can include calling other C++ functions, or even submitting further workloads to the same or other GPU queues. When the CPU wants to check whether the GPU workload is finished, it can use a <strong>Vulkan “Fence” </strong>which is basically a semaphore resource that allows the CPU to be notified when a GPU workload finishes.</p><p>It is important to note that when multiple workloads are submitted to the same queue, even if these are done from multiple C++ threads, the expected execution ordering will still be sequential — at least as of today’s GPU architectures.</p><h4>Parallel Workload Processing</h4><p>Parallel workload processing consists of the concurrent execution of two or more workloads by the GPU. More specifically, if you had two GPU tasks that would take 10 seconds each to process, the theoretical parallel execution would still take 10 seconds for both as they would be carried out at the same time.</p><p>In order for parallel workload processing to be achieved, this is something that first and foremost has to be supported by the underlying GPU. The reason why this is important is because even if you were to submit workloads across different GPU queues, the processing may still be done sequentially by the underlying hardware based on its limitations.</p><h3>Base Sequential Processing Example</h3><p>We will now take a look at the code that we will be using throughout this article. This first version of the code will be the sequential flow — we will be able to then convert it into asynchronous code, and finally into parallel code. We will basically be running a workload where we will be doing the following:</p><ol><li>Creating a Kompute Manager to orchestrate all GPU work</li><li>Create Kompute Tensors in the CPU host that will be used to process data</li><li>Map the Kompute Tensors into GPU Device memory</li><li>Define compute shader which will keep the GPU busy for a few 100s ms</li><li>Run compute shader in the GPU using the Tensors for data processing</li><li>Map results of the Kompute Tensors back into CPU Host memory</li><li>Verify that the operation was successful</li></ol><p>For measuring time we will be using &lt;chrono&gt; from the standard library. We will be mainly using it to calculate the difference across a start and end time retrieved with std::chrono::high_resolution_clock::now() as follows:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/7dbbab57c5b221875d0bee4eb34e5615/href">https://medium.com/media/7dbbab57c5b221875d0bee4eb34e5615/href</a></iframe><p>You can find the <a href="https://github.com/EthicalML/vulkan-kompute/blob/1053cde1f0d27799f0d7dbd8043919656498f8bf/test/TestAsyncOperations.cpp#L8">runnable code in this file</a>, which is part of the Kompute test suite.</p><h4>1. Creating a Kompute Manager to orchestrate all GPU work</h4><p>First we have to create the Kompute Manager, which performs all the required memory management and creates all required Vulkan resources. By default the Kompute Manager will pick GPU Device 0, but you are able to pass the specific device index you would prefer to initialise with, and if preferred you can pass your Vulkan resources if you already have a Vulkan application.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/4bb93252637e5753e66f15bfda3ac91f/href">https://medium.com/media/4bb93252637e5753e66f15bfda3ac91f/href</a></iframe><h4>2. Create the Kompute Tensors in CPU host that will be used to process data</h4><p>We will now be able to create a set of Kompute Tensors. We first initialise the data in the CPU Host, consisting of an array of zeros with length of 10. We will be using two tensors as we’ll be running two algorithm executions. We will be able to check these Kompute Tensors at the end to confirm that the execution has been successful.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/44f27abbf81f6e7e36f3047b97918af6/href">https://medium.com/media/44f27abbf81f6e7e36f3047b97918af6/href</a></iframe><h4>3. Map the Kompute Tensors into GPU Device memory</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*EZl2b3znOUwPBW1a1Zz51g.png" /><figcaption><strong>Stanford CS149 Course </strong><a href="http://cs149.stanford.edu/fall19/lecture/gpuarch/slide_038"><strong>2019 Slides</strong></a></figcaption></figure><p>We are now able to copy the host data of the Kompute Tensors into the GPU Device memory.</p><p>This is an important step as by default the Kompute Tensors use device-only-visible memory which means that a GPU operation will need to copy it with a staging tensor.</p><p>Kompute allows us to create the buffer and GPU memory block, as well as performing a copy with a staging buffer through the kp::OpTensorCreate operation.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/03b4ba718bccf8015722b99603670e30/href">https://medium.com/media/03b4ba718bccf8015722b99603670e30/href</a></iframe><h4>4. Define compute shader which will keep the GPU busy for a few 100s ms</h4><p>The compute shader that we create has a relatively large loop to simulate an “expensive computation”. It basically performs a unit addition for 100000000 iterations and adds the result to the input Tensor.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/82ccd1a014950a7cab15e5aa60694fc0/href">https://medium.com/media/82ccd1a014950a7cab15e5aa60694fc0/href</a></iframe><h4>5. Run compute shader in the GPU using the Tensors for data processing</h4><p>Now we are able to submit the compute shader for execution through the kp::OpAlgoBase operation. This basically allows us to perform a submission of the shader with the respective tensor. This initial implementation runs the execution synchronously, so it will first run the execution of the shader with tensorA, and then the execution of the same shader with tensorB.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/2abddbce812bba0aa8a69abd70f088af/href">https://medium.com/media/2abddbce812bba0aa8a69abd70f088af/href</a></iframe><h4>6. Map results of the Kompute Tensors back into CPU Host memory</h4><p>Finally we want to retrieve the results from the GPU device memory into the CPU host memory so we can access it from C++. For this we can use the kp::OpTensorSync operation.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/b0ff9e334c2b86fd70539c1325501f55/href">https://medium.com/media/b0ff9e334c2b86fd70539c1325501f55/href</a></iframe><h4>7. Verify that the operation was successful</h4><p>Finally we can just check that both resulting kp::Tensor contain the expected value of 100000000.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/ef4cfec3532ca083785000ca53ee8d4d/href">https://medium.com/media/ef4cfec3532ca083785000ca53ee8d4d/href</a></iframe><h3>Extending for Asynchronous Workload Submission</h3><p>The steps that we will need to extend for asynchronous submission in this case are quite minimal. The only thing we need to do is to substitute the evalOpDefault function for the evalOpAsyncDefault function, and then using the evalOpAwaitDefault(&lt;timeInNanoSecs&gt;) to wait until the job is finished. This basically would look as follows:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/cc468a1136c751733bc18c946c5bf37a/href">https://medium.com/media/cc468a1136c751733bc18c946c5bf37a/href</a></iframe><p>As you can see we are able to submit two tasks for processing asynchronously, and then wait until they are finished with the Await function.</p><p>It’s worth pointing out that every time we call evalOpAsyncDefault it creates a new managed sequence, and evalOpAwaitDefault only waits for the most recent default sequence. This means that in the snippet above, we are only waiting for the second asynchronous operation. This isn’t a problem for our example, but this could introduce bugs if we’re now aware. The proper way to do this is with explicitly created “named sequences” — we will do this in the next section.</p><h3>Extending for Parallel Workload Processing</h3><p>Now that we know we are able to execute multiple workloads asynchronously, we are able to extend this to leverage the multiple queues in the GPU to achieve parallel execution of workloads.</p><h4>Running on an NVIDIA 1650 Video Card</h4><p>In order to show a useful example, we will dive into how this would be achieved in an <a href="http://vulkan.gpuinfo.org/displayreport.php?id=9700">NVIDIA 1650 video card</a>. You are able to try this yourself by checking the device report of your video card — namely on the queue families and parallel processing capabilities available.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/680/1*SvqJ5S7FLPnIz_EscgtsGw.png" /><figcaption>Conceptual Overview of Queues in NVIDIA 1650 (Image by Author)</figcaption></figure><p>The NVIDIA 1650 GPU has 3 queue families. Using G for GRAPHICS, T for TRANSFER and C for COMPUTE capabilities, the NVIDIA 1650 has a G+T+C family in familyIndex 0with 16 queues, a T family on familyIndex 1 with 2 queues, and a T+C family on familyIndex 2 with 8 queues.</p><p>As of today (October 2020), NVIDIA does not support parallel processing of workloads when work is submitted across multiple queues within the same family. However it supports parallelizing when workloads are submitted across queue families. This means that workloads between graphics and compute family queues can be parallelized — we will be using this knowledge in our implementation.</p><h4>Implementation of Parallel Workflow Execution</h4><p>So far we have been submitting all GPU workloads to a single queue, namely the GRAPHICS familyIndex 0 using the underlying queue index 0. In our case using the GPU 1650, we will be able to achieve parallel processing if we submit workloads across the GRAPHICS family and the COMPUTE family. The diagram below should provide an intuition on what we will be doing.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*jFMAnHpNFQaAGOFnqWYWSA.png" /><figcaption>Operation Execution in Parallel through Multiple Family Queues (Image by Author)</figcaption></figure><p>In order for us to do this, we will need to modify three key things:</p><ol><li>We initialise Kompute Manager with the respective queues available</li><li>We create two Kompute Sequences with each respective queue allocated</li><li>We run the operations on each respective queue</li></ol><p>We will dive into each of these three points.</p><h4>1. We initialise Kompute Manager with the respective queues available</h4><p>When initialising a manager we are able to pass an array containing the queues that we would like to fetch. In this case, we only fetch one graphics queue and one compute queue, however, based on the hardware specs of the NVIDIA 1650, we would be able to request up to 16 graphics queues (familyIndex 0), 2 transfer queues (familyIndex 1), and 8 compute queues (familyIndex 2).</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/664369ba2b710b115526cec3c4754818/href">https://medium.com/media/664369ba2b710b115526cec3c4754818/href</a></iframe><h4>2. We create two Kompute Sequences with each respective queue allocated</h4><p>Now we are able to explicitly initialise two managed sequences, each allocated to a different queue, referencing the index of the array we passed in the previous step.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/101d8417067ddb9521074c8d90b6066e/href">https://medium.com/media/101d8417067ddb9521074c8d90b6066e/href</a></iframe><h4>3. We run the operations on each respective queue</h4><p>Now we are able to run operations submitting to each respective queue. In this case both of the GPU workloads are submitted in parallel.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/4dadfd67dd6ff63900a3b6e326887fb1/href">https://medium.com/media/4dadfd67dd6ff63900a3b6e326887fb1/href</a></iframe><h3>Parallel Workload Execution Results</h3><p>When running the code provided above, we can see a 2x speed improvement in execution time thanks to the parallel family queue submission of workload. You can also see that if we were to submit to extra queues from the GRAPHICS or COMPUTE queues, we would not see any further speed improvements as intra-queue parallelization is not supported in this NVIDIA 1650 card.</p><p>You can find the full code and run it in this file — instructions on how to run <a href="https://github.com/EthicalML/vulkan-kompute/blob/0b221c9ebd3c8d7c8a81ef2ce80627c9460ec9c2/test/TestAsyncOperations.cpp#L10">the full suite</a> using CMAKE can be found in the <a href="https://github.com/EthicalML/vulkan-kompute">main Kompute repository</a>.</p><p>This is a particularly important result, as based on the recent announcement from NVIDIA coming together with the release of their 300x video cards, there are improvements via the Ampere GA10x architecture that allows for two compute workloads simultaneously. Relative to the example above, this means that we could see a 3x improvement if we were to use one GRAPHICS queue and two COMPUTE queue (together with the extra performance using the TRANSFER queue for transfer operations).</p><h3>Next Steps</h3><p>Congratulations, you’ve made it all the way to the end! Although there was a broad range of topics covered in this post, there is a massive amount of concepts that were skimmed through. These include the underlying Vulkan concepts, GPU computing fundamentals, and more advanced Kompute concepts. Luckily, there are resources online to expand your knowledge on each of these. Here are some links I recommend for further reading:</p><ul><li>“<a href="https://towardsdatascience.com/machine-learning-and-data-processing-in-the-gpu-with-vulkan-kompute-c9350e5e5d3a">Machine Learning in Mobile &amp; Cross-Vendor GPUs Made Simple With Kompute &amp; Vulkan</a>” article with a deeper dive in theory and concepts</li><li><a href="https://axsaucedo.github.io/vulkan-kompute/">Kompute Documentation</a> for more details and further examples</li><li><a href="https://towardsdatascience.com/machine-learning-and-data-processing-in-the-gpu-with-vulkan-kompute-c9350e5e5d3a">Machine Learning in Mobile &amp; Cross-Vendor GPUs Made Simple With Kompute &amp; Vulkan</a></li><li><a href="https://towardsdatascience.com/gpu-accelerated-machine-learning-in-your-mobile-applications-using-the-android-ndk-vulkan-kompute-1e9da37b7617">Supercharging your Mobile Apps with GPU Accelerated Machine Learning using the Android NDK &amp; Kompute</a></li><li><a href="https://towardsdatascience.com/supercharging-game-development-with-gpu-accelerated-ml-using-vulkan-kompute-the-godot-game-engine-4e75a84ea9f0">GPU Accelerated ML using the Godot Engine and Kompute</a></li><li><a href="https://vulkan-tutorial.com/">Vulkan SDK Tutorial</a> for a deep dive into the underlying Vulkan components</li></ul><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=50a38b15a1dc" width="1" height="1" alt=""><hr><p><a href="https://medium.com/data-science/parallelizing-heavy-gpu-workloads-via-multi-queue-operations-50a38b15a1dc">Parallelizing GPU-intensive Workloads via Multi-Queue Operations</a> was originally published in <a href="https://medium.com/data-science">TDS Archive</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Supercharging your Mobile Apps: GPU Accelerated Machine Learning using Android NDK & Vulkan Kompute]]></title>
            <link>https://medium.com/data-science/gpu-accelerated-machine-learning-in-your-mobile-applications-using-the-android-ndk-vulkan-kompute-1e9da37b7617?source=rss-32de426f7278------2</link>
            <guid isPermaLink="false">https://medium.com/p/1e9da37b7617</guid>
            <category><![CDATA[artificial-intelligence]]></category>
            <category><![CDATA[vulkan]]></category>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[android-ndk]]></category>
            <category><![CDATA[mobile-apps]]></category>
            <dc:creator><![CDATA[Alejandro Saucedo]]></dc:creator>
            <pubDate>Tue, 13 Oct 2020 12:50:18 GMT</pubDate>
            <atom:updated>2021-08-21T12:58:55.742Z</atom:updated>
            <content:encoded><![CDATA[<h3>Supercharging your Mobile Apps with GPU Accelerated Machine Learning using the Android NDK, Vulkan and Kompute</h3><h4>A hands on tutorial that teaches you how to leverage your on-device phone GPU for accelerated data processing and machine learning. You will learn how to build a simple Android App using the Native Development Kit (NDK), the Kompute framework and the Vulkan SDK.</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/800/1*dfCkMbt_3Wr61Nq-ii4iDw.gif" /><figcaption>Kompute in Android NDK (Image by Author)</figcaption></figure><p>Some smartphones nowadays pack laptop-level hardware — carrying up to 16GB RAM, high-speed multi-core CPUs, and GPUs that can render high-performance complex graphical applications on 4k displays.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*M4MmQ0BsdGYws9At9aLFUg.png" /><figcaption>Image by Author</figcaption></figure><p>Tapping into that power — especially the GPU processing power — for on-device data processing capabilities becomes growingly important as mobile hardware <a href="https://www.mobilemarketer.com/news/mobile-games-sparked-60-of-2019-global-game-revenue-study-finds/569658/">only continues to improv</a>e. Recently, this has been opening exciting opportunities around <a href="https://en.wikipedia.org/wiki/Edge_computing">edge computing</a>, <a href="https://en.wikipedia.org/wiki/Federated_architecture">federated architectures</a>, <a href="https://arxiv.org/abs/1910.06663">mobile deep learning</a>, and more.</p><p>This article provides a technical deep dive that shows you how to tap into the power of mobile cross-vendor GPUs. You will learn how to use the <a href="https://developer.android.com/ndk"><strong>Android Native Development Kit</strong></a><strong> </strong>and the <a href="https://github.com/EthicalML/vulkan-kompute"><strong>Kompute framework</strong></a> to write GPU optimized code for Android devices. The end result will be a mobile app created in Android Studio that is able to use a GPU accelerated machine learning model which we will write from scratch, together with a user interface that will allow the user to send the input to the model.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*X-q-MWIg4rOlyhBTrJiMUA.png" /><figcaption>Android Studio Running Project in Emulator (Image by Author)</figcaption></figure><p>No background knowledge <strong>beyond programming experience</strong> is required, however if you are curious about the underlying AI / GPU compute concepts referenced, we suggest checking out our previous article, “<a href="https://towardsdatascience.com/machine-learning-and-data-processing-in-the-gpu-with-vulkan-kompute-c9350e5e5d3a">Machine Learning in Mobile &amp; Cross-Vendor GPUs Made Simple With Kompute &amp; Vulkan</a>”.</p><p>You can find the full code in the <a href="https://github.com/EthicalML/vulkan-kompute/tree/master/examples/android/android-simple"><strong>example folder in the repository</strong></a><strong>.</strong></p><h3>Android Native Development Kit (NDK)</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*2rlyOdmPMb70f_qPRntClg.png" /><figcaption>Android NDK Diagram (Image by Author)</figcaption></figure><p>The <a href="https://developer.android.com/ndk">Native Development Kit (NDK)</a> is one of Android’s solution to address the increasing computational demands from mobile apps. The NDK framework enables developers to write low level, highly efficient C and C++ code that can interoperate with the Java/Kotlin application code through the popular <a href="https://en.wikipedia.org/wiki/Java_Native_Interface"><strong>Java Native Interface</strong></a><strong> </strong>bindings.</p><p>This tooling enables mobile application developers not only to write highly efficient code, but also leverage existing optimized frameworks written in C++ for advanced data processing or machine learning.</p><h3>Enter Kompute &amp; the Vulkan SDK</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/512/0*mzEbYNdCJ129eSz3" /><figcaption>Playing “where’s waldo” with Khronos Membership (Image by Vincent Hindriksen via <a href="https://streamhpc.com/blog/2017-05-04/what-is-khronos-as-of-today/">StreamHPC</a>)</figcaption></figure><p><strong>Vulkan </strong>is an Open Source project led by the <a href="https://www.khronos.org/">Khronos Group</a>, a consortium consisting of several tech companies who have come together to work towards defining and advancing the open standards for mobile and desktop media (and compute) technologies.</p><p>A large number of high profile (and new) frameworks have been adopting Vulkan as their core GPU processing SDK. The Android NDK’s main documentation page has a full section <a href="https://developer.android.com/ndk/guides/graphics"><strong>dedicated to Vulkan</strong></a>, together with hands on examples showing how it can be used in Android mobile devices.</p><p>As you can imagine, the Vulkan SDK provides very low-level access to GPUs, which allows for very specialized optimizations. This is a great asset for data processing and GPU developers — the main disadvantage is the verbosity involved, requiring 500–2000+ lines of code to only get the base boilerplate required to even start writing the application logic. This can result in expensive developer cycles and errors that can lead to larger problems. This was one of the main motivations for us to start the <strong>Kompute </strong>project.</p><p><a href="https://github.com/axsaucedo/vulkan-kompute#vulkan-kompute"><strong>Kompute</strong></a> is a framework built on top of the Vulkan SDK which abstracts a lot of boilerplate code required, introducing best practices that expose Vulkan’s computing capabilities. Kompute is the GPU computing framework that we will be using in this tutorial to build the machine learning module in our mobile Android app.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*hkFiBPfbdHQqhHKA.png" /><figcaption>Kompute <a href="https://ethicalml.github.io/vulkan-kompute/">Documentation</a> (Image by Author)</figcaption></figure><h3>Machine Learning in Mobile Development</h3><p>In this post we will be building upon the Machine Learning use-case we created in the “<a href="https://towardsdatascience.com/machine-learning-and-data-processing-in-the-gpu-with-vulkan-kompute-c9350e5e5d3a">Machine Learning in Mobile &amp; Cross-Vendor GPUs Made Simple With Kompute &amp; Vulkan</a>” article. We will not be covering the underlying concepts in as much detail as in that article, but we’ll still introduce the high level intuition required in this section.</p><p>To start with, we will need an interface that allows us to expose our Machine Learning logic, which will require primarily two functions:</p><ol><li>train(…) — function which will allow the machine learning model to learn to predict outputs from the inputs provided</li><li>predict(...)— function that will predict the output of an unknown instance.</li></ol><p>This can be visualised in the two workflows outlined in the image below.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/791/0*ZE0_pQd6kjy0y5I0" /><figcaption>Data Science Process (Image by Author)</figcaption></figure><p>Particularly in app development, this would also be <a href="https://www.forbes.com/sites/forbestechcouncil/2020/08/03/how-machine-learning-is-powering-a-new-generation-of-app-development/#17c728f62a88">a common pattern</a> for machine learning workflows, for both predictive and explanatory modelling use cases. This often consists of leveraging data generated by your users as they interact directly (or indirectly) with the app itself. This data can then serve as training features for machine learning models. Training of new models can be performed through manual “offline” workflows that data scientists would carry out, or alternatively through automated triggers retraining models.</p><h3>Android Studio Project Overview</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/870/1*If89pXcj4hSNJDTEFEdFNw.png" /><figcaption>Project File Structure (Image by Author)</figcaption></figure><p>We will start by providing a high level overview of the core components in the Android Studio project, including the Kompute C++ bindings, the Android User Interface, the App logic build in Kotlin, and the build files required. If you are interested in a particular area you can jump to its respective section below.</p><p>You will need to make sure you install <a href="https://developer.android.com/studio"><strong>Android Studio</strong></a>, and also install the <a href="https://developer.android.com/ndk/guides#download-ndk"><strong>Android NDK </strong></a>— the rest of the dependencies will be installed and configured automatically when opening the project in the IDE.</p><p>Now that you have everything installed, you are able to import the project. For this, you first have to clone the <a href="https://github.com/EthicalML/vulkan-kompute"><strong>full Kompute repository</strong></a> and import the Android Studio project under examples/android/android-simple/. You should now be able to see the project load and configure the build. Once it opens you are able to run it in an emulator or in your own physical Android phone. This project was tested in the Pixel 2 emulator, and in a physical Samsung Galaxy phone.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*mZXTUuwtfxXi1sHQeGmZYg.jpeg" /><figcaption>Final GPU Accelerated Kompute App (Image by Author)</figcaption></figure><p>When you load the project you will notice the following key components in the file structure, which we will be breaking down further in detail in the following sections:</p><ul><li><strong>Android SDK Application </strong>— The Android UI, asset resources, build files, and Kotlin/Java components that provide the relevant application logic that interacts with the UI and C++ Kompute ML bindings.</li><li><strong>Android NDK Kompute ML Module </strong>— The Kompute ML C++ code and bindings configured through Android NDK for GPU accelerated processing.</li></ul><h3>Android SDK Application</h3><p>This section covers the Android SDK Java/Kotlin and User Interface components, which should provide an intuition on how the high level business logic interacts with the native C++ JNI bindings.</p><p>The user interface consists primarily of input text boxes and display text labels to enable users to interact with the GPU Accelerated ML processing C++ core (as shown in the screenshot below). If you are curious on exactly the views used, you can inspect it in your Android Studio project, or directly open the <a href="https://github.com/EthicalML/vulkan-kompute/blob/v0.3.2/examples/android/android-simple/app/src/main/res/layout/activity_kompute_jni.xml">activity_kompute_jni.xml file</a>.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*_TFHzF8zt6xQXvHDsuFlxA.png" /></figure><p>The core of our mobile app can be found in the app/src/main/java/com/ethicalml/kompute folder, inside the <a href="https://github.com/EthicalML/vulkan-kompute/blob/v0.3.2/examples/android/android-simple/app/src/main/java/com/ethicalml/kompute/KomputeJni.kt">KomputeJni.ktfile</a>. This Kotlin file contains the main business logic for our Android app.</p><p>If we look at the shortened version of the class in the code block below we will notice the following key points:</p><ul><li>fun onCreate(…) — This function is called on initialisation of the Android Activity (when the app is loaded)</li><li>fun KomputeButtonOnClick(…)— This function is triggered when the main “KOMPUTE” button gets pressed, and triggers the C++ Jni binding functions using the data from the user interface text boxes.</li><li>external fun initVulkan(): Boolean — This function is one of the C++ JNI functions that will be bound to the Vulkan initialisation C++ function.</li><li>external fun kompute(...): FloatArray — The is the C++ JNI function that will train the ML model and run inference on the inputs provided, returned the inference results.</li><li>external fun komputeParams(...): FloatArray — The C++ JNI function that trains the model and returns the learned parameters weight 1 , weight 2 and bias .</li><li>companion object { ...(&quot;kompute-jni&quot;) } — This is the name you will give to your C++ output shared library, which will contain the compiled C++ source with all relevant binding functions.</li></ul><p>As you will also notice, the external fun functions do not have any definition — this is because the definition is provided in the C++ JNI function bindings, which will be covered in the <a href="http://jni/"><strong>C++ JNI bindings section</strong></a>.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/770bda64b7f036b02442a9ab798da1d1/href">https://medium.com/media/770bda64b7f036b02442a9ab798da1d1/href</a></iframe><p>Now to cover each of the functions in more detail, we start with the onCreate function. This function is in charge of initialising all relevant components in our application. This includes:</p><ul><li>val binding — This is the main object that will allow us to access all the text boxes and elements in the UI.</li><li>val successVulkanInit = initVulkan() — This is our first call to a C++ JNI function, which is primarily in charge of initialising Vulkan. If it’s successful it returns true, and we display the respective success message in a android.widget.Toast popup — an error is displayed otherwise.</li></ul><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/754abec042d9b4bae235d4f9ec0b257b/href">https://medium.com/media/754abec042d9b4bae235d4f9ec0b257b/href</a></iframe><p>Next up we have the KomputeButtonOnClick(...) function. This function gets triggered when the user presses the “KOMPUTE” button in the app. The main purpose of this function is to retrieve the inputs from the input text boxes in the UI, then use the input data to perform an ML train/inference step through the JNI C++ bindings, and finally display the resulting outputs back in the UI text labels. In further detail:</p><ul><li>val &lt;elementname&gt; = findViewById&lt;EditText&gt;(R.id.&lt;elementname&gt;) — This is the format of the steps that create the variable that holds the respective text box with inputs. In this case &lt;elementname&gt; is the name of the element we are interacting with.</li><li>xi, xj and y — The FloatArray elements created from the text in the respective input boxes, which are then used for the ML model processing.</li><li>val out = kompute(xi, xj, y) — Here we run the C++ JNI Binding function kompute which trains and processes the data through the KomputeModelML class we create in C++.</li><li>val params = komputeParams(xi, xj, y) — Here we run the C++ JNI Binding function which trains and returns the learned parameters of the Kompute machine learning model.</li><li>&lt;elementname&gt;.text = &lt;value&gt; — The lines that follow this format basically override the text labels in the UI to display the outputs.</li></ul><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/224935de8f7d1e25307ff598082a9cf0/href">https://medium.com/media/224935de8f7d1e25307ff598082a9cf0/href</a></iframe><p>The last few functions are only explicitly set as external functions to be bound to the C++ JNI bindings. Furthermore the companion object section provides the name of the shared library that will contain the respective bindings referenced in this activity.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/f0979f36a5ccd15ddf2e340259cb87ca/href">https://medium.com/media/f0979f36a5ccd15ddf2e340259cb87ca/href</a></iframe><p>You can find the full file in the <a href="https://github.com/EthicalML/vulkan-kompute/blob/v0.3.2/examples/android/android-simple/app/src/main/java/com/ethicalml/kompute/KomputeJni.kt">KomputeJni.kt file</a> in the repository.</p><h3>Android NDK Kompute ML Module</h3><p>This section covers the <strong>Android NDK Kompute ML Module</strong> files, which includes the build-system, and the C++ source code using the <a href="https://github.com/EthicalML/vulkan-kompute"><strong>Kompute framework</strong></a>.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/462/0*c-LUmEwPLAKgZ3t4.jpg" /><figcaption>Kompute <a href="https://ethicalml.github.io/vulkan-kompute/overview/reference.html">Architecture Design</a> (Image by Author)</figcaption></figure><p>We will be using the core components of Kompute which are outlined in this accompanying diagram. Namely, we will be loading the relevant data in the GPU using Kompute Tensors, processing it with the respective Kompute Operations, and orchestrating this with a Kompute Sequence and a Kompute Manager. We won’t be covering the Kompute architecture in detail but if you want to learn more about the underlying concepts, you can check out the more detailed article on <a href="https://towardsdatascience.com/machine-learning-and-data-processing-in-the-gpu-with-vulkan-kompute-c9350e5e5d3a">the underlying implementation</a>.</p><p>The core components in the Android NDK bindings module consist of the following:</p><ul><li><strong>JNI Binding Functions</strong> — The native functions that can be called from the Java/Kotlin Android SDK application code.</li><li><strong>KomputeModelML Class</strong> — The class that exposes the Kompute GPU Accelerated ML model logic.</li><li><strong>CMake build file </strong>— The C++ build file responsible for compiling and linking all relevant libraries.</li></ul><h4>JNI Binding Functions</h4><p>The JNI bindings in this case are provided via the <a href="https://github.com/EthicalML/vulkan-kompute/blob/v0.3.2/examples/android/android-simple/app/src/main/cpp/KomputeJniNative.cpp">KomputeJniNative.cpp file</a>. The skeleton of the class is below — the function code logic has been redacted for simplicity, and will be explained in more detail below.</p><p>The JNI binding functions have to match the class functions defined in the Java/Kotlin code. The format for the function is:</p><ul><li>Java_&lt;modulepath&gt;_&lt;class&gt;_&lt;functionname&gt;(env, thiz, ...params)</li></ul><p>In our case the class is in the com.ethicalml.kompute module, in the class KomputeJni and its respective function — below the name of the functions will reflect this structure.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/8949d1a2b43564021b0e09f62c1e3b1c/href">https://medium.com/media/8949d1a2b43564021b0e09f62c1e3b1c/href</a></iframe><p>Diving one level deeper, we can now go through each section of the file. Starting with the imports, we can see below the imports together with comments outlining their core functionality.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/72edae8d8a465951245554f3f6fdf15a/href">https://medium.com/media/72edae8d8a465951245554f3f6fdf15a/href</a></iframe><p>In Android applications, we actually need to initialize the Vulkan dynamic library manually (which is something that you normally wouldn’t do outside of Android). The reason why this is required, is because the Vulkan library is not actually linked in Android phones. The reason why Android avoids doing any linking is for backwards compatibility, mainly to ensure the app doesn’t crash if the Vulkan library is not found in older phones.</p><p>This means we need to manually find the library in the C++ code and if found, link each function to its respective memory address pointer so our C++ framework can use it. Fortunately, this is something that Kompute does automatically, and we won’t be covering the details in this article as it probably would require an article in itself, but if you’re interested you can read more about it in <a href="https://marcelbraghetto.github.io/a-simple-triangle/2019/06/16/part-17/"><strong>this post</strong></a>, and you can see how Kompute imports Vulkan dynamically in the <a href="https://github.com/EthicalML/vulkan-kompute/blob/45ddfe524b9ed63c5fe1fc33773c8f93a18e2fac/src/include/kompute/Core.hpp#L5">Core.hpp header file</a> using the <a href="https://github.com/EthicalML/vulkan-kompute/tree/45ddfe524b9ed63c5fe1fc33773c8f93a18e2fac/vk_ndk_wrapper_include">vk_ndk_wrapper_include files</a>.</p><p>Below you can see the implementation of the function that exposes the initVulkan logic—Java_com_ethicalml_kompute_KomputeJni_initVulkan(...). You can see inside this function we run InitVulkan()until the Vulkan library is successfully initialised, or alternatively fails if the maximum number of retries is reached.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/a79aeb2410b7abcc1e39466a634a3b76/href">https://medium.com/media/a79aeb2410b7abcc1e39466a634a3b76/href</a></iframe><p>Once Vulkan has been initialised, it is possible to call the remaining functions. The first one is the kompute function, which is in charge of training the model and running an inference request. The function receives the input Xi and Xj values, together with the expected predictions that the model will learn from. It will then return the prediction treating Xi and Xj as unseen data. The function will basically call the KomputeModelML class train function and predict function.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/8e23df62d8fa3cf589f0d63d07f40e7a/href">https://medium.com/media/8e23df62d8fa3cf589f0d63d07f40e7a/href</a></iframe><p>The last remaining JNI function that will be exposed to the Java/Kotlin code is the komputeParams function, which is in charge of returning the parameters that the machine learning model learns, namely the weight 1 , weight 2 and the bias parameters.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/9bcef7c0185d9a90394c610ec4ff3f31/href">https://medium.com/media/9bcef7c0185d9a90394c610ec4ff3f31/href</a></iframe><p>The only remaining functions are the utility functions that we used in the JNI logic — namely <a href="https://github.com/EthicalML/vulkan-kompute/blob/45ddfe524b9ed63c5fe1fc33773c8f93a18e2fac/examples/android/android-simple/app/src/main/cpp/KomputeJniNative.cpp#L32">jfloatArrayToVector</a> and <a href="https://github.com/EthicalML/vulkan-kompute/blob/45ddfe524b9ed63c5fe1fc33773c8f93a18e2fac/examples/android/android-simple/app/src/main/cpp/KomputeJniNative.cpp#L41">vectorToJFloatArray</a> — these functions are self explanatory, so we’ll leave it to the reader to explore further in the source if interested.</p><h4>KomputeModelML Class</h4><p>Now that we’ve covered the key functions that are bound to the Kotlin / Java class, we can cover the KomputeModelML C++ class that contains the Kompute GPU Accelerated logic.</p><p>The header file for the KomputeModelML class is outlined below, and contains the following key components:</p><ul><li>#include &quot;kompute/Kompute.hpp” — header containing all the Kompute dependencies that we’ll use in this project</li><li>void train(...) —Trains the machine learning model using the GPU native code for the logistic regression model. It takes the input array(s) X, and the array ycontaining the expected outputs.</li><li>std::vector&lt;float&gt; predict(...) —Perform the inference request. In this implementation it is not using GPU code as generally there tends to be less performance gains through parallelization on the inference side. However there are still expected performance gains if multiple inputs are processed in parallel (which this function allows for).</li><li>std::vector&lt;float&gt; get_params() —Returns an array containing the learned parameters in the format of [ &lt;weight_1&gt;, &lt;weight_2&gt;, &lt;bias&gt; ].</li><li>static std::string LR_SHADER— The <a href="https://en.wikipedia.org/wiki/OpenGL_Shading_Language">shader code that will be executed as machine code inside of the GPU</a>. Kompute allows us to pass a string containing the code, however for production deployments it is possible to convert the shaders to binary, and also use the utilities available to convert into header files.</li></ul><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/158e229bc01d5978e5d48aa5978346b9/href">https://medium.com/media/158e229bc01d5978e5d48aa5978346b9/href</a></iframe><p>If you are interested in the full implementation you can find all the files in <a href="https://github.com/EthicalML/vulkan-kompute/tree/v0.3.2/examples/android/android-simple">the repository</a>. Furthermore if you are interested in the theoretical and underlying foundational concepts of these techniques, this is covered fully in <a href="https://towardsdatascience.com/machine-learning-and-data-processing-in-the-gpu-with-vulkan-kompute-c9350e5e5d3a#6c88">our previous post</a>.</p><h4>CMake Build File</h4><p>The <a href="https://github.com/EthicalML/vulkan-kompute/blob/v0.3.2/examples/android/android-simple/app/src/main/cpp/CMakeLists.txt"><strong>CMakeLists.txt build file</strong></a> is a very important component in the Android NDK workflow. This section becomes particularly important if you wish to add Kompute into your own project. The cmake file is quite small so we’ll be covering each of the lines separately.</p><p>First we need to make sure the Kompute library is available. Usually you would run the INSTALL target of the Kompute build to be able to use/import the respective library. However in this case we need to make sure Kompute is built for the right Android CPU architecture —our simplest option is adding the main repository as part of the build, which means that Kompute will also be built for the right mobile architectures. If you want to include this in your project, you just need to make sure the path is relative to the Kompute cloned folder.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/39655df99fc5962b10ba80e15a2a2c53/href">https://medium.com/media/39655df99fc5962b10ba80e15a2a2c53/href</a></iframe><p>We now set the variable VK_ANDROID_INCLUDE_DIR to the vulkan include directory. This contains all the include files we need for Vulkan — for completeness, Kompute uses the vulkan.h header as well as the vulkan.hpp C++ headers.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/f30ff09a3f11a7fd90cebedcc82b1b98/href">https://medium.com/media/f30ff09a3f11a7fd90cebedcc82b1b98/href</a></iframe><p>We are now able to add the library that will be used by the Java/Kotlin Android Studio project, which in this case is the shared library <strong>kompute-jni</strong>.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/11649f51563446dbb2a349d0397f537c/href">https://medium.com/media/11649f51563446dbb2a349d0397f537c/href</a></iframe><p>We now are able to add all relevant include directories. This includes the VK_ANDROID_INCLUDE_DIRwhich we defined above, as well as the VK_ANDROID_COMMON_DIR which contains Android log.h . The single_include is what contains the kompute/Kompute.hpp header from Kompute. Finally, we need to import the Kompute Dynamic library wrapper vk_ndk_wrapper_include which is necessary as the Vulkan library is imported dynamically. The logic behind this could become a series of articles in itself, so we won’t go down this rabbit-hole, but if you’re interested you can read more in <a href="https://marcelbraghetto.github.io/a-simple-triangle/2019/06/16/part-17/"><strong>this post</strong></a>, and you can see how Kompute imports <a href="https://github.com/EthicalML/vulkan-kompute/blob/45ddfe524b9ed63c5fe1fc33773c8f93a18e2fac/src/include/kompute/Core.hpp#L5"><strong>Vulkan dynamically</strong></a>.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/fc5c099f3d79a39ffa8776a8b743c156/href">https://medium.com/media/fc5c099f3d79a39ffa8776a8b743c156/href</a></iframe><p>To compile the project we’ll want to make sure the VK_USE_PLATFORM_ANDROID_KHR is set, as this is what enables the Android configuration. For this project we also disable the Vulkan debug layers with KOMPUTE_DISABLE_VK_DEBUG_LAYERS.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/85bb6732f2852513327a9f1b78bfd5c5/href">https://medium.com/media/85bb6732f2852513327a9f1b78bfd5c5/href</a></iframe><p>Finally, we are able to link the relevant libraries to our kompute-jni shared library target. This includes:</p><ul><li>kompute — This is the library created in the Kompute build.</li><li>kompute_vk_ndk_wrapper — This library also gets created by the Kompute build and contains the code to dynamically load and wrap the Vulkan library.</li><li>log — This is the Android log library, which is required by Kompute to override logging.</li><li>android — This is the Android library which is used by the Android project.</li></ul><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/e3ed88e4a03c8200f5a93f2a91d845f6/href">https://medium.com/media/e3ed88e4a03c8200f5a93f2a91d845f6/href</a></iframe><p>That’s it — you are now able to run the application, which will execute the full build. You should then be able to see the Kompute app in your Android Studio emulator, or in you physical phone, where you’ll be able to trigger the processing on your on-device GPU.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*X-q-MWIg4rOlyhBTrJiMUA.png" /><figcaption>Android Studio Running Project in Emulator (Image by Author)</figcaption></figure><h3>What’s next?</h3><p>Congratulations, you’ve made it all the way to the end! Although there was a broad range of topics covered in this post, there is a massive amount of concepts that were skimmed through. These include the underlying Android Development workflows, Vulkan concepts, GPU computing fundamentals, machine learning best practices, and more advanced Kompute concepts. Luckily, there are resources online to expand your knowledge on each of these. Here are some links I recommend for further reading:</p><ul><li>“<a href="https://towardsdatascience.com/machine-learning-and-data-processing-in-the-gpu-with-vulkan-kompute-c9350e5e5d3a">Machine Learning in Mobile &amp; Cross-Vendor GPUs Made Simple With Kompute &amp; Vulkan</a>” article with a deeper dive in theory and concepts</li><li>“<a href="https://towardsdatascience.com/parallelizing-heavy-gpu-workloads-via-multi-queue-operations-50a38b15a1dc">Parallelizing GPU-intensive Workloads via Multi-Queue Concurrency using Kompute</a>” with more advanced Kompute concepts</li><li><a href="https://axsaucedo.github.io/vulkan-kompute/">Kompute Documentation</a> for more details and further examples</li><li><a href="https://ethical.institute/mle.html">The Machine Learning Engineer Newsletter</a> if you want to keep updated on articles around Machine Learning</li><li><a href="https://developer.android.com/ndk/guides">Android NDK Getting Started Documentation</a> if you want to dive further into the Native Development Kit</li><li><a href="https://github.com/EthicalML/awesome-production-machine-learning/">Awesome Production Machine Learning</a> list for open source tools to deploy, monitor, version and scale your machine learning</li><li><a href="https://www.fast.ai/2018/09/26/ml-launch/">Introduction to ML for Coders course</a> by FastAI to learn further machine learning concepts</li><li><a href="https://vulkan-tutorial.com/">Vulkan SDK Tutorial</a> for a deep dive into the underlying Vulkan components</li></ul><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=1e9da37b7617" width="1" height="1" alt=""><hr><p><a href="https://medium.com/data-science/gpu-accelerated-machine-learning-in-your-mobile-applications-using-the-android-ndk-vulkan-kompute-1e9da37b7617">Supercharging your Mobile Apps: GPU Accelerated Machine Learning using Android NDK &amp; Vulkan Kompute</a> was originally published in <a href="https://medium.com/data-science">TDS Archive</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Real Time Machine Learning at Scale using SpaCy, Kafka & Seldon Core]]></title>
            <link>https://medium.com/data-science/real-time-stream-processing-for-machine-learning-at-scale-with-spacy-kafka-seldon-core-6360f2fedbe?source=rss-32de426f7278------2</link>
            <guid isPermaLink="false">https://medium.com/p/6360f2fedbe</guid>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[realtime]]></category>
            <category><![CDATA[production]]></category>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[scale]]></category>
            <dc:creator><![CDATA[Alejandro Saucedo]]></dc:creator>
            <pubDate>Mon, 05 Oct 2020 23:08:27 GMT</pubDate>
            <atom:updated>2020-10-28T18:08:07.265Z</atom:updated>
            <content:encoded><![CDATA[<h4>A hands on tutorial that covers how to train a machine learning model using the Reddit comment moderation dataset, and deploy it in a scalable infrastructure using Kafka and Seldon Core</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*LuX2T2W-BDiEq9yevrQMAQ.png" /><figcaption>Model Deployment with Seldon Core for Stream Processing (Image by Author)</figcaption></figure><p>In this post, we will cover how to train and deploy a machine learning model leveraging a scalable stream processing architecture for an automated text prediction use-case. We will be using <a href="https://github.com/scikit-learn/scikit-learn">Sklearn </a>and <a href="https://spacy.io/">SpaCy </a>to train an ML model from the Reddit Content Moderation dataset, and we will deploy that model using Seldon Core for real time processing of text data from Kafka real-time streams. You can also find an overview of the content in this post in video form, presented at the NLP Summit 2020.</p><iframe src="https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Fwww.youtube.com%2Fembed%2FT0pPn5KTxFE%3Ffeature%3Doembed&amp;display_name=YouTube&amp;url=https%3A%2F%2Fwww.youtube.com%2Fwatch%3Fv%3DT0pPn5KTxFE&amp;image=https%3A%2F%2Fi.ytimg.com%2Fvi%2FT0pPn5KTxFE%2Fhqdefault.jpg&amp;key=a19fcc184b9711e1b4764040d3dc5c07&amp;type=text%2Fhtml&amp;schema=youtube" width="854" height="480" frameborder="0" scrolling="no"><a href="https://medium.com/media/46402171878fc60e2e59d47fadea2a48/href">https://medium.com/media/46402171878fc60e2e59d47fadea2a48/href</a></iframe><p>You can find the full code for this article in the following links:</p><ul><li><a href="https://docs.seldon.io/projects/seldon-core/en/latest/examples/sklearn_spacy_text_classifier_example.html">Seldon Model Containerisation Notebook</a></li><li><a href="https://github.com/axsaucedo/reddit-classification-exploration/">Reddit Dataset Exploratory Data Analysis Notebook</a></li><li><a href="https://github.com/SeldonIO/seldon-core/blob/master/examples/kafka/sklearn_spacy/README.ipynb">Kafka Seldon Core Stream Processing Deployment Notebook</a></li></ul><h3>Model Training with SpaCy &amp; Sklearn</h3><p>For this use-case we will be using the <a href="https://www.kaggle.com/areeves87/rscience-popular-comment-removal/kernels">Reddit /r/science Content Moderation Dataset</a>. This dataset consists of over 200,000 reddit comments — primarily labelled based on whether the comments have been removed by moderators. We’ll be tasked to train an ML model that is able to predict the comments that would have been removed by reddit moderators.</p><p>SpaCy is one of the most popular and actively used NLP libraries for production text processing use-cases — it provides “industrial-strength” capabilities including tokenization, NER, deep learning integration, and more across a broad range of language models. Similarly, Sklearn is the most popular machine learning toolkit in Python. We’ll be using both of these frameworks for the exploratory data (and model) analysis, as well as for the training of our machine learning model.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*KJTx3Semz8_yMAGxiEMdCA.png" /><figcaption>Exploratory Data Analysis Notebook (Image by Author)</figcaption></figure><p>When it comes to building a machine learning model — especially when it is to be deployed to a production system — it is important that <a href="https://ethical.institute/principles.html">principles and best practices</a> are used to ensure its responsible design, development and operation.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/490/0*LhNvLCBGf6mnGlbb" /><figcaption>Algorithm Comparison Chart (Image by Author)</figcaption></figure><p>You will be able to find best practices and techniques of <a href="https://github.com/axsaucedo/reddit-classification-exploration/blob/master/Exploratory%20Data%20Analysis.ipynb">exploratory data analysis</a> in the <a href="https://github.com/axsaucedo/reddit-classification-exploration/blob/master/Data%20Preprocessing.ipynb">model training</a> notebooks. These covers feature engineering, data distributions, data imbalances, data cleaning, algorithm performance comparison, tokenization approaches, partial dependency plots, etc.</p><p>These techniques enable data scientists and engineers to identify domain specific best practices for the training, deployment and even produciton monitoring of the machine learning models.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*PJ1ysDXkEV2feruW_FOpbQ.png" /><figcaption>Base NLP Pipeline (Image by Author)</figcaption></figure><p>In this tutorial, we will be building the machine learning pipeline using <a href="https://scikit-learn.org/stable/">Sklearn </a>and <a href="https://spacy.io/">SpaCy</a>. As outlined in the image above, we use the following components in the pipeline:</p><ul><li>CleanTextTransformer — Cleans input text by removing relevant characters, symbols and repeated phrases</li><li>SpacyTokenTransformer — Converts the input text into tokens using SpaCy, and removes relevant tokens such as stopwords</li><li>TfidfVectorizer — Converts the tokens into vectors that then can be processed by the model</li><li>LogisticRegression — Logistic regression model trained on our current dataset</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*TJ-tSEOKDGHv77ppzAGq6A.png" /><figcaption>Model Inference Stages (Image by Author)</figcaption></figure><p>For a more intuitive flow of how our model will process the data, the image above shows how a text data input instance would be transformed during each of the stages. More specifically, if we receive a phrase, it will be sent through each of the pipeline components until we’re able to receive a prediction.</p><p>Once we train our model, we will be able to use the same code that was used for the pipeline, and we will export the artifacts of the models trained using pickle so we can load them when deploying the model.</p><h3>Model Containerisation with Seldon Core</h3><p>Seldon Core can be used to convert our model into a scalable microservice and deploy it to Kubernetes using the Seldon cloud native kubernetes operator. Models deployed with Seldon Core support REST and GRPC interfaces, but since version 1.3 it also supports native kafka interface, which we’ll be using in this article.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*LuX2T2W-BDiEq9yevrQMAQ.png" /><figcaption>Model Deployment with Seldon Core (Image by Author)</figcaption></figure><p>Seldon provides several ways to productionise machine learning models. The most common approach is using one of the existing <a href="https://docs.seldon.io/projects/seldon-core/en/latest/servers/overview.html"><strong>prepackaged-model server</strong>s</a>. In this case, however, we will be building our own <a href="https://docs.seldon.io/projects/seldon-core/en/latest/servers/custom.html"><strong>custom model server</strong></a> by extending the default <a href="https://docs.seldon.io/projects/seldon-core/en/latest/servers/sklearn.html"><strong>sklearn pre-packaged server</strong></a> to add SpaCy and its respective English language model.</p><p>To containerise our model with Seldon, we will be following the standard Seldon Core workflow using the <a href="https://docs.seldon.io/projects/seldon-core/en/latest/python/python_wrapping_s2i.html"><strong>Python Language wrapper</strong></a>. As shown in the image below, the standard steps required to containerise a model are:</p><ul><li>Create a <strong>Python Wrapper Class</strong> to expose the model logic</li><li>Add <strong>Python dependencies</strong> via requirements.txt file</li><li>Add <strong>environment parameters</strong> such as protocol, log level, etc</li><li>Optionally add any <strong>further dependencies</strong> with Dockerfile extension</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*0au-EFdV7QLGPO_d6NMi2Q.png" /><figcaption>Standard Steps Required to Containerise Models (Image by Author)</figcaption></figure><p>In our case we will just have to define a Python wrapper that will consist of:</p><ul><li>Importing the code for the ML pipeline used in the training section of this article</li><li>An __init__ method that loads the model artifacts</li><li>The predict method that gets called every time a request is sent</li></ul><p>The code for the wrapper can be found in the image below, or alternatively you can find the full <a href="https://docs.seldon.io/projects/seldon-core/en/latest/examples/sklearn_spacy_text_classifier_example.html">Jupyter notebook </a>to train, containerise and test the Seldon wrapped model.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/816/1*EsNCyUYAsyme0Bb5W7xZZQ.png" /></figure><p>Once we have the wrapper, we are able to simply run the Seldon utilities using the s2i CLI, namely:</p><blockquote>s2i build . seldonio/seldon-core-s2i-python3:1.3.0 nlp-model:0.1</blockquote><p>Now we have fully containerised our model as the image nlp-model:0.1 and we will be able to deploy it for stream processing in the next section.</p><h3>Kafka Stream Processing</h3><p>Seldon models support REST, GRPC and Kafka protocols — in this example we will be using the latter to support stream processing.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*8ethNkBOij8UCbtSC-GBYA.png" /><figcaption>Kafka ML Processing Architecture (Image by Author)</figcaption></figure><p>You are able to find the full <a href="https://github.com/SeldonIO/seldon-core/blob/master/examples/kafka/sklearn_spacy/README.ipynb">jupyter notebook</a> for the example including the deployment files as well as the request workflows.</p><p>We have the following components:</p><ul><li><strong>Kubernetes Cluster </strong>— The Kubernetes cluster where all our components will be deployed to</li><li><strong>Reddit Source</strong> — Component that will produce input data in the form of a stream of reddit comments</li><li><strong>Kafka Queue </strong>— The Kafka cluster component that will process the streams of data provided by consumers and producers</li><li><strong>Seldon ML Service</strong> — Our containerised reddit classifier model deployed using Seldon</li><li><strong>Kafka </strong><strong>model_input topic</strong> — The input topic where the Reddit Source will produce to, and where our Seldon Model will consume from</li><li><strong>Kafka </strong><strong>model_output topic </strong>— The output topic where other applications can consume from</li><li><strong>Other Applications</strong> — In the case of our example this is a single consumer application, however this could include any other applications that want to consume from the model_output topic</li></ul><p>For simplicity we will skip the steps required to set up the kubernetes cluster — including setting up the Kafka brokers and installing Seldon Core — but you can find <a href="https://github.com/SeldonIO/seldon-core/blob/master/examples/kafka/sklearn_spacy/README.ipynb">the full instructions</a> in the notebook example.</p><p>Now we’re able to deploy our model. For this we will just need to define our deployment configuration file following the SeldonDeployment schema:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/820/1*k0XD4oJXiZbL4_dmNhor6A.png" /><figcaption>SeldonDeployment Kafka Configuration File (Image by Author)</figcaption></figure><p>As you can see in the configuration YAML, the structure contains the following key points:</p><ul><li>The name of the model name: reddit-kafka</li><li>The graph.name: classifier is the model we will deploy</li><li>The serverType: kafka enables the deployed microservice with the kafka interface (as opposed to the REST or GRPC protocol)</li><li>KAFKA_BROKER the server for the Kafka broker</li><li>KAFKA_INPUT_TOPIC The name of the input topic to consume from</li><li>KAFKA_OUTPUT_TOPIC The name of the output topic to produce to</li></ul><p>We can now create the model using the kubectl command:</p><blockquote>kubectl apply -f sdep_reddit_kafka.yaml</blockquote><p>Once our model is created we can now send data into the input topic. We can do so by using the kafka-console-producer.sh utility.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/747/1*vEbfCWDfdy4JB92SKR-Hgw.png" /></figure><p>Similarly, we can also listen for the output data that the model produces.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/783/1*jf-Nv6nMFXA1pMF513yVvA.png" /></figure><p>Now when we send the input data to the input topic:</p><blockquote>{”data”: {”ndarray”: [”This is an input”]}}</blockquote><p>We would consequently see the prediction in the output topic stream:</p><blockquote>{“data”:{“names”:[“t:0”,”t:1&quot;],”ndarray”: [[0.6758450844706712, 0.32415491552932885]]},”meta”:{}}</blockquote><p>With this we now have a model deployed in a scalable architecture for real time machine learning processing — more specifically, this architecture allows for horizontal and vertical scalability of each of the respective components. The deployed models can scale to a variable number of replicas, together with autoscaling based on Kubernetes HPA that would horizontally scale based on resource usage. Similarly, Kafka can also scale horizontally through the number of brokers, which enables for large throughputs with low latency. This can be seen more intuitively in the diagram below.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*EzO7Bh1V8hoFkeL-OEMzew.png" /><figcaption>Scaling Replicas and Brokers (Image by Author)</figcaption></figure><h3>What’s next?</h3><p>Now that you have the intuition and understanding of the core architectural components, you are now able to delve into the practical details. For this you will be able to access the code and resources in the following links:</p><ul><li><a href="https://docs.seldon.io/projects/seldon-core/en/latest/examples/sklearn_spacy_text_classifier_example.html">Seldon Model Containerisation Notebook</a></li><li><a href="https://github.com/axsaucedo/reddit-classification-exploration/">Reddit Dataset Exploratory Data Analysis Notebook</a></li><li><a href="https://github.com/SeldonIO/seldon-core/blob/master/examples/kafka/sklearn_spacy/README.ipynb">Kafka Seldon Core Stream Processing Deployment Notebook</a></li></ul><p>If you are interested in further hands on examples of scalable deployment strategies of machine learning models, you can check out:</p><ul><li><a href="https://docs.seldon.io/projects/seldon-core/en/latest/examples/argo_workflows_batch.html">Batch Processing with Argo Workflows</a></li><li><a href="https://docs.seldon.io/projects/seldon-core/en/latest/streaming/knative_eventing.html">Serverless eventing with Knative</a></li><li><a href="https://docs.seldon.io/projects/seldon-core/en/latest/analytics/explainers.html">AI Explainability Patterns with Alibi</a></li></ul><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=6360f2fedbe" width="1" height="1" alt=""><hr><p><a href="https://medium.com/data-science/real-time-stream-processing-for-machine-learning-at-scale-with-spacy-kafka-seldon-core-6360f2fedbe">Real Time Machine Learning at Scale using SpaCy, Kafka &amp; Seldon Core</a> was originally published in <a href="https://medium.com/data-science">TDS Archive</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Supercharging Game Development with GPU Accelerated Machine Learning]]></title>
            <link>https://medium.com/data-science/supercharging-game-development-with-gpu-accelerated-ml-using-vulkan-kompute-the-godot-game-engine-4e75a84ea9f0?source=rss-32de426f7278------2</link>
            <guid isPermaLink="false">https://medium.com/p/4e75a84ea9f0</guid>
            <category><![CDATA[game-development]]></category>
            <category><![CDATA[godot]]></category>
            <category><![CDATA[hands-on-tutorials]]></category>
            <category><![CDATA[vulkan]]></category>
            <category><![CDATA[machine-learning]]></category>
            <dc:creator><![CDATA[Alejandro Saucedo]]></dc:creator>
            <pubDate>Sun, 27 Sep 2020 14:46:29 GMT</pubDate>
            <atom:updated>2021-08-21T13:01:36.250Z</atom:updated>
            <content:encoded><![CDATA[<h4><a href="https://towardsdatascience.com/tagged/hands-on-tutorials">Hands-on Tutorials</a></h4><h3>Supercharging Game Development with GPU Accelerated Machine Learning using the Godot Engine, Kompute Framework and Vulkan SDK</h3><h4>A hands on tutorial that teaches you how integrate GPU optimized ML code in your game development workflows using the Godot Game Engine and the Kompute framework</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/900/1*0b22fD_oJ6vhlWaJkIj4IQ.gif" /><figcaption>Image by Author</figcaption></figure><p>Recently, the world has seen various defining milestones in both, the gaming industry and the AI sector. In only a couple of weeks, we have seen major financial announcements in the gaming industry including <a href="https://venturebeat.com/2020/09/18/unity-technologies-raises-more-than-1-1-billion-in-ipo-at-12-1-billion-valuation/">Unity’s $1.3 Billion IPO</a> and <a href="https://venturebeat.com/2020/08/06/epic-games-unveils-1-78-billion-funding-round-at-17-3-billion-valuation/">Epic Games’ $1.78B Investment</a>. The AI sector has also been catching up with its hype, reaching a <a href="https://www.grandviewresearch.com/industry-analysis/artificial-intelligence-ai-market">$60+ Billion Market in 2020</a>, and bringing mind blowing applications in the intersection of AI and Gaming, including <a href="https://deepmind.com/research/case-studies/alphago-the-story-so-far">AlphaGo’s famous victory over Champion Lee Sedol</a>, as well as deep learning powered games such as <a href="https://play.aidungeon.io/main/landing">AI Dungeon</a> (and <a href="https://en.wikipedia.org/wiki/Artificial_intelligence_in_video_games">many more applications</a>).</p><p>This article provides a technical deep dive into this intersection between the two fields, applied artificial intelligence and game development. We delve specifically into how you can leverage the power of the cross-vendor / mobile GPU frameworks for accelerated processing of machine learning and advanced data processing use-cases.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*B14elpCp2BwwjDIg7Fgu2g.png" /><figcaption>Simple game interface for this tutorial (Image by Author)</figcaption></figure><p>In this tutorial you’ll learn how to use the <a href="https://github.com/EthicalML/vulkan-kompute"><strong>Kompute framework</strong></a> to build GPU optimized code inside the popular open source <a href="https://github.com/godotengine/godot"><strong>Godot Game Engine</strong></a>.</p><p>You will understand how machine learning and advanced GPU compute can be leveraged in game development through the Godot Game engine.</p><p>No background knowledge <strong>beyond programming experience</strong> is required, but if you are curious about the underlying AI / GPU compute concepts referenced, we suggest checking out our previous article, “<a href="https://towardsdatascience.com/machine-learning-and-data-processing-in-the-gpu-with-vulkan-kompute-c9350e5e5d3a">Machine Learning in Mobile &amp; Cross-Vendor GPUs Made Simple With Kompute &amp; Vulkan</a>”.</p><p>You can find the full code in the <a href="https://github.com/EthicalML/vulkan-kompute/tree/master/examples/godot_logistic_regression"><strong>example folder in the repository</strong></a>, together with the <a href="https://github.com/EthicalML/vulkan-kompute/tree/master/examples/godot_logistic_regression/gdnative_shared"><strong>GDNative Library</strong></a> implementation and the <a href="https://github.com/EthicalML/vulkan-kompute/tree/master/examples/godot_logistic_regression/custom_module"><strong>Godot Custom Module</strong></a> implementation.</p><h3>The Godot Game Engine</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*afuXUNGpqOa4-Wh7.jpg" /><figcaption>Image from Godot <a href="https://github.com/godotengine/godot">Open Source GitHub Repository</a></figcaption></figure><p>With over <a href="https://github.com/godotengine/godot">30k github stars and more than 1k contributors</a> Godot is the most popular OSS game engine. Godot <a href="https://www.youtube.com/watch?v=KjX5llYZ5eQ">caters for 2D and 3D development</a>, and has been used for a broad range of mobile, desktop, console and web compatible games / applications. Godot is built in C++ making it fast and light — it’s only <a href="https://godotengine.org/download/windows">a 40MB download</a>.</p><p>Godot is very intuitive for newcomers through <a href="https://docs.godotengine.org/en/stable/getting_started/step_by_step/godot_design_philosophy.html">robust design principles</a>, and support for <a href="https://docs.godotengine.org/en/stable/getting_started/step_by_step/godot_design_philosophy.html#all-inclusive-package">high-level languages</a>, including its domain-specific-language <a href="https://docs.godotengine.org/en/stable/getting_started/scripting/gdscript/gdscript_basics.html">GdScript</a> which has a Python-like syntax, making it very easy to adopt. It is also possible to develop using C#, Python, Visual Scripting, C++, etc.</p><p>In this tutorial we will be building a Game using the editor, using GdScript to trigger the ML training / inference, and C++ to develop the core processing components under-the-hood.</p><h3>Enter Kompute &amp; the Vulkan SDK</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/512/0*mzEbYNdCJ129eSz3" /><figcaption>Playing “where’s waldo” with Khronos Membership (Image by Vincent Hindriksen via <a href="https://streamhpc.com/blog/2017-05-04/what-is-khronos-as-of-today/">StreamHPC</a>)</figcaption></figure><p>Vulkan is an Open Source project led by the <a href="https://www.khronos.org/">Khronos Group</a>, a consortium of a very large number of tech companies who have come together to work towards defining and advancing the open standards for mobile and desktop media (and compute) technologies.</p><p>Large number of high profile (and new) frameworks have been adopting Vulkan as their core GPU processing SDK. The Godot Engine itself is working on a major 4.0 update that will <a href="https://godotengine.org/article/vulkan-progress-report-5">bring Vulkan as its core rendering engine</a>.</p><p>As you can imagine, the Vulkan SDK provides very low-level access to GPUs, which allows for very specialized optimizations. This is a great asset for data processing and GPU developers — the main disadvantage is the verbosity involved, requiring 500–2000+ lines of code to only get the base boilerplate required to even start writing the application logic. This can result in expensive developer cycles and errors that can lead to larger problems. This was one of the main motivations for us to start the <strong>Kompute </strong>project.</p><p><a href="https://github.com/axsaucedo/vulkan-kompute#vulkan-kompute"><strong>Kompute</strong></a> is a framework built on top of the Vulkan SDK, specifically designed to extend its compute capabilities as a simple to use, highly optimized, and mobile friendly General Purpose GPU computing framework.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*hkFiBPfbdHQqhHKA.png" /><figcaption>Kompute <a href="https://ethicalml.github.io/vulkan-kompute/">Documentation</a> (Image by Author)</figcaption></figure><p>Kompute was not built to hide any of the core Vulkan concepts — the core Vulkan API is very well designed. Instead it augments Vulkan’s computing capabilities with a BYOV (bring your own Vulkan) design, enabling developers by reducing boilerplate code required and automating some of the more common workflows involved in writing Vulkan applications.</p><p>For new developers curious to learn more, it provides a solid base to get started into GPU computing. For more advanced Vulkan developers, Kompute allows them to integrate it into their existing Vulkan applications, and perform very granular optimizations by getting access to all of the Vulkan internals when required. The project is fully open source, and we welcome bug reports, documentation extensions, new examples or suggestions — please feel free to <a href="https://github.com/axsaucedo/vulkan-kompute/issues">open an issue</a> in the repo.</p><h3>Artificial Intelligence in Game Development</h3><p>In this post we will be building upon the Machine Learning use-case we created in the “<a href="https://towardsdatascience.com/machine-learning-and-data-processing-in-the-gpu-with-vulkan-kompute-c9350e5e5d3a">Machine Learning in Mobile &amp; Cross-Vendor GPUs Made Simple With Kompute &amp; Vulkan</a>” article. We will not be covering the underlying concepts in as much detail as in that article, but we’ll still introduce the high level intuition required in this section.</p><p>To start with, we will need an interface that allows us to expose our Machine Learning logic, which will require primarily two functions:</p><ol><li>train(…) — function which will allow the machine learning model to learn to predict outputs from the inputs provided</li><li>predict(...)— function that will predict the output of an unknown instance. This can be visualised in the two workflows outlined in the image below.</li></ol><figure><img alt="" src="https://cdn-images-1.medium.com/max/791/0*ZE0_pQd6kjy0y5I0" /><figcaption>Data Science Process (Image by Author)</figcaption></figure><p>Particularly in game development, this would also be a common pattern for machine learning workflows, for both predictive and explanatory modelling use cases. This often consists of leveraging data generated by your users as they interact directly (or indirectly) with the game itself. This data can then serve as training features for machine learning models. Training of new models can be performed through manual “offline” workflows that data scientists would carry out, or alternatively through automated triggers retraining models.</p><h3>Machine Learning Project in Godot</h3><p>We will start by providing a high level overview of how our Kompute bindings will be used in our game in Godot. We will create a simple project and train the ML model we created, which will run in the GPU. In this section we will already have the custom KomputeModelML Godot class accessible — details on how to build it and import it into the godot project are covered in a latter section.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*-fnWpycTHYs6pimF86wUWQ.png" /><figcaption>New Project Screen (Image by Author)</figcaption></figure><p>For this we are using <a href="https://godotengine.org/download">Godot version <strong>3.2.3 stable</strong></a>. Create a new project and a new 2D scene. You should see a blank project with a top level 2D node, as per the image on the left.</p><p>Now we’ll be able to start adding resources to our game. We will start by creating a simple UI interface / menu consisting of inputs for the machine learning model that reflects the workflows covered in the previous architecture diagram.</p><p>We will have two input LineEdit text boxes for the data X_i and X_j, and one input LineEdit text box for the Y expected predictions. The image below shows the node structure used to build the UI. You can also access the <a href="https://github.com/EthicalML/vulkan-kompute/tree/godot_example/examples/godot_logistic_regression">full godot project in the repo</a> by importing the project.godot file. The nodes will be referenced to read the input data and display the output predictions (and learned parameters).</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*bLv-Vx2L2-oUwAppk2qJyw.png" /></figure><p>We will now be able to add the GdScript code in Godot which will allow us to read the inputs, train the model and perform predictions — this script is created under the Parentnode.</p><p>Below is the full script that we are using to perform the processing, which is using the KomputeModelML custom Godot class we build in the next section. We’ll break down each of the different areas of the code below.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/73d8e34655ad987f8421601f2293e743/href">https://medium.com/media/73d8e34655ad987f8421601f2293e743/href</a></iframe><p>First we define variables to make the referencing of Godot Editor nodes simpler. This can be done with the dollar sign syntax $NODE/PATH as implemented in the snipped below.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/f78fbe1335e2c0351fb5b1e8c6a97f03/href">https://medium.com/media/f78fbe1335e2c0351fb5b1e8c6a97f03/href</a></iframe><p>The compute_ml() function below contains the logic for the machine learning training and predictions. We start by reading the inputs from the text boxes using the Godot editor node references.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/700b9ed0fcb15c05360103d54a4c9876/href">https://medium.com/media/700b9ed0fcb15c05360103d54a4c9876/href</a></iframe><p>We now can create an instance from the C++ class bindings that we will be constructing in the next section. This is the class that exposes the training and prediction functions that we will be using for machine learning inference.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/97a6c5b6aa50d4f3a17e90ecfff75c5a/href">https://medium.com/media/97a6c5b6aa50d4f3a17e90ecfff75c5a/href</a></iframe><p>We now can train our model by passing the inputs and the expected predictions. The ML model underneath is <a href="https://towardsdatascience.com/machine-learning-and-data-processing-in-the-gpu-with-vulkan-kompute-c9350e5e5d3a#6c88">a logistic regression model</a>, which will adjust its internal parameters to fit the inputs and outputs best, resulting in a model that is then able to predict unseen datapoints.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/870008e5e165c56a43974c9b1dea9213/href">https://medium.com/media/870008e5e165c56a43974c9b1dea9213/href</a></iframe><p>Now that we have trained our model, we can perform predictions on unseen data-points. For simplicity, we will pass in the same inputs we used for testing, however you could pass completely new arrays and see what predictions it produces. We can then display the results in the preds_node reference node variable that we defined, which will show in the display.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/25de668f24bfda500c4ece82cc78750c/href">https://medium.com/media/25de668f24bfda500c4ece82cc78750c/href</a></iframe><p>Finally we want to also display the learned parameters, which in this case it includes w1 , w2 and the bias . We are able to display the weights and bias in the respective labels.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/38776256dc31e5c31ab5c0b3133f21b8/href">https://medium.com/media/38776256dc31e5c31ab5c0b3133f21b8/href</a></iframe><p>The final thing to set up is to connect the “Kompute Train &amp; Predict” button into the compute_ml function, which can be done by setting up a signal that points to the function itself via the editor.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*UXhGqmr_upim8DBiW0-mUA.png" /><figcaption>Setting up the Signal to the compute_ml method in our script (Image by Author)</figcaption></figure><p>Once this is all set up, we can run the Game and trigger the training and prediction using the provided inputs. We can then see the learned parameters as well as the prediction outputs. It is also possible to see how the learned parameters change as we modify the inputs y, xi and xj.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*7uz9M7sh3H3hzn80VTCudA.png" /><figcaption>Kompute ML Godot interface with resulting parameters (Image by Author)</figcaption></figure><h3>Kompute ML Implementation</h3><p>Now that there is an intuition on what is happening in the game level, we are able to code our underlying C++ class to create the Godot bindings using the <a href="https://github.com/EthicalML/vulkan-kompute"><strong>Kompute framework</strong></a>.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/462/0*c-LUmEwPLAKgZ3t4.jpg" /><figcaption>Kompute <a href="https://ethicalml.github.io/vulkan-kompute/overview/reference.html">Architecture Design</a> (Image by Author)</figcaption></figure><p>We will be following the design principles of Kompute which are outlined in this accompanying diagram showing the different components. We will be following this workflow to load the data in the GPU and perform the training.</p><p>The header file for the Godot class binding implementation is outlined below, which we will break down in detail. As you can see creating a C++ class with bindings is quite intuitive, and you can see the same functions that we used to call from our Godot GdScript above.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/3cf33c243bf497d094f24dbf1b75e124/href">https://medium.com/media/3cf33c243bf497d094f24dbf1b75e124/href</a></iframe><p>The initial section of the class header file includes:</p><ul><li>Import of the Kompute.hppheader containing all the Kompute dependencies that we’ll use in this project</li><li>The top-level Godot.hpp import is required to ensure all Godot components are available.</li><li>Node2D is the resource that we will be inheriting from, but you can inherit from other classes in <a href="https://docs.godotengine.org/en/stable/development/cpp/inheritance_class_tree.html">the inheritance tree</a> depending on how you plan to use your custom Godot class</li><li>For data passing across the application, we will be using a Godot Array which will handle transfer between GdScript and the Naive C++, as well as a Kompute Tensor which will handle the GPU data management.</li><li>The GODOT_CLASS macro definition extends the class by adding extra Godot related functionality. As we will see below, you will need to use GDCLASS instead when building as a Godot custom module.</li></ul><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/1262e597ec7c68b75eafd84670de8427/href">https://medium.com/media/1262e597ec7c68b75eafd84670de8427/href</a></iframe><p>Following the base functionality, we have to define the core logic:</p><ul><li>void train(Array y, Array xI, Array xJ) —Trains the machine learning model using the GPU native code for the logistic regression model. It takes the input array(s) X, and the array ycontaining the expected outputs.</li><li>Array predict(Array xI, Array xJ) —Perform the inference request. In this implementation it is not using GPU code as generally there tends to be less performance gains through parallelization on the inference side. However there are still expected performance gains if multiple inputs are processed in parallel (which this function allows for).</li><li>Array get_params() —Returns an array containing the learned parameters in the format of [ &lt;weight_1&gt;, &lt;weight_2&gt;, &lt;bias&gt; ].</li></ul><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/264116391fc15a85a37da59232748f99/href">https://medium.com/media/264116391fc15a85a37da59232748f99/href</a></iframe><p>We then are able to declare the methods that will be bound between the C++ and the high level GdScript Godot Engine, and accessible to the editor and broader Game in general. We’ll look briefly below at the code required to register a function.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/bcee4b6955648546aea519ec99a70cc8/href">https://medium.com/media/bcee4b6955648546aea519ec99a70cc8/href</a></iframe><p>For data management we will be using Kompute tensors as well as Arrays — in this case we will only need to “learn” and “persist” the weights and biases of our logistic regression model.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/a7fee3b66cf62e8c2e55d59028f5e57d/href">https://medium.com/media/a7fee3b66cf62e8c2e55d59028f5e57d/href</a></iframe><p>Finally, we also define the shader code, which is basically <a href="https://en.wikipedia.org/wiki/OpenGL_Shading_Language">the code that will be executed as machine code inside of the GPU</a>. Kompute allows us to pass a string containing the code, however for production deployments it is possible to convert the shaders to binary, and also use the utilities available to convert into header files.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/386dc6ac9d0051305e88c6f9d17763af/href">https://medium.com/media/386dc6ac9d0051305e88c6f9d17763af/href</a></iframe><p>If you are interested in the full implementation you can find all the files in the <a href="https://github.com/EthicalML/vulkan-kompute/tree/godot_example/examples/godot_logistic_regression/gdnative_shared">gdnative implementation</a> and <a href="https://github.com/EthicalML/vulkan-kompute/tree/godot_example/examples/godot_logistic_regression/custom_module">custom module</a> implementation folders. Furthermore if you are interested in the theoretical and underlying foundational concepts of these techniques, this is covered fully in <a href="https://towardsdatascience.com/machine-learning-and-data-processing-in-the-gpu-with-vulkan-kompute-c9350e5e5d3a#6c88">our previous post</a>.</p><h3>Compiling and Integrating into Godot</h3><p>Now that we have the base code for our GPU optimized machine learning model, we are now able to proceed to running it in our Godot game engine. There are two main ways in which Godot allows us to add C++ code into our projects:</p><ol><li><a href="https://github.com/EthicalML/vulkan-kompute/tree/godot_example/examples/godot_logistic_regression/gdnative_shared"><strong>GdNative Library Build Instructions</strong> </a>— In Godot you can add your own “GdNative scripts” which are basically C++ classes with bindings to GdScript, which means these can be used and referenced dynamically in the project. This approach works with the standard Godot installation and doesn’t require re-compiling the full editor like the 2nd option.</li><li><a href="https://github.com/EthicalML/vulkan-kompute/tree/godot_example/examples/godot_logistic_regression/custom_module"><strong>Custom Module</strong></a> <a href="https://github.com/EthicalML/vulkan-kompute/tree/godot_example/examples/godot_logistic_regression/gdnative_shared"><strong>Build Instructions</strong></a>— In Godot only the underlying core classes are part of the “core” components; everything else — the UI, the editor, the GdScript language, the networking capabilities — are custom models. Writing a custom module is easy, and this is how we are able to expose some of the core Kompute functionality. This option requires the <strong>full Godot C++ project to be recompiled</strong> with the custom module.</li></ol><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*oy4Goki8duQjekmU3tHoQA.png" /></figure><p>Once you set one of these methods through the instructions, you will be able to access the custom object from within Godot. There are some nuanced implementation differences between each approach.</p><p>We won’t be covering specific build details in the blog post, but you will be able to find the exact code and build instructions in the GdNative Library / Custom Module links above.</p><h3>What’s next?</h3><p>Congratulations, you’ve made it all the way to the end! Although there was a broad range of topics covered in this post, there is a massive amount of concepts that were skimmed through. These include the underlying Vulkan concepts, GPU computing fundamentals, machine learning best practices, and more advanced Kompute concepts. Luckily, there are resources online to expand your knowledge on each of these. Here are some links I recommend for further reading:</p><ul><li>“<a href="https://towardsdatascience.com/machine-learning-and-data-processing-in-the-gpu-with-vulkan-kompute-c9350e5e5d3a">Machine Learning in Mobile &amp; Cross-Vendor GPUs Made Simple With Kompute &amp; Vulkan</a>” article with a deeper dive in theory and concepts</li><li><a href="https://axsaucedo.github.io/vulkan-kompute/">Kompute Documentation</a> for more details and further examples</li><li><a href="https://ethical.institute/mle.html">The Machine Learning Engineer Newsletter</a> if you want to keep updated on articles around Machine Learning</li><li><a href="https://github.com/EthicalML/awesome-production-machine-learning/">Awesome Production Machine Learning</a> list for open source tools to deploy, monitor, version and scale your machine learning</li><li><a href="https://www.fast.ai/2018/09/26/ml-launch/">Introduction to ML for Coders course</a> by FastAI to learn further machine learning concepts</li><li><a href="https://vulkan-tutorial.com/">Vulkan SDK Tutorial</a> for a deep dive into the underlying Vulkan components</li></ul><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=4e75a84ea9f0" width="1" height="1" alt=""><hr><p><a href="https://medium.com/data-science/supercharging-game-development-with-gpu-accelerated-ml-using-vulkan-kompute-the-godot-game-engine-4e75a84ea9f0">Supercharging Game Development with GPU Accelerated Machine Learning</a> was originally published in <a href="https://medium.com/data-science">TDS Archive</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Beyond CUDA: GPU Accelerated C++ for Machine Learning on Cross-Vendor Graphics Cards Made Simple]]></title>
            <link>https://medium.com/data-science/machine-learning-and-data-processing-in-the-gpu-with-vulkan-kompute-c9350e5e5d3a?source=rss-32de426f7278------2</link>
            <guid isPermaLink="false">https://medium.com/p/c9350e5e5d3a</guid>
            <category><![CDATA[gpu-computing]]></category>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[vulkan]]></category>
            <category><![CDATA[gpu-machine-learning]]></category>
            <category><![CDATA[gpu-deep-learning]]></category>
            <dc:creator><![CDATA[Alejandro Saucedo]]></dc:creator>
            <pubDate>Sat, 12 Sep 2020 13:46:46 GMT</pubDate>
            <atom:updated>2021-08-21T13:03:34.140Z</atom:updated>
            <content:encoded><![CDATA[<h3>Beyond CUDA: GPU Accelerated C++ for Machine Learning on Cross-Vendor Graphics Cards Made Simple with Kompute</h3><h4>A hands on introduction into GPU computing with practical machine learning examples using the Kompute Framework &amp; the Vulkan SDK</h4><iframe src="https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Fwww.youtube.com%2Fembed%2FXz4fiQNmGSA%3Fstart%3D1%26feature%3Doembed%26start%3D1&amp;display_name=YouTube&amp;url=https%3A%2F%2Fwww.youtube.com%2Fwatch%3Fv%3DXz4fiQNmGSA&amp;image=https%3A%2F%2Fi.ytimg.com%2Fvi%2FXz4fiQNmGSA%2Fhqdefault.jpg&amp;key=a19fcc184b9711e1b4764040d3dc5c07&amp;type=text%2Fhtml&amp;schema=youtube" width="854" height="480" frameborder="0" scrolling="no"><a href="https://medium.com/media/22cc7c593e2dd7d9c626eda5172c47ed/href">https://medium.com/media/22cc7c593e2dd7d9c626eda5172c47ed/href</a></iframe><p>Machine learning, together with many other advanced data processing paradigms, fits incredibly well to the parallel-processing architecture that GPU computing offers.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*M4MmQ0BsdGYws9At9aLFUg.png" /><figcaption>Image by <a href="https://docs.google.com/presentation/d/1WBLcBmk7J04Zu8cD3ugagPkEpKkDuJpNr-HfroNe4X8/edit#slide=id.p">Author</a></figcaption></figure><p>In this article you’ll learn how to write your own ML algorithm from scratch in GPU optimized code, which will be able to run in virtually any hardware — including your mobile phone. We’ll introduce core GPU &amp; ML concepts and show how you can use the <a href="https://github.com/axsaucedo/vulkan-kompute#vulkan-kompute"><strong>Kompute framework</strong></a> to implement it in only a handful lines of code.</p><p>We will be building first a simple algorithm that will multiply two arrays in parallel, which will introduce the fundamentals of GPU processing. We will then write a Logistic Regression algorithm from scratch in the GPU. You can find the repo with the full code in the following links:</p><ul><li><a href="https://github.com/EthicalML/vulkan-kompute">Kompute Repository</a></li><li>GPU Array Multiplication <a href="https://github.com/axsaucedo/vulkan-kompute/tree/4e3802cb9dbf8742f9caf0374ef1612466c5ab1a/examples/array_multiplication#kompute-array-multiplication-example">Repository</a> and <a href="https://github.com/axsaucedo/vulkan-kompute/blob/4e3802cb9dbf8742f9caf0374ef1612466c5ab1a/examples/array_multiplication/src/Main.cpp#L15">Kompute Code</a></li><li>GPU Logistic Regression <a href="https://github.com/axsaucedo/vulkan-kompute/tree/7906406dd1e8bbfb01c1c5d68be44f63587440aa/examples/logistic_regression#kompute-logistic-regression-example">Repository</a>, <a href="https://github.com/axsaucedo/vulkan-kompute/blob/7906406dd1e8bbfb01c1c5d68be44f63587440aa/examples/logistic_regression/src/Main.cpp#L15">Kompute Code</a> and <a href="https://github.com/axsaucedo/vulkan-kompute/blob/7906406dd1e8bbfb01c1c5d68be44f63587440aa/examples/logistic_regression/shaders/glsl/logistic_regression.comp#L7">Shader Code</a></li></ul><h3>Motivation</h3><p>The potential and adoption of GPU computing has been exploding in recent years — you can get a glimpse of the increasing speed in adoption from the charts in the image below. In deep learning there has been a massive increase in adoption of GPUs for processing, together with paradigms that have enabled massively parallelizable distribution of compute tasks across increasing number of GPU nodes. There is a lot of exciting research around techniques that propose new approaches towards <a href="https://mxnet.apache.org/versions/1.7/api/faq/model_parallel_lstm.html">model parallelism</a> and <a href="https://en.wikipedia.org/wiki/Data_parallelism">data parallelism</a>— both which allow algorithms and data respectively to be sub-divided in a broad range of approaches to maximize processing efficiency.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*EZ27EB0hMtEdgm0d" /><figcaption><strong>Ben-Nun, Tal, and Torsten Hoefler. “Demystifying parallel and distributed deep learning: An in-depth concurrency analysis.” <em>ACM Computing Surveys (CSUR)</em> 52.4 (2019): 1–43.</strong></figcaption></figure><p>In this article we outline the theory, and hands on tools that will enable both, beginners and seasoned GPU compute practitioners, to make use of and contribute to the current development and discussions across these fascinating high-performance computing areas.</p><h3>The Vulkan Framework</h3><p>Before diving right in, it is worth introducing the core framework that is making it possible to build hyper-optimized, cross platform and scalable GPU algorithms — and that is the Vulkan Framework.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/512/0*zdzITvlNrxnHSlan" /><figcaption>Playing “where’s waldo” with Khronos Membership (Image by Vincent Hindriksen via <a href="https://streamhpc.com/blog/2017-05-04/what-is-khronos-as-of-today/">StreamHPC</a>)</figcaption></figure><p>Vulkan is an Open Source project led by the <a href="https://www.khronos.org/">Khronos Group</a>, a consortium of a very large number of tech companies who have come together to work towards defining and advancing the open standards for mobile and desktop media (and compute) technologies. On the left you can see the broad range of Khronos Members.</p><p>You may be wondering, <em>why do we need yet another new GPU framework where there are already many options available for writing parallelizable GPU code?</em> The main reason is that unlike some of its closed source counterparts (e.g. NVIDIA’s CUDA, or Apple’s Metal) Vulkan is fully Open Source, and unlike some of the older options (e.g. OpenGL), Vulkan is built with the modern GPU architecture in mind, providing very granular access to GPU optimizations. Finally, whilst some alternatives provide vendor-specific support for GPUs, Vulkan provides <strong>cross-platform</strong>, and <strong>cross-vendor</strong> support, which means that it opens the doors to opportunities in mobile processing, edge computing, and more.</p><p>The Vulkan SDK provides very low-level access to GPUs, which allows for very specialized optimizations. This is a great asset for GPU developers — the main disadvantage is the verbosity involved, requiring 500–2000+ lines of code to only get the base boilerplate required to even start writing the application logic. This can result not only in expensive developer cycles but also prone to small errors that can lead to larger problems.</p><p>This can actually be seen across many new and renowned machine learning &amp; deep learning projects like Pytorch, Tensorflow, and Alibaba DNN — between others — which have either integrated or are looking to integrate the Vulkan GPU SDK to add mobile GPU (and cross-vendor GPU) support. All of these frameworks end up with very similar and extremely verbose boilerplate code, which means they would have benefited (and could still benefit) from using a unified baseline. This was one of the main motivations for us to start the <strong>Kompute </strong>project.</p><h3>Enter Kompute</h3><p><a href="https://github.com/axsaucedo/vulkan-kompute#vulkan-kompute"><strong>Kompute</strong></a> is a framework built on top of the Vulkan SDK, specifically designed to extend its compute capabilities as a simple to use, highly optimized, and mobile friendly General Purpose GPU computing framework.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*xPB9mQd-SDgHQRc29HRz0A.png" /><figcaption>Kompute <a href="https://ethicalml.github.io/vulkan-kompute/">Documentation</a> (Image by Author)</figcaption></figure><p>Kompute was not built to hide any of the core Vulkan concepts — the Vulkan API is very well designed —instead it augments Vulkan’s Computing capabilities with a BYOV (bring your own Vulkan) design, enabling developers by reducing boilerplate code required and automating some of the more common workflows involved in writing Vulkan applications.</p><p>For new developers curious to learn more, it provides a solid base to get started into GPU computing. For more advanced Vulkan developers, Kompute allows them to integrate it into their existing Vulkan applications, and perform very granular optimizations by getting access to all of the Vulkan internals when required. The project is fully open source, and we welcome bug reports, documentation extensions, new examples or suggestions — please feel free to <a href="https://github.com/axsaucedo/vulkan-kompute/issues">open an issue</a> in the repo.</p><h3>Writing your first Kompute</h3><p>To build our first simple array-multiplication GPU computing application using Kompute, we will create the following:</p><ul><li>Two<strong> Kompute Tensors </strong>to store the input data</li><li>One <strong>Kompute Tensor</strong> to store the output data</li><li>A<strong> Kompute Operation</strong> to create and copy the tensors to the GPU</li><li>A <strong>Kompute Operation</strong> with an Kompute Algorithm that will hold the code to be executed in the GPU (called a “shader”)</li><li>A <strong>Kompute Operation</strong> to sync the GPU data back to the local tensors</li><li>A <strong>Kompute Sequence</strong> to record the operations to send to the GPU in batches (we’ll use the<strong> Kompute Manager </strong>to simplify the workflow)</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/462/0*Env2QAL1QVnE-XeR.jpg" /><figcaption>Kompute <a href="https://ethicalml.github.io/vulkan-kompute/overview/reference.html">Architecture Design</a> (Image by Author)</figcaption></figure><p>At the core of Kompute are “Kompute Operations”, which are used for GPU actions, as well as “Kompute Tensor” operations to handle the GPU data and memory. More specifically, this diagram shows the relationship between Kompute components (including explicit memory ownership).</p><p>When interacting with the GPU, you have to send the instructions to the GPU to execute, and you need to make sure that the GPU has all of the relevant data available in GPU memory to begin processing. With Vulkan you send these instructions to the GPU via a queue, so to simplify things intuitively you can think of your GPU as a remote server, where data serialization, resource creation and memory allocation is expensive, and instructions are submitted via a queue — there is still GPU-CPU shared memory but you tend to only use this for data transfer to the GPU.</p><p>Let’s jump into the code. Typically, in a Kompute application we’ll follow the following steps:</p><ol><li>Create a Kompute Manager to manage resources</li><li>Create Kompute Tensors to hold data</li><li>Initialise the Kompute Tensors in the GPU with a Kompute Operation</li><li>Define the code to run on the GPU as a “compute shader”</li><li>Use Kompute Operation to run shader against Kompute Tensors</li><li>Use Kompute Operation to map GPU output data into local Tensors</li><li>Print your results</li></ol><h4>1. Create a Kompute Manager to manage resources</h4><p>First, we’ll create our Kompute Manager, which is in charge of creating and managing all the underlying Vulkan resources.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/68abaaf0e0712867926190e0a44a85c8/href">https://medium.com/media/68abaaf0e0712867926190e0a44a85c8/href</a></iframe><p>As you can see, here we are initializing our Kompute Manager, expecting it to create all the base Vulkan resources on Device 0 (in my case Device 0 is my NVIDIA card, and Device 1 is my integrated graphics card). For more advanced use-cases it’s also possible to initialize the Kompute Manager with your own Vulkan resources (Device, Queue, etc) but this is out of scope of this article.</p><h4>2. Create Kompute Tensors to hold data</h4><p>We will now create the Kompute Tensors that will be used for input and output. These will hold the data required which will be mapped into the GPU to perform this simple multiplication.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/1e00349ed411ee6b022fd62356e42859/href">https://medium.com/media/1e00349ed411ee6b022fd62356e42859/href</a></iframe><p>The reason why Kompute uses std::shared_ptr by design to avoid passing the objects by value, and instead passing them using <a href="https://docs.microsoft.com/en-us/cpp/cpp/smart-pointers-modern-cpp?view=vs-2019">smart pointers</a>.</p><h4>3. Initialise the Kompute Tensors in the GPU with a Kompute Operation</h4><p>Now that we have our Tensors created with local data, we will map the data into the GPU. For this we will use the <a href="https://axsaucedo.github.io/vulkan-kompute/overview/reference.html#optensorcreate">kp::OpTensorCreate</a> Kompute Operation, which will initialize the underlying Vulkan buffer and GPU memory, and perform the respective mapping into the GPU.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/cdafc2af94cae1d1694203c759044b9d/href">https://medium.com/media/cdafc2af94cae1d1694203c759044b9d/href</a></iframe><p>It’s also worth mentioning that it’s possible to shorten the tensor creation steps by leveraging the Kompute Manager buildTensor helper function. This would allow you to skip the need to create the shared_ptr explicitly as well as the kp::OpTensorCreate Operation as outlined below (you can also find the full code implementation of <a href="https://github.com/axsaucedo/vulkan-kompute/blob/2e8a5aa3a6d6172abb51ac038e1b75c5c2d58af9/test/TestMultipleAlgoExecutions.cpp#L274">this variation here</a>).</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/e13e3ccb00718d98275856a715d942f0/href">https://medium.com/media/e13e3ccb00718d98275856a715d942f0/href</a></iframe><h4>4. Define the code to run on the GPU as a “compute shader”</h4><p>Now that we’ve initialized the necessary Kompute Tensor components and they are mapped in GPU memory, we can add the Kompute Algorithm that will be executed in the GPU. This is referred to as the “shader” code, which follows a C-like syntax. You can see the full shader code below, and we’ll break down each of the sections below.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/050fe177459522d98a9e2fac8279b0cf/href">https://medium.com/media/050fe177459522d98a9e2fac8279b0cf/href</a></iframe><p>The #version 450 and layout(local_size_x = 1) in; sections specify the version and parallel thread execution structure (which we’ll look at further down the article). We then can see the GPU data inputs and outputs defined in the format:</p><blockquote><em>layout(binding = &lt;INDEX&gt;) buffer &lt;UNIQUENAME&gt; {float &lt;VARNAME&gt;[]};</em></blockquote><ul><li>&lt;INDEX&gt; — index that maps Tensors to the GPU input</li><li>&lt;UNIQUENAME&gt; — This must be a unique name for the buffer</li><li>&lt;VARNAME&gt; — This is the variable name to use in the shader code</li></ul><p>These are the parameters that can be used throughout the shader code for processing. Namely in this case, the processing is done inside the main function. The first variable uint index = gl_GlobalInvocationID.x; is the currently parallel execution index which will allow us to process each data input.</p><p>We then come into the core of this algorithm which is the multiplication o[index] = a[index] * b[index]. This part is quite self-explanatory — we multiply the elements of the GPU arrays a[] and b[] , then store the output on the array o[].</p><h4>5. Use Kompute Operation to run shader against Kompute Tensors</h4><p>In order to run the shader above we will create the Kompute Operation <a href="https://axsaucedo.github.io/vulkan-kompute/overview/reference.html#opalgobase">kp::OpAlgoBase</a>. The parameters required for this Kompute Operation includes the Tensors to bind into the GPU instructions, as well as the shader code itself.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/55ad8bc292f34a81a3cb61064d03cae0/href">https://medium.com/media/55ad8bc292f34a81a3cb61064d03cae0/href</a></iframe><p>It’s worth mentioning that Kompute allows the user to also pass the shader through a file path, or alternatively there are also Kompute tools that will allow you to convert the shader binaries into C++ header files.</p><h4>6. Use Kompute Operation to map GPU output data into local Tensors</h4><p>Once the algorithm gets triggered, the result data will now be we held in the GPU memory of our output tensor. We can now use the <a href="https://axsaucedo.github.io/vulkan-kompute/overview/reference.html#optensorsynclocal">kp::OpTensorSyncLocal</a> Kompute Operation to sync the Tensor GPU memory as per the code block below.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/44fffe584be46d4e9623d8b86e88b3b4/href">https://medium.com/media/44fffe584be46d4e9623d8b86e88b3b4/href</a></iframe><h4>7. Print your results</h4><p>Finally, we can print the output data of our tensor.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/8eb1486df02a152cc6b444869a844fb0/href">https://medium.com/media/8eb1486df02a152cc6b444869a844fb0/href</a></iframe><p>When you run this, you will see the values of your output tensor printed. That’s it, you’ve written your first Kompute!</p><p>You can also find the full example in the repo so you can run it and extend it as desired. You can find the full standalone example in <a href="https://github.com/axsaucedo/vulkan-kompute/tree/4e3802cb9dbf8742f9caf0374ef1612466c5ab1a/examples/array_multiplication#kompute-array-multiplication-example">this repository</a> which includes the instructions on how to build it as well as the <a href="https://github.com/axsaucedo/vulkan-kompute/blob/4e3802cb9dbf8742f9caf0374ef1612466c5ab1a/examples/array_multiplication/src/Main.cpp#L15">Kompute C++ code</a>.</p><p>Although it may not seem obvious, the above introduced some intuition around core concepts and design thinking in GPU computing, whilst still abstracting a couple of the more in-depth concepts. In the following sections we will be providing more concrete terminology and we’ll be scratching the surface of some of the more advanced concepts such as threads, blocks, memory strides and shared memory (although a lot will be provided as further reading).</p><h3>Diving into the Machine Learning intuition</h3><p>Let’s look at a more advanced GPU compute use-case, specifically implementing the hello world of machine learning, logistic regression. Before we cover the implementation we will provide some intuition on the theory, and the terminology that we’ll be using throughout.</p><p>In machine learning we always have two stages, training and inference. In the diagram below you can see the two simplified flows. At the top is the training flow, where you identify some training data, extract some features, and train a model until you are happy with the accuracy. Once you have a trained model, you persist the model “weights” and deploy the model into the second workflow, where the model would perform inference on unseen data.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/791/0*qjzd2Q5vinPtO5c6" /><figcaption>Data Science Process (Image by Author)</figcaption></figure><p>In this case we will have an input dataset X , where each element is a pair xi and xj . Our input data will be the following:</p><ul><li>xi = { 0, 1, 1, 1, 1 }</li><li>xj = { 0, 0, 0, 1, 1 }</li></ul><p>With this input data, the expected target value Y to be predicted will be the following:</p><ul><li>Y = {0, 0, 0, 1, 1}</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/485/1*C9Sl7wor6-pydyD7NXcC5Q.png" /><figcaption>Logistic Regression Example from <a href="https://www.datasciencecentral.com/profiles/blogs/why-logistic-regression-should-be-the-last-thing-you-learn-when-b">DS Central</a></figcaption></figure><p>Our primary objective in machine learning is to learn using this data to find the function (and parameters) that will allow us to predict values Y from just using X as input.</p><p>It’s worth noting that the predicted values are defined as ŷ , which are specifically the values computed with our inference function, distinct to the “true” or “actual” values of Y that we defined above.</p><p>The functions that we will be using for logistic regression will be the following:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/108/0*ueYCdrdgQqwteXNe.png" /></figure><p>Let’s break down this function:</p><ul><li>z — is our linear mapping function</li><li>ŷ —is the resulting predicted outputs</li><li>Xᵀ —The transpose of the matrix containing our inputs xi and xj</li><li>σ — The sigmoid function which is covered in more detail below</li></ul><p>And the parameters that we’ll be looking to learn with our machine learning algorithm are:</p><ul><li>W— The weights that will be applied to the inputs</li><li>b — The bias that will be added</li></ul><p>There is also the surrounding function σ which is the sigmoid function. This function forces our input to be closer to 0 or 1, which could be intuitively seen as the probability of our prediction to be “true”, and is defined as following:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/117/0*B4vJKaqNBrPgeAG_.png" /></figure><p>This is now the inference function that will allow us to process predictions from new data points. If we say for example that we have a new unseen set of inputs X = { (0, 1) }, and we assume that the learned parameters were W = (1, 1), b = 0 after running our machine learning algorithm through our training data (which we’ll do later on), then we’ll be able to run this through our prediction function by substituting the values as follows:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/463/0*vPB55aWa874FYfwa.png" /></figure><p>In this case the prediction is 0.73..., which would be a positive prediction. This of course is just to demonstrate what our inference function will look like once we learn the parameters W and b.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/368/1*e88JKNWAFok3vpjeuPfHig.gif" /><figcaption>Gradient descent visualized from <a href="https://mi-academy.com/2018/10/04/the-history-of-gradient-descent/">ML Academy</a></figcaption></figure><p>The way that we will be learning the parameters is by performing a prediction, calculating the error, and then re-adjusting the weights accordingly. The method used to “re-adjust” the weights based on the “prediction error” will be done by leveraging gradient descent. This will be repeated multiple times to find more accurate parameters.</p><p>For this we will need to use the derivatives of each of the formulas. The first one, which is the derivative of our linear mapping function z is:</p><ul><li>∂z = z(x) - y</li></ul><p>Where the variables are defined as follows:</p><ul><li>∂z — The derivative of the linear mapping function z(x)</li><li>z(x) — the result of the linear mapping function applied to input x</li><li>y — the actual value label expected for that input x</li></ul><p>Similarly the derivatives for w and b respectively are the following:</p><ul><li>∂w = (x - ∂z)/m</li><li>∂b = ∂z/m</li></ul><p>In this case m is the total number of input elements.</p><p>We will now be able to re-adjust the parameters using the above as follows:</p><ul><li>w = w - θ · ∂w</li><li>b = b - θ · ∂b</li></ul><p>In this case θ is the learning rate, which as the name suggests controls the ratio by which the parameters will be modified on each iteration. Intuitively, the smaller, the more iterations it will be required for the algorithm to converge, however if the learning is too big, it will overshoot, leading to never being able to converge (from the image above you can imagine it will keep bouncing from side to side never reaching the bottom).</p><p>In order for us to calculate loss, we will be using the log loss function, known also as cross-entropy loss function. This function is defined as follows:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/207/0*TgKVkmzh9rpbnlaD.png" /><figcaption>Log loss (cross entropy loss) function</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*SQh95-Fn58JTDgzX.png" /><figcaption>Intuitive diagram to visualize cost function <a href="https://machinelearningmastery.com/how-to-score-probability-predictions-in-python/">from ML Mastery</a></figcaption></figure><p>The function itself is set up such that the larger the difference between the predicted class and the expected class, the larger the error (you can see how much it punishes if the predicted class is on the complete different label).</p><p>The loss function will provide us an idea of the improvement of our algorithm across iterations.</p><p>Finally, one of the most important points here will be the intuition behind how we can leverage the parallel architecture of the GPU to optimize computation. In this case, we’ll be able to do it by processing multiple input parameters at the same time, referred to as a micro-batch, and then re-adjusting the parameters in batch. This is known as data-parallelization, and is one of many techniques available. In the next section we will see how this is implemented, namely passing a mini-batch of inputs, storing the weights, and then re-adjusting them before the next iteration.</p><blockquote>Note: In this post we won’t delve into much detail, nor best practices on machine learning, however at the end of the article we will be listing a broad range of sources for people interested to take their machine learning (or GPU compute) knowledge to the next level.</blockquote><h3>Machine Learning GPU Code Implementation</h3><p>Now that we have covered some of the core concepts, we will be able to learn about the implementation of the shader, which is the code that will be executed in the GPU.</p><p>First we need to define all the input and output buffers as follows:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/03037ea847cda3b9112f07fefcff4417/href">https://medium.com/media/03037ea847cda3b9112f07fefcff4417/href</a></iframe><p>If you remember, at the end of the last section we mentioned how we will be leveraging the concept of micro-batches in order to use the parallel architecture of GPU processing. What this means in practice, is that we will be passing multiple instances of X to the GPU to process at a time, instead of expecting the GPU to process it one by one. This is why we see that above we have an array for xi, xj, y, wOuti, wOutj, andbOut respectively.</p><p>In more detail:</p><ul><li>The input X as arrays xi and xj will hold the micro-batch of inputs</li><li>The array y will hold all the expected labels for micro-batch inputs</li><li>The two input weight parameters wini and woutj will be used for calculating predictions</li><li>The input parameter b which will be used for calculating the predictions</li><li>The output weights wouti and woutj contains weights and will store the derivative of W for all micro-batches that should be subtracted</li><li>Similarly the output bias array contains the derivatives of b for all micro-batches that should be subtracted in batch</li><li>Finally lout contains the output array where losses will be returned</li></ul><p>We also receive the constant M, which will be the total number of elements — if you remember this parameter will be used for the calculation of the derivatives. We will also see how these parameters are actually passed into the shader from the C++ Kompute side.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/a7e8fc9d64acada87dc8877e4f880f22/href">https://medium.com/media/a7e8fc9d64acada87dc8877e4f880f22/href</a></iframe><p>Now that we have all the input and output parameters defined, we can start the main function, which will contain the implementation of our machine learning training algorithm.</p><p>We will first start by keeping track of the current index of the global invocation. Since the GPU executes in parallel, each of these runs will be running directly in parallel, so this allows the current execution to consistently keep track of what iteration index is currently being executed.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/7644c4d3b0e592c3269d81643aff84b7/href">https://medium.com/media/7644c4d3b0e592c3269d81643aff84b7/href</a></iframe><p>We now can start preparing all the variables that we’ll be using throughout the algorithms. All our inputs are buffer arrays, so we’ll want to store them in vec2 and float variables.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/e3348bd7f3d82faf63ab95c1506c39dd/href">https://medium.com/media/e3348bd7f3d82faf63ab95c1506c39dd/href</a></iframe><p>In this case we’re basically making explicit the variables that are being used for the current “thread run”. The GPU architecture consists of slightly more nuanced execution structures that involve thread blocks, memory access limitations, etc — however we won’t be covering these in this example.</p><p>Now we get into the more fun part —implementing the inference function. Below we will implement the inference function to calculate ŷ, which involves both the linear mapping function, as well as the sigmoid function.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/93c448e155fb98af7cb7594b93e982b4/href">https://medium.com/media/93c448e155fb98af7cb7594b93e982b4/href</a></iframe><p>Now that we have yHat, we can now use it to calculate the derivatives (∂z, ∂w and ∂b), which in this case are the derivative of the currently-executed index input element.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/08439e5f09e0e615297e9746499d4fa1/href">https://medium.com/media/08439e5f09e0e615297e9746499d4fa1/href</a></iframe><p>We can now pass the derivatives as outputs, so the parameters can be re-adjusted for the next iteration.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/87d34f24f07c41aad25616422f908574/href">https://medium.com/media/87d34f24f07c41aad25616422f908574/href</a></iframe><p>Finally we’re able to calculate the loss and add it to the output lout array.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/16ece67730c92bb9f0b77c3d59852ca6/href">https://medium.com/media/16ece67730c92bb9f0b77c3d59852ca6/href</a></iframe><p>That’s it, we’ve now finished the shader that will enable us to train a Logistic Regression algorithm in the GPU — you can find the <a href="https://github.com/axsaucedo/vulkan-kompute/blob/7906406dd1e8bbfb01c1c5d68be44f63587440aa/examples/logistic_regression/shaders/glsl/logistic_regression.comp#L7">full code for the shader</a> in the GPU logistic regression <a href="https://github.com/axsaucedo/vulkan-kompute/tree/7906406dd1e8bbfb01c1c5d68be44f63587440aa/examples/logistic_regression#kompute-logistic-regression-example">example repository</a>.</p><p>Now we’ll cover the Kompute code required to run this code against a dataset to train our first model and find the parameters.</p><h3>Machine Learning Orchestration from Kompute</h3><p>In order to run the shader we created above in the GPU using Kompute, we will follow the following steps:</p><ol><li>Import Kompute and create our main function</li><li>Create all the Kompute Tensors required</li><li>Create the Kompute Manager and initialize a Kompute Sequence</li><li>Execute the Kompute Tensor GPU initialization via Kompute Sequence</li><li>Record batch algorithm execution in Kompute Sequence</li><li>Iterate 100 times: Run micro-batch execution &amp; update weights</li><li>Print resulting parameters to use for further inference</li></ol><p>As you can see this is more involved than the simpler example we used above. In this case we will use the Kompute Sequence instead of the Kompute Manager directly, as we want to have deeper control on the commands that can be recorded to send in batch to the GPU. We will discuss this in more detail as we cover each of the steps. Let’s get started.</p><h4>1. Import Kompute and create our main function</h4><p>We will be importing the single header of Kompute — it’s also possible to use the more granular class-based headers if required. We will also create some of the base configuration variables; namely ITERATIONS and learningRate which will be used in latter code blocks.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/5e1aff3e89436facbea1d21162f62036/href">https://medium.com/media/5e1aff3e89436facbea1d21162f62036/href</a></iframe><h4>2. Create all the Kompute Tensors required</h4><p>Now we’ll be creating all the tensors required. In this sub-section you will notice that we will be referencing all the buffers/arrays that are being used in the shader. We’ll also cover how the order in the parameters passed relates to the way data is bound into the shaders so it’s accessible.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/49482a356a6d1409f2250cca2b1b1625/href">https://medium.com/media/49482a356a6d1409f2250cca2b1b1625/href</a></iframe><p>We also store them in a parameter vector for easier access:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/6c658747c69da6b44947ab00666e8a12/href">https://medium.com/media/6c658747c69da6b44947ab00666e8a12/href</a></iframe><h4>3. Create the Kompute Manager and initialize a Kompute Sequence</h4><p>If you remember from the previous example, we were able to execute commands directly using the Kompute Manager. However we are able to use the Kompute Sequence resource if we want further granularity to record command batches that can be submitted and loaded into the GPU before processing. For this, we will create a Kompute Manager, then create a Kompute Sequence through it.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/25e745251a0cb49d77808f99e17df638/href">https://medium.com/media/25e745251a0cb49d77808f99e17df638/href</a></iframe><h4>4. Execute the Kompute Tensor GPU initialization via Kompute Sequence</h4><p>We can now start by running instructions on GPU resources — namely we will start by initialising and map the Tensors with their respective GPU memory. Here you will see how Kompute Sequences provide you with further granularity on the command execution, but it won’t be until the ML inference section that you will see the flexibility of Kompute Sequence.</p><p>Let’s get started by recording commands, namely the OpTensorCreate command, and then evaluating the operation across all the tensors above. This operation will create the respective Vulkan memory/buffer resources.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/1a81e94a158dbc9e98d8861e96fcb9ea/href">https://medium.com/media/1a81e94a158dbc9e98d8861e96fcb9ea/href</a></iframe><h4>5. Record batch algorithm execution in Kompute Sequence</h4><p>In this section we will want to clear the previous recordings of the Kompute Sequence and begin recording a set of sequences. You will notice that unlike the previous section, in this case we won’t be running the eval() straight away as we’ll have to run it multiple times, together with extra commands to re-adjust the parameters.</p><p>You will also notice that we will be recording three types of Kompute Operations, namely:</p><ul><li><a href="https://axsaucedo.github.io/vulkan-kompute/overview/reference.html#optensorsyncdevice">kp::OpTensorSyncDevice</a> — This operation ensures that the Tensors are synchronized with their GPU memory by mapping their local data into the GPU data. In this case, these Tensors use Device-only memory for processing efficiency, so the mapping is performed with a staging Tensor inside the operation (which is re-used throughout the operations for efficiency). Here we’re only wanting to sync the input weights, as these will be updated locally with the respective derivatives.</li><li><a href="https://axsaucedo.github.io/vulkan-kompute/overview/reference.html#opalgobase">kp::OpAlgoBase</a> — This is the Kompute Operation that binds the shader that we wrote above with all the local CPU/host resources. This includes making available the Tensors. It’s worth mentioning that the index of the tensors provided as parameters is the order in which they are mapped in the shaders via their respective bindings (as you can see in the shaders each vector has the format layout(binding = NUMBER).</li><li><a href="https://axsaucedo.github.io/vulkan-kompute/overview/reference.html#optensorsynclocal">kp::OpTensorSyncLocal</a>— This Kompute Operation performs a similar set of instructions as the sync operation above, but instead of copying the data to the GPU memory, it does the converse. This Kompute Operation maps the data in the GPU memory into the local Tensor vector so it’s accessible from the GPU/host. As you can see we’re only running this operation in the output tensors.</li></ul><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/635e32ceabf9ab511f30565333a94a25/href">https://medium.com/media/635e32ceabf9ab511f30565333a94a25/href</a></iframe><h4>6. Iterate 100 times: Run micro-batch execution &amp; update weights</h4><p>Now that we have the command recorded, we can start running executions of these pre-loaded commands. In this case, we will be running the execution of a micro-batch iteration, followed by updating the parameters locally, so they are used in the following iteration.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/4741b2c8a3828e63e9420efce3d52211/href">https://medium.com/media/4741b2c8a3828e63e9420efce3d52211/href</a></iframe><h4>7. Print resulting parameters to use for further inference</h4><p>We now have a trained logistic regression model, or at least we’ve been able to optimize its respective function to identify suitable parameters. We are now able to print these parameters and use the parameters for inference in unseen datasets.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/f19cf52f6b5190d4af9540bb73ede5d0/href">https://medium.com/media/f19cf52f6b5190d4af9540bb73ede5d0/href</a></iframe><p>And we’re done!</p><p>You are able to find this entire example in the example repository, which you’ll be able to run and extend. You will find all the complete files in the GPU Logistic Regression <a href="https://github.com/axsaucedo/vulkan-kompute/tree/7906406dd1e8bbfb01c1c5d68be44f63587440aa/examples/logistic_regression#kompute-logistic-regression-example">example repo</a>, including the <a href="https://github.com/axsaucedo/vulkan-kompute/blob/7906406dd1e8bbfb01c1c5d68be44f63587440aa/examples/logistic_regression/src/Main.cpp#L15">Kompute C++ Code</a>, and the <a href="https://github.com/axsaucedo/vulkan-kompute/blob/7906406dd1e8bbfb01c1c5d68be44f63587440aa/examples/logistic_regression/shaders/glsl/logistic_regression.comp#L7">Shader File</a>.</p><h3>What next?</h3><p>Congratulations, you’ve made it all the way to the end! Although there was a broad range of topics covered in this post, there is a massive amount of concepts that were skimmed through. These include the underlying Vulkan concepts, GPU computing fundamentals, machine learning best practices, and more advanced Kompute concepts. Luckily, there are a broad range of resources online to expand your knowledge on each of these. Some links I recommend as further reading include the following:</p><ul><li><a href="https://axsaucedo.github.io/vulkan-kompute/">Kompute Documentation</a> for more details and further examples</li><li><a href="https://ethical.institute/mle.html">The Machine Learning Engineer Newsletter</a> if you want to keep updated on articles around Machine Learning</li><li><a href="https://github.com/EthicalML/awesome-production-machine-learning/">Awesome Production Machine Learning</a> list for open source tools to deploy, monitor, version and scale your machine learning</li><li><a href="https://www.fast.ai/2018/09/26/ml-launch/">Introduction to ML for Coders course</a> by FastAI to learn further machine learning concepts</li><li><a href="https://vulkan-tutorial.com/">Vulkan SDK Tutorial</a> for a deep dive into the underlying Vulkan components</li></ul><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=c9350e5e5d3a" width="1" height="1" alt=""><hr><p><a href="https://medium.com/data-science/machine-learning-and-data-processing-in-the-gpu-with-vulkan-kompute-c9350e5e5d3a">Beyond CUDA: GPU Accelerated C++ for Machine Learning on Cross-Vendor Graphics Cards Made Simple</a> was originally published in <a href="https://medium.com/data-science">TDS Archive</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Hard work, honesty & strong relationships: Reflecting back on my time at Bloomberg]]></title>
            <link>https://medium.com/@AxSaucedo/reflecting-on-my-time-at-bloomberg-before-taking-the-leap-into-the-startup-world-41ec9428bbab?source=rss-32de426f7278------2</link>
            <guid isPermaLink="false">https://medium.com/p/41ec9428bbab</guid>
            <category><![CDATA[values]]></category>
            <category><![CDATA[entrepreneurship]]></category>
            <category><![CDATA[startup]]></category>
            <category><![CDATA[bloomberg]]></category>
            <category><![CDATA[alejandro-saucedo]]></category>
            <dc:creator><![CDATA[Alejandro Saucedo]]></dc:creator>
            <pubDate>Fri, 10 Mar 2017 16:56:58 GMT</pubDate>
            <atom:updated>2017-03-11T11:54:32.434Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*ax5mPsJEISEj5cuX13m_IQ.jpeg" /></figure><p>I really admire the ethos of Mike Bloomberg — hard work, honesty and strong relationships are certainly critical for success!</p><p>Reflecting back on my time at Bloomberg (feels like a long time ago now!), I have to say that working there provided me with invaluable knowledge that I was able to apply in my startup journey!</p><p>I did have my doubts after university on whether I should have jumped straight into the startup world, or worked somewhere — I do not regret working there before taking the big leap into the startup world!</p><p>Working at Bloomberg provided me with an idea of how a corporate operates internally, as well as what a good company culture looks like — it was really surprising to see how well established Bloomberg was as a large company!</p><p>The support I obtained throughout my time there was incredible — all the way from the day I came in, until the day I left for my startup with with a Tier 1 visa ready to take on the startup world!</p><p>I had massive opportunities to run internal hackathons, work with multiple departments, work with an awesome team, apply cutting edge technologies, learn from the best manager I’ve ever had the pleasure to meet, and build strong friendships that I still hold to this day!</p><p>During my time there I experienced very strong company values, established and flexible internal processes (as far as corporate processes go of course!), and I remember how each of the departments thoroughly believed they were the most important department in the company — which is pretty awesome!</p><p>Although they certainly have a lot of legacy systems (They built routing protocols before the internet existed!), their technology infrastructure is truly impressive — and they have shown to adapt to new technology breakthroughs time after time!</p><p>Their history is even more interesting! How Mike was able to buy Bloomberg from public back to private, how their initial prototype (back in 1981) was hacked up together and debugged right before their first demo, how they have been able to keep the price of their terminals exactly the same since decades ago, how they currently donate so much of their profits to charity!</p><p>I have to say I admire the empire that Bloomberg has been able to build, and how he was able to transform an idea into his own legacy through a strong vision and solid values!</p><p>This ethos really resonates with me — there are many things that I cannot control for success — however the variables of hard work, honesty and strong relationships are certainly within my control, and the more I invest in these, the more opportunities that will arise for success.</p><p>One day…</p><p>#LetsDoThis</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=41ec9428bbab" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[The key for a successful relationship: Aligned values]]></title>
            <link>https://medium.com/@AxSaucedo/the-key-a-successful-relationship-aligned-values-1b826e4931b5?source=rss-32de426f7278------2</link>
            <guid isPermaLink="false">https://medium.com/p/1b826e4931b5</guid>
            <category><![CDATA[startup]]></category>
            <category><![CDATA[relationships]]></category>
            <category><![CDATA[partners]]></category>
            <category><![CDATA[values]]></category>
            <category><![CDATA[vision]]></category>
            <dc:creator><![CDATA[Alejandro Saucedo]]></dc:creator>
            <pubDate>Thu, 01 Dec 2016 20:05:16 GMT</pubDate>
            <atom:updated>2020-10-08T07:17:02.380Z</atom:updated>
            <content:encoded><![CDATA[<p>Aligned values are one the most important things in any relationship.</p><p>Whether it is with your life partner or business partner, aligned or misaligned values will define the success or failure of a relationship from day one.</p><p>I like to visualise this as a pyramid, where at the very top you have your values, and below that, you have your life vision, then your objectives, and then at the bottom, your interests.</p><p>The higher you go in the pyramid, the harder it is for you to change it, and hence, the more important it is to be aligned— changing your love for a specific TV show would be easier than changing your profession, which would be easier than changing how much you value family, money, fame, etc.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*WWQm4Ga1sk3HUJ9GaqoxbA.png" /></figure><h3>The Challenge</h3><p>Even though most people can agree that aligned values are so important, it can be really hard to identify when someone else has similar values to you.</p><p>I mean, sometimes you might think you got it all sorted:</p><ul><li>We both love doing X — <em>[Aligned Interests]</em></li><li>We both want to invest <em>on Y — [Aligned Objectives]</em></li><li>We both want to <em>establish long term in Z — [Aligned Vision]</em></li></ul><p>But as strange as this may sound — you might align on interests, objectives, vision, but if the very top of that pyramid is not aligned, it will just not work in the long term.</p><p>Values play a critical role throughout your life, and when you go against your values, <strong>your body knows </strong>— you go against yourself <strong>at a cellular level</strong>. This can be reflected in feeling unhappy, unproud or unfulfilled.</p><p>…and when I say values, I don’t mean just words like <em>“Respect”, “Transparency”, “Solidarity”</em>, etc.</p><p>In simple terms, I see values as the line you draw where you feel something is either right or wrong — the beliefs that define your priorities and drive your decisions:</p><ul><li>How do I believe people should be <strong>treated</strong>?</li><li>How much do I value <strong>money</strong>? or <strong>knowledge</strong>? or <strong>impact</strong>? or <strong>respect</strong>?</li><li>How much importance do I give to <strong>friends</strong>? or <strong>family</strong>? or <strong>fun</strong>? or <strong>commitment</strong>?</li></ul><p>And to make it even harder — having an aligned set of values is not enough. You also need to be aware about the priorities across your values. More specifically, if someone gives conscious or unconscious priority on money over happiness, or friends over family, then viewpoints will eventually clash.</p><p>This is why it is important to ensure you have aligned values for any strong, long-lasting relationship to work.</p><ul><li><strong>Partner values?</strong> They are the building blocks for the rest of your life with your partner.</li><li><strong>Company values?</strong> They are the founder’s personalities reflected in the company culture.</li></ul><p>Something REALLY important to note: <strong>Different personality does not mean different values.</strong></p><p><strong>People can have a completely different personalities, yet have aligned values — a good example of this is </strong><a href="http://thecontextofthings.com/2015/06/02/senior-management-needs-similar-values-different-personalities/"><strong>Steve Jobs and Steve Wozniak</strong></a><strong>.</strong></p><p>So next time you’re in a journey to find your soulmate — whether on business, friends or life partners— make sure <strong>your core values</strong> are fully aligned.</p><h3>Knowing your values</h3><p>Now we arrive to the most important part of this article: knowing your values, understanding your objectives, and your vision. I have put together <a href="https://medium.com/@AxSaucedo/2016-life-vision-motivations-values-7ea35c15b508#.b6ng1194j">a document</a> outlining my values, my vision, and my motivations. I encourage you to put one together for yourself.</p><p>This document would not be strategy, or a life plan — the things you write today will definitely change as time goes by. But it is important that you are able to understand yourself, what do you value, what drives you. Only then you will be able to evaluate your ongoing situations in life and very honestly question yourself on whether whatever you are doing, and whoever you are with truly aligns with what you value.</p><p>What values do you abide by? Where do you draw the lines that state what is right or wrong for you?</p><p>I recommend you to read <a href="https://www.amazon.co.uk/dp/B005SZ1PJW/ref=dp-kindle-redirect?_encoding=UTF8&amp;btkr=1">“Fierce Conversations” by Susan Scott</a>, as well as <a href="https://www.amazon.co.uk/dp/B00XGX17IM/ref=dp-kindle-redirect?_encoding=UTF8&amp;btkr=1">“Doing Good Better” by William MacAskill</a>— they are of great help when defining values, vision and motivations.</p><p>Let’s make sure we stay true to ourselves!</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=1b826e4931b5" width="1" height="1" alt="">]]></content:encoded>
        </item>
    </channel>
</rss>