<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Eliiza-AI - Medium]]></title>
        <description><![CDATA[AI &amp; Machine Learning in Melbourne, Australia - Medium]]></description>
        <link>https://medium.com/eliiza-ai?source=rss----d3822484c236---4</link>
        <image>
            <url>https://cdn-images-1.medium.com/proxy/1*TGH72Nnw24QL3iV9IOm4VA.png</url>
            <title>Eliiza-AI - Medium</title>
            <link>https://medium.com/eliiza-ai?source=rss----d3822484c236---4</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Sun, 22 Nov 2026 07:54:57 GMT</lastBuildDate>
        <atom:link href="https://medium.com/feed/eliiza-ai" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[Getting Started with MLOps]]></title>
            <link>https://medium.com/eliiza-ai/getting-started-with-mlops-d10301cef521?source=rss----d3822484c236---4</link>
            <guid isPermaLink="false">https://medium.com/p/d10301cef521</guid>
            <category><![CDATA[mlops]]></category>
            <category><![CDATA[devops]]></category>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[machine-learning]]></category>
            <dc:creator><![CDATA[Angus White]]></dc:creator>
            <pubDate>Wed, 05 May 2021 04:36:22 GMT</pubDate>
            <atom:updated>2021-05-05T04:36:22.722Z</atom:updated>
            <content:encoded><![CDATA[<p>MLOps is a hot topic among the data science world at present, but adopting this philosophy can be an intimidating task. There is an abundance of tools and frameworks out there that claim to make such a transition easy, so how does one know what’s actually going to be useful? In this post, I’ll demystify the situation a little bit by highlighting what problems we’re actually trying to solve with an MLOps workflow. I’ll do this by taking you through a “case study” of a fictitious company. This will hopefully give you a sense of how to solve some common problems that plague early-stage data science teams.</p><p>A quick note about the approach I’ve taken with this case study: the task of adopting an MLOps culture can start by introducing new process first, or new tooling. Process should always inform choice of tooling, not the other way around, so the focus of the case study is the workflow. We want to introduce new processes in small, digestible ways to maximise the chance of them actually being adopted. So at each stage, I’ll explain the problem, how we’ll solve it, and provide <em>just enough</em> tooling to support that change.</p><p>There’s no one-size-fits-all with this sort of thing, so I’ve been fairly opinionated in my choice of implementation — it’s easier to follow this with concrete examples. The solutions should translate well to any number of environments and applications, though. The important takeaway is recognising the problems and general solution approach.</p><h3>The Brief</h3><p>Here’s our imaginary situation:</p><p>Daintree™ is an online store which has just begun to use data science in its operations. A few POCs have been implemented, and it’s becoming clear that there is business value in deploying Machine Learning models, so they’re looking to scale up those operations. The business runs as a microservice architecture, and services are run on a Kubernetes cluster. They use Python as their primary model development language.</p><p>One of the models their data scientists have developed enables more effective product recommendations by combining collaborative filtering data and the Word2Vec algorithm. (<a href="https://arxiv.org/vc/arxiv/papers/1603/1603.04259v2.pdf">See here for more info on how this works!</a>) Here’s a quick-and-dirty implementation that uses a dataset from a <em>different</em> online retailer named after a rain-forest:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/52e9c5a7984e482e988a22d343f578c6/href">https://medium.com/media/52e9c5a7984e482e988a22d343f578c6/href</a></iframe><p>The data science team developed this model by experimenting in notebooks on their laptops, and then handing the code off to the ops team for deployment.</p><p>At this point, the team structure looks like this:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/679/1*-ha9GNMocAlDWiISpVsI1A.png" /></figure><h3>Improvement 1: Reproducibility</h3><p>At present, the handoff between the data science team and the ops team is a pain point. The ML experiments are all ad hoc, which gives the data scientists a lot of freedom and flexibility, but makes it hard to actually operationalise any of the results. The main problem boils down to reproducibility: it’s difficult to share logic and dependencies, so there is no guarantee that the code will actually run on the ops team’s machines, or on a server. The ops team are aware of the value of containerisation: the rest of the microservices run on Kubernetes, after all.</p><p>The data scientists are given the responsibility of containerising their experiments. This means much of the heavy lifting of actually deploying the app as a service is already done by the time it reaches ops.</p><p>It’s important to make this transition easy, so we want to provide our data scientists with as many resources as possible to encourage containerisation by default. The following minimal Dockerfile is a good start:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/93e3552f0c76ae16cb6c27de6e433b74/href">https://medium.com/media/93e3552f0c76ae16cb6c27de6e433b74/href</a></iframe><p>Let’s say Daintree has an existing container registry set up at daintree.container-registry.example, so we&#39;ll make use of that as the registry for their ML experiments.</p><p>We should also provide one-liners for building, pushing, pulling and running containers:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/ed468f2f3adbe1f1ed5072c6416a10fd/href">https://medium.com/media/ed468f2f3adbe1f1ed5072c6416a10fd/href</a></iframe><p>All the image tags are dev, because these scripts are designed to be run locally. When model versions are finalised, the tags should be updated to a more unique identifier. The git commit hash is a good choice! To this end, githooks can be set up to do all the building and pushing automatically on every git commit. A script like this might work as a post-commit hook:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/23bbd045750f223e75fab93c71709307/href">https://medium.com/media/23bbd045750f223e75fab93c71709307/href</a></iframe><p>The team that created the Item2Vec model above would build their container images with docker build -t daintree.container-registry.example/ml/item2vec:dev . while still in development. Any other member of the team should be able to docker pull daintree.container-registry.example/ml/item2vec:latest or docker pull daintree.container-registry.example/ml/item2vec:&lt;commit-hash&gt; and run a working version of the project in a container locally if they want to use it themselves.</p><p>It’s really important that the data science team feels like they are actually deriving value from this change (and any of the suggested changes in this post). We also want to avoid introducing too much too soon. Improving the reproducibility of the code is a great place to start — it’s digestible, and should hopefully demonstrate its value early on.</p><p>Containers are just one way to achieve this goal: Python virtual environments or VMs serve a similar purpose (although I’d argue that virtual environments should be used <em>with</em> containers). Containerisation is my preferred route, but choose whatever fits your business best.</p><p>The team structure now looks like this:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/679/1*r6IpcTkhXxj4KQ3cUYGfBw.png" /></figure><h3>Improvement 2: Modularity</h3><p>Daintree has been collecting data for a long time, and their storage pattern is fairly mature — data is stored in .csv files in their preferred cloud storage solution. In the interests of keeping this blog post cloud-agnostic, let&#39;s say they run <a href="https://min.io/">MinIO</a> in their Kubernetes cluster. Data is being loaded from roughly the same sources for every experiment, and even a lot of the preprocessing and cleaning steps are repeated. All the logic to perform these steps is being rewritten again and again, and we&#39;d much prefer single unified implementations.</p><p>We want our data scientists to shift their thinking from a “monolithic” architecture to a “pipeline” architecture. This is a significant shift in thought process, but is often the mark of a mature data science team.</p><p>In practise, one way to facilitate this is to containerise individual modules of data processing logic, and run experiments as a sequence of container tasks. The containers need to share input and output data, so we could mount a volume at some agreed-upon location, and read/write data from/to that location. It’s important, then, that the file format of the data is consistent, so that there is a standard interface by which pipeline components communicate with one another. Given that the existing data storage format is .csv files, this seems as good a format to use as any.</p><p>We should provide the team with templates to make adopting this mentality as easy as possible:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/d3dd9a2ee68f8550e5250c150457ec03/href">https://medium.com/media/d3dd9a2ee68f8550e5250c150457ec03/href</a></iframe><p>It’s worth noting that it’s just the <em>intermediate</em> representation that needs to be uniform. Connectors can be written for any number of data sources to perform the initial conversion to the common interface.</p><p>The Item2Vec model has fairly well-defined stages: loading, cleaning, preprocessing and training. These would be good candidates for a series of modules. For example, the data cleaning module might look like:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/267e2bbc5061400a6868742bed89f9d0/href">https://medium.com/media/267e2bbc5061400a6868742bed89f9d0/href</a></iframe><p>However, the dataset we load for this project is actually a .json file that we download from a URL, so doesn&#39;t fit with this pattern. The benefit of templates is that they aren&#39;t rigid — we can adapt the template to work for this use case:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/282c8b84a07b203b32ef4ff28e2b7e3c/href">https://medium.com/media/282c8b84a07b203b32ef4ff28e2b7e3c/href</a></iframe><p>And <em>voila</em>, we’ve created a general data connector for downloading line-separated .json files and converting them to our compatible file-format. Push this as a containerised app, and everyone can use the same implementation of this logic.</p><p>As mentioned above, now our containers need to mount volumes to be able to share data between stages. We can update the provided docker run one-liner:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/1739dd5678763d130b7f3360e00b7b5c/href">https://medium.com/media/1739dd5678763d130b7f3360e00b7b5c/href</a></iframe><p>Now any data placed in the local data directory will be available to the script.</p><p>We could also improve the structure of Daintree’s ML git repository to reflect this new pipeline architecture:</p><pre>DaintreeML<br>└── components<br>    ├── common<br>    ├── project1<br>    │   ├── task1<br>    │   └── task2<br>    └── project2</pre><p>We could fairly easily amend the earlier build-and-push script to operate on every subdirectory, and push new containers for each component that has been changed as part of a CI/CD process.</p><p>Of all the changes I’m proposing in this post, this one is probably the most dramatic. But it’s also the one from which you’ll likely see the most value. Adopting a pipeline architecture for your ML experiments is crucial to ongoing success — every ML experiment can build on the resources of previous ones, which should dramatically speed up iteration in your team.</p><p>Again, containers are perhaps not strictly necessary for this, you could play around with creating pip packages for your library code as an alternative, although containers are my preferred solution.</p><p>This is the new structure:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/679/1*Zt8npIVX5ohcWEexSBNa-w.png" /></figure><h3>Improvement #3: Centralised Caching</h3><p>The large datasets required by ML applications mean each run of the pipeline can take hours to complete from start to finish. This only serves to slow down the iteration speed of the data science team, especially when they’re usually only developing one component at a time. To speed up the development cycle, it makes sense to cache intermediate results and only run the sections of the pipeline that have changed. Thanks to the changes above, the team are sharing logic between themselves, but it’s likely that many of their intermediate results are also reusable. Hence, instead of mounting docker volumes locally, we should organise a central repository to store input and output data.</p><p>A good choice for this is a cloud storage service like S3 or GCS. As I mentioned above, Daintree uses MinIO as their storage service. We should create a bucket in which to store our ML artifacts — let’s call it ml-artifacts - and come up with a sane prefixing strategy so that it&#39;s easy to find what we&#39;re looking for. Maybe a good prefixing strategy would be &lt;component-name&gt;/&lt;component-version&gt;/&lt;year&gt;/&lt;month&gt;/&lt;day&gt;/&lt;timestamp&gt;/.</p><p>The script template becomes:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/2a2238f0ecfdd2db70fe76f632095297/href">https://medium.com/media/2a2238f0ecfdd2db70fe76f632095297/href</a></iframe><p>NB: MinIO’s API is drop-in compatible with Amazon S3, which is where the s3:// prefix comes from.</p><p>The run command now no longer requires the -v data:/application/data flag, but does require environment variables to be set i.e. -e MINIO_ACCESS_KEY -e MINIO_SECRET_KEY.</p><p>Daintree’s data scientists can now more easily share inputs and outputs of their processing logic with one another. If a particular team member is developing one stage in isolation, they are able to bootstrap their development by using the last cached inputs and outputs. This is also useful for debugging purposes — if a stage isn’t producing expected results, there is an auditable trail of input/output pairs that may be used to determine under what conditions the bug appears.</p><p>You might notice that the prefixing scheme isn’t actually enforced in the template. Sure, it’s important to have consistency across experiments, but there’s also every possibility that the original structure isn’t actually the best choice, and the team might decide down the track that they’d like to change. Since this is a template that may be copied and reused again and again, changing this prefixing scheme becomes a mammoth task of updating the code of every component in the repo. I’d preference trusting the team to follow the standard, at least initially.</p><p>Now our team looks like this:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/735/1*ICWvBVrQfcJTFCHQoZ-sRg.png" /></figure><h3>Improvement #4: Scalability</h3><p>The data science team are running training and inference locally on their laptops at present. This works fine for comparably small datasets and jobs, but there comes a point where it’s necessary to scale up the experiment. To achieve this, there should be a mechanism for the data scientists to offload tasks to an elastically scalable pool of compute resources.</p><p>Given that the company uses Kubernetes, a good solution is to schedule processing stages as pods on the cluster. The stages form cohesive pipelines, so let’s deploy a workflow manager to run experiments end-to-end. <a href="https://argoproj.github.io/projects/argo/">Argo Workflows</a> is Kubernetes-native, and has a fairly simple operating model, so it’s my preferred choice, but there are plenty of alternatives (AWS Step Functions, Luigi, Airflow, Google Cloud Workflows).</p><p>Argo Workflows supports configuring an artifact repo where intermediate results between workflow steps can be stored. This sounds an awful lot like the MinIO repo set up in the previous step, so Daintree configures their MinIO service as the artifact repo. With this change, they also no longer need to explicitly interface with the MinIO API when running pipelines on Argo. They update the previous template to provide a command line option to store inputs and outputs ‘locally’, which they’ll use when running pipelines on Argo.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/86e14cb561743d53f7e501550df916f9/href">https://medium.com/media/86e14cb561743d53f7e501550df916f9/href</a></iframe><p>Argo declares workflows as Kubernetes resources, but we’d really prefer not to have to train the entire team in .yaml syntax and the Kubernetes operating model. If we make a few basic assumptions around what most of the pipelines will look like, we can write a script that does most of the heavy lifting.</p><p>Let’s assume that the pipelines are sequential, without any parallel steps, and each step passes a single directory of data to the next. We’ll let the user configure the path to the original data source, and the path to which the output of the final stage will be written.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/563bb649ed13dcdad06df981633a28f3/href">https://medium.com/media/563bb649ed13dcdad06df981633a28f3/href</a></iframe><p>It’s a pretty quick-and-dirty script, but ought to work for the overwhelming majority of cases. The pipeline itself is configured with a .json file. The config for the Item2Vec project would look like this:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/56efc6c69dddd44ec9145673f847d7d8/href">https://medium.com/media/56efc6c69dddd44ec9145673f847d7d8/href</a></iframe><p>The neat thing is that these pipeline specs could be stored in an entirely different git repository to the components: the glue between them is the container registry. In fact, it’s not a bad idea to keep component logic and pipeline config separate, because pipelines won’t necessarily reflect the latest versions of the component code.</p><p>We could (and should) encourage the data scientists to gradually familiarise themselves with Argo’s API, and make small changes to the generated workflow document. Eventually, it would make more sense for the Workflow manifests to be written manually — this script should be considered nothing more than a tool to make the transition easier.</p><p>Once a valid workflow manifest has been created, the data science team submits to the cluster with argo submit workflow-manifest.yaml, and can access the Argo UI to monitor execution. The basic idea here is that the team should still experiment locally with small subsets of the data, and when ready to train on the entire dataset, the pipeline can be run with the workflow manager. This means they are no longer as constrained by data sizes, or the time taken to run the pipeline.</p><p>The team now no longer needs to interact with the registry directly, but just send requests to the compute cluster:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/679/1*zgcdBVKiNtvfdu6CH4GhKQ.png" /></figure><h3>Conclusions</h3><p>It’s a completely made-up example, but hopefully some or all of these problems seem familiar.</p><p>To summarise what we achieved here, the data scientists are now:</p><ul><li>Containerising their application code</li><li>Versioning models according to git commit</li><li>Structuring ML projects as pipelines of discrete components</li><li>Caching results in a centralised repository of data</li><li>Running large-scale pipelines in a shared compute cluster</li><li>Much more effectively sharing code and data</li></ul><p>Beyond experimentation, improvements like these also enable more effective monitoring and deployment of models. Possible next steps for building on this framework include specifying a common interface and template for APIs that serve model results; or using a workflow engine to automatically retrain the model based on some sort of event trigger.</p><p>It’s worth mentioning that there are existing frameworks and toolkits that provide similar functionality to what we’ve ended up with in this example. In fact, this architecture is effectively a lightweight implementation of <a href="https://www.kubeflow.org/docs/components/pipelines/overview/pipelines-overview/">Kubeflow Pipelines</a>, which actually uses both MinIO and Argo workflows under the hood. However, all the tooling in the world won’t help if the processes aren’t in place to use it effectively. The point of these building blocks is each one introduces new (and possibly unfamiliar) process to a data science team in a digestible way.</p><p>The tooling is deliberately basic, so that it hides as little complexity as possible. This way, the team has the flexibility to change anything that isn’t working for them. New process should never be too prescriptive — if kept flexible, eventually the workflow settles to a point where it’s clear what is and isn’t actually useful. That’s the point where it’s easiest to assess the value of different frameworks, and perhaps consider deploying something a bit more industrial-strength.</p><p>So if you’re wanting to elevate the operations of your ML workflow, consider starting by taking these steps (or something similar to them). If you’d like some help with implementing tooling or process in your data science team, get in touch with Eliiza, and we might be able to help you out.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=d10301cef521" width="1" height="1" alt=""><hr><p><a href="https://medium.com/eliiza-ai/getting-started-with-mlops-d10301cef521">Getting Started with MLOps</a> was originally published in <a href="https://medium.com/eliiza-ai">Eliiza-AI</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Building a successful Data strategy — Users and User Types (part 1).]]></title>
            <link>https://medium.com/eliiza-ai/building-a-successful-data-strategy-users-and-user-types-part-1-3ab8167500da?source=rss----d3822484c236---4</link>
            <guid isPermaLink="false">https://medium.com/p/3ab8167500da</guid>
            <category><![CDATA[data-strategy]]></category>
            <category><![CDATA[programming]]></category>
            <category><![CDATA[data-engineering]]></category>
            <category><![CDATA[data-architecture]]></category>
            <category><![CDATA[data]]></category>
            <dc:creator><![CDATA[Mathew Taylor]]></dc:creator>
            <pubDate>Fri, 02 Oct 2020 04:27:18 GMT</pubDate>
            <atom:updated>2020-10-31T01:29:43.263Z</atom:updated>
            <content:encoded><![CDATA[<h4>Building a successful Data strategy</h4><h3>Building a successful Data strategy — Users and User Types</h3><h4>Stop focusing on the data and start focusing on your users</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*EeCfHWi1W49q207O" /><figcaption>Photo by <a href="https://unsplash.com/@lukechesser?utm_source=medium&amp;utm_medium=referral">Luke Chesser</a> on <a href="https://unsplash.com/?utm_source=medium&amp;utm_medium=referral">Unsplash</a></figcaption></figure><p>Data is now at the forefront of corporate strategies, with the importance of data to drive success only growing over the last decade. Companies are talking about ‘<em>data enablement</em>’, ‘<em>building data platforms</em>’, ‘<em>creating data democracies</em>’ and making ‘<em>data-driven decisions</em>’.</p><p>However, implementing these buzz-words is proving to be challenging. As reported in the ‘New Vantage Partners, Big Data and AI Executive Survey 2019’, 77% of executives report that business adoption of Big Data/AI initiatives is a major challenge, and only a minority say that they have managed to create a data-driven organisation (31.0%), or have forged a data culture (28.3%) [1]. In the same report, 93% of respondents identify people and process issues as the main obstacle [1].</p><h3>What’s the problem?</h3><p>Over the past few years, there’s been a shift in the desired outcome of data projects from reactive, problem-focused deliveries (using data to fix issues one at a time), to proactive, user-focused deliveries (putting the data and the tools in the hands of the users so they can solve problems independently). That’s what’s behind the buzzwords ‘<em>data enablement</em>’, ‘<em>data democracies</em>’ and ‘<em>data-driven decisions</em>’.</p><p>This is all great, but what it means is that we need to change the way we provide the solutions. As engineers/developers we typically analyse things from a purely technical or problem-specific viewpoint, a ‘this is the problem we need to solve, here is the solution’ approach, rather than looking at the needs of the users or our customers. Whilst over the last decade software engineering as a whole has moved towards a customer or user-centric approach, data engineering has remained in the dark ages of using a technical or problem-centred methodology.</p><p>This is evident in a lot of data projects we see. Most data strategy projects fall under one of two scenarios; the first is the <strong>problem-focused approach</strong>:</p><ol><li>We need a data strategy</li><li>Find a problem which can be solved with data</li><li>Solve the problem</li><li>Find another problem which can be solved by data</li><li>Solve the problem</li><li>Repeat</li></ol><p>And whilst this does (eventually) help solve issues within the company, it’s incredibly reactive. Problems are first identified, assessed and then solved one at a time. Data access and use is bottle-necked as projects and data engineers focus on one issue at a time, which in turn leads to siloed solutions. Each solution tends to only solve a particular problem for a particular user (or user group), rather than leading to wider engagement of data by users.</p><p>The second is the <strong>technical approach</strong>. This approach tends to bunch all users in one giant user case and typically leads to a large ‘build-it-and-they-will-come’ data transformation project. Technical advice, articles and best practices are sought out and a cathedral type solution is architected. This is usually a data lake of some sort, or sometimes an all-encompassing data vault or data warehouse. Whilst this is exciting stuff, for the most part, it predominantly ends in a budget-breaking, deadline smashing, technical monolith which only a small subset of users can navigate or use.</p><h3>What can we do?</h3><p>We need to shift our mindset and start looking at things from a user-centric viewpoint: focus on the types of problems that our users come across and need to solve, what data and tools they need, and how we can get that data to them. <strong>We want to implement a data strategy that allows our users to solve their problems for themselves so that they really can become self-sufficient and data-driven</strong>.</p><h3>Identifying our user groups</h3><p>The first step on this path is to identify our data users and user groups in an organisation. This is where data projects get interesting: typical software engineering projects touch a limited number of user groups, whereas data projects usually have an impact over a larger user base.</p><p>When we look closely at our users, we realise how unique they are. They all have different use cases and requirements for data, which varies based on their role, experience and knowledge.</p><figure><img alt="List of different groups of data users in a company." src="https://cdn-images-1.medium.com/max/960/1*0hBtyMcAgMV4jMPGZv_Kzg.png" /><figcaption>Different groups of data users in a company — Illustration made by the author/eliiza.com.au</figcaption></figure><p>As the list above shows, there is a huge range of business users, from the very technical (data scientists and data engineers) to the non-technical (C-level executives). From internal business users with strong domain knowledge to external customers who don’t care how it all works; and from human to non-human users, such as other software applications and machine learning models.</p><h3>User Types</h3><p>Given we have all these different users how do we help solve their particular problems without either</p><p>a) Creating specific solutions for each user or area (which gets us back to the early reactive single problem-single solution issues)?</p><p>Or:</p><p>b) Trying to create a one-size-fits-all solution that doesn’t solve all our use cases (which gets us to the technical approach)?</p><p>We can examine our users and group them into <strong>three specific user types </strong>based on their use of data in their day-to-day work.</p><figure><img alt="The three data user types; explorers, operators &amp; analysers, readers" src="https://cdn-images-1.medium.com/max/960/1*d6xSZg1nupXh2zRX7rVZ6w.png" /><figcaption>The three data user types — Illustration made by the author/eliiza.com.au</figcaption></figure><p><strong>Data explorers </strong>— Data explorers are users whose day-to-day job is to search and wrangle the source or raw data. This group’s needs are fairly simple: their main concern is around getting access to the data. Knowing how and where to find the data, and then easily getting their hands on it, is key to their success.</p><p><strong>Data operators and analysers</strong> — These users are our technical users and domain experts. <em>Analysers</em> usually have strong domain knowledge about the data and use that to join data from multiple sources to answers questions: for example, working out what has gone wrong with a customer’s campaign, or finding new patterns in the data to solve business problems. <em>Operators</em> have strong technical and programming skills and use the data as input into applications they are building. They might need to do some modelling or manipulation of the data to get what they need but they aren’t interested in the data itself.</p><p>Core needs for these users are cleanliness for the data and standardisation of the data and related tools. The users in this group don’t want to worry about whether the source data is accurate, whether they need to convert date formats to match or have to reconcile data between different systems (<em>e.g.</em> different types of user id’s). They also want the data given to them in a standardised format (e.g. parquet) that they can work with using common tools (e.g. SQL).</p><p><strong>Data readers</strong> — Data readers operate in the business domain/layer and use data to drive decisions in their daily work. They want specific answers to specific questions and they don’t want to hunt for these answers or manipulate data to get the results. Accuracy and timeliness are the key requirements for these users, and they need data in an easily consumable and interpretable fashion. The requirements for the users in this group are often specific to the individual use case.</p><h3>Data maturity</h3><p>We can also look at our users in terms of data maturity, <em>i.e</em> how they understand data and how much we can trust them with data.</p><figure><img alt="Data maturity of three user types" src="https://cdn-images-1.medium.com/max/960/1*HLggTC4FL0DehsrqFnidCg.png" /><figcaption>Data maturity of user types — Illustration made by the author/eliiza.com.au</figcaption></figure><p><strong>Data explorers</strong> are experienced data users and have a high level of maturity through their use of and understanding of data and its flaws. They are aware of inherent inaccuracies in the data (<em>e.g.</em> fraud and duplication) and so can be trusted to be sceptical regarding the results they get from queries. Part of a data explorer’s job is to verify that their query is accurate and not to just accept the results.</p><p><strong>Data operators and analysers</strong> also understand the limitations of the data they are using. They are (to a certain extent) aware of the technical or business domain behind the data and so are used to working to get the results they need. Whilst they don’t want to deal with the inherent inaccuracies they will test and check their assumptions and results from the data. Due to this awareness, we can have a medium trust with these users.</p><p><strong>Data readers</strong> however are not experienced with data. They are typically non-technical or management/business users and so technical or domain-specific knowledge should be hidden away from these users. With this group, there is no tolerance for the limitations of the data or discrepancies in the data, and there are high consequences for inaccurate data. As you can imagine, if members of this group are making decisions based on the data provided, then there is no leeway for incorrect results. An example would be providing an incorrect bank account balance to a customer or revenue figures to a CEO which included fraud.</p><h3>Categorising our users</h3><p>So far we’ve identified three groups based on their daily uses and requirements of data, but how does this tie back into implementing a successful data strategy?</p><p>As discussed earlier there are two common problems with how data strategies are implemented; they either are too specific to a single user problem and so bottleneck wider data engagement, or they are too one-size-fits-all which don’t fit any individual user’s data requirements.</p><p>What we need to do instead is look at the users we identified earlier and see how they fit into our user types. We can then focus resources on creating solutions for these types which are tailored enough to the users to provide the data and tools to fulfil their requirements but are also broad enough that we can solve for groups of users rather than individual use cases.</p><p>If we take the list of data users that we identified earlier and apply the user types to categorise them, we can see how they fall under the different types:</p><figure><img alt="Image showing categorising users into the three user types" src="https://cdn-images-1.medium.com/max/960/1*ruNNik5zicpK_Pdikp9yyg.png" /><figcaption>Categorising users into user types — Illustration made by the author/eliiza.com.au</figcaption></figure><p>The more data-specific technical roles fall under the <em>data explorer</em> type due to their requirement for access to source data and their high level of ‘data maturity’. Under the technical <em>data operators </em>are Software engineers and ML engineers who use data as an input to their applications or ML models. Users with strong domain knowledge such as account managers and business analysts are under the <em>data analysers</em> and then furthest to the right under <em>data readers</em> are our non-technical, non-domain decision-makers such as executives, finance, HR and sales.</p><p>So by looking at the makeup of users in a company and how they affect or are affected by the data strategy we can then determine what solution(s) we need to implement to support them.</p><p>For example, if our strategy is to enable ‘<em>data-driven decisions’, </em>that is to enable our executives, managers or sales teams to be able to make informed business decisions, then we need to focus on fulfilling the requirements of our <em>data readers</em>. To achieve success in this strategy we need to build solutions which give theses users accurate and timely data in a way which is easily consumable (e.g dashboards and reports). We’ll need to make sure that the quality of the data is high so we don’t give the users false information and to a certain extent determine what questions will be asked so we can supply the needed information without them having to search for it.</p><p>On the other side of the scale, our strategy might be to use data to drive innovation and create new products or systems. For example, being able to identify and target specific customers or users of our product, or building systems to prevent fraud. Here our initial target group would be <em>data explorers</em>, we would invest in <em>data scientists</em> to identify patterns from the data and then harness the power of machine learning to build models to apply those findings. A successful implementation here would involve collecting data from as many sources as possible and making it accessible in its raw state for the data scientists to use.</p><p>Depending on the size or complexity of the company, our data strategy might involve multiple (or all) of the user types. In this case, we would need to build different solutions to cater for the different user types. For example in the first example with our <em>data readers, </em>in a company with complex or multiple business domains, <em>business analysts</em> in the <em>data analysers</em> group might be needed to provide the domain and analytical knowledge to join different datasets to provide the relevant information to the <em>data readers.</em></p><p>Similarly in the second example, the <em>data scientists </em>would at some point need to get their ML models into production. In a small startup, they might handle this themselves, however, in a larger company, there might be a dedicated team of ML engineers. So we would also need to implement a solution which catered to the requirements of our <em>data operators:</em> cleaning and curating data into a standardised format so it’s easily manipulated into the inputs needed to reliably build and train our models.</p><h3>To sum up</h3><p>In this article I’ve highlighted the issues in trying to implement a data strategy from a technical or problem-centric methodology, and how, if we focus our attention on the users and the requirements they have around data, we can start to design solutions which allow them to fulfil the strategic goals of becoming ‘<em>data-enabled</em>, ‘<em>data democratised</em> and ‘<em>data-driven’.</em></p><p>In my next post, I’ll be showing how we can move from the high-level/non-functional requirements we’ve identified, and drill down into the detailed technical requirements for each of our user types. These technical requirements can then guide the design and technology choices of our solutions.</p><p>[1] New Vantage Partners, Big Data and AI Executive Survey 2019, <a href="http://newvantage.com/wp-content/uploads/2018/12/Big-Data-Executive-Survey-2019-Findings-Updated-010219-1.pdf">http://newvantage.com/wp-content/uploads/2018/12/Big-Data-Executive-Survey-2019-Findings-Updated-010219-1.pdf</a></p><p>Thanks to Sophie and Emma for all their help and patience!</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=3ab8167500da" width="1" height="1" alt=""><hr><p><a href="https://medium.com/eliiza-ai/building-a-successful-data-strategy-users-and-user-types-part-1-3ab8167500da">Building a successful Data strategy — Users and User Types (part 1).</a> was originally published in <a href="https://medium.com/eliiza-ai">Eliiza-AI</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Data Science in Business]]></title>
            <link>https://medium.com/eliiza-ai/data-science-in-business-7e163876ee4d?source=rss----d3822484c236---4</link>
            <guid isPermaLink="false">https://medium.com/p/7e163876ee4d</guid>
            <category><![CDATA[artificial-intelligence]]></category>
            <category><![CDATA[data]]></category>
            <category><![CDATA[business]]></category>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[data-science]]></category>
            <dc:creator><![CDATA[Eike Germann]]></dc:creator>
            <pubDate>Thu, 04 Jun 2020 03:06:48 GMT</pubDate>
            <atom:updated>2020-06-04T03:12:39.173Z</atom:updated>
            <content:encoded><![CDATA[<blockquote>Concept Drift — a hypothetical case study</blockquote><h3>Introduction</h3><p>Many say that the impact of Covid-19 will “change the world forever”, while others are trying to bring the world back “to normal” and to restore a thriving economy.</p><p>How does that affect data science? Surely, the models and methods we use are “scientific”, that means they use maths and therefore produce quantitative outcomes based on logic and data, they don’t get sick, take time off, get stressed from working at home while there are children about — if one thing is reliable in these crazy times, it’s our machines, right?</p><p>As my colleague <a href="https://medium.com/@patrickrobotham">Patrick Robotham</a> outlined in <a href="https://medium.com/eliiza-ai/why-your-models-might-not-work-after-covid-19-a00509e4920b">his article on concept drift,</a> the ways data can change are manifold and subtle. Most often, the data changes because its causal processes change.</p><p>This scenario is an exploration of how our machines can be fallible and what is needed to address the problem.</p><h3>Changing Behaviour</h3><p>During the isolation required to handle the Covid-19 pandemic, many businesses closed, including offices. This shifted the consumption of energy both in where, but also when it was most in demand.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/413/1*865BRhv8RyqHDssRi1Cycw.jpeg" /><figcaption>Ethereal Energy is a fictional energy retailer</figcaption></figure><h3>The Situation</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*qEznMbt2c7HdnpGO" /><figcaption>Lara is a supervising business analyst at EE. (Image: <a href="https://www.pexels.com/@divinetechygirl">Christina Morillo</a>)</figcaption></figure><p>Let’s take an imaginary energy provider, Ethereal Energy. Lara is a supervising business analyst at Ethereal Energy, and her team is in charge of billing. Some customers have smart meters, so their consumption is available to both the customer and the retailer in near real-time. But customers without a smart meter have to rely on company estimates of their consumption until their meter can be read. As the restrictions around the spread of a pandemic similar to Covid-19 lift, Lara’s field operatives can visit their clients again and read the meters accurately to update their bills. Often, the predicted consumption is much lower than the value read by the field agent and Lara’s team is beset by angry customers. Ethereal has a large portfolio of renewable energy and has positioned itself in the market as the underdog, a fair and ethical provider on the side of the customer and now its customers are upset. People already have to make do with less, and now their electricity bills hit them out of the blue. Desperate phone calls, frustrated eMails and raging social media posts make life hard for the billing team.</p><p>Lara realises that the changes coming with the pandemic are not limited to the regulations being lifted. There are predictions of secondary waves, speculations about lasting changes to economy and culture and she finds herself worrying about the billing estimates failing so many vulnerable customers again.</p><p>She consults with her supervisor and arranges for the data science team to look into options to make the estimates better. One of the data scientists, Sienna, is meeting with her to discuss the current estimate modelling and future options.</p><h3>Initial consultation</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*WH4yK5O5uRb5P-hs" /><figcaption>Sienna is a data scientist at EE. (Image: <a href="https://www.pexels.com/@marcus-aurelius">Marcus Aurelius</a>)</figcaption></figure><p>Sienna has prepared an overview of the current model used for the estimation of consumption by the customer.</p><p>“It’s a very straightforward linear model,” she explains. “It has some added features for the change of seasons, but it falls apart when things change as widely as they did in the pandemic response.”</p><p>“Right,” Lara nods. “So the model isn’t broken, it’s just not quick enough? Can’t we just, you know, speed it up?”</p><p>Sienna smiles.</p><p>“Not quite. It’s not the speed of the model, it’s the speed and type of changes. If you input for example the temperature, because you’d expect people to put heaters on. But because of the pandemic, people stay home and put the heaters on at different times of the day. And not just their heaters, they make tea — and they bake. I saw your sourdough loaf on the internal channel, it looked pretty yummy.”</p><p>Lara laughs.</p><p>“It is! Yeah, we’ve been cooking a lot.”<br>“So people are using much more electricity at home at the same temperatures as before, where they’d be in the office, using the office kettle and sandwich press and so on. The model can’t know that.”</p><p>“Right. I see.”</p><p>Lara frowns.</p><p>“But this is all meant to be AI now, isn’t there some artificial intelligence thing you can do? You know, add another input for how things change?”</p><p>Sienna sighs.</p><p>“I think if it was that easy, we’d all be sitting at the beach sipping cocktails with machines doing all the work. I mean, even groups of human specialists have a hard time predicting the effects of this change, and our machine learning algorithms just aren’t that far yet. If it’s something we’ve never seen before, they don’t stand a chance.”</p><p>“Okay, so what can we do instead?”</p><p>“We can take the new data we have and retrain the model.”</p><p>Lara looks at Sienna.</p><p>“That sounds like there’s a ‘but’ coming.”</p><p>Sienna sighs again.</p><p>“Yeah, there is. It’s not as easy as taking the new data and everything will be as before. Things are still changing — restrictions are lifting, so we know the new data won’t be accurate for long. But no one knows what’s coming after and how people will behave. Not to mention that with the short period of time since the pandemic, we don’t have a lot of data to start with.”</p><p>“Can’t you make the training part of the model? Or just retrain it every 3 months or so?”</p><p>“Yeah, we could. We’re retraining the model right now with the data we have to address its current issues. The thing is, we don’t know how much and when we need to retrain yet. We can’t just set a timeline and waste time and money.”</p><p>“Upset customers also cost us money, and we’ve got to address this. The future seems pretty uncertain at this point. How about this: I’ll talk to the risk team and check with accounting and marketing, see how they estimate customer reaction to the challenges ahead. I should be able to give you a price you can work with. And you look into options to retrain the models, costs and requirements and how to future proof this thing.”</p><p>“That sounds good to me,” Sienna agrees. “Let’s do that.”</p><h3>Modelling Options</h3><p>A few weeks later, Lara and Sienna meet again to discuss their findings.</p><p>Lara has spoken to her team and the risk and marketing departments to get an estimate for the impact in reputation and possible loss of customers. She forwarded the numbers to the data science team and Sienna and her team have prepared a range of options to prevent the model results from upsetting the customers again.</p><p>Lara is rubbing her hands.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*l9lYf2RsZ2Okd-cY" /><figcaption>Lara is keen to see Sienna’s ideas. (Image: <a href="https://www.pexels.com/@divinetechygirl">Christina Morillo</a>)</figcaption></figure><p>“Alright, what have you got?”</p><p>Sienna smiles and brings up a slide.</p><p>“So first of all, as you know, we’ve deployed a new model already to address the issues we’ve had with billing before. It is working. For now.”</p><p>“Oh, start with the bad news, please,” Lara tries to put on a smile.</p><p>“We’re monitoring it and comparing it to a few test setups and it’s already deteriorating.”</p><p>“That sounds… bad? Is it going to break?”</p><p>“Well, it’s a slow change. It’s just losing its accuracy as the restrictions are loosening more and the new legal frameworks around prevention are put in place.”</p><p>“Okay, let’s look at the upside. That sounds like we were on the right track about the retraining — what did you come up with?”</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*S5-kNO_xak2JF8Ck" /><figcaption>Sienna has narrowed things down. (Image: <a href="https://www.pexels.com/@marcus-aurelius">Marcus Aurelius</a>)</figcaption></figure><p>Sienna straightened and forwarded to the next slide.</p><p>“We’ve got two options: We can keep running a model until we reach a threshold to retrain it — that could be the deviation from predicted billing, or number of customer complaints, we’ve used the numbers you sent me to establish and test this, it can be reported and triggered automatically. Our second option is a bit more expensive, but more future proof. It involves a continuous machine learning pipeline. We’d have a data silo that continuously gathers information and an integrated system to train and deploy a new model regularly to be up to date with the latest developments.”</p><p>“Sounds good, what are the pros and cons here?”</p><p>“Well, the triggered retraining is cheaper. The alerts can be set up automatically, but it might not be fast enough. Because it’s reacting to a trigger that’s based on something out of the ordinary, it could be that it’s behind the curve when things are changing fast. The machine learning pipeline has a much better chance of catching that, but it incurs some continuous costs because it is always on. But when things change, it changes with them, so it should soften the blow of quick changes significantly.”</p><p>Lara eyes the screen.</p><p>“I see you’ve got estimates for required development and maintenance in hours as well as provider costs on there. I can take this and present it upstairs. That’s great work, thanks Sienna.”</p><p>“You’re welcome, Lara.”</p><h3>Presenting Upstairs</h3><p>Lara has a meeting with the senior stakeholders of the billing and customer service representative departments to discuss options for how to deal with the estimate modelling.</p><p>She has prepared a short set of slides using Sienna’s numbers and projections about the model effectiveness and customer retention based on the more accurate estimates.</p><p>She concludes her presentation with a summary of the two models.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/960/0*CnYjL677n-ZB6D0-" /><figcaption>The two approaches have complementary benefits and drawbacks</figcaption></figure><p>“So, in short, we have a model that retrains as needed, triggered by specific events, such as the model failing beyond a specified dollar amount too often. This option is cheaper and does not require any in-between maintenance, but it only kicks in when things are already changing and damage has already been done.”</p><p>She triggers an animation that highlights the other half of the slide.</p><p>“On the other hand, we have a continuous training pipeline. It requires a little more set up than the triggered model, but it retrains constantly and can prevent and buffer some of the impact of changes like the pandemic restrictions, most of all prevent customer loss. For that, it has an ongoing cost.”</p><p>She brings up a graph with several projections.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/960/0*qLiOXjXn4FBBrLDa" /><figcaption>The cost profiles of the two approaches differ depending on the future</figcaption></figure><p>“I’ve been checking with the risk teams about the impact the first failure of the billing estimate model had. These graphs are showing how much the options cost compared to the cost incurred for us if circumstances change again, with three options for a worst case, a conservative estimate and a smooth, low-impact case.”</p><p>The model strengths are clear from the graph, in the worst case scenario, the machine learning pipeline is expected to buffer much of the change and save much more than its initial and ongoing cost. In the conservative estimate, however, the cost from incurred damages and triggering retraining are still lower than the total savings and ongoing costs for the pipeline model. In all cases, doing nothing is an unacceptably costly baseline.</p><p>She leaves the meeting set to presentation with the slide up and checks the chat for questions. There are no comments in the chat and the meeting is quiet.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*nQ1qxmPDj7gPnVXc" /><figcaption>Mark is a customer service director at EE. (Image: <a href="https://www.pexels.com/@charles-wundengba-1976175">Charles Wudengba</a>)</figcaption></figure><p>Mark, one of the customer service directors, speaks up first.</p><p>“So, we’ve got the choice between these two, that’s what you’re saying? There’s no alternative?”</p><p>Lara clears her throat.</p><p>“Yes. These are the most cost-effective methods.”</p><p>“This is a good overview, Lara. And it’s scary as hell.”</p><p>Mark laughs.</p><p>Lara scratches her head.</p><p>“Thanks, Mark, I guess?”</p><p>“All good, Lara — you’ve gotten onto it quickly and thoroughly, that’s a good thing. What would you do?”</p><p>Lara swallows and takes a deep breath.</p><p>“I think customer retention takes precedence over immediate cost. We can control cost, we can’t control the social media message. My team has been drowning in angry customers. I’d take the pipeline.”</p><p>Mark nods.</p><p>“Fair call, I can see where you’re coming from. I guess that leaves the rest up to us. You’ve shared the slides with us all, we’ll review your proposal and get back to you.”</p><p>“Sounds good, thanks Mark.”</p><p>“Thanks Lara, see you.”</p><p>One by one, the senior stakeholders leave the online meeting.</p><p>Lara closes the browser tab and leans back.</p><h3>Conclusion</h3><p>A few days later, the senior stakeholders email Lara with a few more questions about details and after some clarifications agree to a trigger based model with a low threshold for a test period. If the trigger doesn’t appear to be sensitive enough, they will reconsider and look at options for a machine learning pipeline.</p><p>Lara informs the data science team and her own crew and everyone is relieved that they are doing something to be prepared for the future.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*Ja4OJpfjb3i-Rk45" /><figcaption>Lara ponders the dependence of humans on machines and machines on humans. (Image: <a href="https://www.pexels.com/@divinetechygirl">Christina Morillo</a>)</figcaption></figure><p>“The only constant thing is change,” Lara thinks as she jots a few notes down in her journal. “and even change changes.”</p><p>She archives the slide deck and considers her conversations with Sienna.</p><p>“Their machines have moving parts,” she thinks. “and they need maintenance, too. If the wrong data comes in, it’s like pouring sugar in the tank.”</p><p>She closes her laptop and takes another sip of coffee. Images drift through her mind of changing inputs, different beans in her coffee, different milk. Different paper in a printer.</p><p>“The machines aren’t clever enough yet,” she shrugs. “But that, too, will change.”</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*pGK8JpxMal2mTVQjjG5rtQ.jpeg" /><figcaption>Lara takes notes to summarise what she learned from this project. (Images: <a href="https://thenounproject.com/noomtah/">Nithinan Tatah,</a> <a href="https://thenounproject.com/creativepriyanka/">Priyanka,</a> <a href="https://thenounproject.com/Maludk/">Laymik</a> at the <a href="https://thenounproject.com/">Noun Project</a>)</figcaption></figure><p>If you have any questions — aren’t sure about your models, want to future-proof them and want to know about the most cost-effective methods to do so:</p><p>Talk to us. <a href="http://eliiza.com.au/">Get in touch with us at Eliiza. We’d love to help.</a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=7e163876ee4d" width="1" height="1" alt=""><hr><p><a href="https://medium.com/eliiza-ai/data-science-in-business-7e163876ee4d">Data Science in Business</a> was originally published in <a href="https://medium.com/eliiza-ai">Eliiza-AI</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Data Science in Business]]></title>
            <link>https://medium.com/eliiza-ai/data-science-in-business-b712953383b4?source=rss----d3822484c236---4</link>
            <guid isPermaLink="false">https://medium.com/p/b712953383b4</guid>
            <category><![CDATA[metrics]]></category>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[business]]></category>
            <category><![CDATA[case-study]]></category>
            <category><![CDATA[machine-learning]]></category>
            <dc:creator><![CDATA[Eike Germann]]></dc:creator>
            <pubDate>Tue, 12 May 2020 03:42:47 GMT</pubDate>
            <atom:updated>2020-05-12T03:44:39.064Z</atom:updated>
            <content:encoded><![CDATA[<blockquote>Precision and Recall — a hypothetical case study</blockquote><h3>Introduction</h3><p>In my <a href="https://medium.com/eliiza-ai/modelling-rare-events-c169cb081d8b">blog post on imbalanced data</a>, I introduced some ways to measure the effectiveness of predictions for rare events, specifically the metrics precision and recall. Now, these terms are common jargon in the data science community, but can feel confusing to someone just stepping into the field.</p><p>I have put together an imaginary case study that illustrates how these concepts can be applied in a business context to capture and weigh off complex business priorities.</p><h3>Credit Card Fraud</h3><p>According to the <a href="https://www.auspaynet.com.au/sites/default/files/2019-08/AustralianPaymentCardFraud2019_0.pdf">Australian Payment Network’s report</a>, fraudulent transactions made up less than 0.1% of credit card transactions in 2018 but amounted to a value of more than $574 million. This is clearly a rare event and makes an ideal case to apply our concepts of precision and recall.</p><h3>The situation</h3><p>Let’s take an imaginary bank, we’ll call it ExampleBank.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/960/1*PhT6KdyoqUIsDXuzsgcRNg.jpeg" /><figcaption>Credit Card fraud is an excellent example of imbalanced data with a difficult field of priorities.</figcaption></figure><p>As part of their growth strategy, ExampleBank have entered into a new partnership with a major credit card company. Their analysis shows that the proportion of fraudulent transactions for the recent years has been almost double the national average, about 0.2%.</p><p>Lucy is the business strategist at ExampleBank in charge of asset security. She is dissatisfied with the results of the rules-based model ExampleBank is using to prevent credit card fraud. She has heard a lot about machine learning and wants to find out whether machine learning could help ExampleBank get better results.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*hbV7xpp27ln0-j8k" /><figcaption>Lucy is a business strategist at ExampleBank. (Image: <a href="https://www.pexels.com/@shvetsa">Anna Shvets</a>)</figcaption></figure><p>ExampleBank has a data science unit, so Lucy invites two of their data scientists, Ehsan and Janet, to discuss possible solutions.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*3ZSim7XG3Sl_m1IM" /><figcaption>Ehsan and Janet are data scientists working for ExampleBank (Image: <a href="https://www.pexels.com/@rebrand-cities-581004">Rebranded Cities</a>)</figcaption></figure><h3>Initial consultation</h3><p>To begin, she has a few questions about the performance of the current model.</p><p>“So, the fraud defense unit tells me the model has an accuracy of 98%. If that’s true, why are we not catching all of those fraudsters?”</p><p>Ehsan and Janet explain.</p><p>“We have put together a more detailed report on the current model,” Ehsan says, advancing a slide.<br>“It details what happens exactly. The accuracy of the model might be 98%, but its precision is about 20% and its recall is about 52%.”</p><p>Janet points at the line with precision.<br>“This means that out of all the cases the current model classifies as fraud, only about 20% are actually fraud. And this,” she shifts down to the line with recall, “means that the model only catches about half of the fraud cases that occur.”</p><p>“Wait,” Lucy holds up a hand. “It catches only half? But it’s 98% accurate? How can that be?”</p><p>“Accuracy also counts how many valid transactions the model classifies correctly, and that number is really large,” Ehsan replies.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/499/1*C1n27qzhLAcjdUPSoKv_MA.png" /><figcaption>Janet explains the different impact of what is measured by precision and recall (Image: <a href="https://www.pexels.com/@rebrand-cities-581004">Rebranded Cities</a>)</figcaption></figure><p>“Yes,” Janet agrees.”Accuracy is like a headline. It sums it up, and that can miss detail.”</p><p>Lucy nods.<br>“Okay. I see how that works. But how do we get more detail? I don’t want to do a whole lot of math when looking at a result or presenting my findings to the board.”</p><p>“We’ll look at what the model does from a different angle,” Janet explains while Ehsan prepares another slide.<br>“A precision of 20% means that for each fraud, we also falsely suspect 4 customers of fraud.”</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*gJFxXfGJHGSbHVw5Qw6MVQ.png" /><figcaption>Graphics by NounProject, angry by Adrien Coquet and gangster by wira wianda</figcaption></figure><p>“And a recall of 50% means that we catch half of the fraud cases out there.”</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*5AOybdSD1fMkZJoP7bVRbQ.png" /><figcaption>Graphics by NounProject, fraud by Bartama</figcaption></figure><p>“Okay,” Lucy agrees. “I’ve got to hand it to you, that’s a lot more solid information than 98% accuracy. But I still don’t really get it. Why can’t we just catch all the fraud and not bother any customers?”</p><p>Ehsan sighs.<br>“I wish we could. We could just suspect everyone of fraud, and that would catch all the fraudsters, but we probably would not have any customers left after that.”</p><p>“What Ehsan means,” Janet says, ”is that the two aren’t as related as it seems. That’s why we use the two separate metrics and why the two numbers are so different. It’s because of what type of mistake the different metrics count, whether we falsely suspect someone or let something slip through by mistake.”</p><p>“If you can put a dollar value on angry customers, though, we can compare those two as well,” Ehsan adds. “Then we can compare them by cost.”</p><p>Lucy’s face lights up.<br>“That’s the number I need. I’ll get in touch with the risk team and get you that dollar value. If you can rate the effectiveness of the model in dollars, I can understand it. And I can present it to the board.”</p><h3>Candidate Models</h3><p>A few weeks later, Lucy meets up with Ehsan and Janet again.</p><p>During that time, Ehsan and Janet have tested a variety of model types on the data gathered by ExampleBank’s fraud defense division. Lucy has requested details on the risk analysis of the customer experience with fraud interventions and forwarded the information to Ehsan and Janet.</p><p>While Ehsan sets up the presentation, Janet summarises their work of the past weeks.<br>“I think we’ve got something promising for you,” she says. “we’ve taken the report from the risk analysis team and put together a comparison of some candidate models.”</p><p>Ehsan brings up the first slide.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/960/1*EeGP5Mm5vmkKCh7y3NHLTQ.jpeg" /><figcaption>Estimated costs for missing a fraud case and upsetting a customer at ExampleBank</figcaption></figure><p>“The report outlined an expected risk between $900 and $1500 for an angry customer affected by fraud interventions, which they average to $1200. We found the average value of a fraud transaction in the last year to be $298, so we can conservatively round this up to $300. In other words, preserving the customer experience is 4 times as important as catching a fraudulent transaction.”</p><p>He forwards to the next slide.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/960/1*uJ3LJhEXOpAMyOsNPGAiwQ.jpeg" /><figcaption>Ehsan and Janet have prepared two candidate models with very different attributes</figcaption></figure><p>“There is a combined metric called the F1-score that gives us a value for how efficiently the models balance these priorities.”</p><p>Lucy frowns.<br>“Wait. Those are just numbers. I thought we were going to express everything in dollars?”</p><p>“We still are,” Janet explains, ”the F1-score uses a weight that’s based on the dollar value per fraud divided by the dollar value per customer, that’s just a number. It compares how well the model balances the two values, where higher is better.”</p><p>“Okay,” Lucy nods. “that makes sense. Carry on.”</p><p>Ehsan points at the slide.<br>“Our base model has a precision of 20%, a recall of 52%, that means its F1-score becomes 0.20751. That’s the number we’ve got to beat, or we’re losing money somewhere.”</p><p>“Yes, I remember,” Lucy says. “It gets half the fraud, but has to go through four customers to get to a fraud case.”</p><p>“Exactly,” Ehsan confirms. “The first new variant we built has a similar precision at 22% and its recall is 68%. It has an F1 score of 0.22912.”</p><p>Lucy frowns again. “That seems like it’s not doing very much. What else have you got?”</p><p>“Our third model has a precision of 13%, and we managed to get the recall up to 82% on this one.”</p><p>“That sounds promising,” Lucy says. “That’d sure catch a lot more fraud.”</p><p>“That’s true,” Janet agrees, “it’s got an F1-score of 0.13677, though. That means overall it would perform worse than even the model we are currently using.”</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*gkHIhNI0p_2b6KgG" /><figcaption>Lucy is trying to decide on the right model to pursue with her team. (Image: <a href="https://www.pexels.com/@shvetsa">Anna Shvets</a>)</figcaption></figure><p>Lucy stares at the screen, trying to make sense of the numbers.<br>“I don’t get it. Why is that? It’s got like 30% more recall, that should catch way more fraud cases. The difference in precision is so small.”</p><p>“It’s because of the cost,” Janet explains. “Instead of 5 out of 10, it will now catch 8 out of 10 fraud cases, but for each 10 suspected cases, 9 will be honest customers. That’s more than twice as many, and those cases are more expensive.”</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*SfQzk1HbBpENxF3PY7rO8Q.png" /><figcaption>Graphics by NounProject, angry by Adrien Coquet and gangster by wira wianda</figcaption></figure><p>Lucy frowns at the screen for a bit longer, then nods.<br>“Yes. Yes, I see now. Okay, that sounds to me like we’ve got something to work with.”</p><p>The three agree for the team to continue working and to report regularly on their findings.</p><h3>Implementation</h3><p>Lucy, Ehsan and Janet meet again after the development period for the proof of concept has concluded. Ehsan and Janet have prepared a few slides to help Lucy report the results of the project to her supervisor.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/396/1*XvCYMyUM2eN--8D1fsemOw.png" /><figcaption>Ehsan can be proud of the latest model. (Image: <a href="https://www.pexels.com/@rebrand-cities-581004">Rebranded Cities</a>)</figcaption></figure><p>“So, what’s the latest result of the model development?”</p><p>Ehsan and Janet both smile.</p><p>“We managed to get both the precision and the recall up a little, to 23% and 76% respectively.” Janet says, and Ehsan adds:<br>“That means the F1 score is now up from 0.22912 to 0.23984.”</p><p>Lucy lets her eyes wander over the figures on the screen.<br>“Good work! And this means the customer experience has not gotten worse, right?”</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*gJFxXfGJHGSbHVw5Qw6MVQ.png" /><figcaption>Graphics by NounProject, angry by Adrien Coquet and gangster by wira wianda</figcaption></figure><p>“Yes,” Ehsan confirms.</p><p>“And we catch 50% more fraud cases than before?”</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*LmRYVV8uqQG8yeJkSv-f0A.png" /><figcaption>Graphics by NounProject, fraud by Bartama</figcaption></figure><p>“Correct.”</p><p>Lucy smiles.<br>“Great. You’ve given me some great tools to show the performance of the model. I’m really chuffed about what we’ve achieved. I’ll add your data to my presentation and let you know how we go.”</p><p>The board is impressed to have tangible, real world effects as part of the description of the model performance and applaud Lucy for a well-prepared presentation.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*KMtyeEM-EqJUee2q" /><figcaption>The board is impressed with Lucy’s effort and wants her project to proceed. (Image: <a href="https://www.pexels.com/@shvetsa">Anna Shvets</a>)</figcaption></figure><p>A week later, Lucy receives the confirmation that the project has the green light to proceed into productionisation.</p><h3>Conclusion</h3><p>The project finishes as a great success. Lucy is commended for helping ExampleBank realise their growth strategy and she continues to have a great working relationship with the data science team. She keeps the presentations she has made as a memento of her success and a quick note on how to interpret Precision and Recall in her personal files.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*nDRL1CfWQfyH67AmxyThcw.png" /><figcaption>Lucy’s note with a graphic from the NounProject: money value by Eucalyp</figcaption></figure><p>“Accuracy isn’t everything. Always bring it down to the dollar value,” she writes down, with a little set of scales drawn next to it. “If I can understand it, I can explain it.”</p><p>She puts the notes aside and turns to her laptop. Time to get on to the next project.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=b712953383b4" width="1" height="1" alt=""><hr><p><a href="https://medium.com/eliiza-ai/data-science-in-business-b712953383b4">Data Science in Business</a> was originally published in <a href="https://medium.com/eliiza-ai">Eliiza-AI</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Modelling Rare Events]]></title>
            <link>https://medium.com/eliiza-ai/modelling-rare-events-c169cb081d8b?source=rss----d3822484c236---4</link>
            <guid isPermaLink="false">https://medium.com/p/c169cb081d8b</guid>
            <category><![CDATA[metrics]]></category>
            <category><![CDATA[business]]></category>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[imbalanced-data]]></category>
            <dc:creator><![CDATA[Eike Germann]]></dc:creator>
            <pubDate>Mon, 11 May 2020 03:14:18 GMT</pubDate>
            <atom:updated>2020-05-12T03:51:58.405Z</atom:updated>
            <content:encoded><![CDATA[<h3>Introduction</h3><p>In some of my previous posts, I talked about <a href="https://medium.com/eliiza-ai/anomaly-detection-part-1-92f45bdc5729">anomaly detection</a> — ways of finding the odd one out, the unexpected little thing that’s rare, mostly undefined and often unwanted.</p><p>What if you were looking for a specific, rare thing? What if you wanted to find the nugget of gold in a pile of river mud, the needle in a haystack?</p><p>In a situation where we have lots of examples for one or more cases, and few examples of others, we call the dataset “imbalanced”.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/510/0*czIkDW7St0R972n8" /></figure><p>For machine learning, these datasets pose a particular challenge. It is hard to learn from rare examples, especially considering that machine learning thrives on repeated exposure to large quantities of data.</p><p>Now, even with imbalanced datasets your ML model can still be successful and extremely precise. If the distinction between the common examples and the rare types is strong enough (think one is a Zebra while the other is a garden hose), the right model will be able to distinguish the different types of data very well. But often, the data is noisy or the common and rare cases are so similar that they are hard to distinguish even for human experts.</p><p>Medical or financial applications come to mind: We want to be able to diagnose rare diseases with the same certainty as common ones and businesses want certainty before they accuse their customers of illicit dealings.</p><h3>Measuring Imbalance</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/667/0*27F94zWP8my1EteH" /><figcaption>Image: <a href="https://xkcd.com/license.html">XKCD</a></figcaption></figure><p>So how imbalanced is “imbalanced”? While a lot of that is really up to the dataset itself, as a rule of thumb any difference greater than 1:10 between two classes can be considered imbalanced.</p><p>But the difference in the number of examples is not really the clincher. It’s how well your model can deal with it. And that’s where imbalance can trip you up:</p><p>Accuracy is a very common metric for machine learning models. It’s a simple number that tells us how much of its predictions the model got “right” — and that sounds like what you want. But for imbalanced data, this can be very deceiving. If youwanted to build a “vintage-car-detector” and only 1 in 1000 cars is a vintage model, then you could simply have the model output “modern car” by default without any learning and it would be 99.9% accurate, but useless. Not only would you miss out on all the vintage cars, the model could result in losing vintage cars irrevocably because their number is limited.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*Zx7VEIN0a2dDY_7a" /><figcaption>Avoid bad classifiers.</figcaption></figure><p>Another way that a badly calibrated model could result in a bad business outcome is misclassifying a modern car as vintage and upon attempting to sell it causing the customer to accuse you of fraud. This can be costly in both money for possible lawsuits as well as immeasurable damage to your reputation.</p><p>What we need is a way to express the needs of the business in terms of getting the answer right or wrong.</p><p>If this feels daunting or just annoying because it looks mathy, you can have a look at this example of a hypothetical use case that uses the metrics I’m introducing in a practical setting: <a href="https://medium.com/eliiza-ai/data-science-in-business-b712953383b4">Data Science in Business — Precision and Recall</a>.</p><p>The easiest way to show how confused a model is about its results is called a “confusion matrix”. For a binary/yes-no case, it’s a simple 2x2 grid comparing the real true (let’s call it 1) and false (let’s call this one 0) values against the true and false values given by the model. That results in four possibilities:</p><ol><li>The true value is 1, the model says it’s 1.</li><li>The true value is 0, the model says it’s 0.</li><li>The true value is 1, the model says it’s 0.</li><li>The true value is 0, the model says it’s 1.</li></ol><p>The technical terms for these cases are True Positive (TP), True Negative (TN), False Positive (FP) and False Negative (FN). The 2x2 grid has one field for each of these:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/761/0*qcZiIhTGruM7MfsU" /><figcaption>Binary confusion matrix</figcaption></figure><p>Now we can distinguish the business critical cases of missing an event or falsely predicting one.</p><p>We can take these values and their ratios to create new metrics that give more detailed information about the model’s performance, especially when the number of cases in the dataset are not equal. The two most important metrics in this regard are called “<a href="https://en.wikipedia.org/wiki/Precision_and_recall#Recall">Recall</a>” and “<a href="https://en.wikipedia.org/wiki/Precision_and_recall#Precision">Precision</a>”. Recall is also called the True Positive Rate or just Hit Rate, because it represents the fraction of positive target values that have been identified correctly. Precision is the measure of the <em>correctly identified</em> positive values in all <em>proposed</em> positive values. Precision is often also called the Positive Predictive Value, because it is an indicator of a model’s reliability. While they might sound very similar, they describe very different behaviours of the model. Imagine our example from before, the vintage-classifier, in reverse: If it classified every car as vintage, it would have perfect recall, but its precision would be horrible. And it would get worse the more data points you add!</p><p>These two metrics are much better suited to capture a model’s performance in relation to the priorities of the business. If you were evaluating vital machine parts for possible faults, it might be more acceptable to discard a few false positives than to risk machine failure. In other investigations, it might be more important to reduce the number of interventions due to false classifications, be it to minimise cost or customer inconvenience.</p><p>In the former case, we would tune the model prioritised on recall, in the latter, we would prioritise precision. For the examples we look at below, we will compare precision and recall against accuracy.</p><p>There is a plethora of other combinations that give further insight into a model’s behaviour, of course. If you want to dive deeper into this, a good start is the wikipedia page on the <a href="https://en.wikipedia.org/wiki/Confusion_matrix">confusion matrix</a>.</p><h3>Balancing Acts</h3><p>I’ve taken a dataset from Kaggle, <a href="https://www.kaggle.com/danielpe/earthquakes/metadata">Daniel Perico’s Earthquake data</a>, and added a column with a boolean for whether the event is an earthquake or not (it replaced the “type” column that has various other possibilities, such as quarry explosions or sonic booms). About 3% of the dataset are non-earthquake events. If this was a predictive model, we could imagine that we would have to decide between options such as expensive evacuations in case of false alarms and injury or loss of life in case of missed events.</p><p>I dropped any columns that had a time component or a unique identifier of the event, scaled the continuous data and converted the categorical columns to one-hot encoded features. All experiments except where specified were run using a default Random Forest algorithm, because it is popular, easy to use and has cross validation “built in” due to its bootstrapping functionality.</p><p>Most of the examples here are using the <a href="https://imbalanced-learn.readthedocs.io/en/stable/">imbalanced-learn library</a> which has a collection of utilities to optimise imbalanced datasets for machine learning.</p><p>The jupyter notebooks with the examples discussed here (and many more) can be found on <a href="https://github.com/eliiza/imbalanced-data">github</a>.</p><p>There are various ways to address the imbalance in a dataset that differ in the amount of change made to the dataset. The simplest way to address imbalance is of course to <strong>do nothing</strong>, and this will form our baseline results to compare against other measures.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*xS8rVfZ8sEIRN1xO" /><figcaption>You can try filling in the missing pieces.</figcaption></figure><p>The next least destructive measure is to try and salvage missing data by training a model on the complete dataset to “guess” missing values to fill them in. This option is called <strong>imputation</strong>. (This is, of course, not limited to imbalanced data. However, I feel it is important to include it as a possible comparison point for what effect “more data” would have on the results.)</p><p>The next degree of change to the training data is <strong>undersampling</strong>. This refers to using fewer samples of the majority class than available in the dataset to achieve a better ratio between the classes. There are a variety of options from randomly choosing to excluding specific data points from the majority class in different ways to achieve a better distinction between the classes. The latter is generally referred to as <strong>cleaning undersampling</strong>.</p><p>The opposite of undersampling is <strong>oversampling</strong>. In the simplest form of oversampling, the available data points of the minority class get multiplied in a copy-paste fashion to balance out the class ratio. In more elaborate methods, an algorithm determines specific characteristics of the minority class to generate new, similar but not equal data points to bolster the minority class. Examples of these types of algorithms include <a href="https://en.wikipedia.org/wiki/Oversampling_and_undersampling_in_data_analysis#SMOTE">SMOTE</a> or <a href="https://en.wikipedia.org/wiki/Oversampling_and_undersampling_in_data_analysis#SMOTE">AdaSYN</a>. The field of synthetic data is developing all the time, so it is worth looking for the latest findings and experiments — a competition to find the best oversampler is currently underway as part of the <a href="https://github.com/analyticalmindsltd/smote_variants">smote-variants</a> package!</p><p>A significant difference between imputation and resampling methods is that imputation is, in principle, an attempt at getting more data overall. While the imputation could be limited to fix only minority cases, it can also serve to give more majority examples. There can be very little control over the ratio of minority to majority cases with this method, and it can never produce more minority cases than were part of the dataset to begin with.</p><p>The resampling methods allow to control and aim for a ratio of majority to minority cases, and depending on the modelling algorithm used, the ratio might not need to be 1:1 to make a good model.</p><p>Of course there are ways to mix and match between these options as well. However, keep in mind that these steps are not independent — trying to e.g. create synthetic data on top of imputed values requires these values to be highly accurate, else the synthetic data will propagate the errors made by the imputation.</p><p>The table below shows results for some of the tests run with the simplified earthquake data:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*k5R7BG00gHovvs989qU9Dg.png" /><figcaption>Interventions on imbalanced data by type. Best results are highlighted dark grey, worst light grey.</figcaption></figure><p>These results show some important features of tools that can be used to address imbalanced data. The first thing to notice is that the only metric that could be significantly increased here is the <a href="https://en.wikipedia.org/wiki/Precision_and_recall#Recall"><em>recall</em></a>.</p><p>This makes sense, since capturing a rare case in general is easier than distinguishing it correctly from another case. That is why all increases in recall come at a cost in precision. Still, this could be a boon: a detailed look at the cases the model keeps getting wrong could give some clues for e.g. feature engineering to improve the classifier. I won’t go further into this topic because feature engineering is a general practice in machine learning and not limited to imbalanced data.</p><p>Some of the treatments were quite fast, while others required a lot of computation or were not compatible with the dataset altogether. It depends on the business case what balance is required, how much data is available and how much computing power and time are allocated for model development.</p><p>(NB: There is a combined measure, called the <a href="https://en.wikipedia.org/wiki/F1_score">F1-score</a>, that evaluates precision and recall together and allows for weighting the importance of one against the other.)</p><p>So far, the interventions we’ve looked at have dealt with the imbalance at a preprocessing level. Couldn’t we just tell the model that these minority cases are more important? Expressing the priority of classes for the modelling algorithm is generally called <strong>cost-sensitive</strong> modelling and can be done in a variety of ways. Many algorithms, such as Random Forest, have built-in support for imbalanced data in that way. The classes receive an associated weight to penalise the algorithm more strongly for missing more important cases.</p><p>If a desired technique is not included in the modelling algorithm, they often provide options to add this functionality by hand. A custom loss function can make the algorithm cost-sensitive, or a tool to measure the difference between classes can be implemented as a new decision criterion for the algorithm. One such example is the <a href="https://github.com/EvgeniDubov/hellinger-distance-criterion">Hellinger distance criterion</a> that can replace the built-in decision criteria in Random Forest.</p><p>The <a href="https://imbalanced-learn.readthedocs.io/en/stable/">imbalanced-learn library</a> contains a few standard algorithms that have been enhanced through built-in balancing measures.</p><p>If this all looked really confusing, here’s an example of a hypothetical scenario that uses the metrics introduced above in a realistic setting: <a href="https://medium.com/eliiza-ai/data-science-in-business-b712953383b4">Data Science and Business — Precision and Recall</a>.</p><h3>Getting it straight</h3><p>The most important aspect of any method to address your imbalance is the same as in all business propositions — how do we measure<em> success?</em></p><p>We have seen that the methods to address data imbalance can change the dataset by a lot, so you have to make sure that your validation set is untouched by those methods.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*tlp-I9KoJslholDz" /><figcaption>Save some of your pure materials for later.</figcaption></figure><p>Algorithms with built-in methods to address imbalanced training data make this very easy, of course. Here, you can just keep a validation set as usual.</p><p>However, with oversampling methods such as AdaSYN or SMOTE, you are creating new data. One the one hand, you want to give these algorithms the whole dataset as a basis to produce the best results. On the other hand, you need to make sure that you can still separate out a section of unmodified data points to create your validation set.</p><p>If you don’t, your validation set won’t represent naturally occurring data. It’ll represent the natural data plus the interpretation of the imbalanced data algorithm. Depending on your situation, that algorithm might have used a lot more compute and data than your model will have, but it’s still an approximation of the world and any misconceptions it has will translate into the data it generates.</p><p>This can be annoying, because many oversampling algorithms have no easy way to separate out the synthetic data from the original data. It’s tempting to manipulate the entire dataset and split the complete result into train, test and validation data, but that is misleading and dangerous. Only when your model is tested on what made your dataset so challenging in the first place are you getting an impression of its real performance.</p><p>So make sure that you always test your model on a validation dataset that is purely natural data, most importantly, that the ratio between classes is similar to what you would expect your model to have to deal with when in production.</p><p>We haven’t discussed time series here, but this is especially true for them. Even if you are training a simple classifier, make sure that you are validating your model on more recent data than your training data. This ensures that your model is tested on data that most closely resembles the real world data it will work on eventually. One reason why this is important is concept drift — a change in the underlying patterns in your data that your model won’t recognise because it is trained only on data that doesn’t contain the new patterns. To find out more about this, read this blog post by my colleague <a href="https://medium.com/@patrickrobotham">Patrick Robotham</a>:</p><p><a href="https://medium.com/eliiza-ai/why-your-models-might-not-work-after-covid-19-a00509e4920b"><strong>Why your models might not work after COVID-19</strong></a></p><h3>Keeping it level</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*B4Qjh2l5NFbzPMwp" /><figcaption>Keeping the balance is difficult, but can be achieved.</figcaption></figure><p>To sum up, imbalanced data can be a tough challenge for a machine learning project. And while there are many options to address the issue, there is no one-size-fits-all solution to the problem.</p><p>One of the reasons for that is that the most important aspect to identify when dealing with imbalanced datasets is where the priorities of the underlying business questions are:</p><blockquote>Can you deal with a few more falsely classified items as long as you get most of your targets?</blockquote><blockquote>Put the emphasis on recall.</blockquote><blockquote>Would misidentifying your subjects be more harmful than missing a few targets?</blockquote><blockquote>Focus on precision.</blockquote><p>Depending on the makeup of your dataset, the restrictions on your model and the requirements of your business case, different approaches can give you a different balance between the two.</p><p>I have illustrated a variety of algorithmic approaches with a special focus on a hierarchical view of manipulations on the dataset, ordered by the changes made to the original dataset to obtain a training set. That is, of course, only one way to view these techniques. To get a deeper understanding, I invite you to download the repository of examples from <a href="https://github.com/eliiza/imbalanced-data">github</a> which includes details and options not discussed here.</p><p>If you have any further questions or would like to discuss a specific business problem, get in touch with us at <a href="https://eliiza.com.au/">Eliiza</a>.</p><p>We have experience with imbalanced data across a variety of industries and applications from anomaly detection to sentiment analysis or computer vision and we’d love to help.</p><h3>Further reading:</h3><p><a href="https://pdfs.semanticscholar.org/239b/2210b3fbc1f4b8246437a88a668bf9a0d2c0.pdf">An overview of classification algorithms for imbalanced datasets</a>, V. Ganganwar 2012</p><p><a href="https://arxiv.org/pdf/1711.00837.pdf">Oversampling for imbalanced learning based on K-means and SMOTE</a>, F. Last et. al. 2017</p><p><a href="https://www.ele.uri.edu/faculty/he/PDFfiles/ImbalancedLearning.pdf">Learning from imbalanced data</a>, H. He &amp; E. A. Garcia, 2009</p><p>Data Science and Business — Precision and Recall (a hypothetical case study), Eliiza AI, Medium</p><p><a href="https://en.wikipedia.org/wiki/Confusion_matrix">Confusion Matrix</a>, Wikipedia</p><p><a href="https://en.wikipedia.org/wiki/F1_score">F1-score</a>, Wikipedia</p><p><a href="https://imbalanced-learn.readthedocs.io/en/stable/">imbalanced-learn</a>, readthedocs</p><p><a href="https://github.com/eliiza/imbalanced-data">Imbalanced Data examples</a>, <a href="https://eliiza.com.au/">Eliiza</a> on Github</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=c169cb081d8b" width="1" height="1" alt=""><hr><p><a href="https://medium.com/eliiza-ai/modelling-rare-events-c169cb081d8b">Modelling Rare Events</a> was originally published in <a href="https://medium.com/eliiza-ai">Eliiza-AI</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Why your models might not work after Covid-19]]></title>
            <link>https://medium.com/eliiza-ai/why-your-models-might-not-work-after-covid-19-a00509e4920b?source=rss----d3822484c236---4</link>
            <guid isPermaLink="false">https://medium.com/p/a00509e4920b</guid>
            <category><![CDATA[pipeline]]></category>
            <category><![CDATA[training-data]]></category>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[coronavirus-covid19]]></category>
            <category><![CDATA[data]]></category>
            <dc:creator><![CDATA[Patrick Robotham]]></dc:creator>
            <pubDate>Fri, 03 Apr 2020 03:58:46 GMT</pubDate>
            <atom:updated>2020-04-03T03:58:46.371Z</atom:updated>
            <content:encoded><![CDATA[<h3>An introduction to concept and data drift.</h3><p>I stumbled across the following tweet last week:</p><h3>Peter Skomoroch on Twitter</h3><p>Drift is real in normal times. If your company is not already retraining daily or weekly it&#39;s very likely your models are going sideways as consumer behavior changes drastically. https://t.co/5nJtC94cua</p><p>Why would our models start failing? What does he mean by “drift”? How do we know if we have to retrain? I’ll try and answer these questions in this blog post, as well as how we can help.</p><h3>Concept Drift</h3><p>When we model a phenomenon, we assume that our data is generated by some underlying process. Concept drift occurs when the world changes so that the target variable we’re interested in predicting now comes from a new process. From a statistical modelling point of view, this is like playing a game where the rules can change while you play.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/306/0*bpGKhcAVhJbsa9W1" /></figure><p>Concept drift is more likely to occur in domains of high complexity, especially when they involve people’s behavior. If you use ML for image recognition, chances are you won’t have to deal with concept drift very much (unless there are changes to the sensor or new kinds of object pop up). On the other hand if you use it for, say, demand forecasting, it would be a good idea to retrain your model with training data that takes the Covid-19 outbreak into account (e.g. by throwing out data before the outbreak).</p><p>Business decisions made using models suffering from concept drift are likely to be flawed and impact the bottom line. You may waste money on inappropriate marketing, overstock or understock items, or offer the wrong prices.</p><h3>Toilet Paper Shortages: an example of concept drift.</h3><p>Australian supermarkets are currently experiencing a <a href="https://www.sbs.com.au/news/the-feed/local-councils-beg-sydney-residents-to-stop-stealing-toilet-paper-from-public-toilets-amidst-coronavirus-shortages">shortage of toilet paper</a>. This is due to concept drift. The amount of an item that’s in stock depends on how much is forecasted to sell. Usually forecasted sales are based on how much sold previously. However the COVID-19 outbreak led to a lot of hoarding and demand for toilet paper spiked. There was then a lag as suppliers and manufacturers struggled to catch up with the new demand. Concept drift may hit supermarkets on the way down as well, as consumers don’t buy any toilet paper at all because they have a <a href="https://i.pinimg.com/736x/2e/83/70/2e8370549f6ab7c8d11aff6cd826a3c2--prepping-funny-sayings.jpg">6 month supply</a>.</p><p>Below is a (fictional) illustration, based on a moving average forecast. You can see how the old data leads to much greater error than usual.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*9l_4bhmfyNcaafm_TUTL1A.png" /><figcaption>The spike in demand for toilet paper is a great example of concept drift. A naive forecasting model will assuming that this rise in demand is permanent.</figcaption></figure><h3>Data Drift</h3><p>Another cause of model production problems is data drift. While concept drift is about changes in the target variable (an output like forecasted toilet paper sales), data drift occurs when there are changes in the input data. An example of data drift would be a sudden sensor malfunction.</p><p>To recognise data drift, log and analyse incoming data. For example, if column averages change drastically, chances are that something is going on with the way data are collected.</p><p>Another thing that can happen is that a model is set up to pull some of its data from external sources, but one of the sources is no longer operational or has changed in nature. This would result in a much larger fraction of missing data than the model is set up to cope with. Again this would show up in logs.</p><p>To remedy concept drift and data drift once it’s detected, you will have to retrain your model on data generated by the new process.</p><h3>Fighting Drift</h3><p>The best tool we have against our models becoming unexpectedly inaccurate is a good model monitoring system. In particular, you should</p><ol><li><strong>Log inputs. </strong>This is to prevent data drift. A model may be trained for data of one kind, and then start behaving weirdly when it encounters data that was completely out of scope during training.</li><li><strong>Log model predictions</strong>. If your models start delivering very different outputs than was historically the case, chances are that the domain has changed and you may be subject to concept drift.</li><li><strong>Log model accuracy</strong>. For some applications of machine learning (e.g. forecasting), you may be able to examine predictive accuracy directly. If predictive accuracy starts decreasing substantially, chances are you have concept drift.</li><li><strong>Log upstream behaviour</strong>. If your model matters, it will have an effect on the real world. For example a recommendation system will affect consumer behavior. If this behavior suddenly changes, then your model may no longer be appropriate.</li></ol><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*SXlRZ2xxvZ-7_W2-3KgeLg.png" /></figure><h3>When do I retrain?</h3><p>There are two options for model retraining:</p><ol><li>Continual retraining, or</li><li>Retraining after an alert is triggered.</li></ol><p>To continually retrain a model, you need a <a href="https://www.youtube.com/watch?v=32hRmUQaBt4">machine learning pipeline</a>. This will automate the collection and preprocessing of training data, model training, cross validation and deployment.</p><p>If you decide to retrain only when an alert is triggered, you need to find appropriate thresholds. The idea is to view the problem as one of <a href="https://www.statisticsviews.com/details/feature/5663861/Control-charts--tools-for-understanding-variation.html">quality control</a>, we can view concept drift as being analogous to a malfunctioning machine part, which would produce greater than usual variance.</p><p>A product manager or domain expert can help decide an acceptable model accuracy threshold. If, on the other hand, you alert based on changes in a metric, a starting threshold could be 4 standard deviations away from historical average. If metrics are collected daily, then you would expect a 4 standard deviation change to only happen about once every 43 years.</p><h3>Which data should I retrain on?</h3><p>When concept drift occurs, you should not retrain the model naively on old data that was generated by a different process than the one you’re modelling. Sometimes there will be a dramatic event that signals concept drift (e.g. a pandemic outbreak!), and sometimes your model may just gradually get less and less accurate.</p><p>There are three different approaches for handling this:</p><ol><li>Exclude all training data prior to concept drift, this would result in much less training data available and is only viable in the “dramatic event” case.</li><li>Weighting your training data by recency, so that the most recent data is weighted the most heavily.</li><li>Creating a new model that explicitly corrects for the source of concept drift. An example of this would be forecasting models that allow for different behavior during holidays.</li></ol><h3>Getting help</h3><p>As we have seen, drift is a real issue for systems dependent on machine learning or statistics, such as inventory management, pricing, advertising, or trading. If you think you have a problem with concept drift or data drift, we at Eliiza can help you, through advice, designing custom drift monitoring and retraining pipelines, as well as integrating automated services into your cloud platforms. Please <a href="/service/mailto:info@eliiza.com.au">get in touch</a> if you want to know more. <a href="https://docs.aws.amazon.com/sagemaker/latest/dg/model-monitor.html">AWS</a>, <a href="https://docs.microsoft.com/en-us/azure/machine-learning/how-to-monitor-datasets">Azure</a> and <a href="https://cloud.google.com/solutions/machine-learning/mlops-continuous-delivery-and-automation-pipelines-in-machine-learning">Google Cloud Platform</a> all offer tools for model monitoring, and we have experience integrating them into machine learning pipelines</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=a00509e4920b" width="1" height="1" alt=""><hr><p><a href="https://medium.com/eliiza-ai/why-your-models-might-not-work-after-covid-19-a00509e4920b">Why your models might not work after Covid-19</a> was originally published in <a href="https://medium.com/eliiza-ai">Eliiza-AI</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[What I Learned at the Tensorflow Dev Summit 2020 — Part 1]]></title>
            <link>https://medium.com/eliiza-ai/what-i-learned-at-the-tensorflow-dev-summit-2020-part-1-c9d4fc829b24?source=rss----d3822484c236---4</link>
            <guid isPermaLink="false">https://medium.com/p/c9d4fc829b24</guid>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[dataset]]></category>
            <category><![CDATA[ai]]></category>
            <category><![CDATA[tensorflow]]></category>
            <dc:creator><![CDATA[Patrick Robotham]]></dc:creator>
            <pubDate>Wed, 01 Apr 2020 04:44:27 GMT</pubDate>
            <atom:updated>2020-04-04T05:32:49.553Z</atom:updated>
            <content:encoded><![CDATA[<h3>What I Learned at the Tensorflow Dev Summit 2020 — Part 1</h3><h3>Spotting weeds with Tensorflow Datasets</h3><h3>Introducing Tensorflow Datasets</h3><p>One of the biggest obstacles to learning the techniques of deep learning is finding a good dataset. A lot of examples just cover handwriting recognition (the famous <a href="https://www.tensorflow.org/datasets/catalog/mnist">MNIST </a>dataset). Some online data is published in a weird proprietary format and you spend just as much time extracting the data as you do with any kind of machine learning. Enter Tensorflow Datasets: go from a URL to vectors in one line of code. And you’re not stuck with MNIST!</p><p>Google discussed <a href="https://www.tensorflow.org/datasets/overview">Tensorflow Datasets</a> at the <a href="https://www.youtube.com/watch?v=_lsjCH3fd00&amp;list=PLQY2H8rRoyvzuJw20FG82Lgm2SZjTdIXU&amp;index=1">2020 Tensorflow Dev Summit</a>. They’ve made available a lot of well documented, labelled datasets that are ideal for deep learning projects. All kinds of data are <a href="https://www.tensorflow.org/datasets/catalog/overview">available</a> such as:</p><ul><li>Audio,such as <a href="https://www.tensorflow.org/datasets/catalog/librispeech">books read aloud </a>and <a href="https://www.tensorflow.org/datasets/catalog/groove">13.6 hours of tempo-aligned expressive drumming</a>.</li><li>Images, such as photos of <a href="https://www.tensorflow.org/datasets/catalog/celeb_a">10,117 different celebrities</a> or mappings from <a href="https://www.tensorflow.org/datasets/catalog/aflw2k3d">photos to facial landmarks</a></li><li>Text, such as <a href="https://www.tensorflow.org/datasets/catalog/natural_questions">reading comprehension quizzes</a>, <a href="https://www.tensorflow.org/datasets/catalog/scientific_papers">summaries of scientific papers</a>, and <a href="https://www.tensorflow.org/datasets/catalog/imdb_reviews">movie reviews</a></li><li>Video, including a mapping from<a href="https://www.tensorflow.org/datasets/catalog/robonet"> a video of a robot to how the robot is controlled</a>.</li></ul><p>For this blog, we’ll look at the <a href="https://www.tensorflow.org/datasets/catalog/deep_weeds">deep_weeds</a> dataset. It consists of images of different species of weeds native to Queensland. Neural Networks are great at image classification, and an interesting application of this is weed control. It’s easier to survey crops than ever thanks to drones, and so you could imagine using a weed classifier as part of a crop management system.</p><p>I chose this dataset because it’s local to Australia and I like the idea of using deep learning to help Agriculture. Please <a href="/service/mailto:patrick.robotham@eliiza.com.au">get in touch</a> if you work in agriculture and think machine learning would be helpful.</p><p>This blog post will be about showing some code to explain how to use these datasets.</p><h3>Downloading Datasets</h3><p>Downloading datasets is straightforward.</p><pre>import tensorflow_datasets as tfdsdata, info = tfds.load(&quot;deep_weeds&quot;, with_info = True)</pre><p>The infoobject provides metadata describing the dataset in question. From it we learn that the dataset consists of 17509 images with shape 256x256x3, and that it has not been split up into training vs test sets.</p><p>Let’s do that now:</p><pre>data_train, info = tfds.load(&quot;deep_weeds&quot;, with_info=True, split=&#39;train[:80%]&#39;)<br>data_valid  = tfds.load(&quot;deep_weeds&quot;,split=&#39;train[80%:90%]&#39;)<br>data_test = tfds.load(&quot;deep_weeds&quot;, split=&#39;train[90%:]&#39;)</pre><p>We can preview the dataset with the show_examples function.</p><pre>tf.show_examples(data_train)</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/516/0*_oZvJ-TvTRP-CdSo.png" /></figure><h3>Working with tf.data.Datasets</h3><p>The objects that we’ve just created are Datasets. Unfortunately they don’t behave like arrays. This is because if we do away with the notion of ordering, we can parallelize processing more easily.</p><p>data_train[0] # Throws an error.</p><p>If you want to “poke inside” a dataset, I recommend taking a sample and casting it to a list. Here’s us confirming the shape of our data is 256x256x3.</p><pre>&gt;&gt;&gt; my_data = list(data_train.take(5))</pre><pre>&gt;&gt;&gt; type(my_data[0])<br>dict</pre><pre>&gt;&gt;&gt; my_data[0].keys<br>dict_keys([&#39;image&#39;, &#39;label&#39;])</pre><pre>&gt;&gt;&gt; my_data[0][&#39;image&#39;].shape<br>TensorShape([256, 256, 3])</pre><p>You could also convert the entire dataset to a list, but this is slow and uses a lot of memory.</p><p>To preprocess datasets, use the map method. The function below scales pixel values between 0 and 1 and converts to the (data, label) format that keras expects.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/1c56407cd7ba311088470d5582b4758d/href">https://medium.com/media/1c56407cd7ba311088470d5582b4758d/href</a></iframe><p>We can now train an image classifier. I use <a href="http://torch.ch/blog/2016/02/04/resnets.html">ResNet50 </a>as the base because that’s what was used in the <a href="https://www.nature.com/articles/s41598-018-38343-3">deep weeds paper.</a> The paper trained an image classifier to achieve 95% accuracy on the validation set.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/40721293d32b7bd800ad4d2b298762b9/href">https://medium.com/media/40721293d32b7bd800ad4d2b298762b9/href</a></iframe><p>After 5 epochs, I got a training accuracy of 95%, but a validation accuracy of 50%. This is a classic case of overfitting and we’ll improve it in the next section.</p><h3>Image Augmentation on Datasets</h3><p>A standard method for dealing with overfitting is to feed in more data. Since going out and taking pictures of weeds would be time consuming and involve going outside, we’ll just generate new images synthetically with image augmentation.</p><p>I usually use the keras <a href="https://keras.io/preprocessing/image/">image augmentor</a>, which has a bunch of different augmentations built in. Unfortunately it doesn’t work well with Tensorflow Datasets.</p><p>Because the default keras image augmentor is based on numpy arrays, we have to use the tf.numpyfunction to use it. This <a href="https://stackoverflow.com/questions/59648804/how-can-i-combine-imagedatagenerator-with-tensorflow-datasets-in-tf2">stackoverflow post</a> elaborates. This approach is fairly slow however, and I wouldn’t recommend it. Instead it’s better to<a href="https://www.tensorflow.org/tutorials/images/data_augmentation#augment_a_dataset_and_train_a_model_with_it"> write your own </a>image augmentation pipeline with the tf.image library.</p><p>Below is an example which applies random augmentations such as flipping, rotation and adjusting brightness to the images. Note that we should only apply image augmentation to the training set, not the validation set. This is because the validation set gives a measure for the performance of the classifier and this measure won’t change even if we adjust the training method.</p><p>Below is a sample image augmentation pipeline that takes an image, randomly adjusts brightness, randomly flips the image, adjusts the hue, contrast and rotates it by a random number of degrees.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/2caccc17e5d6ac67b28cfd1dc6ded93b/href">https://medium.com/media/2caccc17e5d6ac67b28cfd1dc6ded93b/href</a></iframe><p>After augmenting the images and training for a long time (50 epochs). I was able to achieve a validation accuracy of 90%! We’re catching up to the paper! The full training code is available on <a href="https://github.com/eliiza/deep_weeds/blob/master/model_training.py">github</a>.</p><pre>438/438 [==============================] - 77s 176ms/step - loss: 0.1353 - accuracy: 0.9527 - val_loss: 0.3281 - val_accuracy: 0.9092</pre><h3>Speeding things up with prefetching</h3><p>Preprocessing data can be expensive, especially if you’re using data augmentation and need to do it during the training loop. I spend a lot of my time training models waiting for things to finish training, so any features that speed things up are really helpful.</p><p>A benefit to using Datasets is that Tensorflow provides a <a href="https://www.tensorflow.org/guide/data_performance">number of options </a>for making data processing faster. The prefetch method will overlap data preprocessing (on the CPU) and model training (on the GPU) in order to minimize the amount of time the GPU spends idle.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*7zu7eZffN6u-YETtWZl8zQ.png" /><figcaption>Sequential Processing without prefetch</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*vJrSrBtmZ3QsPSdsNdC-Aw.png" /><figcaption>Interleaved processing with prefetch. Notice the shorter gaps between training.</figcaption></figure><p>To use prefetching, just use the .prefetch method. Note that we don’t augment the validation set.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/ed7eca44f19d5dcc39bcd16474bf83f5/href">https://medium.com/media/ed7eca44f19d5dcc39bcd16474bf83f5/href</a></iframe><p>Without prefetching, training on a frozen Resnet model took 50 seconds per epoch in Colab. With prefetching, training took 30 seconds. The image augmentation takes time that could be spent training the model, so augmenting the next batch of images while we train is really useful!</p><p>In this blog we’ve:</p><ul><li>Looked at the Deep Weeds dataset from tensorflow datasets</li><li>Trained an image classifier,</li><li>Wrote an image augmentation pipeline,</li><li>Used prefetching to speed up model training.</li></ul><p>In part 2 we’ll look at the Profiling Feature in Tensorboard and how to download pretrained models online.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=c9d4fc829b24" width="1" height="1" alt=""><hr><p><a href="https://medium.com/eliiza-ai/what-i-learned-at-the-tensorflow-dev-summit-2020-part-1-c9d4fc829b24">What I Learned at the Tensorflow Dev Summit 2020 — Part 1</a> was originally published in <a href="https://medium.com/eliiza-ai">Eliiza-AI</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[A tour of awesome features of spaCy (part 2/2)]]></title>
            <link>https://medium.com/eliiza-ai/a-tour-of-awesome-features-of-spacy-part-2-2-d7bd628a81ce?source=rss----d3822484c236---4</link>
            <guid isPermaLink="false">https://medium.com/p/d7bd628a81ce</guid>
            <category><![CDATA[python]]></category>
            <category><![CDATA[spacy]]></category>
            <category><![CDATA[transfer-learning]]></category>
            <category><![CDATA[naturallanguageprocessing]]></category>
            <category><![CDATA[machine-learning]]></category>
            <dc:creator><![CDATA[Nuszk]]></dc:creator>
            <pubDate>Thu, 30 May 2019 00:24:08 GMT</pubDate>
            <atom:updated>2019-08-28T09:18:01.079Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*xGYsNSBR3oILg24XFR3RdA.png" /></figure><p>In <a href="https://medium.com/eliiza-ai/a-tour-of-awesome-features-of-spacy-part-1-2-58b32425954f">the first part of this overview</a> of spaCy we went over the features of the large English pretrained model that spaCy comes with. In this part I would like to discuss using pretraining to transfer learning to subsequent machine learning tasks. Full code for the post can be found on <a href="https://github.com/eliiza/spacy_tour/blob/master/spacy_tour_part2.ipynb">GitHub</a>. If you are new to NLP perhaps you might consider reading <a href="https://heartbeat.fritz.ai/nlp-chronicles-intro-to-nlp-with-nltk-b2c369fbb9a7">this intro</a> first.</p><p>The plan for this post is to train a TextCategoriser by first pretraining the model with the text from the training data. Pretraining will give us weights that we can use to initialise the pipeline component we want to train later. We will use the vectors-only model for pretraining, so run the following command in terminal to download it.</p><pre>python -m spacy download en_vectors_web_lg</pre><p>Even though pretrained models do not come with a TextCategorizer component spaCy has a blank native spot reserved for it with relevant infrastructure already in place. I will train a TextCategorizer pipeline component using a dataset that I derived from <a href="https://www.kaggle.com/harrisonjansma/medium-stories">Medium tags data</a> published on Kaggle. The dataset that I used consists of 23833 Medium story titles and tags of 1 or 0 depending on whether a story is related to ML/AI or not, respectively. The classes are balanced. One third of these went into the training set and the rest into the testing set. I used the training set titles (over 120,000 words total) for pretraining.</p><h3>Pretraining</h3><p>The pretraining bit was added recently with the <a href="https://explosion.ai/blog/spacy-v2-1">release of spaCy 2.1</a> and allows the use of <em>Language Modelling with Approximate Outputs</em> (LMAO), or training of a transformer for the ‘token-to-vector’ (tok2vec) layer of pipeline components with the language modeling objective of predicting word vectors rather than words.</p><p>Following the spaCy 2.1 release notes I ran two pretraining jobs, the first one can be used with any model that does not use vectors, such as en_core_web_sm or a blank model, and the second one can be used with en_core_web_md/lg.</p><pre>python -m spacy pretrain texts.jsonl en_vectors_web_lg ./pretrained-model</pre><pre>python -m spacy pretrain texts.jsonl en_vectors_web_lg ./pretrained-model-vecs --use-vectors</pre><p>The texts.jsonl has to be in a line separated json format with {&#39;text&#39;:&#39;training text&#39;} on each line. I used jsonlines library for this while spaCy’s website recommends using srsly.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/9be11df43f417290f1eb16272ccdfed0/href">https://medium.com/media/9be11df43f417290f1eb16272ccdfed0/href</a></iframe><p>I used the default parameters that can be checked with the command below. These parameters align with &#39;simple_cnn&#39; architecture that is used for training text categoriser after pretraining. The default number of iterations is 1000 but some of the interim models are saved during the pretraining process. So, below I report results for the final models and for models after only 50 iterations.</p><pre>python -m spacy pretrain -h</pre><h3>Training</h3><p>The pretraining returns weights that can be used to initialise a training model. There are a few steps to set up a training job. I started with <a href="https://spacy.io/usage/training#textcat">the code from spaCy TextCategorizer user guide</a> and adapted it as needed. First we have to load a model that we are going to train, create a pipeline component to train and add it to the pipeline as well as add categories to train for.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/5cc6f1e03c91cc170b253a92b103b8af/href">https://medium.com/media/5cc6f1e03c91cc170b253a92b103b8af/href</a></iframe><p>My training and test sets come in the format below. But the data has to be a list of (text, category dictionary) pairs. So we do a bit of preprocessing.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/722/1*_URZ6WwErqFQ9x-rwXVZJQ.png" /></figure><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/1647c9d1ebca8925b57ae277ed8c1a2b/href">https://medium.com/media/1647c9d1ebca8925b57ae277ed8c1a2b/href</a></iframe><p>Now we are ready to train the TextCategorizer. We just disable the other pipeline components as we train.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/30457816af3d93e4f0e0b46ea99fbcfc/href">https://medium.com/media/30457816af3d93e4f0e0b46ea99fbcfc/href</a></iframe><p>I trained six models: three blank models and three en_core_web_lg models. For each group one model was trained without pretraining, one with corresponding pretrained weights of 50 iterations and one with corresponding pretrained weights of 1000 iterations. Then I evaluated training loss and accuracy, precision, recall and F1 scores on the test set for each of the five training iterations. Here we report only loss, accuracy and F1 scores but the full results as well as the full training code can be found on <a href="https://github.com/eliiza/spacy_tour/blob/master/spacy_tour_part2.ipynb">GitHub</a>.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*RF7GE6gOxHtNeldIP9yJXA.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*YQRkJ8fdXFEwbF3W52FUTQ.png" /></figure><p>We notice the following:</p><ol><li>There is a small but consistent improvement with pretraining for both the blank model and en_core_web_lg model.</li><li>Doing only 50 iterations of pretraining does not seem to have as much effect for the large model as for the blank model.</li><li>Pretraining improves the blank model more than the large english model.</li><li>Starting from blank model without pretraining, just using a larger model and just using the pretraining give comparable results.</li><li>The best model is the large English model with pretraining.</li><li>The scores decrease with iterations which suggests that the models are overfitting the training data and looking at the training loss also supports this.</li></ol><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*fNkVuhVvBBre_dOiRf3xqQ.png" /></figure><p>Considering that our pretraining data was not very large, only about 120,000 words, and that it was given as sentence (titles, that might not even be sentences) chunks rather than recommended paragraph-sized chunks, the pretraining with 1000 iterations results in a reasonable improvement.</p><p><strong>Conclusion</strong></p><p>spaCy is a fast and flexible NLP library. We can add a new pipeline component, replace the existing ones or fill in one of the native empty component spots. One such spot is reserved for a TextCategorizer with related infrastructure already in place. Even though it is a little hard to navigate through documentation when it comes to training and pretraining there are examples out there to learn from and follow. While each of the pretraining jobs took about 5.5 hours (~6,000 words per second) on a four core CPU, LMAO is still feasible to experiment with. The pretraining done with only about 120,000 words resulted in a slight but consistent improvement in the TextCategorizer performance.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=d7bd628a81ce" width="1" height="1" alt=""><hr><p><a href="https://medium.com/eliiza-ai/a-tour-of-awesome-features-of-spacy-part-2-2-d7bd628a81ce">A tour of awesome features of spaCy (part 2/2)</a> was originally published in <a href="https://medium.com/eliiza-ai">Eliiza-AI</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[A tour of awesome features of spaCy (part 1/2)]]></title>
            <link>https://medium.com/eliiza-ai/a-tour-of-awesome-features-of-spacy-part-1-2-58b32425954f?source=rss----d3822484c236---4</link>
            <guid isPermaLink="false">https://medium.com/p/58b32425954f</guid>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[transfer-learning]]></category>
            <category><![CDATA[spacy]]></category>
            <category><![CDATA[naturallanguageprocessing]]></category>
            <dc:creator><![CDATA[Nuszk]]></dc:creator>
            <pubDate>Thu, 30 May 2019 00:20:26 GMT</pubDate>
            <atom:updated>2019-08-28T09:17:36.247Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*vc7ZrlbLFYg316e5RtOBRg.png" /></figure><p>A few weeks ago I started working on a text summarisation project and I needed a Natural Language Processing library with comprehensive features. The project had several potentially computationally expensive components where I wanted to try out different things. So I needed something flexible that was also fast enough for me to experiment with. The search brought me to <a href="https://spacy.io/">spaCy</a>. spaCy was very helpful and I decided to summarise its features in this two part guide. Full code for the post can be found on <a href="https://github.com/eliiza/spacy_tour/blob/master/spacy_tour_part1.ipynb">GitHub</a>. If you are new to NLP perhaps you might consider reading <a href="https://heartbeat.fritz.ai/nlp-chronicles-intro-to-nlp-with-nltk-b2c369fbb9a7">this intro</a> first — it explains many concepts refered to in this post.</p><p><strong>Why I like spaCy</strong>:</p><ul><li>It is fast because of <a href="https://medium.com/huggingface/100-times-faster-natural-language-processing-in-python-ee32033bdced">Cython’s data structures and parallelism magic</a>: so I can experiment with different approaches.</li><li>It is accurate: there are three pretrained english models to choose from, small, medium and large. As accuracy and speed change accordingly, this gives flexibility to balance them based on a given task.</li><li>It is user-friendly: the documentation is good for most of the package and there is a built-in explain function for quick access.</li><li>There is a free interactive four hour <a href="https://course.spacy.io/">course</a> to get one started.</li><li>It has mild learning curve: it was possible to get started after taking the course and learn as needed on the go.</li></ul><p><strong>Features</strong>:</p><ul><li>Preprocessing: tokenisation, sentence segmentation, lemmatisation, stopwords</li><li>Linguistic features: part of speech tags, dependency parsing, named entity recogniser</li><li>Visualisers for dependency trees and named entity recogniser</li><li>Pre-trained word vectors and models</li><li>Flexibility: can augment or replace any pipeline component or add new components such as TextCategorizer.</li><li>Transfer learning with BERT-style pretraining</li></ul><p><strong>Downsides</strong>:</p><ul><li>The documentation becomes rather hard to navigate through as one gets to training and pretraining of models.</li><li>There is some common NLP functionality missing, such as scikit-learn-style vectorisers for term-document or TFIDF matrices. Even though these are not necessary if you are training your models with spacy, they are still handy if you want to combine spaCy with other tools.</li></ul><h3>Getting started</h3><p>First we need to install spaCy and download a pretrained model. There are three english language models, small, medium and large as well as the model with only GloVe words vectors. The medium and large models also come with GloVe words vectors. All en_core_web_* models come with tokeniser, tagger, parser and entity recogniser components but accuracy improves with model size. I will use the large model here. To get started run the following commands in terminal.</p><pre>pip install spacy<br>python -m spacy download en_core_web_lg</pre><p>Next we create an nlp object and load a model in it. The nlp object now has a tokeniser, tagger, parser and entity recogniser in its pipeline and we can use it to process a text and get all of those features.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/947c668082cd767cb0ab20dc6e85e07c/href">https://medium.com/media/947c668082cd767cb0ab20dc6e85e07c/href</a></iframe><h3>Preprocessing</h3><p>At this point some of the usual text preprocessing tasks are a breeze. The doc can be sliced with token indices to get single tokens or sequences of tokens (spans) and various token attributes such as text, lemma, index, pos, tag and etc. can be accessed. Some attributes extend to spans as well. Sentence segmentation is also available.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/da15a73fb5bd64ac93b0de73b157b917/href">https://medium.com/media/da15a73fb5bd64ac93b0de73b157b917/href</a></iframe><p>The medium and large english models also come with GloVe vectors and the vectors can be accessed through token/span/doc .vector attribute. Vector of a span or a doc is calculated by taking the average of vectors for all tokens in the span. spaCy also has built in similarity function.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/631b40480d2efe997d8a105a66218d83/href">https://medium.com/media/631b40480d2efe997d8a105a66218d83/href</a></iframe><p>The similarity function is the same as the cosine similarity, or the cosine of the angle between two vectors. Cosine similarity ignores vector lengths and, in the two extreme cases, vectors pointing in the same direction have similarity 1, while vectors pointing in the opposite direction have similarity -1. So larger the value higher the degree of similarity.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/d003f38486d4d2e59c6709c2feeb53b9/href">https://medium.com/media/d003f38486d4d2e59c6709c2feeb53b9/href</a></iframe><h3>Linguistic features</h3><p>We could also retrieve some linguistic features such as noun chunks, part of speech tags and dependency relations between tokens in each sentence. In order to understand what various tags such as token.pos_, token.tag_ or token.dep_ mean, we can use spacy.explain() that will access annotation specifications.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/17dc44c2caf3800ed5e8e87c9fe3a478/href">https://medium.com/media/17dc44c2caf3800ed5e8e87c9fe3a478/href</a></iframe><p>Notice that token.tag_ carries extra morphological information compared with token.pos_. These attribures, as well as some others, come in pairs with and without an underscore, corresponding to a unicode and integer values respectively. The .dep_ attribute is for dependency relations between words and is best understood via spaCy’s built-in customisable visualiser.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/a96aa0434fa79e262967d17b25cff9e8/href">https://medium.com/media/a96aa0434fa79e262967d17b25cff9e8/href</a></iframe><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*vc7ZrlbLFYg316e5RtOBRg.png" /><figcaption>Dependency tree</figcaption></figure><p>Retrieving and visualising named entities is done very conveniently in spaCy.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/a5ed126a0235371c061845b947d4c1c1/href">https://medium.com/media/a5ed126a0235371c061845b947d4c1c1/href</a></iframe><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*FVLg_nIrFKuX0vk-kl6EXw.png" /></figure><p>You could introduce some custom options for visualisation part.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/41e8e87a10988d82b042202af8cdc204/href">https://medium.com/media/41e8e87a10988d82b042202af8cdc204/href</a></iframe><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*PhFYSBFFD3FXMgjTIHUqyg.png" /></figure><h3>spaCy universe</h3><p>spaCy is very flexible. It is possible to add new pipeline components or replace existing ones. People have been building on top of spaCy and there is a myriad of packages in the <a href="https://spacy.io/universe">spaCy universe</a>. I will only mention two of the pipeline extentions, <a href="https://spacy.io/universe/project/spacy-langdetect">spacy_langdetect</a> and <a href="https://spacy.io/universe/project/neuralcoref">neuralcoref</a>, but there are many other packages worth spending time to play with.</p><p>Since a lot of text data comes from mostly uncurated sources such as the web the language detector functionality comes especially handy. For example, combining the lang_detect with with spaCy’s .lang_ attribute (which points to the language of the model of the nlp object used to process the text) one could ensure that the correct language model is used. Notice ._. is used to access extensions.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/8e84b90b331b0c8444541656be6f787e/href">https://medium.com/media/8e84b90b331b0c8444541656be6f787e/href</a></iframe><p>Another common feature often necessary in text processing is Coreference Resolution, or linking together all phrases that mention the same thing. For example, in “I like ice cream because it is tasty” both “ice cream” and “it” refer to the same thing. Coreference Resolution is a key feature of language understanding and while humans are pretty good at it, machines are not. The neuralcoref extension integrates this functionality in spaCy as another pipeline component. We need to re-process doc objects that were created before adding new pipeline components to make neuralcoref available.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/ef67cdf63fcdb25454a147fdc7e74eb4/href">https://medium.com/media/ef67cdf63fcdb25454a147fdc7e74eb4/href</a></iframe><p>We see that ‘it’ is incorrectly resolved while resolution for ‘he’ is correct and the corresponding scores show that the model wasn’t as confident in it’s decision for ‘it’ as it was for ‘he’. We can tweak the greediness of the model to make it stricter about resolving coreferences. The default greediness is .5 and if we decrease it to .45 we see that the coreferences are now resolved correctly.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/85640a6c70ac17a17885041f077feaad/href">https://medium.com/media/85640a6c70ac17a17885041f077feaad/href</a></iframe><p>The neuralcoref also comes with a <a href="https://huggingface.co/coref/">visualization client</a>.</p><h3>Conclusion</h3><p>spaCy has almost all of the common preprocessing and linguistic features used for text processing. It is user-friendly and one can start using it with minimal initial prep. There is a considerable ecosystem of additional packages built on top of spaCy. The documentation is very helpful for using these features and there is a really handy spacy.explain function to quickly access annotation specifications. Several of the features also come with built-in visualisers. We discuss BERT-style pretraining and pipeline component training examples in <a href="https://medium.com/eliiza-ai/a-tour-of-awesome-features-of-spacy-part-2-2-d7bd628a81ce">the second part of this tour</a>.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=58b32425954f" width="1" height="1" alt=""><hr><p><a href="https://medium.com/eliiza-ai/a-tour-of-awesome-features-of-spacy-part-1-2-58b32425954f">A tour of awesome features of spaCy (part 1/2)</a> was originally published in <a href="https://medium.com/eliiza-ai">Eliiza-AI</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Utility meter reading on mobile devices]]></title>
            <link>https://medium.com/eliiza-ai/utility-meter-reading-on-mobile-devices-dfb376c83465?source=rss----d3822484c236---4</link>
            <guid isPermaLink="false">https://medium.com/p/dfb376c83465</guid>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[utility-infrastructure]]></category>
            <category><![CDATA[computer-vision]]></category>
            <category><![CDATA[neural-networks]]></category>
            <category><![CDATA[ai]]></category>
            <dc:creator><![CDATA[Eike Germann]]></dc:creator>
            <pubDate>Fri, 03 May 2019 03:41:29 GMT</pubDate>
            <atom:updated>2019-05-03T04:22:30.952Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*0CfL2TlQi-NbQ_bnpfNEIQ.jpeg" /></figure><p>Australia is a vast country. From the sprawling metropolis with the highest living standard on the planet to the isolated rural settlement cut off from any power grid or network — Australia has it all. This wide range of environments makes it a challenge to maintain or upgrade infrastructures like the electricity or gas network.</p><h3>Reading meters today</h3><p>Some states, like Victoria, are ahead of the curve with mandated upgrades and have already phased out analog electricity metering devices. All electricity meters in Victoria are digital smart meters that can collect more than just the reading of the energy consumption and often connect directly to the provider via the Internet of Things, providing usage data every 30 minutes. Modern contracts that have pricing based on time of day, day of the week and more can be easily facilitated and measured for with devices like this, a single device can record energy consumption along with a timestamp and the rest is worked out by computer calculation. The regular updates significantly reduce the chances for any discrepancy between the value billed for and the energy consumed.</p><p>However, Victoria is an outlier. In other states, the energy providers have to have physical access to the customer’s meter to read it. If the providers cannot read the customer’s meter, they provide an estimated bill. With the arrival of more extreme weather events in recent years in both summer and winter, these estimates can diverge from the actual consumption and frustrate the customer when they receive an updated bill with a higher value than expected. To reduce these sudden gaps, some energy providers have the option for customers to read their meter themselves and supply the reading to the provider to update the estimate to the correct value.</p><p>Unfortunately, reading analog gas and electricity meters is packed with challenges for the consumer. Even if the customer can identify which meter is theirs (see image below),</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*oGJVnhIZ7duA6m8SowIQvg.jpeg" /><figcaption>Typical meter array in an apartment block (CC BY 2.0 <a href="https://www.flickr.com/photos/marktee/9437074650/in/photolist-fnVtvC-9LJSPz-49gR7d-9zqqEd-nJdhjb-TJehn3-rvgWQ-hLmvY-dWAA7A-72VZwv-e3ud9A-4YbkiU-udRcVz-YbHDts-5AvSJX-8LzFXt-7ic3za-61QLQk-8TZg4D-nzk1Qr-">Mark Turnauckas Photography</a>)</figcaption></figure><p>knowing how to read the meter, how many digits are required and how this affects their bill can be near impossible for non-experts (try it yourself on the examples below!).</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*4HSp3NxrtDrwsLFpFEnyjg.jpeg" /><figcaption>Meters are tricky. Can you spot the difference? (credit: AGL)</figcaption></figure><p>AGL approached us to find a way to improve the customer experience by giving the customer more control and information when they are reading their meters in a way they can easily use and understand.</p><h3>The project</h3><p>AGL were looking for an application that would be a game changer for its customers — a solution that would enable an interactive meter read, in real time. The objective was to reduce unexpected bills as much as possible and give the customer more control over their energy consumption.</p><p>The initial project timeline had two stages: A Proof of Concept (POC) would determine the feasibility of a solution and, if completed successfully, lead into a field trial to test the tool ‘in the wild’.</p><p>The challenges</p><p>To begin finding a solution, we had to understand the problem space. The meters, both digital and analog, come in various makes and models:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*lnydIwrCBXbGz8mHbfjldw.png" /><figcaption>A few examples of meter types and makes (credit: AGL)</figcaption></figure><p>AGL has a <a href="https://www.agl.com.au/help/meters-connections/how-to-read-your-gas-electricity-meter">guide</a> to help customers read their meters manually. These instructions divide the meters into four different groups for both electricity and gas meters. While this information gave us a starting point, we did not know much about the proportions of these groups among the meters in use and what challenges the individual types might pose.</p><p>AGL provided us with a large dataset of images from all groups of meter types which we had to classify manually in separate stages. Regular discussions with subject matter experts at AGL were invaluable to set priorities and identify pitfalls and opportunities in classifying and reading the individual meters.</p><p>The images supplied by AGL were a genuine reflection of the customers’ challenges and so it quickly became obvious that the analog electricity meters with clock faces were the most difficult to read. At the same time, they are the most ubiquitous non-digital model still in use. We therefore decided that the POC should demonstrate how the most complex and most widespread problem can be tackled successfully and focussed on developing a solution to identify all meters and read clock faced meters specifically.</p><p>It wasn’t just the meter read that needed to be captured through the app, it also had to locate the label on the meter that connects to the specific customer.</p><p>Furthermore, the app had to locate the region on the meter that would identify it in relation to the customer. This was especially important to help the customer manage the complexity of modern peak/off-peak contracts that require multiple analog meters to record power consumption for different times of the day.</p><p>We realised that due to the complexity of the problem space and the number of data labels we would require, we had to not only understand how to read the meters ourselves, we had to find ways to quickly teach large groups of people how to do the same to distribute the labelling process. The transfer of human knowledge and understanding into a machine readable format is still one of the most time-consuming and error-prone steps in the process of machine learning. A machine learning model will only be calibrated as well as the data it is trained with, and it is tempting to take shortcuts because the labelling task is repetitive and tedious to humans. We solved this problem in collaboration with AGL, having multiple short labelling sessions with a limited number of volunteers who focussed on an individual task or label type to speed up the learning process and minimise confusion and risk of error.</p><h3>The solution</h3><p>Given the target of an interactive, real-time application on a mobile platform, we decided to use the live camera output of the phone as an input stream of images for the app to read sequentially. This greatly improved the app’s resilience to glare, reflections or other effects that could make a single image unreadable.</p><p>The objects we wanted to detect were very complex, so we chose to divide the model into smaller, specialised subunits that would only be used if required. The first component of the architecture would therefore be an object detector model that identified elements of each frame and filtered out the relevant information to pass on to the specific subunit. A final logic would statistically evaluate the results from multiple frames in real time to reduce errors and produce a result with a detailed confidence value for each component.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*C2KF2ZNj_dssqn1YkhkgyA.png" /><figcaption>A basic outline of the solution architecture to read clock face meters</figcaption></figure><p>We developed the initial POC as a mobile app on the iOS platform. From the beginning, it was our goal to not only provide AGL with a well developed technology, but also to enable AGL to maintain and upgrade the product on their own.</p><p>For implementation we chose to use Tensorflow as the machine learning framework. It is widely accepted and can be run on any platform, be it locally, on a virtual machine or with cloud GPU support in AWS, GCP or Azure. Our preferred machine learning cloud environment for this project was Google Cloud Platform, hosting our training data and training on Google ML Engine. Tensorflow also supports the ability to export to both CoreML and TensorFlow lite for applications on iOS and Android which secured the necessary compatibilities beyond the POC stage.</p><p>Another advantage of using Tensorflow was the range of pre-trained models available as part of the Tensorflow object detection API. Using a pre-trained version of a neural network optimised for mobile devices, meant we could quickly train and iterate through various versions of our object detector without having to completely start over every time.</p><p>Throughout the process, we documented the procedures, tools and techniques used for labelling, training and model deployment and met several times with the internal data science team at AGL. It was important for us to keep the client informed of our progress as well as sharing our reasoning behind choices for design, architecture and tooling. AGL’s data science team were very receptive of our continuous involvement and greatly impressed by the capabilities and portability of the Tensorflow approach.</p><h3>Results</h3><p>The goal to increase the customer’s control over their power bill provided us with a straightforward metric: measure the accuracy of the meter reader in actual dollars. This had the added advantage that the metric was relatable for both developers and stakeholders.</p><ul><li>The field trial had a sample size of more than 300 measurements and 74% were within a range of modest deviation of $5</li><li>The region linking the meter with the customer was found in more than 99% of the images.</li><li>The average time to take these measurements was 5–6 seconds, less than half the time taken by an average human reader.</li></ul><p>This is a great result for a proof of concept and a field trial. Not only did we develop a standalone app that holds great promise of increased control and insight for AGL’s customers, we discovered various possibilities of further improvement and increased performance during our trials.</p><p>This prototype has the capability to give both AGL and its customers a more valuable, real time insight into gas and energy consumption and provide more control for better pricing as well as conscious spending and saving money.</p><blockquote>“We were looking for a partner with deep machine learning expertise to develop a high-performing model quickly but also with the engineering expertise to get the model into the hands of customers at scale. Eliiza’s rapid ideation approach, cross-functional data science and engineering teams, and collaborative approach to working with our internal data science and engineering teams made them an ideal partner for AGL.”<em><br></em><strong><em>Hamish McMichael — Head of Digital Transformation, AGL</em></strong></blockquote><figure><img alt="" src="https://cdn-images-1.medium.com/max/106/1*7PCp3vpd5TmLIyU6vQkNJA.png" /></figure><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=dfb376c83465" width="1" height="1" alt=""><hr><p><a href="https://medium.com/eliiza-ai/utility-meter-reading-on-mobile-devices-dfb376c83465">Utility meter reading on mobile devices</a> was originally published in <a href="https://medium.com/eliiza-ai">Eliiza-AI</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
    </channel>
</rss>