<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Jaco du Toit on Medium]]></title>
        <description><![CDATA[Stories by Jaco du Toit on Medium]]></description>
        <link>https://medium.com/@jacowp357?source=rss-815098f8de27------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/0*LN2UcTlEJBiIdvF3.jpg</url>
            <title>Stories by Jaco du Toit on Medium</title>
            <link>https://medium.com/@jacowp357?source=rss-815098f8de27------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Sun, 22 Nov 2026 18:05:16 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@jacowp357/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[Where is the Value of Data Science?]]></title>
            <link>https://medium.com/@jacowp357/where-is-the-value-of-data-science-45f19290b5f2?source=rss-815098f8de27------2</link>
            <guid isPermaLink="false">https://medium.com/p/45f19290b5f2</guid>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[value-proposition]]></category>
            <category><![CDATA[prioritization]]></category>
            <dc:creator><![CDATA[Jaco du Toit]]></dc:creator>
            <pubDate>Fri, 28 Apr 2023 06:34:53 GMT</pubDate>
            <atom:updated>2023-04-28T06:46:48.044Z</atom:updated>
            <content:encoded><![CDATA[<p>A guide for prioritising data science initiatives</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*e1wryTj6CnKzEbgrD5DR6w.png" /><figcaption>Muizenberg (Image by author)</figcaption></figure><p>Whether your starting a data science team in a startup or planning your next data science roadmap in an established organisation, the critical questions are: what initiatives should the data science team be focusing on and what value can be expected from those initiatives? In this post, I will share a useful tool that should help identify opportunities for data science initiatives, and we will discuss ways of evaluating its impact (although it was not originally intended for this purpose). By “initiatives” I mean the focus is on using data to build predictive models to solve problems or use data for optimisation. These are often referred to as data products for which a product team with complimentary skills is required during its development.</p><p>Before I introduce the tool, let’s first examine some characteristics of a typical data science project. This should provide more depth when we use the tool to refine initiatives and it should help to set more realistic expectations by considering the operational effort. So, what can be expected during a typical data science project?</p><ol><li><strong>Domain knowledge:</strong> For a chance at developing long term solutions in a specific domain (e.g., retail, healthcare, energy, marketing, engineering, law, medicine, finance, etc. — this list seems to grow more and more), it is crucial to have domain experts collaborate with data scientists so they can thoroughly understand the problems, work within any legal and regulatory requirements, and more rapidly understand domain specific datasets and business rules. Instances may arise where domain experts need to help label data that is too complex (or risky) for data scientists to label themselves. Domain experts are also required to provide feedback on the accuracy and reliability of data products, which is crucial to validate and improve a solution. In my experience, what works best is to nominate a domain expert as a product owner, since continuous input and time commitments will need to be dedicated to the data product.</li><li><strong>Integration:</strong> Some data products (especially those developed for internal use) may need to be integrated into existing systems or software so that users can use it seamlessly in their day-to-day operation. Such solutions may require software developers, since this is outside your typical data scientist’s skill profile. The alternative in my view, is to develop new interfaces (see point 4) due to extra dependencies created on (and development work required from) external companies responsible for existing systems or software.</li><li><strong>Support:</strong> Code is like a building. Once written and productionised, it will need to be maintained. A dedicated support and maintenance team need to ensure the stability of productionised data products. In my experience, so many companies lack this function. A data scientist is responsible for developing the model (or algorithm), as such they have the best understanding of their portion of code and should ideally be involved in providing very specific ongoing support related to the data product. I often observe data scientists supporting multiple initiatives that consumes a significant amount of their time and hinder their focus on new proof of concepts.</li><li><strong>UI development:</strong> Some data science models may require APIs and a front-end in which case a software developer may be required. This typically happens with newer initiatives of which existing (internal) systems or software are either not available or limited. A user-friendly front-end is crucial to ensure user engagement and in obtaining quality feedback from users (labelled data as well). This is useful for gauging the impact of the solution and for making model improvements more rapidly.</li><li><strong>Data validation:</strong> Data is continuously accessed, explored and monitored by data scientists to establish its relevance, quality and maturity for initiatives. Ideally, the data needs to be relevant, complete, consistent, and accurate as possible. In my experience, the “volume of data” is often confused with the “statistical value of data “— you can’t get water from a stone. Since data can make or break a project, time may be required to explore the data before committing the entire product team to the initiatives.</li><li><strong>By-products:</strong> During the development cycle of a data product, internal staff members may identify that their productivity could be enhanced by gaining access to additional data sources, such as new data sets or information from other domains. This realisation can potentially lead to high-impact spin-off projects that build on the initial data science work. As teams become more familiar with the data science process, they may also generate novel ideas and concepts for other projects. In my experience, the visualisation of data is most often the catalyst for initiating data science by-products.</li><li><strong>Collaboration and automation:</strong> Building on the preceding point, it is noteworthy that internal staff members may initiate requests for reports or analyses from the data scientists, given their specialised skills, knowledge with open-source code and access to company wide datasets. In certain cases, such reporting or analyses may demand more sophisticated data science expertise, which the data scientists can proactively facilitate (e.g., by providing the code, queries, business logic, or statistics). My observation here is that there are usually plenty of opportunities to automate tedious and repetitive tasks (some more sophisticated than others), which data scientists should also contribute to in part. The value of such initiatives can be multiplicative, since you’ll get more done by more people, more accurately, with the same amount of effort.</li></ol><p>To shed light on the various resources typically required when developing a data product (from what I was exposed to at least), it is essential to consider the skill demand across different development stages. The ratios of these requirements (and time ranges) will inevitably vary based on factors such as skill levels, project complexity, data and system maturity, and so on. This context (with titles aside!) will serve as a useful backdrop as we proceed to introduce the tool next.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*zLWl_12J37fGaMp85mxxWg.png" /><figcaption>High-level overview of data product development stages and resource requirements. (Image by author)</figcaption></figure><p>The Osterwalder business model canvas is a widely-used visual framework in strategic management, designed to facilitate the creation or comprehension of business models. This template can also serve as a powerful tool for identifying potential data science initiatives. It simplifies a business model into nine essential building blocks, we can effectively pinpoint domains where data can be harnessed to bolster efficiency, curtail costs, improve customer satisfaction, and improve growth. Let’s break it down!</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*7Ta4ghRsQh0y_FpaC1NGUQ.png" /><figcaption>Osterwalder business model canvas. (Original source: <a href="https://www.strategyzer.com/">https://www.strategyzer.com/</a>)</figcaption></figure><ol><li><strong>Customer segments:</strong> The collection and analysis of data can provide valuable insights into customer behaviour and preferences, thereby informing product development and marketing strategies. By identifying shared characteristics among customer segments, such as interests, behaviours, purchasing power, demographics, and the like, data science initiatives can group customers in a multidimensional manner and uncover key segments for personalisation (targeted offerings). Another avenue is to develop propensity models for different products or services from the hidden regularities in the customers. These can have a positive impact on enhancing customer satisfaction, fostering loyalty, and boosting profitability. Other examples are sentiment and churn analysis to understand customers and act pre-emptively. The financial impact of initiatives in this block is a function of customer volumes, which make it easier to quantify the revenue generation, cost of acquisition, lifetime value, etc. (compared to some of the other blocks we’ll visit later). Note that data science initiatives in this block are mostly customer-facing, which may require more software skills, data engineering skills, and ongoing support and maintenance of live (“real-time”) systems that subsequently need to be factored into the expected return on investment. I reckon the main objectives of data science initiatives here is to find, grow and keep customers!</li><li><strong>Customer relationships:</strong> This block relates to the interactions and connections that a business establishes with its customers in order to create and maintain long-term relationships. Some examples are: personal assistance from a call centre, self-service online, tutorials, community forums, or automated services like chatbots, virtual assistants, or automated email responses. Data can be collected and analysed to establish how effective these interactions with customers are or how satisfied customers are with products or services. A lot of detail could be required when mapping out customer journeys. From here, business processes, products, or services can be optimised with the objective of growing and keeping customers.</li><li><strong>Channels:</strong> This block represents the channels that are used for reaching and engaging with each of the customer segments. With the use of data, analytics and machine learning the key channels can be identified for reaching and engaging with the target customer segments.</li><li><strong>Value proposition:</strong> A company’s unique value proposition can be better understood and tailored to continuously address specific needs of customers. This follows naturally from point 1 and involves analysing customer feedback, market research, and identifying areas where the company’s products or services can be differentiated from competitors. The aim of data science initiatives in this block is to create novel value propositions that resonates with the target customer segments driven by data.</li><li><strong>Key activities:</strong> This block refers to the primary activities that a company needs to perform in order to deliver its value proposition to customers. Data science initiatives in this block can help optimize key activities by identifying inefficiencies, automating manual processes, and improving overall operational efficiency. Measuring the impact of data science initiatives in this block is not always trivial. Metrics to keep in mind are: time and resources required before (i.e., completing it manually) and after automation, reduction in errors, impact of automation on customer experience, cost avoidance, improved job satisfaction, among others.</li><li><strong>Key resources: </strong>This block relates to the key resources that a company needs in order to deliver its value proposition and key activities. Data science initiatives can help companies optimise their supply and demand of resources by identifying areas where resources are being underutilised, or where additional resources may be needed.</li><li><strong>Key partners:</strong> This block refers to the key external partners that a company needs in order to deliver its value proposition and key activities. Data science initiatives can help identify potential partners, evaluate their suitability, and optimise partnerships over time. For example, data science can be used to identify potential suppliers or distributors, evaluate their performance over time, and optimise the terms of partnerships.</li><li><strong>Cost structure:</strong> This block refers to the costs that a company incurs in order to deliver its value proposition and key activities. Data science initiatives can help optimise costs by identifying areas where costs can be reduced, or where investments can be made to improve efficiency. For example, data science can be used to optimise marketing spend, reduce production costs, or optimise distribution channels. With a limited budget, where should it be spent to optimally meet multiple strategic objectives.</li><li><strong>Revenue streams:</strong> This block relates to the sources of revenue that a company generates from its value proposition and key activities. Data science initiatives can help optimise revenue streams by identifying areas where revenue can be increased, or where new revenue streams can be developed. For example, data science can be used to identify new customer segments to target, optimise pricing strategies, or develop new products or services to offer to customers. See point 1.</li></ol><p>It is important not to view data science in isolation, but a group effort in which complimentary skills are required— which need to be considered in the cost benefit analysis. The supplementary benefits (e.g., potential by-products), described as the characteristics of a data science project above, should also be considered among the strategic objectives.</p><p>My experience is that the Osterwalder business model canvas, with an understanding of the nature of a typical data science project, is useful in initiating holistic thinking around a data science strategy, motivating for initiatives in different areas of a business, and getting an idea of cost saving potential. Hope it also helps you!</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=45f19290b5f2" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Hiring Exceptional Data Scientists]]></title>
            <link>https://medium.com/data-science/hiring-exceptional-data-scientists-c454110742f?source=rss-815098f8de27------2</link>
            <guid isPermaLink="false">https://medium.com/p/c454110742f</guid>
            <category><![CDATA[exceptional]]></category>
            <category><![CDATA[hire-data-scientists]]></category>
            <category><![CDATA[interview]]></category>
            <dc:creator><![CDATA[Jaco du Toit]]></dc:creator>
            <pubDate>Tue, 28 Feb 2023 16:45:38 GMT</pubDate>
            <atom:updated>2023-04-11T13:52:09.079Z</atom:updated>
            <content:encoded><![CDATA[<h4>The outliers: going beyond just technical skill</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*hr-F_HVvvlOKkvyjJDbBGw.png" /><figcaption>Bloubergstrand (Image by author)</figcaption></figure><p>With the waning of the initial excitement surrounding applied data science, it is pertinent to reflect on the factors that contributed to the success of some initiatives during the peak of the hype cycle. It is worth considering what was instrumental in surpassing the mostly unrealistic expectations set during this period. However, the question at hand may be better framed as “who?” rather than “what?”. In this post, I share my observations and thoughts on what I think makes an <strong>exceptional</strong> data scientist — those that kept impressing and demonstrating business impact with their all-round skill and endless passion.</p><h3>Active self-learner</h3><p>An exceptional data scientist will demonstrate a deep sense of curiosity and a keen drive for ongoing learning. In my experience, most data scientists are faced with problems that require novel solutions that are not the run-of-the-mill supervised or unsupervised methods. Solutions for such problems require active research on topics not always directly relevant to the problem at hand, exploring online resources, keeping up-to-date with the latest from the open-source communities, reading books, engaging in conversations with peers, or attending data science meetups where they get to externalise ideas. This type of individual may also contribute to the field by developing tools or algorithms (although not necessarily enrolled in a postgraduate degree). They will also engage with academia to keep up-to-date with research and substantiate their own learning, which is an invaluable resource in my opinion.</p><p>Self-learning data scientists can critically assess new information and associate it to their ongoing projects or those of their peers. Furthermore, they exhibit an open-minded attitude and remain receptive to new ideas and perspectives, and possess a willingness to evaluate and challenge their own assumptions as they continue to learn and expand their skills. Such individuals exhibit a high degree of self-efficacy, which enables them to be highly motivated to learn and persevere. I’ve spoken to exceptional data scientists that also practice meta-cognition. They will reflect on their own cognitive processes and learning methods, which is highly beneficial in regulating and directing their own thoughts and knowledge towards a specific purpose. In my view, this contributes towards enhancing their learning efficacy and their ability to problem-solve more effectively.</p><p>How I would recognise such individuals is by looking at their public repositories where they implemented difficult theoretical concepts or algorithms derived from scientific papers. How did they approach the problem at hand? What literature and resources did they refer to? How did they ultimately develop a viable solution? Such insights can provide a useful understanding of their thinking process and approach towards problem-solving. I would also look out for those data scientists that use the learning-by-teaching method where teaching others help reinforce their own understanding and knowledge on the subject matter. This type of candidate is highly beneficial for a team, since they are open to share their learnings! As part of my interview process, I ask candidates about the methods or algorithms that they would like to learn (or better understand) and why. What is on their wish-list?</p><h3>Effective communicator</h3><p>Exceptional data scientists have a vocabulary and communication style that allows them to clearly articulate both business and technical information, taking into account the needs and perspectives of their audience<em>. </em>They may use active listening skills such as restating or summarising what a speaker has said and ask follow-up questions. They are able to put themselves in their audience’s shoes demonstrating empathy. This is often necessary when data scientists need to explain difficult concepts to non-technical business stakeholders. They also have belief in their own ability, but also able to convey confidently where they have uncertainties. They can keep an audience engaged during presentations and use storytelling to make their message memorable and impactful. They are great collaborators with product owners, business stakeholders and peers and are aware of group dynamics, which is crucial in recognising potential obstacles and opportunities for group success. They exhibit trust in others and are adept at building robust working relationships, maintaining a high level of interdependence within the group and resolving conflict in a positive and constructive manner. This also enhances group cohesion and facilitates successful outcomes.</p><h3>Creative problem-solver</h3><p>Exceptional data scientists possess a high level of fluid intelligence, which enables them to solve novel problems, think abstractly, and adapt to new situations. This facilitates their inventiveness and grasp of new concepts, often leading to the generation of new business ideas (an entrepreneurial mindset if you will) that can be transformed into new practical products or solutions. Some may display divergent thinking, capable of generating a wide range of ideas in response to a challenge or problem, which involves thinking beyond conventional approaches, making novel connections and considering multiple perspectives. They are imaginative, curious, and open to new experiences.</p><p>To identify such individuals, I would pay attention to their cognitive flexibility when asking them to come up with different approaches to a problem. The focus here is not necessarily for the candidate to come up with correct solutions, but rather to assess their capacity for original thinking by connecting seemingly unrelated concepts to generate meaningful associations relevant to solving the problem.</p><h3>Domain professional</h3><p>Apart from their technical skill, some may possess considerable knowledge and expertise in a particular field or industry that is pertinent to your recruitment objectives. Distinguished data scientists may have a good understanding of business (e.g., business strategy, objectives, etc.), which help them identify and work autonomously on core problems related to business. The same goes for other fields (e.g., engineering, manufacturing, finance, etc.). These data scientists have an advantage in using their domain knowledge in understanding the nuances of the data they are working with and to develop models or insights that are relevant and actionable. They would also know what model assumptions represent the underlying dynamics of the domain best, which can be gauged during interviews. Hiring domain professionals can accelerate data science initiatives in a particular domain.</p><h3>Technical proficiency</h3><p>Here are some key technical areas that exceptional data scientists take seriously.</p><p><strong>Cogent<em> </em>model assumptions: </strong>The ability to identify and explain underlying logic, principles and assumptions mathematically, using coherent arguments and evidence to support their position. The exceptional data scientist can explain their tools and identify the ideal scenarios to leverage them.</p><p><strong>Systems orientated reasoning: </strong>They will approach problems by considering the larger end-to-end system in which they are embedding a solution. They meticulously analyse the inter-relationships among various data and model components of a system, taking into account how these components are being consumed or interacted with. They also consider the impact of changes in one part of the system on the system as a whole and are interested in understanding the underlying structure and feedback loops that give rise to system behaviour. I’ve also observed exceptional data scientists working closely with end-users of the system to leverage user-experience feedback.</p><p><strong>Metrics driven:</strong> They place a strong emphasis on using data and quantitative measures (appropriate statistics) to drive decision-making and to evaluate performance. They are focused on tracking and analysing KPIs and using that information to make informed decisions to improve models or systems. For example, they will know exactly what metrics to use for specific models, how to interpret them, and what cognitive biases and statistical biases (e.g., Goodhart’s law, Simpson’s paradox, Berkson’s paradox, etc.) to be aware of.</p><p><strong>Data orientated:</strong> They are highly focused on the collection, analysis, and interpretation of data in order to inform decision-making and gain insights into various phenomena. The individual is comfortable working with data, is skilled in data processing, data analysis, and data visualisation. They are detail-oriented in their data exploration process and will search for explanations for anomalous discoveries — never sweeping hidden knowledge under the carpet.</p><p><strong>Open-source projects:</strong> They may have their own public open-source projects or repositories where they and other contributors frequently contribute. This individual actively contributes code towards a common goal and their contributions are valued and accepted by the community.</p><p>There are different ways in which interviews can be conducted. Personally, I do not prefer assigning time limit coding tasks to data science candidates. Doing so, undermines almost everything written in this post. Instead, I would review their public code repositories to assess their coding style, contributions, and problem-solving approaches. By examining the comments within their code and analysis, one can gauge their thought process and observations. Follow-up questions during interviews related to their public code can help clarify gaps. Alternatively, asking a data science candidate to talk through an interesting project they did (with the protection of confidential information), or providing them with a predetermined case study to solve conceptually (not with code) should provide ample opportunity for a hiring team to identify most of the characteristics mentioned in this post.</p><p>P.S. Please ensure that your job requirements are realistic and representative of the actual role; otherwise, exceptional candidates will lose interest in your company. Know what practices have been successful within your present team, what your company/team culture is like, what complimentary skills your team requires, <strong>and how you will retain and grow the exceptional data scientists after you’ve hired them!</strong></p><p>To end on a humours note — an exceptional data scientist is like a Kalman filter ;)</p><ul><li><strong>Adaptable:</strong> they should be able to adapt their models and analyses as new information or data becomes available.</li><li><strong>Robust:</strong> they should be robust to noisy and uncertain data (or people). They should be able to utilise uncertainty and filter out noise and other irrelevant information to focus on what matters.</li><li><strong>Prediction:</strong> they should be able to make accurate predictions based on past data (and current observations), identify trends and patterns in the data and use them to make informed predictions/recommendations.</li><li><strong>Balance accuracy and complexity:</strong> they should find the best model to fit a given set of data and extract the most useful information from it while keeping computation time reasonable by using simple (often linear) assumptions.</li></ul><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=c454110742f" width="1" height="1" alt=""><hr><p><a href="https://medium.com/data-science/hiring-exceptional-data-scientists-c454110742f">Hiring Exceptional Data Scientists</a> was originally published in <a href="https://medium.com/data-science">TDS Archive</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[12 Steps Towards Implementing a Successful Data Science Solution]]></title>
            <link>https://medium.com/@jacowp357/12-steps-towards-implementing-a-data-science-solution-d2ce196f22c1?source=rss-815098f8de27------2</link>
            <guid isPermaLink="false">https://medium.com/p/d2ce196f22c1</guid>
            <dc:creator><![CDATA[Jaco du Toit]]></dc:creator>
            <pubDate>Mon, 30 Jan 2023 10:05:49 GMT</pubDate>
            <atom:updated>2024-02-19T07:44:58.590Z</atom:updated>
            <content:encoded><![CDATA[<p>A quick guide keeping you focused on what matters</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*aHbu9wmfmdrQNoxgBZ9CHw.jpeg" /><figcaption>Cape Town (Image by author)</figcaption></figure><p>Starting a new data science project can be overwhelming. The business wants results yesterday, maintenance is scheduled on the server, your other model is unexpectedly drifting — you’re a deer caught in the headlights! Where do you begin? What should you be doing first? Or maybe you’re new to data science.</p><p>Here are 12 steps to help remind you.</p><h4><strong>Address the following 5 non-technical steps:</strong></h4><ol><li><strong>Write a well-defined problem statement:</strong> make sure this captures your real goal (not an indirect goal), clearly defined as you can manage. Talk to people that will help you achieve this.</li><li><strong>Describe your ideal outcome:</strong> leave out the technical data science terms, write a few sentences describing your 1st prize outcome. Think of the larger objective. Make sure this outcome solves the problem.</li><li><strong>Describe your success metrics:</strong> leave out the technical data science terms, what business performance indicators can be used to measure the impact of your outcome. Is this measurable and how will you measure it. When is your solution a success, and when is it a failure.</li><li><strong>Describe your output:</strong> leave out the technical data science terms, write a sentence that describes what your solution will output. Make sure this output solves the problem.</li><li><strong>Describe how the output will be used:</strong> where will your output be made available, what is the latency of the requirement, how will it be consumed, by whom will it be consumed?</li></ol><p>After you made a few notes, review them with members in your team, product owners and business stakeholders. Remain rational, realistic, data-driven, factual, and scientific. Make sure everyone is on the same page and balance the expectations. During this phase, you may need to revisit some steps and make compromises, maybe even opt for a 2nd or 3rd prize outcome. Note that some familiarity with the data is a prerequisite, if the data is new, an exploratory data analysis (EDA) will be required beforehand to help you make more informed decisions.</p><h4>Address the following 7 technical steps:</h4><ol><li><strong>Describe logic that could solve the problem:</strong> if you didn’t use fancy data science models how would you solve the problem? What would the rules/logic look like even for an approximate solution? Maybe this is good enough considering the success metrics?</li><li><strong>Describe your solution as a model:</strong> using fancy data science terms, frame your solution as a model that uses some input data to predict some output that solves your problem. What models are you considering and why? Consider online vs batch learning. Consider computational cost.</li><li><strong>Cast your problem as a simpler model:</strong> what would a solution using a simple model look like? Consider simplifying the target variable or data. To what extent does this solution differ from the original problem statement?</li><li><strong>Design your data for the model:</strong> write out the input data (target data included if applicable) you need for the model. Also, think about data that will help improve the model accuracy over time, and data that will help measure the model’s business impact.</li><li><strong>Determine data risks:</strong> in tabular form write out the sources of the data, its consistency, its reliability, when instances are available for training and evaluation, and for prediction. Determine if the data has biases, what are the implementation risks and complexities.</li><li><strong>Easily obtained data:</strong> identify a subset of data and subset of features that you can use to develop a simple first solution (or at least a first end-to-end run). Consider the risks of the data.</li><li><strong>Define your end-to-end system:</strong> draw a flow diagram of the end-to-end solution. What your model depends on, what it outputs, where it outputs to, and how it is consumed.</li></ol><h3>Next Steps</h3><p>Speak to a system/data architect, data engineers, members in your team, and product owners to get design feedback. Make changes with the objective to simplify things as much as possible. Share your plan, make sure everyone is on the same page and balance expectations. By this stage you may be considering a 4th or 5th prize solution as your first implementation should be based on a simplified model and solution using a few easily obtained inputs with minimum associated risk. Once this basic setup is working, you can iterate with the aim of improving business value (not necessarily model accuracy) by steadily introducing more complexity (e.g., more data, non-linearity, more parameters, more dependencies, etc.). Knowing when to stop iterating is also something to keep in mind. Make your thoughts known and review them with peers. Be flexible, but practical!</p><p><a href="https://github.com/jacowp357/templates/blob/main/data-science-model-development-template.ipynb">Here</a> is a notebook template that may help get you started.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/684/1*pn27PQ5bfX9MMyOuYVmHbQ.png" /><figcaption>Model development template. (Image by author)</figcaption></figure><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=d2ce196f22c1" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Entropy, KL divergence, Cross-entropy, and Mutual information]]></title>
            <link>https://medium.com/@jacowp357/entropy-kl-divergence-cross-entropy-and-mutual-information-519075a2e3fa?source=rss-815098f8de27------2</link>
            <guid isPermaLink="false">https://medium.com/p/519075a2e3fa</guid>
            <category><![CDATA[cross-entropy]]></category>
            <category><![CDATA[mutual-information]]></category>
            <category><![CDATA[entropy]]></category>
            <dc:creator><![CDATA[Jaco du Toit]]></dc:creator>
            <pubDate>Thu, 26 Jan 2023 11:15:21 GMT</pubDate>
            <atom:updated>2023-01-26T11:42:38.117Z</atom:updated>
            <content:encoded><![CDATA[<p>Information theory concepts used in Machine learning</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*YE0h8fhfNxfIk1ekXmU1vQ.jpeg" /><figcaption>Mossel bay (Image by author)</figcaption></figure><h3>What is Entropy?</h3><p>From the field of Information Theory, thanks to Claude Shannon, entropy captures information value.</p><p>How many bits<strong><em> k</em></strong> do we need to distinguish perfectly between <strong><em>M</em></strong> different possible outcomes if one bit is limited to a <strong><em>b</em></strong> number of different values (or elements in its alphabet). For example, if our alphabet size is <strong><em>b</em>=2</strong> (binary), and we want to distinguish between <strong><em>M</em>=4</strong> different outcomes, we need to communicate <strong><em>k</em>=2</strong> bits. The number of bits required can also be interpreted as the optimal coding length, which in this case is 2 bits long. This is given by the following well known logarithm function.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/256/1*s1wnRq7YM4UeJ0FSFTofFw.png" /><figcaption>Logarithm function. (Image by author)</figcaption></figure><p>If you receive a message that you expected, the message carries very little information content. If you receive a message that surprises you, the message carries a higher degree of information content. For a random variable, entropy provides a measure of the average amount of information needed to represent an event drawn from a probability distribution. In discrete form it looks like this:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/548/1*_veKDz4G_0L2yxyfSAStHQ.png" /><figcaption>Entropy equation. (Image by author)</figcaption></figure><p>We can think of the <strong>-log(<em>p</em>(<em>x</em>))</strong> term as measuring the optimal coding length (or uncertainty). It is large when the probability <strong><em>p</em>(<em>x</em>)</strong> of an event is small (i.e., we have more uncertainty when an event is less likely so we need a larger coding length). If an event has a high probability the term approaches zero (so we have less uncertainty and require a smaller coding length). By taking the expectation of this term with respect to the distribution <strong><em>p</em>(<em>x</em>)</strong> we get the average cost of sending messages using the optimal coding length, which is the entropy <strong>H(x)</strong>. Entropy is the average coding length required to communicate an event/message drawn from the probability distribution <strong><em>p</em>(<em>x</em>)</strong> (or the average amount of information for the random variable). The negative sign ensures that the result is non-negative. Note that <strong><em>p</em>(<em>x</em>)log(<em>p</em>(<em>x</em>))=0</strong> when <strong><em>p</em>(<em>x</em>)=0</strong>.</p><p>If all outcomes <strong><em>M</em></strong> are equally likely <strong>[0.25, 0.25, 0.25, 0.25]</strong> (uniformly random with probability <strong>1/<em>M</em></strong>), we need to use <strong>H(X)=2</strong> bits to communicate a message on average. If the outcome distribution is <strong>[0.25, 0.25, 0.5, 0.0]</strong> we know that some messages are more likely than others (so our expectation of the outcome is different). This reduces the number of bits required (or optimal coding length required) to <strong>H(X)=1.5</strong> bits on average. In other words, our uncertainty about the outcome message has decreased.</p><p>Now we know something about the quantification of information for a random variable using entropy. That is the average optimal coding length required to represent an event (or message) drawn from a probability distribution.</p><p>Note in this example we used logarithm base 2, which corresponds to the number of bits needed to represent a value drawn from the random variable.</p><h3>What is Kullback-Leibler divergence?</h3><p>The Kullback-Leibler (KL) divergence is used to measure the difference between probability distributions. KL divergence is scattered throughout the field of Machine learning and you won’t have to look far to see it. When it comes to data, KL divergence is useful to measure data drift, which is the change in a data-generating distribution over time.</p><p>Typically, <strong><em>p</em>(<em>x</em>)</strong> represents a true distribution of data or a precisely calculated theoretical distribution where <strong><em>q</em>(<em>x</em>)</strong> typically represents a theory, model, or approximation of <strong><em>p</em>(<em>x</em>)</strong>. In discrete form the KL divergence looks like this:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*IGbNP-UtZWyT8oVQ1gXhLQ.png" /><figcaption>Kullback-Leibler divergence equation. (Image by author)</figcaption></figure><p>The <strong><em>p</em>(<em>x</em>)log(<em>p</em>(<em>x</em>)) </strong>term is the entropy of <strong><em>p</em>(<em>x</em>)</strong>, and -<strong><em>p</em>(<em>x</em>)log(<em>q</em>(<em>x</em>)) </strong>is the cross-entropy. Cross-entropy is the average cost of sending messages from <strong><em>p</em>(<em>x</em>)</strong> using the optimal coding length of <strong><em>q</em>(<em>x</em>)</strong>. If <strong><em>p</em>(<em>x</em>)</strong> and <strong><em>q</em>(<em>x</em>)</strong> match, the cross-entropy should equal the entropy and the result will be zero. Note the small difference between KL divergence and cross-entropy that is the extra entropy term, which is a constant (hence a redundant term when minimising for the cross-entropy loss).</p><p>The KL divergence measure (sometimes called a distance, which it isn’t really) has a few unique properties to be mindful of:</p><ol><li>The KL divergence is always <strong>non-negative</strong>,</li><li>The KL divergence is an <strong>asymmetric</strong> measure, meaning that the KL divergence between two distributions <strong><em>p</em>(<em>x</em>)</strong> and <strong><em>q</em>(<em>x</em>)</strong> is different from the KL divergence between <strong><em>q</em>(<em>x</em>)</strong> and <strong><em>p</em>(<em>x</em>)</strong>,</li><li>The KL divergence is <strong>zero</strong> if the two distributions are <strong>equal.</strong></li></ol><p>When minimising the KL divergence,<strong> D(<em>p</em>(<em>x</em>)||<em>q</em>(<em>x</em>))</strong> is called the forward KL divergence, where <strong><em>p</em>(<em>x</em>)</strong> is the true fixed probability distribution and <strong><em>q</em>(<em>x</em>)</strong> is controllable with parameters.</p><p>Consider what happens if <strong><em>p</em>(<em>x</em>)=0</strong> for a particular <strong><em>x</em></strong>. As <strong><em>p</em>(<em>x</em>)</strong> is the weight, the <strong>log(<em>p</em>(<em>x</em>)/<em>q</em>(<em>x</em>))</strong> term receives no attention hence there is no need for<strong><em> q</em>(<em>x</em>)</strong> to change. If <strong><em>p</em>(<em>x</em>)&gt;0 </strong>the <strong>log(<em>p</em>(<em>x</em>)/<em>q</em>(<em>x</em>)) </strong>term starts contributing and minimising the difference between <strong><em>p</em>(<em>x</em>)</strong> and <strong><em>q</em>(<em>x</em>) </strong>becomes more important.</p><p>Forward KL divergence is known as <em>zero avoiding</em>, since <strong><em>q</em>(<em>x</em>)</strong> is not penalised for having high probability mass where <strong><em>p</em>(<em>x</em>)</strong> has low probability mass. Wherever <strong><em>p</em>(<em>x</em>) </strong>has high probability, <strong><em>q</em>(<em>x</em>)</strong> must also have high probability. This is known as<strong> </strong><em>moment-seeking</em> behaviour, because <strong><em>q</em>(<em>x</em>)</strong> must cover all high probability areas in <strong><em>p</em>(<em>x</em>).</strong></p><p>The backward KL divergence is <strong>D(<em>q</em>(<em>x</em>)||<em>p</em>(<em>x</em>)) </strong>where <strong><em>p</em>(<em>x</em>)</strong> is the true fixed probability distribution and <strong><em>q</em>(<em>x</em>)</strong> is controllable with parameters. Consider what happens if <strong><em>q</em>(<em>x</em>)=0</strong> for a particular <strong><em>x</em></strong>. As <strong><em>q</em>(<em>x</em>)</strong> is the weight, the <strong>log(<em>q</em>(<em>x</em>)/<em>p</em>(<em>x</em>))</strong> term receives no attention hence there is no penalty when we ignore <strong><em>p</em>(<em>x</em>)&gt;0</strong>. If <strong><em>q</em>(<em>x</em>)&gt;0</strong> the <strong>log(<em>q</em>(<em>x</em>)/<em>p</em>(<em>x</em>)) </strong>term starts contributing and minimising the difference between <strong><em>q</em>(<em>x</em>)</strong> and <strong><em>p</em>(<em>x</em>) </strong>becomes more important. This is known as <em>mode-seeking</em> behaviour, since it is better to fit just some portion of <strong><em>p</em>(<em>x</em>)</strong>.</p><p>Now we know something about measuring the difference between two probability distributions using KL divergence.</p><h3>What is Mutual information?</h3><p>Mutual information is a measure of the amount of information that one random variable contains about another random variable. It is a way to quantify the amount of dependence or association between two variables and is often used in training decision trees and for feature selection. Mutual information is more general than other statistical correlation measures, since it focuses directly on information itself using entropy. Another way to think about it is: how much does one random variable<br>tell me about another random variable?</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*f1Z_rK4dY04LT8zZSNjwcg.png" /><figcaption>Mutual information visual. (Image by author)</figcaption></figure><p>From probability theory we know that if a joint distribution of two random variables factorises into the product of the two marginal distributions, the two random variables are statistically independent <strong><em>p</em>(<em>x</em>,<em>y</em>) = <em>p</em>(<em>x</em>)<em>p</em>(<em>y</em>)</strong>. We also know how to measure the difference between two probability distributions using the KL divergence measure. Mutual information is the KL divergence between the joint distribution <strong><em>p</em>(<em>x</em>,<em>y</em>)</strong> and the product of the marginal distributions <strong><em>p</em>(<em>x</em>)<em>p</em>(<em>y</em>)</strong>. It can be expressed in terms of entropy as:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/526/1*pHBuyKrf1YjRMH6h43pl_w.png" /><figcaption>Mutual information equation. (Image by author)</figcaption></figure><p>Now we know something about measuring dependence between two random variables using mutual information.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=519075a2e3fa" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Reflecting Back on a Decade of Practicing Data Science]]></title>
            <link>https://medium.com/data-science/reflecting-back-on-a-decade-of-practicing-data-science-b06783d7bddf?source=rss-815098f8de27------2</link>
            <guid isPermaLink="false">https://medium.com/p/b06783d7bddf</guid>
            <category><![CDATA[careers]]></category>
            <category><![CDATA[work]]></category>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[personal-growth]]></category>
            <category><![CDATA[data-science]]></category>
            <dc:creator><![CDATA[Jaco du Toit]]></dc:creator>
            <pubDate>Wed, 11 Jan 2023 08:51:59 GMT</pubDate>
            <atom:updated>2023-02-02T21:17:53.203Z</atom:updated>
            <content:encoded><![CDATA[<h4>Journey, thoughts &amp; interests, and a possible future</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*x-VzEecy7DcuHtcNJIUPzA.jpeg" /><figcaption>Betty’s Bay (Image by author)</figcaption></figure><h3>How it started?</h3><p>I first discovered machine learning in 2012 during my master’s degree in Electrical and Electronic Engineering through conversations with professors and colleagues in our research lab. I was fascinated by what I learned and decided to take online courses that included the popular Andrew Ng and Daphne Koller material on coursera.org. Andrew’s machine learning course stood out as the most engaging, practical, and enjoyable. This opened up a new and exciting world of endless possibilities for X to Y mappings.</p><p>I was left with more questions after taking the online courses and wondered if this is how the mind actually functions, if there’s a universal learning algorithm that can achieve human-like intelligence, what learning and inference truly are, and how they can be expressed mathematically. I was also interested in the inspiration behind the creation of these algorithms and their potential to be exploited psychologically. To gain more insight, I read books and research papers by authors such as Jeff Hawkins, Geoffrey Hinton, Daniel Kahneman, Dan Ariely, V. S. Ramachandran, Claude Shannon, Pedro Domingos, Steven Pinker, and Ian Goodfellow.</p><p>By 2017/2018, I had a good understanding of the basic learning algorithms and more specifically Neural networks but, after extensive discussions with my supervisor and much consideration, I decided to shift my focus to Bayesian methods during the peak of the Deep learning trend. This shift in focus changed my course from weights, neurons and activation functions to <strong>uncertainty</strong>. And much of it! I read more books and research papers by authors such as David Barber, Kevin Murphy, Christopher Bishop, Tom Minka, John Winn, Judea Pearl, Norman Fenton, David MacKay, Daphne Koller, and Edwin Thompson Jaynes. I’m still finding this field intellectually stimulating as it requires a different way of thinking compared to other machine learning approaches—often leading to introspection.</p><h3>What was interesting over the past 10 years?</h3><p>As an industry professional, the past decade has been filled with opportunities to practice data science and implement machine learning algorithms across various industries: electricity utility, digital and social media, fintech, and mobile telecommunications. My aim has always been to achieve more with less and to share the learnings. Some of the most impactful algorithms I’ve applied (and variations thereof) include PCA, Logistic Regression, Naïve Bayes, Gaussian Mixture Models, Gaussian Process, Latent Dirichlet Allocation and simple Neural Networks (sequential and feed-forward). The versatility and broad applicability of the Gaussian distribution for unsupervised, semi-supervised, and supervised tasks continues to impress me.</p><p>I particularly enjoyed working on engineering-type problems more than customer-facing problems, partly due to the physics that underlie the data generation processes. Analysing and interpreting human behaviour from data or metadata was complex, outside of my expertise, and Apophenia a “tool” that I was not comfortable using.</p><p>During my time working at digital and social media companies, I found Deep learning to be highly applicable, largely due to the reasons outlined in Chapter 11 of Ian Goodfellow’s book on Deep learning. From my experience, this was a “safe and effective” form of data science, particularly for text and image problems. There are plenty of tools, example code, online courses, and research papers available, and apart from the tedious process of data preparation, it was often like choosing from a candy store before going to the movies. However, I had to be familiar with the underlying theory of the tools I was using, especially when pre-built solutions were not sufficient. Nando de Freitas once compared it to playing with Lego, which is an accurate analogy and the most valuable take-away from my experience with Deep learning. Breaking complex problems down into smaller, manageable parts, an age-old wisdom.</p><p>In all of the industries where I worked, I found Bayesian modelling to be appropriate for a wider range of problems compared to other methods. These problems often involved limited data, missing data, the need to incorporate domain knowledge, the need for custom models, the need to incorporate uncertainty, and the need for probabilistic and causal interpretation. However, it was not always easy to convince management to invest in such methods. Barriers to adoption included a lack of specialized skills, limited relevant online resources, and longer model development times. Developing Bayesian models meant burning the midnight oil. Despite these challenges, I managed to make Bayesian modelling work for some problems, but for most I had to keep a straight face while “panel beating” the data. Things changed once I discovered Infer.NET, a framework for developing bespoke Bayesian models (open-sourced in 2018). It took a lot of effort to get familiar with the framework, but it was well worth it!</p><p>The field of Bayesian modelling is intellectually satisfying as it frequently uncovers hidden shortcomings in our own reasoning and perceptions, which is surprising considering the basic principles (and philosophy) at its core. The results obtained can often be counter-intuitive, unexpected, and have a significant impact. It’s also intriguing to note that many widely-used machine learning algorithms can be understood as special cases derived from a more general Bayesian perspective. Why limit oneself when one can have it all (approximately)?</p><p>Over the past decade, I observed many advancements in technology driven by the contributions of AI researchers and open-source communities, as well as the support provided by the industry in terms of infrastructure. This has made model development and data management faster, cheaper, and more accessible, which in turn has made it possible to label more data and train models with high computational costs almost “instantly”. While many of us were focused on joining Pandas dataframes, organizations like OpenAI, DeepMind, and others created ground-breaking text-to-image, text-to-video, and language models that have become widely used. Who knows the limits of ChatGPT yet? Additionally, it’s worth mentioning the impressive accuracy achieved by AlphaFold and the historic (bittersweet) AlphaGo championship. Makes me wonder which field of study will ultimately provide the most accurate understanding of human intelligence and consciousness.</p><p>Like many others, I have encountered unusual job titles such as “Big Data Engineer.” I once overheard a recruiter asking “How big should the candidate be?” This comment, though figurative, highlights the lack of a clear definition and ongoing evolution of responsibilities and overlap of these emerging roles. This has made hiring and applying for these positions challenging. The Drew Conway Venn diagram, which was widely popular, currently appears to be a fitting depiction of the complexity of the data science role. However, I still believe that the Ouroboros would be a more apt representation, as it symbolizes the cyclical and self-referential nature of the field.</p><h3>What worked well?</h3><p>Data science has become an integral part of many organizations’ operations, but in my experience effective leaders overseeing such initiatives were scares. To develop a strong support structure and business model for data science projects requires a deep understanding of the various components involved, including balancing the needs of the business with the latest technology trends. From what I have observed, the most successful leaders in this field had a practical approach and were well-versed in both the industry and the science. Success in data science projects often depended on the ability to balance project intent, necessary skills, subject matter expertise, and systems and data. Unfortunately, those who were overly ambitious or overly focused on the latest technology trends often struggled, while those who lived by principles such as Occam’s Razor and Gall’s Law tended to be more successful, often with a bit of luck as well!</p><p>Seeing the financial impact that a model or solution developed by yourself or by others can make on an organization is a truly satisfying experience. One approach that has worked for me in achieving this is to start small and allow for enough time for development and refinement cycles while keeping the end-to-end product in mind throughout the process. This includes keeping well documented findings, presenting information in a clear and easy-to-understand manner, seeking feedback from peers, actively collaborating with subject matter experts across different business units, approaching the problem from different perspectives, experimenting with different techniques, asking many questions, and carefully listening to user feedback. The process of developing a prototype and bringing it to production is a challenge in itself, but it’s important to remember that even the best solution will be of little use if people don’t understand how to use it. Something not easily resolved without making compromises.</p><h3>What’s next?</h3><p>In the coming years, I believe the focus of AI will shift from creating artificial general intelligence, to the development of systems that are able to perform well on specific tasks and maintain a high degree of accuracy in handling edge cases, making them more practical and reliable for human consumption. I think we’ll see more realistic valuation of AI companies, more focus on enterprise level data as a product, more focus on A/B testing using the Bayesian methodology, the integration of blockchain technology to allow customers to monetise their data, and more advancements with multi-modal models.</p><p>Personally, I think more is needed now more than ever to make the field of Bayesian modelling (causal modelling in particular) practical and attractive to students, researchers, scientists, and industry practitioners. An “Andrew Ng style Bayesian inference course” if you will. I stress this point especially after what happened during the 2020–2022 period, which appears to be the biggest biological misstep in humanity. It seems that many Covid studies are plagued by statistical fallacies such as Simpson’s Paradox and Berkson’s Paradox, and other statistical manipulation that influenced the Covid narrative (see examples <a href="https://www.normanfenton.com/">here</a>). It’s quite frustrating to see this happening, especially when the proper use of statistics and probability are crucial to understanding and addressing a pandemic. I mean “How dare you?”.</p><p>While data science skills and data science leaders are in high demand and scarce, I hope to see more support provided by organisations for professional development in the field. I guess one way to tackle this would be to encourage more collaboration between industry and academia, as well as more initiatives to develop and retain skilled professionals.</p><p>Labelled data will continue to be important, so we will probably see organisations offering incentives for capturing and managing clean and reliable data. This should be interesting!</p><p>This is my personal experience practising data science over the past decade. These reflections, opinions and “forecasts” are my own. Please reach out and do let me know what you’ve experienced!</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=b06783d7bddf" width="1" height="1" alt=""><hr><p><a href="https://medium.com/data-science/reflecting-back-on-a-decade-of-practicing-data-science-b06783d7bddf">Reflecting Back on a Decade of Practicing Data Science</a> was originally published in <a href="https://medium.com/data-science">TDS Archive</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Bayesian Gaussian mixture models (without the math) using Infer.NET]]></title>
            <link>https://medium.com/data-science/bayesian-gaussian-mixture-models-without-the-math-using-infer-net-7767bb7494a0?source=rss-815098f8de27------2</link>
            <guid isPermaLink="false">https://medium.com/p/7767bb7494a0</guid>
            <category><![CDATA[probabilistic-programming]]></category>
            <category><![CDATA[gaussian-mixture-model]]></category>
            <category><![CDATA[bayesian-inference]]></category>
            <dc:creator><![CDATA[Jaco du Toit]]></dc:creator>
            <pubDate>Wed, 09 Sep 2020 19:12:14 GMT</pubDate>
            <atom:updated>2020-09-28T08:11:49.361Z</atom:updated>
            <content:encoded><![CDATA[<h4>A quick practical guide to coding Gaussian mixture models in Infer.NET</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/868/1*X2OZtkSZQ3JNaYH8jedo5g.png" /><figcaption>(<em>Image by author</em>)</figcaption></figure><p>This post provides a brief introduction to Bayesian Gaussian mixture models and share my experience of building these types of models in <a href="https://dotnet.github.io/infer/default.html">Microsoft’s Infer.NET</a> probabilistic graphical model framework. Being comfortable and familiar with k-means clustering and Python, I found it challenging to learn c#, Infer.NET and some of the underlying Bayesian principles used in probabilistic inference. That being said, there is a way for Python programmers to integrate .NET components and services using <a href="https://pythonnet.github.io/">pythonnet</a>, which I will cover in a follow-up post. My hope is that the content of this post will save you time, remove any intimidation that the theory may bring and demonstrate some of the advantages of what is known as the <a href="http://www.mbmlbook.com/">Model-based machine learning</a> (MBML) approach. Please follow the guidelines provided in the <a href="https://github.com/dotnet/infer">Infer.NET documentation</a><strong> </strong>to get set up with the Infer.NET framework.</p><p>Bayesian Gaussian mixture models constitutes a form of unsupervised learning and can be useful in fitting multi-modal data for tasks such as clustering, data compression, outlier detection, or generative classifiers. Each Gaussian component is usually a multivariate Gaussian with a mean vector and covariance matrix, but for the sake of demonstration we will consider a less complicated univariate case.</p><p>We begin by sampling data from a univariate Gaussian distribution and store the data in a .<em>csv</em> file using Python code:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/552079e6af61e16d989dd749973a4f0b/href">https://medium.com/media/552079e6af61e16d989dd749973a4f0b/href</a></iframe><p>This is what our data looks like:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/640/1*riqF863Y83ATmb6vrnCBFg.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/640/1*-NCTisuFY45v9STsSp_-Fg.png" /><figcaption>Left: a plot of 100 data points sampled from a Gaussian distribution with mean=5 and precision=10. Right: a histogram of the data. (<em>Images by author</em>)</figcaption></figure><p>Let us pretend for a moment we did not know the distribution that generated our data set. We visualise the data and make the assumption that the data was generated by a Gaussian distribution. In other words, we hope that a Gaussian distribution can sufficiently describe our data set. However, we do not know the location or the spread of this Gaussian distribution. A Gaussian distribution can be <a href="https://en.wikipedia.org/wiki/Normal_distribution">parameterised</a> by a mean and variance parameter. Sometimes it is easier mathematically to use a mean and precision, where precision is simply the inverse of variance. We will stick with precision for which the intuition is that the higher the precision the narrower (or more “certain”) the spread of the Gaussian distribution.</p><p>Firstly, we are interested in finding the mean parameter of this Gaussian distribution, and will pretend that we know the value of its precision (we set the precision=1). In other words, we think our data is Gaussian distributed and we are unsure what its mean parameter is, but we feel confident that it has a precision=1. Can we learn its mean parameter from the data? It turns out we need a second Gaussian distribution to depict the mean of our first Gaussian distribution. This is known as a <a href="https://en.wikipedia.org/wiki/Conjugate_prior">conjugate prior</a>. Here is a graphical representation of learning the unknown mean (using a Gaussian prior with parameters mean=0, precision=1):</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/928/1*U71KPX4xdtwprzpr7lJExA.png" /><figcaption>Bayes network for learning the mean of data with known precision. (<em>Image by author</em>)</figcaption></figure><p>Notice the difference between the mean random variable and the known precision in the graph. Here is the code in Infer.NET:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/87655b099c7a5b96d3e0ec6e127607cb/href">https://medium.com/media/87655b099c7a5b96d3e0ec6e127607cb/href</a></iframe><p>After observing only 100 data points we now have a posterior Gaussian distribution, which depicts the mean of our data <strong><em>x</em></strong>. We have learned something useful from the data! But wait… we can also learn something about its precision, without having to pretend it is fixed at 1. How? We do the same thing we did to the mean and place a distribution over the precision (effectively removing our “infinitely confident” knowledge that it was equal to 1 by replacing it with something resembling our “uncertainty”). The conjugate prior for precision is the <a href="https://en.wikipedia.org/wiki/Gamma_distribution">Gamma distribution</a>. We update our graphical representation of the model by including the Gamma distribution (with prior parameters shape=2, rate=1) over a new precision random variable:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*ixJ7PQHuftiPv349LowUjw.png" /><figcaption>Bayes network for learning the mean and precision of data. (<em>Image by author</em>)</figcaption></figure><p>Here is the code in Infer.NET:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/db2ccc29b90bcbe586e1b02fc379ec88/href">https://medium.com/media/db2ccc29b90bcbe586e1b02fc379ec88/href</a></iframe><p>A recap of our assumptions up to this point (referring to the figures below):</p><ol><li>the data <strong><em>x </em></strong>is Gaussian distributed,</li><li>we pretended to have full knowledge of its precision (precision=1) and learned its mean by using a Gaussian prior,</li><li>we then stopped pretending to “know” the precision and learned the precision by using a Gamma prior. Notice the difference this makes in the figure on the left below. The first model could not learn the precision due to the restriction we imposed (shown in green).</li><li>the parameters for an unknown mean and unknown precision are Gauss-Gamma distributed. After learning from the 100 data points, the prior distribution over these parameters (shown in red) updated to the posterior distribution (shown in blue in the figure on the right below).</li></ol><figure><img alt="" src="https://cdn-images-1.medium.com/max/640/1*XDOhuHqp-NWEVFtzWVueNA.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/640/1*BpFaOHbR9dwTVkMlqxEzvA.png" /><figcaption>Left: a Gaussian distribution with known precision (green) and unknown precision (blue). Right: a prior Gauss-Gamma distribution over the mean and precision parameters (red) and posterior distribution after learning from 100 data points (blue). (<em>Images by author</em>)</figcaption></figure><p>Infer.NET can produce a Factor graph of our model when setting <em>ShowFactorGraph = true</em>. Factor nodes are shown in black boxes and variable nodes are shown in white boxes. This graph shows our data <strong><em>x</em></strong> (the observed variable array at the bottom), which depends on a Gaussian factor. The Gaussian factor depends on a random variable called mean, and a random variable called precision. These random variables depend on a Gaussian prior and a Gamma prior respectively. The parameter values of both prior distributions are shown at the top of the graph.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/424/1*Ar1NKKj0IBxvvgL0LhzAZw.png" /><figcaption>Infer.NET produced factor graph of our model. (<em>Image by author generated</em> by Infer.NET)</figcaption></figure><p>We made certain assumptions in order to learn what the mean and precision of the Gaussian distribution are. In MBML, learning and inference is essentially the same. You can read more on the supported Infer.NET inference techniques and their differences <a href="https://dotnet.github.io/infer/userguide/Working%20with%20different%20inference%20algorithms.html"><strong>here</strong></a>. The examples in this article make use of <a href="https://en.wikipedia.org/wiki/Variational_message_passing">Variational message passing</a> (VMP). In order to learn more complex distributions (i.e., multi-modal densities) this model will not be expressive enough and should be extended by introducing more assumptions. Ready to mix things up?</p><p>As with many things in life if one Gaussian is good, more should be better, right! First, we need a new data set and use the same Python code introduced at the start of the post. The only difference is that we set p=[0.4, 0.2, 0.4]. This means that 80% of the data should be sampled from the first and third Gaussian distribution, while 20% of the data should be sampled from the second. The data can be visualised:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/640/1*q21P0Jnp1EnJIKcoMH6CzA.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/640/1*QFr_Tc60FVRo6yBkIVeo-g.png" /><figcaption>Left: a plot of 100 data points sampled from three different Gaussian distributions with means=[1, 3, 5] and precisions=[10, 10, 10]. Right: a histogram of the data. (<em>Images by author</em>)</figcaption></figure><p>To create the model we will use <strong><em>k</em></strong>=3 number of Gaussian distributions, also known as components, to fit our data set. In other words, we have three <strong><em>mean</em></strong> random variables and three <strong><em>precision</em></strong> random variables that we need to learn, but we also need a latent random variable <strong><em>z</em></strong>. This random variable has a discrete distribution and is responsible for selecting the components that best describe its associated observed <strong><em>x</em></strong> value. For example, more weight should be assigned to state one of <strong><em>z</em></strong>₀ if the observed <strong><em>x</em></strong>₀ is best explained by Gaussian component one. In this example, we will pretend to know the mixture weights responsible for all data points and use a uniform assignment (w₀=1/3, w₁=1/3, w₂=1/3) as shown in the graph below.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*cS2UADqggaG1BylmErD0RQ.png" /><figcaption>Bayes network for learning a mixture of Gaussian distributions with known mixture weights. (<em>Image by author</em>)</figcaption></figure><p>Here is the code in Infer.NET:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/a73e984ddb1e080d456375d76c477a32/href">https://medium.com/media/a73e984ddb1e080d456375d76c477a32/href</a></iframe><p>The three Gaussian distributions/components learned from the data are plotted below:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/640/1*QJjB_zMt0uz3-rq3G0ueeg.png" /><figcaption>The three learned Gaussian distributions and their sum with known weights set to 1/3. (<em>Image by author</em>)</figcaption></figure><p>Hold on…we claimed to have full knowledge of the component weights, but what if we do not? Can we also learn the weights from the data? Indeed, but we need a prior! A <a href="https://en.wikipedia.org/wiki/Dirichlet_distribution">Dirichlet distribution</a> is the conjugate prior for the discrete/categorical distribution. The graph below is updated showing a random variable for the unknown weights with its accompanying Dirichlet prior.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*WLrWYv8VWPiFwza4zAbs4A.png" /><figcaption>Bayes network for learning a mixture of Gaussian distributions with unknown mixture weights. (<em>Image by author</em>)</figcaption></figure><p>Here is the code in Infer.NET:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/d73c4453ae59527ff56588a25bdde2b9/href">https://medium.com/media/d73c4453ae59527ff56588a25bdde2b9/href</a></iframe><p>The three Gaussian distributions/components learned from the data and their learned weights are illustrated below:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/640/1*IkYuVv9-w3LIaQRp_RNi5w.png" /><figcaption>The three learned Gaussian distributions and their sum with unknown weights = {0.45, 0.16, 0.37}. (<em>Image by author</em>)</figcaption></figure><p>In summary, we started this journey by assuming our first data set can sufficiently be described by the Gaussian distribution. We were able to learn the mean parameters and the precision parameters of the Gaussian distribution using the observed data and VMP inference in Infer.NET. Our second data set used a more complex generating mechanism, which requires a more expressive model. We then introduced a latent variable <strong><em>z</em></strong> and Dirichlet prior, which allows us to learn mixtures of Gaussian distributions and their mixture weights. All steps are provided in c# code using Infer.NET and can be accessed <a href="https://github.com/jacowp357/bayesian-gaussian-mixtures"><strong>here</strong></a>.</p><p>For a more formal treatment the following books and links come recommended:</p><ol><li><a href="https://dotnet.github.io/infer/InferNet101.pdf">https://dotnet.github.io/infer/InferNet101.pdf</a></li><li>Bishop, Christopher M. <em>Pattern recognition and machine learning</em>. springer, 2006.</li><li>Murphy, Kevin P. <em>Machine learning: a probabilistic perspective</em>. MIT press, 2012.</li><li><a href="http://mbmlbook.com/index.html">http://mbmlbook.com/index.html</a></li><li><a href="http://www.jmlr.org/papers/volume6/winn05a/winn05a.pdf">http://www.jmlr.org/papers/volume6/winn05a/winn05a.pdf</a></li><li><a href="https://en.wikipedia.org/wiki/Exponential_family">https://en.wikipedia.org/wiki/Exponential_family</a></li></ol><p>Important concepts that were not mentioned in this post:</p><ol><li>appropriate prior parameters,</li><li>identifiability,</li><li>the predictive distribution,</li><li>breaking symmetry,</li><li>message-passing (variational message passing (VMP) &amp; expectation propagation (EP)),</li><li>Wishart conjugate prior of the inverse covariance-matrix,</li><li>Occam’s razor and the Dirichlet prior.</li></ol><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=7767bb7494a0" width="1" height="1" alt=""><hr><p><a href="https://medium.com/data-science/bayesian-gaussian-mixture-models-without-the-math-using-infer-net-7767bb7494a0">Bayesian Gaussian mixture models (without the math) using Infer.NET</a> was originally published in <a href="https://medium.com/data-science">TDS Archive</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
    </channel>
</rss>