<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Abhishek Mukherjee on Medium]]></title>
        <description><![CDATA[Stories by Abhishek Mukherjee on Medium]]></description>
        <link>https://medium.com/@abhi.mukh?source=rss-4f4cabff3c74------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/1*Z5b-zUjPX2KYrMV_UJjIgA.jpeg</url>
            <title>Stories by Abhishek Mukherjee on Medium</title>
            <link>https://medium.com/@abhi.mukh?source=rss-4f4cabff3c74------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Wed, 18 Nov 2026 10:21:01 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@abhi.mukh/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[Minimum viable domain knowledge in data science]]></title>
            <link>https://medium.com/data-science/minimum-viable-domain-knowledge-in-data-science-5be7bc99eca9?source=rss-4f4cabff3c74------2</link>
            <guid isPermaLink="false">https://medium.com/p/5be7bc99eca9</guid>
            <category><![CDATA[domain-knowledge]]></category>
            <category><![CDATA[information-modeling]]></category>
            <category><![CDATA[business-value]]></category>
            <category><![CDATA[data-science]]></category>
            <dc:creator><![CDATA[Abhishek Mukherjee]]></dc:creator>
            <pubDate>Mon, 15 Apr 2019 21:33:10 GMT</pubDate>
            <atom:updated>2019-04-15T21:33:10.784Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*chcbY6p00JNujkEBGhnjFg.jpeg" /><figcaption>Image by <a href="https://pixabay.com/users/geralt-9301/?utm_source=link-attribution&amp;utm_medium=referral&amp;utm_campaign=image&amp;utm_content=1632912">Gerd Altmann</a> from <a href="https://pixabay.com/?utm_source=link-attribution&amp;utm_medium=referral&amp;utm_campaign=image&amp;utm_content=1632912">Pixabay</a></figcaption></figure><p>When I was finishing my PhD in physics almost a decade ago, there was a steady stream of physicists moving to finance, biology and other fields where quantitative skills were in high demand. Such moves were, in large part, motivated by the belief, tinged with a hint of arrogance, that the modeling expertise that one acquires while doing a doctorate in physics is transferable to other domains. Some of these moves were successful, some were not. To the best of my knowledge, the physicists who ended up making successful moves were the ones who had the humility to recognize that productive modeling depends quite crucially on the semantics of a field, and had the tenacity to learn the relevant domain knowledge.</p><p>In the years after I finished my PhD, the stream turned into a wave. There is now a mass migration from quantitative fields into various industry and research sectors. Sometime after finishing my PhD, I also joined this mass of domain immigrants — the data scientists. From my own experience and from those of my peers, I am quite sure that what was true for my physicist peers from a decade ago, is also true for data scientists. It is simply not possible to do useful data science without sufficient domain knowledge.</p><p>This is not a new realization. The most ancient <a href="http://drewconway.com/zia/2013/3/26/the-data-science-venn-diagram">archetype of a data scientist is from Drew Conway</a> where he paints a data scientist as someone who is well versed in mathematics/statistics, programming (hacking skills) and domain knowledge (substantive expertise). However, while there are numerous books, articles, courses and self-help guides dedicated to the mathematical/statistical and programming aspects of data science, very little is written about the crucial domain aspect of data science. In this piece I will present my own opinionated view on this topic, informed by my experience and those of my peers.</p><p>Since “data scientist” was <a href="https://hbr.org/2012/10/data-scientist-the-sexiest-job-of-the-21st-century">declared to be the sexiest job of the 21st century</a>, data science has come to mean different things to different people. I think of data science as the act of building computational models for complex systems while leveraging moderate to large amounts of data. Domain knowledge provides the context for building these models. There are three main distinguishable, yet interrelated, aspects to domain knowledge: the problem context, the information context, and the data collection mechanism.</p><p><strong>The problem context</strong></p><p>One has very little chance of solving a problem if one does not understand what one needs to solve. But what does it really mean to understand the problem context in data science? Consider the example of a recommender system. No one wants to build a recommender system just for the sake of building one; perhaps a business wants to increase its revenue, and believes that building a recommender system will help towards that goal.</p><p>The first step is to specify and formalize the goal, e.g. increase revenue. The model that a data scientist would build would be used to calculate something. The data scientist needs to have the domain knowledge to clearly articulate the domain specific assumptions that can be used to relate the problem goal to the calculated quantity. In the recommender system example, the model might calculate the affinity that a user has towards a product. Using this calculation one can show a user the products towards which (s)he has the most affinity (as calculated by the model). The assumption here is that if a user is shown the products that they have the most affinity towards, then they have a higher likelihood of buying them, hence increasing the revenue for the business.</p><p>More often than not, it is not possible to directly measure whether a model is achieving its goal or not. Rather, one needs to make this judgement based on proxies — the evaluation metrics. A data scientist needs to be able to provide a reasoning (preferably quantitative and data driven) as to why the chosen evaluation metric is a suitable one. Typically, this boils down to a question of attribution. For example, one might use the click through rate as an evaluation metric for an online product recommender system. One then needs to have a plausible model for attributing a part of the revenue to the clicks generated by the recommendations. This would require the data scientist to understand how users browse webstores.</p><p>Given a model and a dataset one can calculate quantities only up to a certain level of accuracy. Typically, it is relatively easy to build models that have moderate to good accuracy, but large investments are required to increase the accuracy beyond that point. One the other hand, the level of accuracy that is actually required depends very heavily on the problem at hand. A data scientist should have a very clear picture of the value generated by incremental improvements in accuracy. In many cases the value vs accuracy graph would look like an S. There would be a threshold below which the model would generate almost zero or negative value, after which there would be a range where small increments in accuracy will result in proportional or exponential returns in value, and then there would be a threshold after which any further increase in accuracy will result in diminishing returns. Other profiles of the value vs accuracy graph are also possible. See <a href="https://towardsdatascience.com/how-to-build-an-ai-moat-386c81a79900">this excellent piece</a> that elaborates on this topic. It is only when a data scientist has a good understanding of the payoff, i.e. additional value generated, from increased accuracy can (s)he make any educated decision on what level of accuracy is good enough.</p><p>A data science problem is almost never an unconstrained optimization problem — there are always constraints. Some of those constraints will be technical in nature, such as an upper bound on the amount of computational resources available. Other constraints will be domain related, such as constraints related to fairness and privacy or constraints related to user experience. A data scientist needs to have the domain knowledge to understand the bounds set by these constraints, and ensure that the model stays within those bounds — otherwise there is little hope of the model ever seeing the light of day in production.</p><p>A data scientist would have understood the problem context if (s)he could</p><ul><li>formalize the problem goal and relate it to the calculated quantities and the evaluation metrics of the model</li><li>draw at the very least a semi-quantitative version of the value vs accuracy graph</li><li>show that the model is consistent with the problem’s constraints</li></ul><p><strong>Information context</strong></p><p>The problem context deals with the output of the model. The information context deals with its input. The hype surrounding big data and machine learning has given rise, in some quarters, to the <a href="https://www.wired.com/2008/06/pb-theory/">idea</a> that one can simply throw raw data at one end of an algorithm and generate meaningful insights at the other. <strong>That algorithm does not exist</strong> — deep learning or otherwise.</p><p>One domain where deep learning has been quite successful is image recognition (although one should be careful to not overstate the case given the susceptibility of deep neural networks to adversarial attacks). Nowadays we even have pre-trained models for image recognition that one can download and use with very little to no fine-tuning.</p><p>Let us consider for a moment the reasons behind this success. Images always consist of pixels. The pixels have the attributes of color and intensity, and they are always arranged on a regular grid. For the purposes of image recognition, we expect the answer to be unchanged under certain transformations such as translations, rotations and change of lighting. None of this pieces of information were obtained from math/stat or hacking. Rather they are part of the domain knowledge of images. The machine learning models that are successful in image recognition, such as those that use convolutional neural networks, are built on top of this domain knowledge.</p><p>In some sense, image recognition is an “easy” domain. Images are geometric objects, and hence it is easier to formalize the domain knowledge about images. Natural language processing has also seen some recent success in building universal models. Once again, although written text is not as structured as images, one can still identify words as the basic entities of a language, with a document being a ordered list of words — i.e. there is some natural structure to the data.</p><p>Unfortunately, the data in most domains do not come with such natural structure. Neither are they big or interesting enough for large multinational companies or research institutions to invest in building domain-specific algorithms. In these domains, it is up to the data scientists working in the trenches to determine the information context of the data.</p><p>One might think that the information context is nothing but the structure in the data. But it is much more than that. The training data for algorithms or inputs for models such as a set of vectors, or tensors, or an ordered list of tokens — they all have structure. But, by themselves they do not have any information context. The information context essentially consists of the recognizable concepts within a domain — entities, relationships and their attributes. For example, pixels (entities), their colors and intensities (attributes), and their relative positions in a grid (the relationships) form the information context within the image domain. A data scientist should be able to determine the relevant information context and assign it to the data.</p><p>One of the reasons for emphasizing the information context is interpretability. Remember, interpretability is always domain specific. If one does not understand the concepts of the domain, one has very little chance of being able to build interpretable models. But, it goes beyond model interpretability and into model improvability. Although, models work with data, modeling happens at the conceptual level. For a “data scientist” to live up to the second word in that title, she should be able to design sensible strategies for incrementally improving a model — beyond trial and error based (hyper)parameter tweaking. It is simply not possible to devise such strategies without a conceptual understanding of the domain.</p><p>A data scientist would have understood the information context if (s)he could</p><ul><li>formalize the relevant concepts of the domain in terms of entities, relationships and their attributes</li><li>map the training data set for the algorithm, and the inputs/outputs of the resulting model to the the aforementioned conceptual formalism.</li></ul><p><strong>Data collection mechanism</strong></p><p>Most, if not all, data-driven modeling methods make some assumptions about the representativeness of the available dataset with respect to the overall population that one is interested in. In practice, this assumption is rarely satisfied which limits the trust that one can place in the resulting model. The degree of non-representativeness of the dataset depends acutely on the data collection mechanism which is domain dependent. <a href="https://towardsdatascience.com/trust-and-interpretability-in-machine-learning-b7be41f01704">Elsewhere</a>, I have discussed at length how this representativeness assumption can be violated in different scenarios. I will not repeat that discussion here. Suffice to say that a data scientist needs to have a clear understanding of the data collection mechanism to understand the robustness of the model’s outputs.</p><p>Understanding the data collection mechanism is also important for another reason. In general, there are two ways of improving model performance — (i) more powerful algorithms, (ii) better quality and/or more quantities of data. A data scientist needs to have enough of an understanding of the data collection mechanism to determine if there are any levers available to get better quality or more data.</p><p>Consider the scenario where we are asked to classify images that are all generated in a room (including the labelled data). Imagine we have built a moderately accurate model. Any further improvement based on more powerful algorithms will require a big investment in algorithmic development. However, suppose we could control the lighting in the room. In that case we could possibly tweak the lighting to get better quality images, and hence better model without the huge investment in building more powerful algorithms.</p><p>A data scientist would have understood the data collection mechanism if (s)he could</p><ul><li>determine the limitations of the model resulting from the non-representativeness of the dataset</li><li>devise techniques to mitigate the non-representativeness problem, e.g. by having an exploration budget for showing random predictions</li><li>suggest tweaks to the data collection mechanism that might result in better model performance.</li></ul><p>—</p><p>I will end this piece by reiterating what I said in the beginning. Modeling skills that one learns in quantitative fields are highly transferable to other domains. But, you can only build the model right, if you know what is the right model to build. And all the math or all the hacking in the world will not be enough to fix that. Taking the time to really understand what you are trying to model, will. Context is king!</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=5be7bc99eca9" width="1" height="1" alt=""><hr><p><a href="https://medium.com/data-science/minimum-viable-domain-knowledge-in-data-science-5be7bc99eca9">Minimum viable domain knowledge in data science</a> was originally published in <a href="https://medium.com/data-science">TDS Archive</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Trust and interpretability in machine learning]]></title>
            <link>https://medium.com/data-science/trust-and-interpretability-in-machine-learning-b7be41f01704?source=rss-4f4cabff3c74------2</link>
            <guid isPermaLink="false">https://medium.com/p/b7be41f01704</guid>
            <category><![CDATA[modeling]]></category>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[algorithms]]></category>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[testing]]></category>
            <dc:creator><![CDATA[Abhishek Mukherjee]]></dc:creator>
            <pubDate>Sat, 30 Mar 2019 16:28:47 GMT</pubDate>
            <atom:updated>2019-03-30T16:28:47.932Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*1LI9TzwDU1l6IyJFBRcULw.jpeg" /></figure><p>Do machine learning models always need to be interpretable? Given a choice between an interpretable model that is inaccurate and a non-interpretable model that is accurate, wouldn’t you rather choose the non-interpretable but accurate model? In other words, is there any reason for sacrificing accuracy at the altar of interpretability?</p><p>Before going any further we should try to clarify what makes a model interpretable. Often, interpretability is equated with simplicity. This definition is obviously ambiguous; what is simple for one person may not be so for another. More importantly, advanced machine learning is most useful for modeling complex systems. These systems are called complex for a reason — they are not simple! Demanding that useful models for such systems should be simple, by any measure, makes very little sense.</p><p>Perhaps, we can go further if we consider how we think about interpretability in a traditional modeling setting. Consider the mathematical model of an internal combustion engine in a car. I doubt if anyone would consider that to be simple. But, on the other hand, I also doubt if anyone would consider the model of an internal combustion engine to be non-interpretable, either. This is primarily because we can derive this model in a deductive manner from well established physical theories such as thermodynamics and fluid dynamics.</p><p>Why not use this as our definition of interpretability? A model should be considered to be interpretable if it can be derived (or at least motivated) from a trustworthy theory. This definition of interpretability serves the dual purpose of understanding and trust. It helps us understand the model because we tend to understand things in a deductive manner — by going from the known to the unknown. Also, with such a definition, the trust in the model is derived from the trust that we place in the underlying theory.</p><p>Indeed, there are situations where both understanding and trust are necessary — scenarios where we are interested in determining the causal factors behind the behavior of a system. In such scenarios, we must insist that the corresponding models must be interpretable according to the above definition. Most models in the realm of physical sciences belong to this category. One can argue that purely inductive blackbox models are not suitable for such scenarios.</p><p>However, there are many other situations where understanding might be nice to have, but by no means is it a must have. In these situations what really matters is the ability to make trustworthy predictions. In these situations, if we could provide an alternate source of trust, then our models need not be bound by the definition of interpretability given above. This is a common argument, and there is merit to it. Remember, machine learning is a way of systematically building models from (preferably) large amounts of data using inductive reasoning. Constraining these models to be interpretable in a deductive manner can seriously limit their accuracy.</p><p>So then the question becomes how can we generate trust in a blackbox model where we have little to no insight into its inner workings. A credible basis for trust could be <a href="https://hackernoon.com/explainable-ai-wont-deliver-here-s-why-6738f54216be">testing</a>. After all, testing forms the basis of our trust in regular software. But to test a model we need be able to formalize our expectations about it. If we could formalize our expectations completely then that would correspond to a complete specification of the model itself. In that case, we would not really need machine learning or any other modeling methodology. What we really need to be able to do is to formalize our expectations about the aspects of the model that we consider important. This is not easy either, because many of the concepts that we care about, such as <a href="https://vimeo.com/248490141">fairness</a>, do not lend themselves to a convenient mathematical treatment.</p><p>It is worth pointing out that significant progress has been made in developing testing methodologies for testing machine learning models. I personally find the idea of using <a href="https://ieeexplore.ieee.org/document/8493260">metamorphic relations for formalizing expectations</a> to be particularly promising. But, we are still a long way from having concrete methodologies that will allow us to perform comprehensive testing of blackbox models, and this inability of ours contributes to a trust deficit in blackbox models.</p><p>One could question the efficacy of such expectation-based comprehensive testing. After all, the goal of machine learning is to find undiscovered patterns in data. By insisting that the models meet our expectations amounts to pre-defining the model, which defeats the whole purpose. Following this line of reasoning, one would argue that as long as the data is representative and our algorithms are powerful enough to capture the patterns, there is little reason not to trust the model — we should expect the model results to generalize to the overall population, and the extent to which we should expect them to generalize is encapsulated in the model’s performance (accuracy) scores.</p><p>Thus, in essence we are asked to delegate our trust to the trifecta of data, algorithms and performance scores. We first need to dissuade ourselves from the notion that a single performance (accuracy) score can form sufficient basis for trusting the model. A performance score is usually a point estimate of how a model is expected to generalize on an average over a population given the current data. Trust, on the other hand, is a nuanced multidimensional concept that cannot be encapsulated in such a single coarse grained score. One can imagine defining more granular performance scores— e.g. by population segments. But, that would require a certain level of understanding of the population and determining what we consider important — this is not very different from forming expectations.</p><p>Let us examine the data aspect of this argument. It is, indeed, quite easy to convince oneself that if the data is representative of the population we are interested in, then it should contain all the relevant patterns and no spurious ones. Unfortunately, that is rarely the case. The degree to which the data can be non-representative depends quite acutely on the situation. Nonetheless, we can identify certain high level scenarios.</p><p>In the first scenario, we would have a good understanding of the population and complete control over the data collection mechanism. In this scenario, we can choose our data to be representative, and with a high degree of confidence we can expect our resulting model’s predictions to be applicable to the overall population. However, note that having a good enough understanding of the population to be able to draw a representative sample for the task at hand means that we already have some understanding of which features are important for the prediction. Hence, in this case it is debatable if blackbox models are terribly useful. Opinion polling for predicting election results is a good example of this scenario.</p><p>In the second scenario, we do not have complete control over the data collection, but our predictions do not affect the data collected. In this scenario, if we assume that the data collection mechanism is unbiased then were we to wait long enough, we would have a representative sample of the population. Of course, there are a lot of ifs and buts that go with this assumption. Firstly, one does not know how long is long enough. Thus one needs to assume that the time scale over which the data is collected is long enough to produce a representative sample. Furthermore, the population itself might change in the meantime. Thus, an additional assumption is that the time scale over which population changes is much longer than the time scale over which a representative sample is generated. As long as we can justify those assumptions, then the estimated performance will be reliable. A model for predicting the stock prices is an example of such a scenario — as long as we are not making investments that are large enough to tip the market as a whole, the decisions that we make as the result of the predictions should not affect the stock prices.</p><p>The third scenario is one where the data collection is impacted by the predictions, but we have a moderate to high risk appetite for wrong predictions. An example of this is a product recommender system. The model for a recommender systems will be trained on data consisting of ordered lists of products that different users have bought/clicked on. Based on this data the model will predict what a user is most likely to buy/click on and based on the model’s predictions the system will decide what the user gets to see, which limits what (s)he can buy/click-on. Thus the prediction biases the data collection. In product recommender systems, one can circumvent this problem, somewhat, by keeping an exploration budget — for a fraction of the cases the system shows the user a random set of products regardless of the prediction of the model. The observations resulting from these randomized predictions can then be used to estimate the performance of the model. One still has to address the concerns of the aforementioned second scenario in order to access the reliability of these estimates.</p><p>In the fourth and final scenario, the data collection is impacted by the predictions, but we have little to no risk appetite for wrong predictions. For example, suppose we have to build a model to predict whether someone will default on their mortgage loan payments. The mortgage loan will be approved or not based on the prediction. If the prediction is that the person will default, then the loan will not be approved, and in that case there is no way of knowing whether this person would have actually defaulted or not. It is difficult to imagine a situation where an institution would randomly approve (or otherwise) a loan for the sake of data exploration. In these situations, it is very difficult to gauge the reliability of the estimated performance of the resulting model without additional information.</p><p>Thus, it is not such a great idea to blindly expect the data to be representative of the population. In most scenarios, given the constraints of the problem at hand, it simply might not be possible to get an unbiased representative sample. Understanding the limitations of one’s data collection mechanism, being able to deduce the implications of those limitations, and having the honesty to report those as a part of the model’s results goes a long way in building trust.</p><p>Let us now consider the algorithm aspect of the argument. It is a widespread belief that the more flexible an algorithm is the better it is, because flexibility equips an algorithm to capture more complex patterns. But if the history of the actual successful applications of machine learning are anything to go by, then this belief would appear to be utterly misplaced. In computer vision, success came when we were able to encode the symmetries in pictures into models in the form of convolutional neural networks. In natural language processing we are now able to build extremely accurate cross-purpose language models because we could encode our knowledge about languages, including structure and word context, into these models. In recommender systems — most collaborative filtering algorithms including matrix factorization methods, make strong assumptions about the affinity of a user towards an item.</p><p>Whether we would like to slap the label of interpretability on these models or not, it is an objective fact that we build better models when we understand the domain and the context in which the model needs to operate. The best models do not come from the most flexible algorithms, they come from algorithms that are well constrained by domain knowledge and have just the right amount of flexibility to capture the relevant patterns in the data.</p><p>We have seen the word understanding being used quite a few times in the above discussion. What we should have realized by now is that it is difficult to build trust without understanding. In the end, it boils down to how one perceives machine learning. Yes, machine learning is an incredibly powerful inductive modeling technique. When combined with big data and big compute, it allows us to model systems and solve problems that were previously out of our reach. But the entry of machine learning should not imply the exit of everything else, including common sense. Machine learning is one element in the wider modeling family that includes deductive modeling as well as domain knowledge. The better we understand and leverage the interconnections between these elements, the further we will go towards robust complex system modeling.</p><p>Trust is contextual and trust can have multiple sources, but eventually it flows from knowledge and integrity; specifically in our trust in the knowledge and integrity of the individuals who are building the models. Trust as well as adoption of models will come, in my opinion, only when the wider audience is convinced that the modelers have the knowledge to understand the limitations of their models (machine learning or otherwise), and the integrity to report them.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=b7be41f01704" width="1" height="1" alt=""><hr><p><a href="https://medium.com/data-science/trust-and-interpretability-in-machine-learning-b7be41f01704">Trust and interpretability in machine learning</a> was originally published in <a href="https://medium.com/data-science">TDS Archive</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Scalable methods for explaining machine learning]]></title>
            <link>https://medium.com/data-science/scalable-methods-for-explaining-machine-learning-f28f2b8f9f13?source=rss-4f4cabff3c74------2</link>
            <guid isPermaLink="false">https://medium.com/p/f28f2b8f9f13</guid>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[algorithms]]></category>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[complexity]]></category>
            <dc:creator><![CDATA[Abhishek Mukherjee]]></dc:creator>
            <pubDate>Sun, 09 Sep 2018 17:28:58 GMT</pubDate>
            <atom:updated>2018-09-11T06:08:53.875Z</atom:updated>
            <cc:license>http://creativecommons.org/licenses/by/4.0/</cc:license>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*EMw0UhuPJk0xUQBU999EDQ.jpeg" /></figure><p>Machine learning is often called the part of AI that works. Yet, there is a growing <a href="http://www.argmin.net/2017/12/05/kitchen-sinks/">unease</a> that we do not really understand why it works so well. There is also the fear of <a href="https://www.amazon.com/Weapons-Math-Destruction-Increases-Inequality/dp/0553418815">algorithms gone wild</a>. These criticisms are not entirely fair. The stalwarts in the field who designed some of the most successful algorithms <a href="https://www.facebook.com/yann.lecun/posts/10154938130592143">do understand</a> why things work. However, this is a form of understanding that is as amorphous as it is deep. Machine learning algorithms are complex, and <a href="https://techcrunch.com/2018/06/14/the-problem-with-explainable-ai/">explaining complex objects is never easy</a>. But as machine learning matures as a field and becomes ubiquitous in software, it is perhaps time to talk about more tangible forms of understanding and explanation.</p><p>In this post I will discuss why our conventional methods of understanding are not very useful for machine learning. And then I will discuss an alternate method of understanding that might be more suitable for the specific context of machine learning. Nothing that I discuss in this post is new. It should all be very familiar to the seasoned practitioners. This is merely my attempt to make explicit some of the implicit wisdom in the field.</p><h3>Decision making in complex systems</h3><p>Machine learning is most useful for decision making problems in complex systems. The goal would be to predict the most suitable action given a certain context in order to elicit the most desirable reaction. In real life, the problem is almost never stated in terms of observable quantities. Thus the first step is to formulate the problem in terms of observable quantities. This is far from trivial, but this is not a machine learning step. You would need to take this step no matter which problem solving approach you choose.</p><p>Once you have formulated the problem in terms of observables, the next step is to develop a mathematical formulation for the solution. The solution is invariably formulated as a mathematical function that describes the complex system. This function is called the model of the system.</p><p>The inputs and the outputs of the model will depend on your formulation. For example, the input can be a context action pair with the output being the expected reaction, or the input can be the context with the output being the optimal action and expected reaction or you might choose a probabilistic formulation in which case the input can be context action reaction triplets with the output being the probability of occurrence of such a triplet. There are no universal guidelines for choosing the formulation. Which formulation will work best is heavily dependent on the specifics of the problem.</p><p>Once you have chosen your mathematical formulation, then you are left with the core task of determining the model. The model must be consistent with our perception with respect to the complex system. Our perception can be divided into two elements: assumptions, and data driven insights.</p><p>Assumptions are statements that you make about the system independent of any available data. The assumptions can be further divided into two elements: known truths and guesses. The known truths are those facts about the system that everyone agrees on and is not up for debate. The guesses are those assumptions that are specific to you and that can be challenged. The guesses might come from your prior experience with similar systems, in which case they will be educated guesses and will be defensible. They might also simply come from your gut feeling, in which case they will most likely be bad guesses and will probably be less defensible.</p><p>When it comes to data-driven insights, broadly speaking there are two ways of extracting insights from data: manual and automated. The manual way primarily consists of analysing trends in data to determine functional relationships between variables. The quality of the manual insights are heavily dependent on the competence and the biases of the person deriving those insights. But they do have the benefit of being more easily explainable because the person deriving the insights should be able to state the logical steps that went into deriving the said insights. The automated way mainly consists of using mathematical optimization to determine functional relationships between variables. Given the choice of the optimization objective, the quality of automated insights are independent of any person. But this also makes them more difficult to explain because one does not know the exact logical steps that were taken by the optimization method to obtain those insights; if one did know the exact steps then there would be no need for the automated method, one could simply derive them manually. There is no sharp dividing line between the manual and automated modes. Rather, there is a continuum between fully manual on one end and fully automated on the other.</p><p>In general, the stronger and more numerous your assumptions are, the less of a role the data-driven insights will play in determining the model. If you want to reduce the influence of a specific person or persons on your model then you would want to reduce the role of guesses and manual insight extraction. Let us first look at the two extreme ends of this program: the data-free and the theory-free zones.</p><p>In the data-free zone you would remove the dependence of data completely, and make decisions based on known truths. To be able to do so, the known truths alone should be able to provide a complete description of the complex system. In other words, you would need a complete practical theory of the complex systems. In interesting real life problems this is never the case — either we do not have a complete theory of the system or even if we do, we do not have the means to calculate the consequences of the said theory.</p><p>In the theory-free zone you would get rid of assumptions altogether, and construct the model based on data alone. Mathematically speaking, this would mean that you need data on all possible inputs and outputs of the model. Apart from the fact that such data is never available, this beats the whole purpose of building the model in the first place. You would want to build a model so that it can generalize from a sample to the whole population. If you had data on the whole population then there is not need to build a model.</p><p>Thus, building models solely based on assumptions or solely based on data is neither feasible nor useful. The valhalla of machine learning is to eliminate the role of guesses and manual insights completely from the decision making process and provide a solution based solely on known truths and fully automated data driven insights. Unfortunately, we are not there yet. At this time, every machine learning model will include some guesses and manual data driven insights.</p><p>As we discussed, known truths require no explanation. Guesses and manual insights are relatively easy to explain. The crux of explaining machine learning lies in explaining the automated data driven insights. What would such an explanation entail? An explanation is intimately tied to the concept of expectation. Typically, we say that we have explanation for something if we can somehow show that its behavior is consistent with our expectations. Thus, explaining is contingent on being able to form expectations.</p><p>To explain a machine learning model we need to be able to form expectations about it, independently of the process that was used to construct it. Also, this method of forming expectations should be simpler that the process that was used to construct the model, otherwise we would need another method to explain the expectation forming method — which would beat the whole purpose of explanations. This immediately rules out the possibility of exact and comprehensive expectations. If we did have a simpler method that we could use to predict the behavior of the model exactly in all possible scenarios, then we would simply use that method for constructing the model instead of the original method. Thus the best we can aim for is approximate and/or partial explanations.</p><h3>How not to explain</h3><p>Before we discuss an explanation method that might work, let us first discuss some explanation methods that will almost certainly not work. We can think of these methods as a-posteriori methods. In other words, these methods somehow try to break down the final optimized model into parts and explain the parts. For example, we might want to think of the output of the model as somehow being independently impacted by the inputs with some weights. This is very dangerous and flat out wrong for strongly correlated systems. <a href="https://hackernoon.com/why-feature-weights-in-a-machine-learning-model-are-meaningless-b0cd22a4c159">Here</a> is a nice explanation of why judging variable importance based on feature weights in a machine learning model is not a good idea. Thus, this method simply does not work if the input variables are correlated.</p><p>Another option would be to tile up the domain of the model with patches such that on each patch you can have an accurate local approximation the model in terms of a simple function. We can then think of the overall model as a patchwork of these simple local approximations, and ask if the behavior of these local approximations is consistent with our expectations. This method is quite similar to how traditional software is generally understood and tested using <a href="https://en.wikipedia.org/wiki/Equivalence_partitioning">equivalence class partitioning</a>.</p><p>The question is what do we consider to be a simple function? For most people it is difficult to visualize beyond a three-dimensional space which means that a simple function can at most have two independent variables. Also, we would find it difficult to reason about functions with more than one or two extrema. When we impose these constraints into the definition of a simple function, we will see that the number of patches needed grows exponentially with the number of input variables and the number of extrema of the model.</p><p>If we operationally define complexity of a function as some combination of the number of inputs, the number of extrema and the strength of the correlations between the variables, then the above explanation methods do not work because they do not scale well in terms of complexity. These methods do not scale because they try to build a-posteriori reductive approximations. A scalable method would need to look not at the final model, but at the optimization method itself. In order to scale, it would need to build global approximations for how decisions are made, and not approximations for each individual decision.</p><h3>Start with the symmetries</h3><p>First we need to elaborate our terminology. In machine learning, one starts with the known truths, guesses and manual data-driven insights. These are mathematically expressed as a partially defined multivariate function called an algorithm. In addition to the inputs and outputs, an algorithm will also have a set of undetermined parameters. The parameters are fixed by training the algorithm with data. Training an algorithm is essentially extracting insights from data in an automated manner, and it typically involves solving an optimization problem. The result of training an algorithm with data is a fully defined function called the trained model. The models that we have been discussing so far are these trained models. The objective function for optimization is setup in such a manner that minimizing it would imply that the trained model captures the stable patterns in the data that are relevant for this specific problem.</p><p>The first step towards explanations is clarity. Before we can explain anything we need to be clear about the starting points. In machine learning the starting points are the known truths, guesses and the manual data-driven insights. For any given problem, we can only provide an explanation given these starting points. Therefore, they need to be stated very clearly and comprehensively.</p><p>A problem is almost always stated in the language of the domain. You would then translate it to a mathematical problem using a representation. The complexity of the problem shows up when you make this translation. Any explanation would also have higher power if it can be provided in terms of the expectations of the problem domain rather than in terms of the specific representation chosen for the solution. Thus, the starting points have a better chance of scaling with complexity and they would have higher explanatory power if they are stated in terms of the language of the domain, rather than the language of the representation.</p><p>But what do we mean exactly when we say that the starting points should be stated in the language of the domain? In the case of machine learning the starting points are often statements about what is relevant for the problem. Let us consider the example of what is perhaps one of the most successful applications of machine learning: image recognition. Deep learning, which is a type of machine learning, has proven to be incredibly successful in classifying images. Not only that, we actually have a fairly good understanding of why deep learning works so well for image recognition. One of the main reasons for this success and understanding is that we devised ways to encode the relevant symmetries of the domain of images into the relevant algorithms (deep neural networks). In this context, a symmetry is essentially a transformation that leaves the outcome unchanged. For images, these symmetries are changes in resolution, rotation, translation etc. A convolutional neural network is essentially an encoding of these symmetries in the language of neural networks.</p><p>Symmetries are the most economic way of stating our knowledge about relevance in a holistic manner. Symmetries tell you what is certainly not relevant. Everything else, by exclusion, might be relevant. They enable us to identify what is relevant vs what is irrelevant for a given domain in a data independent manner. They place constraints on the functional forms of the algorithms, rather than making statements about local behavior of the model. The class of algorithms chosen for a task must be powerful enough to accommodate the symmetries of the domain. If the training data is of sufficiently high quality and quantity, then the main reason for a model to misbehave would be that the wrong symmetries were unwittingly encoded into the algorithm.</p><p>With image recognition, we have a clear example of the fact that when symmetries are used as their basis the resulting algorithms are both successful and understandable. Without a doubt, formalizing the relevant symmetries of a domain and encoding them into machine learning algorithms is very non-trivial. Image recognition is an “easy” example because the symmetries of images are intuitively familiar to us. I would argue, however, that symmetries are vital to understanding machine learning. Whenever possible the starting points for an algorithm should be stated as symmetries of the domain.</p><h3>Method of successive global approximations</h3><p>Symmetries will put some constraints on the parameter space of an algorithm. The remaining underdetermination of the parameters will be removed by training. We now need a path to connect the symmetries to the model. This is equivalent to finding a path connecting the partially determined parameters in the algorithm after the imposition of the symmetries and the fully defined parameters in the model. The method that we will use is that of successive global approximations.</p><p>Let me illustrate with an example. Suppose you ask a mapping service (such as Google maps) to find the best route between two locations in a city in terms of travel time. The algorithm from the mapping service will, more often than not, produce a route consisting of many turns in order to save time. If you did not have the mapping service handy and if you are like most people, then you would have probably taken the route of least confusion, i.e. the route with the least number of turns. In essence, you were solving a constrained version of the problem given to the mapping service algorithm. You would first check if there is a route with no turns at all. If so then you would pick that route. If such a route does not exist, then you would look for route with one turn. If such a route exists and and is unique, then you would pick that route. If there are multiple such route then you would choose the one that would take the least amount of time. If no such route exists then you would try a route with two turns, and so on.</p><p>One way to understand the route produced by the mapping service will be to continue the above process until you reach the route produced by the mapping service. Suppose your solution has two turns, while the solution produced by the mapping service has five. You can now try to successively look for solutions with three, four, five and six turns. In each case, you will be able to see how much time is saved by adding a turn. When going from five to six turns you should be able to see how adding a turn does not save any more time and perhaps increases the travel time.</p><p>Notice that in the above method we were not trying to understand each turn produced by the mapping service in isolation. Rather we focussed on understanding the whole solution as the terminal of a series of approximations. Generally, the human mind is not very well equipped at grasping the behavior of complex functions as a whole. But it is quite good at understanding “effective theories” based on the differences between two complex functions. This is the reason why the approach above works as a useful tool for understanding.</p><p>Similar approaches are used in many different disciplines engaged with the study of complex systems. In fact, even in machine learning we use similar ideas when benchmarking a model against baseline models. For example, when trying to determine how good a classifier is, one would compare its performance to that of a random classifier. The benchmark models are nothing but solutions to the optimization problem with additional constraints imposed to make the solution “simpler”. We can repurpose this benchmarking methodology into an explanation methodology by explicitly constructing successive benchmarks as solutions to constrained optimization problems, where at each successive step we remove some of the constraints. In other words, we are explaining the optimized solution as the end result of a series of successive approximations.</p><p>The number of successive approximations should be scalable in the face of complexity, which in this context means that this number should be independent or at worst weakly dependent on the number of parameters in the algorithm. One way to ensure this would be to frame the constraints leading to the approximation in a representation that is adjoint to that of the parameters — a generalized Fourier space. The successive approximations can then be viewed as progressively including higher order fluctuations in the solution.</p><p>Similar to the symmetries, constructing successive global approximations is not easy. One needs to have very deep expertise to be able to think globally rather than locally about a complex system. But complex systems are called complex for a reason, and it will be naive to assume that we will be able to understand them and convey such understanding in the form of explanations without the requisite expertise.</p><p>Machine learning is a game changer. It has already proven itself to be incredibly successful and it is, without a doubt, the most powerful method that we currently know of for building software for complex systems. But, after success must come wisdom. Our traditional methods of understanding software are inadequate for understanding machine learning software because they do not scale well with complexity. On the other hand, symmetries and successive global approximations do scale well with complexity. I have tried to discuss when used together, how they can provide a scalable method for explaining machine learning.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=f28f2b8f9f13" width="1" height="1" alt=""><hr><p><a href="https://medium.com/data-science/scalable-methods-for-explaining-machine-learning-f28f2b8f9f13">Scalable methods for explaining machine learning</a> was originally published in <a href="https://medium.com/data-science">TDS Archive</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Lean startup and machine learning]]></title>
            <link>https://medium.com/data-science/lean-startup-and-machine-learning-39a96efecb2a?source=rss-4f4cabff3c74------2</link>
            <guid isPermaLink="false">https://medium.com/p/39a96efecb2a</guid>
            <category><![CDATA[lean-startup]]></category>
            <category><![CDATA[a-b-testing]]></category>
            <category><![CDATA[scientific-method]]></category>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[machine-learning]]></category>
            <dc:creator><![CDATA[Abhishek Mukherjee]]></dc:creator>
            <pubDate>Sun, 19 Aug 2018 22:03:25 GMT</pubDate>
            <atom:updated>2018-08-20T17:58:38.265Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*YcmluY665tbTo0Nm1SgXOA.png" /></figure><p>The term Lean startup was <a href="http://www.startuplessonslearned.com/2008/09/lean-startup.html">coined</a> about ten years ago. Since that time it has grown to become one of the most influential methodologies for building startups, especially those that fall in the category of web-based software companies. Lean came of age during the internet revolution. We now sit on the cusp of a different revolution — one ushered in by machine learning algorithms. It is safe to assume that most or all software in the near future will contain some element of machine learning. But how compatible is Lean with machine learning, in principle and in practice? (See <a href="https://x.ai/blog/why-anti-lean-startups-are-back/">here</a> for an interesting perspective)</p><h3>Validated learning and machine learning</h3><p>According to the Lean startup philosophy, most startups fail not because of the quality of the product but because of incorrect assumptions about customers, i.e. startups end up building the perfect product for a non-existent customer. In order to avoid this fate, startups should first validate and refine their assumptions by running experiments to learn about their customers. This process is called validated learning. In Lean, validated learning is accomplished by performing repeated hypothesis testing.</p><p>The hypotheses are always assertions about the response of the customers given the functional definition of the product. In order to conduct an experiment, one would first need to define the product functionally (basically a list of features) — this defines a viable product. In addition, one also needs to define the quantitative measures of the customer’s response to the product. The functional product definition still leaves many (primarily technical) choices open. These choices should be made in a way that minimizes the expenditure of the most valuable commodity in a startup (usually time); such choices define the minimum viable product (MVP). One would then proceed to build the MVP, bring it to the customers, record their response, and determine the validity of the hypothesis. The experiments are usually set up as split tests with the current baseline product serving as the control and the proposed change serving as the treatment.</p><p>Notice, that there is no scope for product side uncertainty in Lean — given a functional definition, one knows with certainty whether a product that meets that definition can be built or not. This is true for traditional software. The behavior of a traditional software product is completely determined by the instructions you provide (i.e. the code). Therefore, given a full specification of its behavior you can work “backwards” and figure out what instructions you need to provide to build a piece of software with the desired behavior. An MVP in some sense is basically the minimal set of instructions you need for this purpose. A behavior of a piece of software can exhibit different variations based on the context. However, the number of such possible variations is usually rather small, and it makes sense to provide a functional definition as the sum total of all such variations.</p><p>However, machine learning is fundamentally different. In machine learning, the product is a model which is generated by combining a set of instructions (an algorithm) with some data (training data). The behavior of the model is thus determined not just by the algorithms but also by the training data. In machine learning, it is not possible to make precise assertions about the behavior of a product in advance (before having access to the training data). In addition, machine learning algorithms learn complex rules from massive amounts of data. The set of possible variations of their behavior based on context is very large. Thus, even if one had access to the training data, it still not feasible or sensible to functionally define a machine learning product by <strong>manually</strong> listing all possible variations in its behavior — if you already knew exactly what an algorithm would learn from data then you would not need to train the algorithm in the first place.</p><p>This presents a <a href="https://hackernoon.com/the-challenge-of-product-data-fit-92543078551b">basic problem</a> for validated learning when it comes to machine learning products. Simply stated, an experiment cannot be used to test the validity of a hypothesis if the preconditions for the hypothesis cannot be guaranteed to have been satisfied — one would not be able to tell whether the test failed because the hypothesis is invalid or because the preconditions were not met. In statistical hypothesis testing, this is known as confounding.</p><p>In their unvarnished form, Lean experiments are not suitable for validated learning in the domain of machine learning or any other domain with significant product side uncertainty. As we discussed above, the root cause of this conundrum is due to the fact that Lean proposes hypotheses validation via repeated <a href="http://www.startuplessonslearned.com/2008/09/one-line-split-test-or-how-to-ab-all.html">split testing</a> as the means of validated learning.</p><h3>Split testing was the MVP of validated learning</h3><p>The goal of validated learning is really to understand the customer from empirical evidence. Split testing is an incredibly restrictive tool for that purpose. So why was it chosen in the first place? I believe, this is a case of the Lean startup methodology finding its own MVP.</p><p>When Lean startup was created as a concrete methodology, its primary “customers” were web-based software companies. In this domain, split testing is the ubiquitous tool for product optimization (making small changes to the product given that you know your customer very well). This familiar tool was simply repurposed for validated learning.</p><p>At that time, the most visible domains with significant product side uncertainty were mostly moonshots such as finding a cure for cancer. It was unclear then, as it is now, whether Lean principles are useful for such moonshots. Thus, it it made very little sense in complicating the formalism with less well known tools. In other words, split testing is the MVP of validated learning.</p><h3>It is time for a pivot</h3><p>Machine learning changes everything — the primary goal of software is changing from executing instructions to learning from data. For the Lean startup methodology this represents a dramatic shift in its most important customer segment. The incompatibility of split testing with machine learning products means that soon it will no longer remain a “viable product” for the purpose of validated learning. It is time to look for other options; it is time for a pivot.</p><p>For inspiration, let us turn to the original source of inspiration for validated learning — scientific research. Lean borrows much of its methodology of learning through experiments from the scientific method. In science, however, experiments are expensive. Using experiments as a means of statistical hypothesis testing is only ever done in few cases, such as when one needs approval from a supervisory body (e.g. clinical trials for drugs) or when there is no real underlying theoretical foundation.</p><p>In more mature fields of science one usually does not move from experiment to experiment trying to validate hypotheses, but instead one tries to build theories. I will provide a simplistic description of this process in the following paragraph. What follows is somewhat of a caricature; not entirely accurate, but serves our purpose.</p><p>A theory is basically an explanation of a wide range of experimental facts. But usually the simplest explanations are not obtained in terms of objects that are directly observable but rather in terms of certain conceptual objects. For example, atomic theory can explain a variety of experimental facts. However, what you actually observe in experiments are not the atoms themselves but some blinks on a detector. Now we need a way, a map, to connect the atoms to the blinks. The map is the bridge between the theory world (consisting of atoms) and the observable world (consisting of blinks).</p><p>The map would be used to convert the results from the theory into predictions about observable quantities for that specific experimental setup. If the predictions from the theory are consistent with all existing observations and there are some new predictions, then new experiments might be commissioned depending on the importance of the new predictions. If any of the predictions from the theory is found to be inconsistent with existing observations then the theory needs to be modified.</p><p>Maps such as the one above are not simple tables, but sophisticated entities. They are accurate models of the experimental setup; such as a detector in the above example. In fact, detector physics is a sub-field on its own.</p><p>An important question to ask is why do we need such maps? What don’t we simply go from theory to experiment? There are many reasons for that, but the most important one, and the one most relevant for our discussion is that these maps provide a separation of concerns. The goal of a theory is to provide the simplest possible explanations; one should be able to test the validity of a theory with experiments conducted in Shanghai, San Francisco or Sao Paulo, without having to worry about the details of each experimental setup. On the other hand, the role of experiments is to be able to compare multiple competing theories trying to explain the same phenomenon. If the burden of including the details of each experimental setup was placed on the theory itself, then none of the above would be possible. Placing a map between theory and experiment provides the necessary decoupling between the theory world and the real world.</p><p>Now in the above discussion if we replace theory world with the product world and the real world with the customer world, we will immediately notice the similarities with validated learning. The goal of validated learning is to find a map connecting product configurations with customer responses. For the lack of a better term I will simply call this the customer map. Finding the product/market fit is equivalent to finding a good first approximation to the customer map. During the product optimization phase one uses this customer map to build the right products for the right customers. The customer map is also further refined during the product optimization phase.</p><h3>A new MVP of validated learning</h3><p>How does one go about building a customer map? In general, it is terribly inefficient to conduct an experiment for each product configuration for which one wants to know the customer response. This is, more or less, the approach that repeated split testing takes. What is needed instead is a tool that can generalize from data. As it so happens, in the decade since Lean startup was introduced, we have perfected a tool that does just that — this tool is called machine learning.</p><p>Machine learning is the perfect tool for validated learning in a Lean startup. To make this claim more concrete let me provide an example of how this might work conceptually. As we already discussed, a customer map is supposed to connect product configurations to customer responses. A recommender system is a machine learning algorithm that does just that. The quality of the mapping in the recommender system will obviously depend on the quality of the data that was used to train it. Thus in the first phase the product configurations should be chosen to reduce the uncertainty in the mapping, while in the latter phase we use the map to generate the best product configurations given a customer. This is precisely the problem that contextual bandits solve. Contextual bandits is a reinforcement learning algorithm. Reinforcement learning is a sub-field of machine learning. In the contextual bandit algorithm the two phases are known as explore and exploit. Thus, recommender systems in combination with contextual bandits can be a tool for validated learning. This setup is a far superior replacement for repeated split testing.</p><p>The approach to validated learning that I just described above is fully compatible with machine learning products because it does not require one to set up hypotheses that are free from product side uncertainty. As such, it does not have the problem of confounding inherent to repeated split testing.</p><p>The approach also does not require functional definitions of products for it to work. The product configurations in this approach can simply be technical specifications. For machine learning algorithms this will amount to choices of algorithm types, parameter choices, training datasets etc. Nevertheless, it is useful to have a functional definition of the product to understand its relationship with the customer. And indeed, it is possible to provide an functional product definition operationally within the approach. The functionality of a product is fully defined by the responses it generates when used with the current customer map — essentially this is our best guess our how the customers will react to the product. This is not a list of features as in a traditional product definition. But it is potentially far more useful because it can be analyzed using the tools of behavioral analytics to gain insight into how customers perceive the product.</p><p>To summarize, machine learning products are fully compatible with the principles of Lean. However, to realize this compatibility in practice we need to abandon some of the antiquated tooling that is used in validated learning and replace them with modern machine learning tools. The central entity that emerges during this migration is the customer map whose determination becomes the primary goal of validated learning, replacing hypotheses validation via repeated split testing.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=39a96efecb2a" width="1" height="1" alt=""><hr><p><a href="https://medium.com/data-science/lean-startup-and-machine-learning-39a96efecb2a">Lean startup and machine learning</a> was originally published in <a href="https://medium.com/data-science">TDS Archive</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[What can machine learning do for theoretical science?]]></title>
            <link>https://medium.com/data-science/what-can-machine-learning-do-for-theoretical-science-ed16fbe8dc8e?source=rss-4f4cabff3c74------2</link>
            <guid isPermaLink="false">https://medium.com/p/ed16fbe8dc8e</guid>
            <category><![CDATA[deep-learning]]></category>
            <category><![CDATA[physics]]></category>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[big-data]]></category>
            <category><![CDATA[machine-learning]]></category>
            <dc:creator><![CDATA[Abhishek Mukherjee]]></dc:creator>
            <pubDate>Thu, 26 Jul 2018 17:06:37 GMT</pubDate>
            <atom:updated>2018-07-26T17:17:47.178Z</atom:updated>
            <cc:license>http://creativecommons.org/licenses/by/4.0/</cc:license>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Mlh6cj1jPMDwIueYvpsQqg.jpeg" /></figure><p>Scientific theories are what make the world comprehensible, at least for most of us. But then we heard a <a href="https://www.wired.com/2008/06/pb-theory/">rumor</a> that there is a new game in town: machine learning. Along with its sibling, big data, they threatened to drive scientific theories out of town. Machine learning, and especially deep learning, have already become a magic box for building ever more accurate predictive models. Using it, one could make predictions based on patterns found in previous observations. Traditionally, making predictions was a complicated business, involving, amongst other things, developing underlying theories for understanding how things work. But now you could throw enough data at a large enough neural network and you will have predictions coming out from the other side. So why bother with theories at all?</p><p>The rumor dissipated soon enough, because it was based on the false premise that the goal of science is to churn out predictions. It is not. The goal of science is to provide understanding. Understanding comes from explanations, and explanations are provided by theories. The whole edifice of modern science stands on the shoulders of a web of interconnected theories.</p><p>The rumor might have died, but its ghost continues to haunt us. Old school theorists tend to regard this new wave of empiricism as an attack on their profession by the plebs. And, many freshly minted data experts, coming from the less analytical lands of our newly democratized landscape, often seem to conflate theory with preconceived bias.</p><p>For me, personally, this rather sorry state of affairs is … somewhat awkward. I started out as a theoretical physicist. Theories are what help me make sense of the world. Yet, I make my living now by tinkering machine learning algorithms. I can appreciate, first hand, the power of these algorithms. Yes, machine learning is a tool, but it is a tool like no other. It fundamentally alters our relationship with information. One way or the other, our conception of what constitutes an understanding of reality will be shaped by the role that machine learning plays in science.</p><p>If rationalism is to survive this deluge of empiricism, then theorists need to find a way to incorporate machine learning meaningfully into their world. Not as a foreign clerk dealing with the mindless drudgery of mining through data, but as a full citizen and guide to the art of building scientific theories.</p><p>It is not such a strange wish. After all, most of the important advancements in how we store, process or convey information, be it new mathematical techniques or electronic computers, have found their use in the development of scientific theories. There is no reason why machine learning should remain the surly exception. The question is, how?</p><p>The template that we use for building theories is derived largely from physics. A theory is essentially a set of rules that can be used to derive predictive models of different aspects of phenomena. The explanatory power of theories comes from their ability to provide holistic pictures of aspects of reality, i.e. in being able to show that disparate phenomena emerge from a small set of simple rules. For example, the same rules of <a href="https://en.wikipedia.org/wiki/Statistical_mechanics">statistical mechanics</a> can be used to calculate the <a href="https://en.wikipedia.org/wiki/List_of_thermodynamic_properties">thermodynamic properties</a> (such as temperature, pressure, density) of any substance in <a href="https://en.wikipedia.org/wiki/Thermodynamic_equilibrium">equilibrium</a>.</p><p>Historically, our belief in being able to explain the universe on the basis of such theoretical frameworks has been motivated largely by the spectacular successes of physics. However, thanks to insights provided by the seminal work by <a href="https://www.nobelprize.org/nobel_prizes/physics/laureates/1982/wilson-lecture.pdf">Kenneth Wilson</a> and others in the last quarter of the previous century, this belief now stands on a healthy foundation of understanding.</p><p>Consider a hierarchy of rulesets, with the initial (bottom level) ruleset representing the mathematical structure of a theory and the final (top level) one representing the mathematical structure of the observed stable correlations in data. One can now think of a transformation such that the rulesets at each level are obtained by applying this transformation to the ruleset at the previous level. This process used to derive higher level rulesets from the lower level ones is called the renormalization group flow (I am using this term very loosely).</p><p>For certain kinds of transformations and rulesets, something quite remarkable and unexpected happens; starting from very different initial rulesets you end up with the same final ruleset. The final ruleset in this case is called a fixed point and the group of initial rulesets that lead to the same fixed point are said to constitute a universality class. The hypothesis of universality (or simply universality for brevity) states that rulesets and transformations that are actually found in nature are of the above kind. (See <a href="https://arxiv.org/pdf/1210.2262.pdf">here</a> for an introduction to universality and renormalization group).</p><p>If universality is true, then it would mean that the observed stable correlations in complex systems would be independent of the details of the underlying theory, i.e. simple theories may be good enough. And, in addition, we should see correlations having the same mathematical structure across various unrelated domains.</p><p>Universality was first observed and studied in the behavior of the thermodynamic variables of disparate systems near continuous <a href="https://en.wikipedia.org/wiki/Phase_transition">phase transitions</a>. Since then it has been observed in a variety of diverse and unrelated places such as <a href="http://barabasi.com/f/432.pdf">the dynamics of complex networks</a>, <a href="http://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.97.9529&amp;rep=rep1&amp;type=pdf">multi agent systems</a>, the occurrence of <a href="https://en.wikipedia.org/wiki/Pink_noise">pink noise</a> and <a href="https://www.quantamagazine.org/in-mysterious-pattern-math-and-nature-converge-20130205/">the bus system of a town in Mexico</a>, to name a few (see <a href="https://arxiv.org/pdf/math-ph/0603038.pdf">here</a> for some interesting examples). There is enough empirical evidence to believe that nature (including many man-made entities) really does indeed favor universality.</p><p>Although theories belonging to a universality class may have very different origins (with respect to the aspect of reality they are trying to explain) and mathematical details, they share some important mathematical properties which puts tight constraints on their mathematical structure. For the universality classes found in physics these properties are usually symmetries, dimensionality and locality. But, in general, they will depend of the specific universality class, and can be determined by carrying out the renormalization group flow of a member of the class.</p><p>Universality, by itself, can only partially explain why the theoretical frameworks in physics are so successful. The second part comes from the observation that the hierarchy of rulesets in physical systems corresponds very nicely with our intuition. In physics the hierarchy of rules is the hierarchy of scales or resolution. Intuitively, we expect that big things (macroscopic objects) have rules and so must small things (microscopic entities). We also know that big things are composed of small things, hence the macroscopic patterns should follow from microscopic theory. And this is exactly what happens in reality. This is the reason why (almost naive) reductionism works so well in most areas of physics.</p><p>The final piece in this puzzle has to do with the timeline of technological development. We started off by observing phenomena at the human scale, and only then started developing the technology, microscopes and telescopes, to observe phenomena at progressively smaller and larger scales. This timeline corresponds very nicely with the hierarchy of rulesets in physical systems. As a result we could develop a very fruitful feedback between theory and experiment. But, even more importantly the starting point was very crucial — for many physical systems the human scale is the one where universality kicks in. What this meant was that the stable correlations were manifest even with small amounts data and manual inspection.</p><p>To appreciate why the above points are so important, consider the situation where instead of measurements of thermodynamic properties we started off with pictures containing the snapshots of all the atoms in a box of gas at different times. How easy would it be to derive thermodynamics or statistical mechanics from this data?</p><p>The situation that we currently encounter in fields such as biology, economics or social sciences is not very different from the above situation. Unlike physics, in these fields we do not have the luxury of knowing what the hierarchy of rulesets corresponds to in reality. Neither do we know at which stage should universality kick in and we should expect to see stable correlations.</p><p>But what we did not have before, and we do have now, is a lot more data and a tool, machine learning, for distilling that data and finding these stable correlations. There is good <a href="https://arxiv.org/abs/1410.3831">reason</a> to believe that deep neural networks essentially perform a version of renormalization group flow, and that one of the reasons why they are so effective is because in many situations generative processes (rulesets) for data generation are <a href="https://link.springer.com/article/10.1007/s10955-017-1836-5">hierarchical</a>. When viewed through the prism of universality, this means that deep neural networks provide us with access to a renormalization group flow in the universality class containing the correct underlying theory, which can then be used to constrain the mathematical structure of the underlying theory.</p><p>Consider a thought experiment where a deep neural network is provided with the snapshots of gas atoms along with the value of some complicated function of the thermodynamic variables; and we train the network with the task of predicting the value from the snapshots. Do we expect thermodynamics to emerge in the final layers of the network? Should we be able to constrain the mathematical structure of statistical mechanics from the weights of the network? There is no reason, in principle, to believe otherwise.</p><p>To come back to the question raised earlier; how can machine learning help theoretical science? Machine learning can provide the mathematical scaffolding for scientific theories, to which theorists will then add meaning and the bridge to reality. However, before we can get there we will need to develop a much better understanding of machine learning. We will need to understand machine learning algorithms from general principles. In other words, what are the analogs of symmetry, dimensionality and locality in machine learning? Perhaps, it is time to start developing a real <a href="http://www.argmin.net/2017/12/05/kitchen-sinks/">theory of machine learning</a>.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=ed16fbe8dc8e" width="1" height="1" alt=""><hr><p><a href="https://medium.com/data-science/what-can-machine-learning-do-for-theoretical-science-ed16fbe8dc8e">What can machine learning do for theoretical science?</a> was originally published in <a href="https://medium.com/data-science">TDS Archive</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Psychographics — the math that may have turned a presidential election]]></title>
            <link>https://medium.com/data-science/psychographics-the-math-that-may-have-turned-a-presidential-election-d4b24363b2a3?source=rss-4f4cabff3c74------2</link>
            <guid isPermaLink="false">https://medium.com/p/d4b24363b2a3</guid>
            <category><![CDATA[towards-data-science]]></category>
            <category><![CDATA[cambridge-analytica]]></category>
            <category><![CDATA[big-data]]></category>
            <category><![CDATA[facebook]]></category>
            <category><![CDATA[psychographics]]></category>
            <dc:creator><![CDATA[Abhishek Mukherjee]]></dc:creator>
            <pubDate>Tue, 26 Jun 2018 21:32:15 GMT</pubDate>
            <atom:updated>2018-06-27T15:46:32.096Z</atom:updated>
            <cc:license>http://creativecommons.org/licenses/by/4.0/</cc:license>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/640/1*PN94HoskXfaSbLh-zQojBg.jpeg" /></figure><p>Unless you have been hiding under a rock or a social media free zone, you must have heard about the <a href="https://www.theguardian.com/news/series/cambridge-analytica-files">Facebook/Cambridge Analytica scandal</a>. If the allegations are true, then the legality (or rather the illegality) of what Cambridge Analytica did is beyond question; the data that they used was obtained illegally (without proper consent). Nevertheless, this incident has also brought into sharp focus the actual algorithmic techniques that they used to transform the data into actionable insights viz. the mysterious art and the subtle science of psychographic profiling.</p><p>The word profiling immediately conjures negative emotions in most of us. We have all heard of profiling based on race, gender or age. No one likes to be put in a box based on characteristics that we cannot control. But, we must not confuse profiling with stereotyping. Racial or gender profiling are examples of stereotyping. Profiling is the act of predicting the choices and actions of a person based on what is known about the individual. Stereotyping is a narrow and completely unscientific form of profiling; it is the act of forming opinions about an individual based on unsupported generalizations stemming mainly from prejudice and ignorance. The relationship between stereotyping and profiling is analogous to that between intelligent design and the study of the origin of species. Both are largely discredited methodologies propped up to support one’s unverified beliefs, and they give a bad name to the whole field.</p><p>The basis of any scientific methodology is learning from evidence. For at least a century now, psychologists have tried to capture human behavior in terms of analytical models. However, the success and the reliability of these models has been rather limited, especially when compared to their counterparts in the physical sciences. This is primarily because human beings are far more complex than atoms or molecules. Any useful model of human behavior must be able to connect observable characteristics about an individual with his/her choices. However, the space of possible characteristics and choices is incredibly large, if not infinite. We now know that one needs complex algorithms to determine which choices are predictable and which characteristics are important predictors for those. An immediate consequence of the complexity of these algorithms is that they require an enormous amount of data and computing power to work.</p><p>The kind of data and computing power that is needed for a quantitative theory of human behavior was not available up until about a decade or two ago. Psychologists and marketeers had been trying to understand behavior on the basis of surveys. Yet, surveys are fundamentally limited as data collection mechanisms. They suffer from self-selection bias (i.e., you only get data from those people who choose to respond to the survey), survey design bias (i.e., you will only ask those questions that you deem are important based on your bias or model), and data scarcity (in the pre-internet era there were severe limitations on the number of people you could reach with your surveys). Thus, the results coming out of such analysis were simply not reliable enough to qualify as quantitative science. As is usual, life moves on when the science is not forthcoming. Some (mainly marketeers) tried to use a cocktail of demographic parameters such as age, gender, race, location etc., mixed with some descriptive statistics and the magic sauce of gut feeling to make oversimplified generalizations about human behavior. This goes by the name of demographic segmentation. Others simply replaced statistics with prejudice, and resorted to stereotyping.</p><p>Everything changed with the dawn of the new century. We moved computing to the cloud, which made it feasible for an ever increasing number of people to train and validate algorithms of ever increasing complexity. And then, came the social media networks. Facebook, Twitter, Instagram and their cousins are veritable laboratories of human behavior. Social media networks are free from the issues that traditional surveys suffer from. Their problem, on the other hand, is that of data explosion. Prima facie, most of the data on Facebook or a Twitter is trivial and might appear as noise. Can the likes on Facebook really say anything interesting about your disposition? Can they be used to predict your future choices? In a series of papers, Michal Kosinski and collaborators found the answer to be a definitive yes. In a <a href="http://www.pnas.org/content/110/15/5802">landmark paper</a>, they convincingly demonstrated that Facebook “likes” can be used to discern personality traits of individuals. In a <a href="http://www.pnas.org/content/112/4/1036">following paper</a> they showed that given enough data, computer algorithms are better at this than humans, including close family and friends. And finally, in another <a href="http://www.pnas.org/content/early/2017/11/07/1710966114.full">important followup</a>, they showed that these personality traits can be used to predict the choices that an individual is likely to make.</p><p>The main contribution of these researchers was not a revolutionary new algorithm. Neither was this the first demonstration that past actions in a domain can be used to predict future actions in the same domain; various companies such as Amazon and Netflix had already demonstrated this in their respective domains. What Kosinski and collaborators showed is that the sum total of a large number of digital footprints consisting of seemingly trivial interactions (such as Facebook “likes”) can be used to predict future choices and hence influence future actions in completely unrelated domains. This is an incredibly powerful realization.</p><p>Cambridge Analytica used these insights to supposedly influence elections. But they are not the first to use psychographics, neither will they be the last. In fact, the potential of psychographics was <a href="https://hbr.org/2009/05/want-to-understand-your-custom.html">intuitively obvious</a> to many way before the Kosinski studies. There are plenty of companies and institutions that are using some form of psychographics in <a href="https://www.cbinsights.com/research/what-is-psychographics/">a perfectly legal manner</a>. The OECD, for example, is conducting <a href="http://www.oecd.org/education/ceri/thestudyonsocialandemotionalskills.htm">a study</a> of the influence of various individual and contextual factors on the social and emotional skills of students in various cities. The goal is to provide policy makers with insights to devise better policy instruments in order to promote these socio-emotional skills amongst students. The study will use the <a href="https://en.wikipedia.org/wiki/Big_Five_personality_traits">five factor model</a> of personality traits; the very same model used by Cambridge Analytica for their modeling. The OECD study is a clear example that psychographics can being used for unambiguous good.</p><p>Profiling has been around since the birth of cognition. Anytime you interact with a human being, or perhaps with a sentient being, you are being profiled. A majority, or almost all such profiling is based on incomplete information and “faulty” algorithms. People are regularly harassed in airports because of their skin-color, or put on no-fly lists because of their last names. It is difficult to argue that unbiased data-driven algorithms will not provide better alternatives in such situations.</p><p>A big part of the hysteria regarding psychographics is rooted in the lack of knowledge about what psychographics is and what it can do. Each one of us would like to believe that they are unique. And in many ways we are. But, in many other ways we are quite predictable. Psychographics is not magic, it is math. It can only learn what is learnable, only ever predict that which is predictable. That which makes one unique will forever remain hidden from psychographic analysis. And even when it comes to effects that can be considered to be predictable, it can only make statistical predictions that are expected to hold most of the time for large groups. In order words, given enough data and powerful enough algorithms, for the predictable aspects of your personality, psychographics will probably get it right most of the time but not all the time.</p><p>It is very important to draw the distinction between the legality of the data collection and the ethics of what we do with it. The questions surrounding the Cambridge Analytica scandal have to do with how the data was obtained in the first place. These questions are the easiest to answer. Almost any sane person would agree that there should be reasonable laws regarding the collection of personal data. What is much more difficult to answer is what are the legitimate uses of that data, even when the data is collected with the broadest possible form of consent.</p><p>Broadly speaking, psychographics can be used in two different (but related) ways. The first is to understand and provide insights about human behavior on a large scale. The OECD study is an example of this kind of use. The second is in microtargeting and persuasion. This kind of use is <a href="http://www.pnas.org/content/early/2017/11/07/1710966114.full">most effective in marketing</a>. Its effectiveness in electoral politics <a href="https://www.vox.com/science-and-health/2018/3/23/17152564/cambridge-analytica-psychographic-microtargeting-what">is possibly marginal</a>, at best. And in both these situations, one can argue that the (mathematical) methods used in psychographics are more ethical (being more unbiased) than the status-quo. Thus, in its present state there are very few ethical issues concerning psychographics beyond data privacy; and data privacy is not an issue unique to psychographics alone.</p><p>It is, however, the future that concerns us. Psychographics, when combined with cognitive science, has the potential of shaping human behavior through persuasion. One can envision the use of such a technique for good, e.g., enhancing our virtuous aspects such as altruism or helping individuals with symptoms of depression, or for evil, e.g. exploiting xenophobia or traits associated with pathological addiction. The ethics of this future psychographics are somewhat similar to <a href="https://www.genome.gov/27569225/what-are-the-ethical-concerns-about-genome-editing/">that of gene editing</a>. And just like gene editing, or atomic energy or the internet, the science of such a future of psychographics will stand innocent of moral judgement. But its uses will not. It is up to us, the inhabitants of the present, to look over the horizon of our collective ethical landscape and shape the boundaries of this future.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=d4b24363b2a3" width="1" height="1" alt=""><hr><p><a href="https://medium.com/data-science/psychographics-the-math-that-may-have-turned-a-presidential-election-d4b24363b2a3">Psychographics — the math that may have turned a presidential election</a> was originally published in <a href="https://medium.com/data-science">TDS Archive</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Machine learning and the law of falling apples]]></title>
            <link>https://medium.com/data-science/machine-learning-and-the-law-of-falling-apples-f6a9cfc06ac3?source=rss-4f4cabff3c74------2</link>
            <guid isPermaLink="false">https://medium.com/p/f6a9cfc06ac3</guid>
            <category><![CDATA[transfer-learning]]></category>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[physics]]></category>
            <dc:creator><![CDATA[Abhishek Mukherjee]]></dc:creator>
            <pubDate>Thu, 03 May 2018 22:06:20 GMT</pubDate>
            <atom:updated>2018-05-04T08:13:57.044Z</atom:updated>
            <cc:license>http://creativecommons.org/licenses/by/4.0/</cc:license>
            <content:encoded><![CDATA[<p><em>Disclaimer: The purpose of this article is not to disparage machine learning in any shape or form. Machine learning is lovely, I make my living off it! The point is simply to explore the edges and try to see what lies beyond.</em></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*3dzCAcvkCxVq5ezTdQhd2w.jpeg" /></figure><p>Imagine a young Isaac Newton sitting under a tree when he notices an apple fall. He thinks about it for a moment and realizes that he has never really seen an apple do anything else but fall straight down. They never go upwards or sideways.</p><p>Now, had Newton known about machine learning and had the actual machines to do the learning, then this is how he might have gone about it. First, he could have set up a classification problem with three class labels “down”, “up” and “sideways”. Then he would collect data on the direction of falling apples. He would have noticed his dataset to be highly imbalanced. But, undaunted, he would have marshalled on and trained his classifier. If his classifier were any good it would predict “down” as the direction of fall in most cases.</p><p>Had he been even more enterprising he would have noticed that the time it takes for the apple to fall to the ground is larger for taller trees. To come up with a better model he would have measured the height of every apple tree he could find. And then he would stand under each one of them waiting for an apple to fall. In each case he would record the time it took for the apple to fall to the ground. After doing some exploratory data analysis he would have realized that he would be able to fit a better linear regression model if he used the square root of the height of the tree as a feature. Finally he would fit this linear regression model and gotten a very good fit.</p><p>Armed with all these insights he would have formulated the “<strong>Law of falling apples</strong>”: <em>Apples almost always fall straight down and the time it takes for them to fall to the ground is approximately proportional to the square root of the height of the tree.</em></p><p>Thankfully for everyone involved, Newton was completely oblivious to machine learning. Instead, he went about it the old fashioned way. He thought hard about the issue and came to the conclusion that apples falling straight down is a manifestation of a deeper principle. This deep underlying principle affects not only apples falling from trees, but everything around us. It equally affects the earth and and the heavenly bodies. It affects everything in the universe. Newton formulated the law of universal gravitation.</p><p>The story of Newton formulating the law of gravitation after seeing an apple fall is probably apocryphal. It is, however, a very good illustration of what really makes science so powerful — its ability to generalize, the ability to find universal truths from limited data. At its core, scientific inquiry is predicated on a set of foundational conjectures regarding the nature of the universe. To a large extent machine learning derives its empirical methodology with science, while replacing human ingenuity, whenever possible, with computational muscle. But how far does this similarity go? To answer this question, let us play the game of analogies.</p><p>The fundamental conjecture of science is that there is order in the universe waiting to be discovered. Although this might sound trivial, without this core belief no scientific research is possible. In the case of science we do not stop to consider the importance of this conjecture because it has been validated over and over again. We simply take it for granted.</p><p>But, what about machine learning? Well, machine learning does not concern itself with the fate of the whole universe, but with data. Machine learning is effectively the art of of function approximation via inductive generalization, i.e., clever ways of “guessing” the form of a function based on data samples. The above statement is manifestly true for supervised learning. With a little thought and elaboration, it can also be seen to be true for reinforcement learning and unsupervised learning. (In the interest of simplicity, I will stay close to the language of supervised learning in the rest of the post).</p><p>In order to guess a function, one needs to assume that a function exists in the first place and a function is nothing but a codification of regularities. Thus, the first fundamental conjecture of machine learning is: <strong><em>it is very likely that observed data will contain regularities waiting to be discovered</em>.</strong></p><p>Or in other words, given an input <strong>X</strong> and an output <strong>Y</strong>, there exists a function <strong>F</strong> such that</p><p><strong>Y = F(X)</strong>.</p><p>Unlike science, the first conjecture of machine learning is not a given, but rather it needs to be validated on each and every data sample. If found to be untrue then machine learning is not of much use for that dataset.</p><p>Regularities are useful because they help predict the unknown from the known. But in order to do so one needs to be able to express them in a language that is powerful enough. In the physical sciences, this language is that of mathematics. The key conjecture being that mathematics provides a sufficient basis for expressing and exploiting the regularities in physical phenomena. Once again, this might look like a trivial observation, but it is far from it. Without its validity much of the grand edifice on which most of modern science and technology rests on will come crashing down.</p><p>The language of machine learning is also a mathematical one, albeit of somewhat narrower scope. The underlying mathematical machinery behind machine learning is that of piecewise differentiable functions in vector spaces (roughly speaking, calculus and linear algebra). There are two very special properties of this machinery. First, it is possible to define the concept of “closeness” and consequently that of a “change” in a concrete manner in a vector space (by defining a distance). Second, for piecewise differentiable functions small changes lead to small effects. Together, these two properties are ultimately responsible for the enormous power of machine learning; its ability to generalize beyond observed data.</p><p>Therefore, in order to successfully apply machine learning to any dataset we should be able to transform the data to a form that is amenable to its underlying machinery,</p><p><strong>Y = F(X) = O(G(I(X)))</strong></p><p>where <strong>I</strong> and <strong>O</strong> are transformations to and from the original representation to one where the machinery can be applied (the <strong>feature space representation</strong>), and <strong>G</strong> is the function or the model that is built using the machinery in the feature space representation.</p><p>The properties mentioned above that make the feature space representation enormously powerful, also make it incredibly restricted. Not every dataset should be expected to have an appropriate feature space representation. However, most do, leading to the second fundamental conjecture of machine learning: <strong><em>if the observed data shows regularities then it is very likely that there exists a representation of the data where small changes give rise to small effects.</em></strong></p><p>The act of transforming raw data into the feature space representation is called feature engineering. According to Andrew Ng — <em>Coming up with features is difﬁcult, time consuming, requires expert knowledge. “Applied machine learning” is basically feature engineering.</em> The success of a machine learning task is severely dependent on being able to find the right transformations <strong>I</strong> and <strong>O</strong>. Very often they are lovingly handcrafted using a combination of deep domain knowledge and arcane witchcraft!</p><p>Deep learning seeks to relieve this burden somewhat by making the process of feature engineering partially automated. Essentially, in deep learning, the transformations I and O are performed by the first and the last few layers of the deep neural network. Thus the mundane drudgery of nonlinear transformations is outsourced to machines while reserving human ingenuity for more impactful insights.</p><p>While we are playing the game of analogies, we are bound to notice that in science there is one final fundamental conjecture. It is the conjecture that universal truths exist and that different phenomena are simply manifestations of those universal truths. It is this conjecture that allows science to generalize from a narrow set of observations to universal laws spanning a multitude of phenomena. To be clear, this conjecture alone does not automatically manifest those universal laws. One needs the genius of Newton to deduce the law of universal gravitation from observing falling apples. But, in the end, it is this conjecture that provides the basis for making those leaps of intuition, elevating science from being an exercise in stamp collecting to the engine of progress and enlightenment.</p><p>Is is possible to make an analogous conjecture in machine learning? Certainly, machine learning does not have any grand designs of discovering universal truths. However, it can, and it must, have the ambition to break free from narrow domain walls. For sure, being able to identify cats in pictures after sifting through millions of pictures with cats, is useful. However, what would be much more useful is if one could use this data to draw some conclusions about how pictures are composed in general. Or, even better, if one could say something about the intentions or emotions of the photographers behind the pictures.</p><p>Notice that this is a different kind of generalization. It is not the kind of generalization that necessarily aims to to be universal. But rather, it is the kind that is transferable. Transferable across domains — from the domain of cat pictures to the domain of visual composition or the domain of human emotion. But how do we find such transferable generalizations?</p><p>What if the the feature space representations were not merely computational crutches, but encoded something deeper? What if the models in these representation (<strong>G</strong>) were not merely operational tools to connect inputs to outputs in this specific domain, but actually revealed underlying structural regularities spanning multiple domains?</p><p>As it turns out, these “what ifs” are not mere wishful thinking. There exists many situations where the observable data do have this feature of transferable generality. This vital observation underpins the fundamental premise of transfer learning. Thus, the third fundamental conjecture of machine learning is : <strong><em>(transfer learning) there exist situations where the observed data are manifestations of underlying (possibly probabilistic and approximate) laws.</em></strong></p><p>As with the previous cases, conjecture alone is not sufficient to make progress. There are many questions that are yet unanswered. Which situations are amenable to transfer learning? How does one know if one has split <strong>F</strong> correctly between <strong>I</strong>, <strong>G</strong> and <strong>O</strong>? After all they are only unique up to a transformation. Is deep learning the only technique that can benefit from transfer learning?</p><p>We are only beginning to appreciate the potential of transfer learning in taking machine learning to the next frontier — cross domain generalization. According to Andrew Ng transfer learning will be the next driver of machine learning success. Such optimism is very well founded. Transfer learning provides machine learning with that elusive bridge to go from falling apples to the law of gravitation.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=f6a9cfc06ac3" width="1" height="1" alt=""><hr><p><a href="https://medium.com/data-science/machine-learning-and-the-law-of-falling-apples-f6a9cfc06ac3">Machine learning and the law of falling apples</a> was originally published in <a href="https://medium.com/data-science">TDS Archive</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Product definition in the age of AI]]></title>
            <link>https://medium.com/data-science/product-definition-in-the-age-of-ai-619a417e3415?source=rss-4f4cabff3c74------2</link>
            <guid isPermaLink="false">https://medium.com/p/619a417e3415</guid>
            <category><![CDATA[product-management]]></category>
            <category><![CDATA[artificial-intelligence]]></category>
            <category><![CDATA[software-development]]></category>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[agile]]></category>
            <dc:creator><![CDATA[Abhishek Mukherjee]]></dc:creator>
            <pubDate>Sat, 21 Apr 2018 21:46:02 GMT</pubDate>
            <atom:updated>2018-04-29T13:42:34.184Z</atom:updated>
            <cc:license>http://creativecommons.org/licenses/by/4.0/</cc:license>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/640/1*yjbAuQdGmjVoECF3NVsI2A.jpeg" /></figure><p>A decade ago, software was eating the world. But, right now it looks like artificial intelligence is eating the world. Not a day goes by without a new algorithm, framework or library being released that makes it ever easier to turn complex math into killer products. However, one aspect that seems to be conspicuous by its absence, is product definition of AI powered software.</p><p>The reason is not too difficult to fathom. Defining AI products is hard, and there are no industry standard methodologies to speak of. In this piece, I will not attempt to offer any silver bullets, but merely share some thoughts and experiences. But before we jump to the specific case of AI based software, let us first look at two other canonical examples from the software world.</p><h4>Conventional software development</h4><p>Most of our intuition and methodologies for product definition come from conventional software development. In an ideal scenario, building software products involves the following steps: starting with a business hypothesis about consumer behavior, deriving a specification from the hypothesis, building software according to the specification, testing if the software meets the specification, deploying the software in production, and determining the validity of the hypothesis.</p><p>An example of a business hypothesis for an eCommerce website may be “If we implement a one-click checkout process, then that will result in a higher conversion rate”. A business hypothesis is a codification of business intuition, which may have been informed by a lot of market research. However, once a hypothesis is formulated, the scope of its antecedent (the part between the if and the then) is usually very narrow and straightforward to work out.</p><p>A product specification is nothing but a representation of the necessary and sufficient conditions for the fulfilment of the antecedent of the business hypothesis. For the above hypothesis, the corresponding specification can be a wireframe for the one-click checkout process. Notice that we could unambiguously derive the specification from the hypothesis. Building the software according to the specification is complicated business. However, once built it is relatively straightforward to check whether it meets the specification or not. The specification provides a sufficient basis for building and testing the software.</p><p>The connection between the specification and testing is especially important. For the purposes of testing, a piece of software is treated as a system that produces outputs given inputs. Traditional software testing relies on the existence of a complete and independent test oracle, i.e. method for enumerating all possible inputs and generating the expected outputs for each input, independent of the software itself. Given this, the goal of testing is to “simply” verify if the software produces the correct output for every input. The test oracle is derived from the specification, which in turn was derived from the antecedent of the hypothesis. Thus, the simplicity and the narrowness of the (antecedent of the) hypothesis plays a determining role in the feasibility of developing software as it is traditionally done.</p><p>In traditional software development, the goal of the development team is to generate value for the business by building software according to business intuition. The business hypothesis and the product specification together form this important bridge between business and software, and a common basis of interaction for everyone. As such, the hypothesis and the specification need to be written in a manner that is understandable to product managers, software developers and testers.</p><p>In the old school way of doing things (aka waterfall), the different steps (such as hypothesis generation or specification) are performed sequentially for the whole product. Each step is performed by a different group of specialists with detailed handovers between different steps. Turns out that this is a fairly inefficient way of doing things. So in the the more modern Agile/Lean methodologies, the product is built up into small slices in short iterations. Nominally, the whole team participates in performing all the steps for each slice. Typically the hypothesis and the specification is codified in a user story or a close relative.</p><h4>Academic scientific computing</h4><p>At the other end of the spectrum is high performance scientific computing common in physics and related fields.Examples include lattice quantum chromodynamics to predict the behavior of subatomic particles, and computational fluid dynamics in many disciplines. In scientific computing a piece of software is written to find the (often approximate) answer to some question.</p><p>Scientific computing rarely allows for the hypothesis/specification split mentioned above. Consider the example of lattice quantum chromodynamics. The underlying hypothesis here is “If we were to calculate physical quantities using the theory of quantum chromodynamics then they will be very close to what we would get from experiments”. A specification for building a piece of software to test this hypothesis might include the equations of quantum chromodynamics and the commandment “build a piece of software to calculate physical quantities according to these equations”. This is actually a reasonable description of lattice quantum chromodynamics, but it is no good as a specification, because it does not help us test the software.</p><p>Testing of scientific computing is performed using a combination of different techniques: exactly solvable cases (there may be some inputs for which the outputs can be calculated exactly using alternate methods), limiting conditions (often the results are easy to calculate in certain limits, such as a input going to zero or infinity), and heuristics (whether the outputs make “sense” given prior experience). The test coverage, i.e., the fraction of the (input, output) pairs included in the testing, is usually much lower in scientific computing as compared to commercial software testing. Also, the test methods require very deep understanding of the domain. Usually, this is not a problem because building and testing of software is usually done by scientists who also happen to be domain experts.</p><p>In general, there is no clear decoupling between the hypothesis and the software in scientific computing. This results in the concept of a specification being of very little use. As a consequence complete and independent test oracles do not exist: the software was written to calculate the outputs (answers) given the inputs (questions). If we had another method for calculating the answers, then we would not need to write the software in the first place!</p><h4>AI-based products</h4><p>At its core, AI software development is very similar to academic scientific computing, in the sense of the hypothesis being very tightly coupled to the software, resulting in the non-existence of test oracles. However, commercial AI-based products are built in a manner that is very similar to traditional software development; driven by business intuition. Moreover, AI-based products are more often than not developed in teams that look very similar to traditional software development teams, with the addition of one or more data scientists. This impedance between its core nature and development methodology makes product definition so challenging for AI-based products.</p><p>To make things concrete, let us consider the canonical example of a recommender system for an eCommerce website. A possible hypothesis based on business intuition might be “If customers are shown relevant products then the conversion rate will increase”. This almost obvious hypothesis does capture the basic motivation behind building recommender systems. In addition, it has the advantage of being easily comprehensible to anyone involved in building eCommerce software. It is also broad enough to support the development of very powerful AI algorithms.</p><p>But, from the perspective of defining a product, this hypothesis is not good. It is not possible to derive an unambiguous specification from it. The problem lies in the vagueness of the word “relevant” — what is relevant in this case? It is simply not very useful when it comes to deriving product specifications, and consequently for building and testing software.</p><p>So where does this leave us? Before we go any further, let us try to understand the different dimensions involved in constructing a business hypothesis. In my opinion, there are two main relevant dimensions that we need to consider.</p><p>The first dimension is the generality of the hypothesis. Conventional software development is predicated on the assumption that it is possible to distil market research and business intuition into a set of narrow but useful hypothesis, which can then be used to build software. On the other hand, the whole power of AI-based algorithms comes from being able to start from very general hypotheses. Consider the previously mentioned case of a recommender system. We might narrow down the hypothesis to “If we show beer to male users trying to buy diapers, then that will increase the basket size.” Clearly in this case we will be able to derive precise specifications from it, and build the software to support it. In that case, congratulations, you have just built your first rules-based system! If we are to leverage the power of AI then such narrow hypothesis is not the answer.</p><p>The second dimension is the comprehensibility of the hypothesis. This comprehensibility is with respect to a common vocabulary that is known to all members of an usual product development team. Consider the hypothesis “Given a context (defined by the user’s history, the user’s features, the product catalog), and two algorithms trained on historical data with similar contexts with similarity being measured in terms of <em>&lt;insert_your_favorite_context_similarity_measure&gt;</em>, if products are shown based on the recommendations from the algorithm that had higher scores for <em>&lt;your_favorite_evaluation_metric_here&gt;</em>, then this will result in a higher <em>&lt;your_favorite_KPI_here&gt;</em>”. Although, the hypothesis above is general enough to support algorithm development, at the current time it will be nearly incomprehensible to non data scientists in product development.</p><p>There is another important constraint that we need to keep in mind. Real business-facing data science talent is scarce. Therefore, outside of the giants such as Google and Facebook, AI-based products will continue to be developed, at least in the near future, in diverse teams of data scientists, business analysts, software developers and software testers who have, at best, a shallow understanding of each others fields.</p><p>Given the above constraint, we cannot sacrifice comprehensibility completely. On the other hand, if we are to leverage the power of AI then we cannot completely sacrifice generality either. The important point to note here, though, is that in most contexts it is not necessary to develop the “best” or most powerful algorithms. Rather what is needed is to develop a solution that is “good enough”. However, when you are building something new it is usually not possible to calculate the effort (in terms of man hours) involved in getting to the “good enough” stage, i.e. you cannot timebox your way to a working solution!</p><p>In fact, the boundaries of what is good enough can, and should, be set in terms of business realities. What is missing is a common vocabulary. In order to appreciate the importance of a shared vocabulary let us consider the example of scientific research, and the scientific method in particular. The scientific method has been popularized within the software development community by the Agile and Lean movements. What is largely underemphasized is the importance of having a common vocabulary within a discipline for the scientific method to work. Typically a the scientific method consists of trying to understand a phenomenon by building a hypothesis, constructing a theory from it, running simulations and experiments to validate it. Each of these steps are usually performed by different groups of scientists who may not understand each other’s work in depth. However, what they do understand in detail are the claims and results made in each work. This is possible because they share a vocabulary. In physics, this vocabulary is usually built on top of mathematics. In chemistry, on top of mathematics and the language of chemical reactions. The same holds true in other scientific disciplines.</p><p>Our current vocabulary for software development is largely set by web development. This vocabulary consisting mainly of wireframes is completely useless when ideating AI-based products, and should probably be the first thing that needs to be thrown out of the window.</p><p>Like in physics, the common vocabulary of AI is also built on top of mathematics, with the more important elements being probability theory and statistics, linear algebra, and optimization principles. The vocabulary of AI product development must consist of a “user friendly” subset of the above. One is able to create and interpret a wireframe, for example, without ever understanding how an API call is made or how market research is done. What is the equivalent of a “wireframe” for AI? I believe that a good starting point is descriptive statistics, i.e. the business hypothesis and product specifications need to be written at least using the vocabulary of descriptive statistics.</p><p>Another piece of received wisdom that needs to be challenged in the age of AI is that market research and software development are completely distinct and decoupled disciplines, that the goal of software is to provide a means for validating a business hypothesis, not to inform it. This assumption is manifestly false for AI-based software. Because of the tight coupling between hypotheses and software in AI-based software it is nearly impossible to separate the learning (or explore) and the doing (or exploit) phases. In light of this, it is important to use AI not just as a tool for validating business intuition but rather as one for building business intuition as well.</p><p>In fact, once one has enough knowledge the relevant action is either straightforward to derive or one can often use out of the box AI algorithms to figure out the relevant actions. As such, the main focus of hypothesis for AI-products should probably be on learning and not necessarily on doing. And what is the piece of information that needs to be learned should be informed by AI as well.</p><p>As with any new paradigm shifting technology, it takes some time before the business processes can catch up. AI is no different. In the long run, I believe that a basic understanding of AI within the wider product development community is the solution. However, in the short run we cannot expect product managers to become data scientists overnight. We can, however, expect them to become data literate. And on the other side of the coin, we should expect data scientists to look up from their algorithms and understand the basics of value creation in a business.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=619a417e3415" width="1" height="1" alt=""><hr><p><a href="https://medium.com/data-science/product-definition-in-the-age-of-ai-619a417e3415">Product definition in the age of AI</a> was originally published in <a href="https://medium.com/data-science">TDS Archive</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
    </channel>
</rss>