<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by John Mark Agosta on Medium]]></title>
        <description><![CDATA[Stories by John Mark Agosta on Medium]]></description>
        <link>https://medium.com/@johnmark54?source=rss-fdb3e1d9846e------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/0*Z4d0whPCBF5-ECDH.jpg</url>
            <title>Stories by John Mark Agosta on Medium</title>
            <link>https://medium.com/@johnmark54?source=rss-fdb3e1d9846e------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Sat, 21 Nov 2026 00:37:06 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@johnmark54/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[Decision Tables for “Thinking Slow”]]></title>
            <link>https://medium.com/decision-analysis/decision-tables-for-kahnemans-type-ii-decisions-aeed1e18aa60?source=rss-fdb3e1d9846e------2</link>
            <guid isPermaLink="false">https://medium.com/p/aeed1e18aa60</guid>
            <category><![CDATA[decision-making]]></category>
            <category><![CDATA[predictions]]></category>
            <category><![CDATA[expected-value]]></category>
            <category><![CDATA[utilities]]></category>
            <dc:creator><![CDATA[John Mark Agosta]]></dc:creator>
            <pubDate>Mon, 26 Jan 2026 20:06:55 GMT</pubDate>
            <atom:updated>2026-01-31T17:08:21.765Z</atom:updated>
            <cc:license>https://creativecommons.org/licenses/by-nc/4.0/</cc:license>
            <content:encoded><![CDATA[<h4><em>Quantifying the science of what it means to be rational, based on </em>Kahneman’s notion of “Thinking fast, thinking slow”</h4><figure><img alt="AI Arta Web generated banner file by the author" src="https://cdn-images-1.medium.com/max/1013/1*asUZxcusVd6AqSDBhF2KpA.png" /></figure><blockquote><strong><em>TL;DR</em></strong><em> A </em>Decision Table<em> is a tool for rational decision-making. It incorporates the fundamentals of how to quantify rational choice. Think of it as means-ends analysis. One makes a table with a row for each decision alternative with the possible predicted outcomes spanning the columns. Solve the table to find the alternative with the greatest expected value. (See, for instance, D. Lindley </em><a href="#fn1"><em>1</em></a><em> and S. French </em><a href="#fn2"><em>2</em></a><em>)</em></blockquote><h3>Combining Predictions and Options</h3><p>What does it mean to make a rational choice? Kahneman <a href="#fn3">3</a> divides decision-making into two kinds; <em>Thinking Fast </em>by immediate pattern matching, and <em>Thinking Slow </em>by deliberative reflection. Decision Tables encompass rational, deliberative decision-making. They are a method for thinking abstractly about hypothetical prospects; so that one’s chosen <em>means</em> lead to one’s desired <em>ends</em>. <em>Means</em> are encoded as a set of <em>alternatives,</em> <em>choices</em>, or <em>options</em>, each indicating a different <em>action</em> by the decision maker.</p><p>Philosophers are satisfied to define rationality simply as means-ends alignment guided by intuition and judgment. However with the rise of a quantitative decision theory we can apply computational methods to put teeth into how to make a choice.</p><p>This idea flies in the face of a commonly held false distinction that there are numbers to describe the objective world, but numbers would not be not applicable or appropriate for measuring “subjective” belief and matters of judgment. Incorrectly some maintain that quantitative “objective” analysis cannot capture sublime human judgment. Consequently any analysis is thought to merely inform a decision, as a recommendation for its unquantifiable refinement. This article shows that decision theoretic methods can encompass the subjective.</p><p>This article is explains a computational method that’s been around for a while, that operationalizes how to make rational choice. In short we can apply probability theory and economics to define rationality in quantitative terms. <em>Means</em> are a set of exclusive and exhaustive decision options to choose from. Similarly <em>ends</em> are the exclusive and exhaustive outcomes. In the unlikely case where means and ends pair up one-to-one, so that each option leads to an unambiguous outcome, things are simple, and one just choses the option corresponding to the most desired outcome. But <em>in real life nothing is certain,</em> so any choice can lead to multiple possible outcomes, all the more so when the choice and outcomes are separated by time and cause. Prospects are uncertain, requiring recourse to probability theory.</p><h3>This article’s argument: The Yin and Yang of Probabilities and Values</h3><p>A Decision Table is a matrix of values, with options labeling the rows and outcomes the columns. The best option is revealed by the row in the table that by combination over all outcomes has the highest value. Rationality entails a principle to assign a value in the face of a range of uncertain outcomes. The quandary is to make the choice not knowing which of the possible outcomes will occur but only their probabilities. We speak of the value as determined by the probabilities as the choice’s <strong>expected value</strong>.</p><p>Table 1 shows the form of a 2-choice by 2-outcome Decision Table. This is a simple example of a prediction of binary outcomes (a *classification*, in ML terms) and an “either or” binary choice. The rows <em>i </em>are are labelled with the choices <em>d</em>₁ <em>d₂</em> The columns <em>j</em> are labelled with outcomes <em>o₁</em> and <em>o₂</em>. Corresponding to each outcome are there probabilities P(<em>o₁</em>) and P(<em>o₂</em>). The combination of the two form a 2 X 2 matrix with the values, called *utilities* <em>u ᵢⱼ</em> in each cell.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/420/1*WSufivXWqD9kfmTiRWoE0w.png" /></figure><p>This rest of this article consists of these sections:</p><ul><li>First, to incorporate uncertainty about the future, one needs to make <strong>predictions</strong>. But predictions specifically that generate probabilities. The good news is that Machine Learning has expanded ways to incorporate data into prediction, but — surprise — most conventional methods don’t output consistent probabilities.</li><li>Second, where do the <strong>values</strong> come from that fill in the cells of the Decision Table? The Decision Table structure assumes there is a source of values as they depend on both choice and outcome. For values we borrow from utility theory, and an elegant way to elicit utilities .</li><li>Then we derive <strong>maximum expected utility (MEU)</strong>, a principled way to combine probabilities and utilities to reduce the Decision Table and find the best choice. We point out that many so-called analytic approaches to decision-making are either coarse approximations to this or just outright inconsistent in comparison.</li><li>Since we’ve demonstrated a principled way to integrate both the objective and subjective quantities when deliberating about a decision, the challenge rests in the psychological methods to <strong>quantify the subjective aspects</strong> of the decision, within the framework of maximum expected utility.</li></ul><h3>Predictions</h3><p>How to predict outcomes follows from probability theory. Specifically we need a method to incorporate data that generates not just a ranked score, but actual probabilities, P(<em>o</em>). There are statistical machine learning methods such as <em>logistic regression</em> that by design converge on probabilities. Statistical predictions by other methods, for example deep learning, can be adjusted by techniques like CalibratedClassifierCV from scikit-learn.</p><p>But why not just use a prediction of the most likely or average outcome, and save the bother of generating a distribution over outcomes? This is an example of the so called “flaw of averages” <a href="#fn4">4</a>. The reason is that with uncertainty the best decision is determined by the tradeoffs between under and over prediction Analogously, what of a web search that returns just one result — do you always expect to be lucky?</p><p>As mentioned before, if one knew the predicted outcome <em>oⱼ</em> for each option <em>dᵢ</em>, the problem would reduce to just finding which option paired with which outcome had the highest value. In the limit of completely accurate predictions this suffices, but in practice going with just the most likely outcome is a poor approximation. Alas, life isn’t so simple, and to be rational a prediction over the range of outcomes and their probabilities of each must be taken into account.</p><p>To be applied, the numerical predictions need to be combined with values. Specifically the range of predictions need to be combined with values assigned over their range. The ultimate value and the best choice depend on how different actions combine with the prediction. Why is this important? In cases where the best decision is not trivially a direct consequence of the prediction due to predictable errors in the prediction then the errors drive the analysis of the best choice, as the Decision Table makes clear.</p><h3>Values</h3><p>The Decision Table consists of the values, in our case utilities <em>u ᵢⱼ </em>assigned to its cells. The table is a framework for assembling the values assigned to each combination of outcome and option to find the best option, but is “value free.” It offers no guidance about how one comes up with these values. Or one could say that the framework does not place constraints on the values in it’s cells. Rather it opens up the possibility to incorporate any source of values into a rational framework.</p><p><em>Utilities</em> express values as deterministic numbers (e.g. do not possess probability distributions) that express the preference for that combination of option and outcome . All uncertainty in the model is captured in outcome probabilities, not the values. The utility numbers express the desirability of each combination, with larger numbers for those more desirable. The scale of the utilities is arbitrary, but given the scale, unit quantities are comparable, in the sense that an equal difference between any two utilities indicates the same change in desirability.</p><p><strong>Valuation methods</strong> Techniques for coming up with utilities are field unto themselves that has spawned it’s own sub-discipline. Here are a few places to start. Using monetary value such as dollars works for some situations, by employing established accounting methods. The disadvantage of this is when there are also “intangible” attributes to values, and dollars alone do not capture desirability.</p><p>To estimate desirability one can run a probability exercise, gauging each utility as a uncertain prospect between two extreme values, the maximum, <em>M,</em> the value of the best combination of choice and outcome, and the minimum set to zero for convenience. Estimation requires the ability (or acceptance) that one can equate a certain outcome (a so-called <em>expected value</em>) with an uncertain prospect of the probability of a larger value. The prospect’s probability is then adjusted to make the equivalence, and this “preference probability” is equated with the <strong>utility</strong> <em>u ᵢⱼ </em>of the prospect — a psychological construct that underlies the existence of subjective values. From here on I will use the term “utility” as this more comprehensive definition of value.</p><p>Despite what I just said about using probabilities to measure utilities, there is a complete and fundamental distinction between the utilities and ooutcome probabilities in a model, which is often confused, despite (or maybe due to) both deriving from subjective estimates. There’s a natural tendency–a bias– to perceive more desirable things to be more probable and vice versa. Keep in mind that all aspects of uncertainty are described by outcome probabilities. Utilities are certain quantities: Consider the hypothetical that if the outcome and the choice would be known, the utility reduces to a single certain number.</p><p>A single utility can comprise a tradeoff. A utility <em>u ᵢⱼ </em>can depend on a set of <em>attributes</em> that can span both tangibles and intangibles. Consider attributes as a different dimension from options and outcomes in the Decision Table. For instance one may assess attributes separately, then aggregate them to derive a cell’s <em>u ᵢⱼ </em>e.g. by adding each attribute’s value. In the general case each <em>u ᵢⱼ </em>can be a comprehensive value model given an outcome and option. Value modeling is an entire field, separate from theory from which the Decision Table is derived.</p><p>For Example consider the tradeoffs in marketing, of reducing short term prices in expectation of improving long-term market share. One can calculate the short term losses, then estimate how price elasticity of market share will increase future revenue. Utility is a function of the sum of these two attributes, one tangible and immediate, the other less tangible and anticipated.</p><h3>A Decision Table for “games against nature”</h3><p>In game theory terms, a Decision Table (See Table 1) is a “game against nature in normal form”. <a href="#fn5">5</a> First one makes one’s choice, then nature makes its move. The outcome is nature’s move, and is not known at the time of the choice, except to the degree that the prediction assigns probabilities to these outcomes. In summary:</p><p>— With this matrix, each cell has a value, as a combination of its choice and outcome.</p><p>— A cell’s value, its <em>utility</em> is a function of both choice and outcome. Utility of a single cell expresses tradeoffs between cost and benefit, and includes risk and time preference. How to do this is the subject of utility theory.</p><p>— The set of outcomes that label the columns are the same for each option. Each option is evaluated over the entire set of outcomes. As mentioned the set of outcomes are exclusive and exhaustive. And the probabilities of the outcomes therefore sum to one, and do not change by option. Instead what changes is the utility, <em>u ᵢⱼ </em>for different options in the same column. The best choice will depend on how utilities differ among rows.</p><p>A similar term, <em>decision matrix</em>, applies to a table with decision <em>attributes</em> labelling the columns, so that each row displays one alternative’s tradeoffs among relevant attributes. It is a tool for value modeling and hence can be useful in combination with a Decision Table. However a Decision Matrix does not represent uncertainty among outcomes. Hence it does not replace a Decision Table as <strong>a technique for how quantified predictions and options are combined using <em>expected value</em> to indicate the preferred alternative.</strong></p><h4>Eisenhower’s D-Day decision</h4><p>Eisenhower’s historic WW II example of when to invade Europe can be drawn up in this Decision Table. Eisenhower’s decision was to either launch the D-Day invasion given what he knew about the likely weather, or to delay to a later date, hoping for better weather, but possibly missing the opportunity had the weather conformed to his hopes.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/755/1*KjHZQd0Qvv4jMru5KxSb2g.png" /><figcaption>Decision Table for the D-Day Landing</figcaption></figure><p>This simple example is sufficient to reveal limitations of poorly conceived decision rules. If Eisenhower’s aim was to <em>minimize regret,</em> he would just avoid options that could result in the worst possible outcome. He would minimize the possibility of the worst loss (his “regret”), by ruling out June 6th’s possible “Impeded operations” outcome. Just focusing on regret would have ignored the opportunity to obtain the best outcome. A better approach takes into account the probability of weather outcomes, balancing the possibility of gain with possible loss, as happens with expected value. With the benefit of hindsight we can assume this is what Eisenhower did, however, by application of expected value, one has the value of foresight into what it means to make a good decision before having the benefit of knowing the eventual outcome.</p><h3>Computing the Best Choice by Maximizing Expected Utility</h3><p>How does one combine utilities in a row to determine which row is the best choice? The best choice has the highest <em>expected utility</em>, obtained by multiplying the value of each cell by the probability of its column, then summing over the row. The row with the highest value is the preferred choice.</p><blockquote><em>Expected Utility as a criterion for decision-making has a principled formal justification that distinguishes it from ad hoc methods. This is a guarantee of what it means to be rational; for one choice to be better than another.</em></blockquote><p>The reason Expected Utility has this guarantee turns on a subtle property previously described of how utilities can be measured using so-called <em>preference probabilities.</em></p><h4>Utilities as probabilities, too.</h4><p><strong>Assumption 1:</strong> Let us make the strong assumption that one can equate the utility of an outcome with the probability it represents of some maximum “target” value, <em>M</em>. We write</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/140/1*sAS-by3o9UGZ88nBfPJPJQ.png" /></figure><p>for the utility depending on decision <em>dᵢ</em> and outcome <em>oⱼ</em> where</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/290/1*W-J3FbSFXYIjpczVU5z0nw.png" /></figure><p>The target value <em>M</em> can be the value of the most desirable <em>u ᵢⱼ</em>, or some hypothetical “bliss” reference state.</p><p><strong>Assumption 2:</strong> When a person assigns a utility by proffering the fraction <em>Pᵢⱼ</em>it means they would accept a chance <em>Pᵢⱼ </em>of receiving the outcome with value <em>M</em> as equivalent to the actual certain outcome <em>u ᵢⱼ</em>. The person is competent in the mental task of equating the certain utility <em>u ᵢⱼ </em>with the probability they designate of an uncertain event. (The pleasure or dislike of engaging in a deal is not a consideration. The like or dislike of gambling per se does not enter into the person’s valuation.)</p><p>Both of these assumptions, known as “continuity” for the choice of the <em>Pᵢⱼ</em>and “substitutability” for equating <em>PᵢⱼM </em>with a certain value depend on the decision maker’s skill to think abstractly and hypothetically on judgments represented as probabilities.</p><p>Given the outcome probabilities <em>P(oⱼ) </em>generated (most likely from data) by the prediction and the utilities expressed as preference probabilities <em>Pᵢⱼ</em>, one can compute the expected utility of option <em>i</em> from <em>P(dᵢ) </em>by the “law of total probability,” which is just</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/358/1*zbkOPgbr5_u-8zPDe9ChYg.png" /></figure><p>By substitution we get the expected utility for each option:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/610/1*PVo_yIsiSSxCRm2fG72oVw.png" /></figure><p>So the best choice is the option, <em>i</em> such that:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/231/1*dEhO7AARPoEgVJq85kkg1g.png" /></figure><p><strong>Thus expected utility criterion for making a decision under uncertainty is just an algebraic consequence of the law of total probability.</strong> This is justified from the utility-probability equivalence (assumption 2). It is important to keep clear these preference probabilities are not probabilities of any anticipated event, and are not to be confused with the outcome probabilities . They are hypotheticals by which utilities can be encoded in a way that allows them to be treated computationally as probabilities.</p><p>Under Assumption 1 may be any increasing continuous function of value (e.g. dollars), not just a linear one. This freedom is used to express risk and time preference by the function for <em>u ᵢⱼ</em>.</p><h4>Computing Maximum Expected Utility (MEU)</h4><p>Here is the MEU calculation shown above applied to Eisenhower’s problem with some conjectured numbers:</p><pre># Generate a 2x2 matrix with values between 1 and 10<br>value_matrix = pd.DataFrame([[100, 1], [45, 82]], \<br>                            index = (&#39;act&#39;, &#39;delay&#39;),\<br>                            columns = (&#39;clear&#39;, &#39;cloudy&#39;))<br>print(f&#39;\n\tvalue matrix\n{value_matrix}&#39;)<br><br># A vector of outcome probabilities<br>outcome_probs = pd.DataFrame([[0.8, 0.2]],\<br>                            columns = (&#39;clear&#39;, &#39;cloudy&#39;))<br>print(f&#39;\n\toutcome_probs\n{outcome_probs.to_string(index = False)}&#39;)<br><br># Take expectation to reduce the rows<br>expected_utilities = value_matrix @ outcome_probs.transpose()<br>print(f&#39;\n\texpected_utilities\n{expected_utilities}&#39;)<br><br># Maximize over the rows<br>maximum_expected_utility = expected_utiities.max()<br>print(f&#39;\n\tmaximum_expected_utility\n\<br>    {maximum_expected_utility.to_string(index = False)}&#39;)</pre><pre>        value matrix<br>       clear  cloudy<br>act      100       1<br>delay     45      82<br><br>        outcome_probs<br> clear  cloudy<br>   0.8     0.2<br><br>        expected_utiities<br>          0<br>act    80.2<br>delay  52.4<br><br>        maximum_expected_utility<br>       80.</pre><p>In this example the option to act on June 6th has utility 80.2 and is preferred to delay.</p><p><strong>Maximum Expected Utility (MEU) decision-making</strong> makes a clear distinction between probability and value of outcomes. Probability comes from predictions, utility comes from preferences. Additionally, since utility is a linear function of probabilities, the effect of changes in prediction accuracy are easy to determine. There’s more detail about this in the section below on <em>Value of Information.</em> Most importantly MEU is an easy to understand, transparent rule for normative decision-making.</p><p>Further exploration of rationality via MEU reveals some other advantages. As shown in Eisenhower’s decision, a “minimum regret” decision rule is one that considers only downside risk. MEU balances tradeoffs between gain and loss. By incorporating a model of risk preference it is revealed that a “minimum regret” choice is just the limit in MEU of extreme risk aversion.</p><p>MEU decision-making is more comprehensive than making choices based simply on measures of statistical significance. Classical statistical tests are based solely on the likelihood of outcomes, consequently the resulting thresholds ignore all subjective aspects, notably values, and the results don’t generalize to multiple option comparisons, making it a poor approximation to a complete analysis. A full exploration of the comparison with MEU can be found in Bayesian statistics, which encompass the principles of rationality.</p><h3>An extended example: Prioritizing Sales Opportunities</h3><p>Here’s an example of how Decision Theory can apply to conventional business operations, and how this might improve on current practice. Consider a sales manager of business to business (B2B) sales, deciding which potential customer deals out of a portfolio are best to pursue. Deals may take substantial time and effort to close, therefore the manager has to decide which have the highest value and probability of closing.</p><p>Consider this Decision Table:</p><ul><li><strong>The two choice</strong>s for the deal are <em>whether to attempt close the deal in this quarter or next.</em> We assume that these are mutually exclusive and assume that if one attempts to close the deal this quarter and it fails, one has lost the opportunity.</li><li><strong>The three outcomes:</strong> : <em>The deal succeeds in this quarter, in next quarter, or not at all.</em> Its value depends on the profit from the sale, minus the cost of trying to close it. The value also depends on when the deal is closed. To accelerate the deal the manager could discount the price, increasing the probability of a close, but decreasing his profit. Thinking of utility, the manager’s <em>time preference</em> determines one’s tradeoff between profit in this quarter or the next. The manager’s <em>risk preference</em> determines the relative value of a small, but high probability deal versus a more risky higher stakes deal. Note that time preference and risk preference are distinct notions that affect her utility computation.</li></ul><p>The result is a 2 x 3 Decision Table. Add the predicted outcome probabilities, typically learned by machine learning from historical details about the customer, product ,and sales team, and one can compute the MEU option.</p><h4>Extending this example — Adding rows and columns</h4><p>Just three outcomes and two alternatives is not realistic. The cherished applied techniques of decision modelling consist in creatively coming up with novel alternatives and postulating hitherto unanticipated outcomes to fully explore all contingencies. How to do this can be worthy of an in depth exercise in Design Thinking. Let’s just consider these cases:</p><p><strong>The default alternative.</strong> There is always the alternative to do nothing by letting the available choices expire before the time a decision needs to be made. The values in the do-nothing row typically are a set of zeros, however there may be outcomes where inaction incurs costs.</p><p><strong>Complete outcomes.</strong> The set of outcomes must be exhaustive and mutually exclusive. The strong assumption we made that a deal will be lost if the attempt to close it fails in this quarter can be relaxed by adding another outcome, of a “failed close but still active” outcome. This instance is by no means the only other outcome worth considering.</p><p>The “failed close but still active” outcome implies that there could be an option contingent on the outcome, to “pursue a previously failed close.” To form this Decision Table requires expanding the “pursue in this quarter” to two rows, one for the combination “option pursued, failed, then pursued in the next quarter” and “option pursued, failed, then dropped.” and the outcomes need to be expanded correspondingly. Zeros as utilities appear in cells that are not possible. As you see this quickly gets out of hand, and the model would be better suited to an alternate technique such as an extensive form representation, such as a decision-probability tree, or a causal decision diagram (See <a href="#fn6">6</a> &amp; <a href="#fn7">7</a> )</p><p>Typically many deals are in play in one quarter, so the full Decision Table would have many rows and columns to include all <em>2</em>ⁿ combinations of deals to pursue in either the first or second quarter. Additionally resource constraints may limit the practical options, making the Decision Table just one component in a larger model.</p><h3>What is the value of a prediction?</h3><p>Another extension of the Decision Table is a kind of probability sensitivity analysis. What if we select the highest <em>u ᵢⱼ </em>in each row — equivalent to changing the order of the game against nature; that is, so one can make their choice <em>after</em> knowing the eventual outcome? Reversing the outcome — choice order is equivalent to treating outcomes as certain, obtaining an upper bound of the value of a perfect prediction.</p><p>If the outcome will be known before the decision is made, then the solution would be simply to chose the alternative in that column with the highest value, and there will be no cases of regret. In any interesting case, the chosen alternatives will differ among rows, instead of their being one preferred choice over all outcomes as there is with the conventional solution. Of course if the decision after knowing the outcome is constant over all columns, then knowing the outcome makes not difference and having the prediction adds no value. (It <em>is</em> possible that one choice is preferred for more than one outcome.) <em>Weighting the utility of the selected choices by each column’s probability gives an expected value of the decision if the outcome would be revealed before having to make the decision.</em> This is the value of a perfect prediction. This incremental expected value between this “perfect information” value and the Decision Table expected value is called <em>Value of Information.</em> This analysis can be extended to compare the incremental value of different predictions.</p><p>We have shown how the Decision Table elucidates the core principles of maximum expected utility for rational decision-making under uncertainty. However, the examples in this article highlight the need to consider also methods for utility elicitation and modelling, and trends in current Bayesian statistical prediction that I didn’t go into detail here. While Decision Tables suffice as an elementary tool, situations with sequences of decisions, possibly contingent on uncertain variables demand more advanced modelling methods. And as Kahneman reminds us, that in the larger scheme of things, means-ends analysis may not be appropriate to all rational behavior.</p><h3>Footnotes</h3><ol><li>D. Lindley, (1971). <em>Making Decisions, 2nd Edition</em>, John Wiley, Section 4.3.<a href="#fnref1">↩︎</a></li><li>S. French (1988). <em>Decision Theory: An Introduction to the Mathematics of Rationality</em> Ellis Horwood, Ltd. <a href="#fnref2">↩︎</a></li><li>Kahneman, D. (2011). <em>Thinking, fast and slow</em>. Farrar, Straus and Giroux.<a href="#fnref3">↩︎</a></li><li>Savage, S. L. (2009). <em>The Flaw of Averages: Why We Underestimate Risk in the Face of Uncertainty</em>. John Wiley &amp; Sons.<a href="#fnref4">↩︎</a></li><li>Jackson, Matthew O. (December 5, 2011). A Brief Introduction to the Basics of Game Theory . Available at SSRN: <a href="https://ssrn.com/abstract=1968579">https://ssrn.com/abstract=1968579</a> or <a href="https://dx.doi.org/10.2139/ssrn.1968579">http://dx.doi.org/10.2139/ssrn.1968579</a><a href="#fnref5">↩︎</a></li><li>J. Pearl, (1995). “Causal diagrams for empirical research.” <em>Biometrika</em>, 82(4), 669–710<a href="#fnref6">↩︎</a></li><li>R. A. Howard, &amp; J. E. Matheson, (1984). “Influence Diagrams”<a href="#fnref7">↩︎</a></li></ol><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=aeed1e18aa60" width="1" height="1" alt=""><hr><p><a href="https://medium.com/decision-analysis/decision-tables-for-kahnemans-type-ii-decisions-aeed1e18aa60">Decision Tables for “Thinking Slow”</a> was originally published in <a href="https://medium.com/decision-analysis">Decision Analysis</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Agentic AI @ UC Berkeley]]></title>
            <link>https://johnmark54.medium.com/agentic-ai-uc-berkeley-600ec4776b23?source=rss-fdb3e1d9846e------2</link>
            <guid isPermaLink="false">https://medium.com/p/600ec4776b23</guid>
            <category><![CDATA[ai-agents-in-action]]></category>
            <category><![CDATA[uc-berkeley]]></category>
            <category><![CDATA[reinforcement-learning]]></category>
            <category><![CDATA[retrieval-augmented-gen]]></category>
            <dc:creator><![CDATA[John Mark Agosta]]></dc:creator>
            <pubDate>Tue, 05 Aug 2025 22:46:05 GMT</pubDate>
            <atom:updated>2025-08-13T18:50:43.931Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Hrk7sjdu49AxIvxPJduDng.png" /><figcaption>Image by author</figcaption></figure><blockquote>tl;dr — UC Berkeley’s Center for <em>Responsible Decentralized Intelligence’</em>s (RDI) one day <a href="https://rdi.berkeley.edu/events">Summit on <em>Agentic AI</em> </a>puts Open Source LLM Agents on the map. AI Agents integrate one or more LLMs with services for applications that take action. This is a community that owns the AI applications they are building. With the current rate of innovation it is almost a full time job just keeping up with the technology.</blockquote><h4>Saturday, on campus</h4><p>I spent a day with a capacity crowd of 2000-plus mostly junior and almost entirely Asian tech folks on the UCB campus — in addition to the claimed 10,000 folks on line. It had the flavor of one of the university’s previous ML or Spark systems summits, and it was led by several familiar CS faculty names, Dawn Song, Ian Stoica, &amp; Matei Zaharia, but it was an order of magnitude larger. The scope and scale of interest in the technology is overwhelming.</p><h4>Agent AI</h4><p>What’s new are multi-agent systems built around one or more LLM services. The software infrastructure for this is quickly evolving, such as MCP (Model Context Protocol), which didn’t even exist a year ago. Multi-agent integration, performance scaling, and efficient training by reinforcement learning (RL) are maturing rapidly, and all in open source.</p><p>I was most impressed by the techniques presented and by the sophistication of the crowd. This was not the place for the typical “AI will change everything v/s AI is all hokey” policy debate. And it was not a drawn out discussion on safety and humanity of AI, although security tools came up several times. I think this is a community that sees AI as an artifact that they create that works for them rather than as AI as an anthropomorphic kind of intelligence that competes with humans, as perceived in the popular press.</p><p>I got some validation for my notion that problem solving by LLMs is best posed not as —</p><ul><li>“Here is a problem — what is the solution”</li></ul><p>but as</p><ul><li>“Here is a problem amenable to such and such a technique — translate it into a formal process that solves the problem.”</li></ul><h4>Presenters</h4><p>I paid more attention to the technical innovations and software packages presented versus the applications. Most applications were student university projects; novel, but not likely to change the course of major commercial AI vendors, who were sparse as invited speakers. Notable presenters where from Deep Mind, Mozilla, AMD, Linux Foundation, Unsloth.ai, Open AI, LlamaIndex, Lambda &amp; others, not to mention UCB faculty.</p><h4>Open Source Software</h4><p>Here’s a list of some packages and by whom (most in python):</p><ol><li><strong>any-agent</strong> by Mozilla. A common API interface for different vendors LLMs, making it easy to A/B test etc.</li><li><a href="https://github.com/rail-berkeley/rlkit"><strong>RLkit </strong>by S. Levine’s UCB lab</a>. An RL framework, but not particular to LLM fine tuning</li><li><strong>dspy</strong><a href="http://dspy.ai/"><strong>.</strong></a><strong>ai</strong> I. Stoica @ UCB.** Prompt tuning with RL</li><li>Software at dev.hub by AMD for running AMD GPUs.</li><li><a href="https://www.notion.so/AI-ML-Stat-b1d1a8fc000745f68d7d0ea2cfd95f12?pvs=21"><strong>AG2</strong> by Chi Wang</a>. Open source AgentOS.</li><li><strong>vLLM</strong> (presentation by Red Hat) — Acceleration and scaling of LLMs for multiple user/ distributed clusters.</li><li><strong>A2A</strong> by Google. Agent to agent protocol</li><li><strong>RLVR</strong> by <a href="http://unsloth.ai/">unsloth.ai</a>/unsloth RL model tuning.</li><li><strong>Ray</strong> from Anyscale (I. Stoica’s company? Is this just repurposing the Ray project from previous UCB work?)</li><li><strong>MCP</strong> To standardize integration of clients and servers for Agents. A community for writing open specs.</li></ol><p>MCP is explained with this diagram:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/960/1*KfvSuW1eZ5yRxvBNv-LpOw.png" /><figcaption>Author supplied graphic</figcaption></figure><p>making it a replacement for the conventional RAG (Retrieval Augmented Generation) model for supplementing chat sessions with search results. Similarly one could cobble together a scalable service pipeline with:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/748/1*js8HIZaRt4SzTInlMsuo_w.png" /></figure><h4>Why not now?</h4><p>If current LLM applications arguably can achieve “AGI” level performance why haven’t they already taken over parts of the workforce? Sherwin Wu of Open AI mentioned four impediments to adoption:</p><ul><li>Connectors are not yet mature (vide supra “MCP”)</li><li>Generalizability. Many applications fail outside of the specific domains for which they were built. (<em>Heard that one before?)</em></li><li>Crafting agents requires skill. Context engineering is evolving.</li><li>Cost of commercial LLM subscriptions is high but coming down.</li></ul><p>Context in terms of persistent memory is a challenge. One would like a personal assistant that improves daily, and didn’t wake up with a blank slate each day.</p><h4>Final words</h4><p>At the end of the day Vinod Khosla spoke and gave his philosophy about tracking trends in technology.</p><p><em>To understand trends see where is the university talent going. Big labs only make progress in narrow-areas, whereas resource-constrained startups are forced to try novel approaches. As for novelty only improbables are important.</em></p><p><em>When the data shows exponential rates of progress one doesn’t need to refer to expert opinion forecasts. It’s evolution in action, often with “punctuated equilibria.”</em></p><p><em>As for when AGI?- not the right question. If 80% of economically valuable jobs can be done by AI ⇒ that’s the relevant sign of progress. But adoption will not be driven by technology only, but by the dynamics of capitalism as constrained by policy.</em></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=600ec4776b23" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[A Data Scientist’s View of Running R in Visual Studio Code]]></title>
            <link>https://medium.com/data-science-collective/a-data-scientists-view-of-running-r-in-visual-studio-code-9107cf6fa7b6?source=rss-fdb3e1d9846e------2</link>
            <guid isPermaLink="false">https://medium.com/p/9107cf6fa7b6</guid>
            <category><![CDATA[r-programming]]></category>
            <category><![CDATA[interoperability]]></category>
            <category><![CDATA[developer-tools]]></category>
            <category><![CDATA[python-programming]]></category>
            <dc:creator><![CDATA[John Mark Agosta]]></dc:creator>
            <pubDate>Thu, 08 May 2025 00:00:45 GMT</pubDate>
            <atom:updated>2025-08-14T15:20:59.737Z</atom:updated>
            <content:encoded><![CDATA[<h4>VS Code combines R and Python development, borrowing features from both languages’ dedicated GUIs. This article reveals several ways to mix the languages in a project.</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/795/1*nGBbKNNqskXdXbHcx8RCjQ.png" /><figcaption>Artwork by the author, Trademarks by the respective owners.</figcaption></figure><h3>A new, universal development environment</h3><p><a href="https://code.visualstudio.com/download">VS Code</a> is Microsoft’s open-source software development environment that can be customized for any language. It is arguably the general-purpose GUI programming tool of choice when working across multiple computer languages. As a data scientist, its R &lt;-&gt; Python integration deserves careful consideration. Its ecosystem is rich with <em>Extensions</em> to add language-specific features, including features for machine learning specific notebook and visualization tools. For R language development, the <em>RStudio</em> development environment by <a href="https://posit.co/download/rstudio-desktop/">Posit</a> (the creator of <em>RStudio</em>) still sets the standard. VS Code has appropriated most features available from <em>RStudio</em>, including the <em>Quarto</em> coding and publishing system. While <em>RStudio</em> is more “Quarto native” — with all its document publishing ability — <em>VS Code</em> makes debugging and Cloud integrations possible for more extensive programming projects. In this post we will be taking advantage of its file management and editing, its ability for symbolic debugging, and the chameleonic <em>VS Code</em> Extensions by which it can be adapted to many languages. If you’re not familiar with it, you should give it a try.</p><h4>Running VS Code</h4><p><em>VS Code</em> has a visual coding and development GUI that runs locally. It works at the level of a folder where your project code and other file artifacts live, similar to the way you’d organize a project with Git. In fact, it integrates smoothly with Git. As a Microsoft product, it is positioned for Azure Cloud inter-operability, making it possible to build Cloud systems at scale - an entirely different topic covered in my article <a href="https://medium.com/data-science/simply-just-python-in-the-cloud-78274389ab00">Simply Python</a>.</p><p>So download <a href="https://code.visualstudio.com/download"><em>VS Code</em></a> for your Windows, Mac or Linux machine if you don’t have it already. This is its default layout:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*4Do3LKk85khT-Z-x.png" /><figcaption>The VS Code GUI. Artwork by the author.</figcaption></figure><p>The interface is partitioned into <em>panes</em> and <em>bars</em>. The main <em>editor pane</em> also shows previews and can be split horizontally and vertically ad infinitum. Below it is the <em>terminal</em>, which can also show other scrolling output. The icons on the far left compose the <em>activity bar</em> that determines what is displayed in the pane immediately to its right, which here shows the <em>File Explorer and Outline.</em> For a tour of the interface and its basic features, see the <a href="https://code.visualstudio.com/docs/getstarted/getting-started">Getting Started</a> page on the <em>VS Code</em> website.</p><h4>How to run R with Python in VS Code</h4><p>There is more than one way to run R in VS Code. These are the various ways covered in this article:</p><ol><li>Use <em>VS Code</em>’s R <em>Extension</em> and Debugger to run .R files.</li><li>Run R code in Jupyter Notebooks in <em>VS Code</em>.</li><li>Run R code using rpy2 in Python Jupyter Notebooks.</li><li>Run .qmd Quarto markdown documents.</li></ol><h3>1. Running R files</h3><h4>First, check your current R executable.</h4><p>First, you need to set up your current R executable environment. Assume you have a current working R version that runs in the terminal. Check when you run</p><pre>.libPaths()</pre><p>that it returns a valid path. On my Mac, this is what appears:</p><pre>[1] &quot;/Library/Frameworks/R.framework/Versions/4.4-arm64/Resources/library&quot;</pre><p>On the Mac, <em>VS Code</em> finds the R executable and library paths automatically. You can check this by opening a shell (e.g., zsh or bash) in <em>VS Code</em> and invoking R.</p><h3>Integration requires these installs:</h3><ul><li>First, in R add the <a href="https://github.com/microsoft/vscode-languageserver-node">languageserver</a> package:</li></ul><pre>install.packages(&quot;languageserver&quot;)</pre><p>This R package exposes an API for common text editors not specific to <em>VS Code</em>.</p><p><strong>VS Code Extensions</strong></p><ul><li>Then, install the <em>R Extension for Visual Studio Code</em> Extension. Find the Extension by selecting the</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/98/1*7WgSF3unCeJGFFFa-urQtw.png" /><figcaption>The VS Code Activity Bar. Artwork by the author</figcaption></figure><p>icon from the VS-Code <em>Activity Bar</em>, found in a column on the far left. (Don’t confuse this Extension with “R Tools” or any of the several R-related extensions that are available.) It needs to be enabled:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/619/0*axuOc3V001JUzSgc.png" /><figcaption>The R language Extension. Artwork by the author.</figcaption></figure><p>Configuring the Extension is done by the drop-down menu from its “gear” icon. Find <em>Settings</em> under the Extension header in a drop-down menu attached to the gear icon. However the default settings typically work fine.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/353/0*FWfrV2nHXHq-XINO.png" /><figcaption>THe Extension Settings menu. Artwork by the author.</figcaption></figure><p>The R Extension also recommends installingthe <a href="https://github.com/ManuelHentschel/VSCode-R-Debugger/releases">R Debugger Extension</a> to enable source code debugging.</p><p>Now, when you do a New File... from the <em>VS Code</em> menu, you&#39;ll have a choice of both an .R file and .Rmd file option. R files include &quot;intellisense&quot; popup suggestions for command completion in addition to other smart editor features. For now, try the .R file feature; .rmd Rmarkdown files need additional extensions enabled. The R Extension also adds an &quot;R terminal&quot; to the list of choices in the lower right terminal dropdown:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/301/0*_5r20oxaCgtMb2Uu.png" /><figcaption>The Terminal Dropdown menu. Artwork by the author.</figcaption></figure><p><em>You could just copy-paste your code into the terminal to run R. But you don’t need to create an active R terminal to execute R code</em>.</p><h4>How to execute .R files</h4><ul><li>The R Extension enables the keyboard shortcut cmd-return (Mac) to send the current line or file selection to the R process running in the current terminal window. Or it creates a process to run it if none exists. <em>I&#39;m not sure if this always finds the right R process. To ensure that the R Extension has an attached R process, I suggest you start an R terminal before executing any R code to get the attached features to work.</em></li><li>Of course, you can interact directly with the R process at the prompt in the terminal. (This does not require you to load the aforementioned languageserver library in the R process.)</li><li>An alternate command is the Run code (keyboard shortcut: cmd-option-n) in the &quot;arrow&quot; drop-down in the file title bar or in the file&#39;s right-click context menu to run the current file via Rscript. Be sure to save the file first.</li></ul><p>Since there can be several active R processes, the current process PID is shown on the right in the Status bar, under the Terminal pane. Either method to run code will pop up a graphics pane if necessary, which can be saved as a .png file.</p><p>Other actions are analogous to those familiar in <em>RStudio</em>, but the interface is adapted to <em>VS Code</em>.</p><h4>Executing .R code in the source debugger</h4><p>Yet another way to run .R code is to use the <em>VS Code</em> debugger from the Run menu (keyboard shortcut: F5). This gives you the familiar breakpoint, variable inspection, and debug console features of <em>VS Code</em>. <em>Apparently, &quot;Run without Debugging&quot; calls the debugger just the same.</em></p><p>Additionally, for any “attached” process, in the far left side pane, the R language icon brings up the “Workspace” with a browsable list of global R objects similar to the <em>RStudio</em> Environment feature — a more full-featured inspector than the <em>VS Code</em> variable inspector, but not available with the <em>VS Code</em> debugger.</p><blockquote>Honestly, with the several execution methods provided, possibly attached to different R processes, it can get complicated to understand which process is running one’s code and what the current process state is.</blockquote><h4>Help Pages</h4><p>Below the workspace inspector in the side pane is the help page tool menu. Similarly a file’s right-click context menu “Open help for selection” will bring up the documentation file for an R object.</p><h3>2. Running Jupyter R Notebooks</h3><p>The second engine for running R uses the Jupyter Notebook integration with <em>VS Code</em>. Conventional Jupyter notebooks can run R if they have an R kernel installed that connects them with the R executable that runs on your machine. To set this up, at the R prompt install the <a href="https://irkernel.github.io/installation/">IRkernel</a> package:</p><pre>install.packages(&#39;IRkernel&#39;) <br># to register the kernel, making it visible to Jupyter.<br>IRkernel:: installspec()</pre><p>Now restart <em>VS Code</em> and create a Jupyter Notebook from the New File... menu item. To switch to the R kernel, click the &quot;Select Kernel&quot; button above the notebook to the right. Then, in the menu-bar popup, select the R executable, and the &quot;MagicPython&quot; dropdown menu on the notebook cell will change to R. If no R kernel is shown, Choose &quot;Select Another Kernel...&quot; to search the file system for one. Once you&#39;ve found one, it will appear next time in the kernel selection menu.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/593/0*4K3AdsQFqiu_a80I.png" /><figcaption>The VS Code command (Cmd-shift-P) prompt. Artwork by the author.</figcaption></figure><p>As with Python, you have a choice with new cells to create them as either R or Markdown cells. In a notebook, a cell’s graphic output will appear below the cell rather than in a separate pane.</p><h3>3. Notebooks for R &lt;-&gt; Python interoperability</h3><p>Alas, your R notebook is for R code only; there is no option to mix notebook cells of different languages. This section discusses how it is possible to create notebooks that can switch back and forth between cells running different languages in the same notebook.</p><h4>Polyglot notebooks</h4><p>You may notice that cells in <em>VS Code</em> Python notebooks do give you a choice of languages. This is Microsoft’s <a href="https://code.visualstudio.com/docs/languages/polyglot#:~:text=and%20signature%20help.-,Variable%20Sharing%20and%20Variable%20Explorer,visit%20the%20Variable%20Sharing%20documentation.">Polyglot Notebooks Extension</a> built on top of .NET. When running a <em>VS Code</em> Python Notebook, here’s the drop-down for cell types. Sadly, R is not a choice.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/210/0*HU7ogPOgegaqc2Qj.png" /><figcaption>The polyglot notebook language dropdown, Artwork by the author.</figcaption></figure><h4>Solution: The rpy2 Python module</h4><p>As I’ve promised in this post, there is a way you can work freely between R and Python. There’s no need to choose one over the other.</p><p>Polyglot <em>VS Code</em> Python notebooks running a Python kernel do allow mixtures of cells of different languages but unfortunately, this feature does not include R. You either choose an R kernel or a Python one. The solution is to load this special Python module that mediates between R and Python.</p><p>In Jupyter notebooks running the Python kernel, one can use “R magics” exposed by the rpy2 module to execute native R code. This is a feature built on the <a href="https://rpy2.github.io/">rpy2 Python package</a>. This is a way to insert R code in a native Python notebook. <a href="https://rpy2.github.io/doc/v3.5.x/html/generated_rst/notebooks.html">Several cloud services use the package</a> by having built-in rpy2 &quot;magics&quot; to transfer variables between the two languages. Think of these &quot;magics&quot; as commands that read a variable from one process and write it into the other. If you subsequently change a transferred variable&#39;s value in a cell in one language, it needs to be re-imported to the other language to gain access to the changes.</p><h4>To use rpy2 in VS Code notebooks.</h4><p>Since “rpy2 magics” are not built in to <em>VS Code</em> kernels, you need to load rpy2 into the Python kernel. Of course, your Python environment needs the package installed:</p><p>First, configure your Python notebook by loading rpy2 with this cell magic in one of the notebook (preferably the first) cells:</p><p>Then in a cell where you want to run R and import an existing Python variable into the cell, say my_df, preface the cell with this magic command:</p><p>Similarly, to export an R variable from an R cell so it’s visible in Python, preface it with:</p><p>The analogous single “%” R magic %R -i my_df imports from R into a Python cell. To see documentation on the set of magic commands run %magic in a Python cell. The authors of the rpy2 Python package continue to support it for sharing data frames between notebook cells in the two languages. <a href="https://blog.revolutionanalytics.com/2016/01/pipelining-r-python.html">This blog</a> explains how.</p><h3>4. Quarto: An alternative to multi-lingual notebooks</h3><p>Quarto is designed to build beautiful documents interspersed with embedded, functioning code. In contrast, Jupyter notebooks are intended to be a programming environment, with a sequence of code cells alternated with text. The line between them gets blurred as more features are added to both. Quarto evolved from R Markdown, as a multi-purpose document generation tool, combining the best of existing tools with the ability to run code. In fact, .Rmd &quot;R Markdown&quot; files, an extension of standard Markdown, find a natural extension with Quarto .qmd files. Quarto can be run from the command line as a batch task that knits .qmd files into HTML, PDF, or other formats. However, it is fully integrated with <em>RStudio</em> and <em>VS Code</em> so that one never needs to resort to the command line.</p><h4>Reticulate: Integrating Python and R with Quarto</h4><p>An alternative to running R with Python in Jupyter Notebooks is the competitive feature of running code in Quarto’s .qmd files by using the reticulate R library. It supports code blocks in multiple languages that are embedded in an extended, full-featured version of Markdown. This is a preferred solution for sharing R and Python data frames in a document. See <a href="https://www.r-bloggers.com/2023/01/combining-r-and-python-with-reticulate-and-quarto/">this article</a> about <em>reticulate</em>.</p><h4>Python in .qmd markdown files</h4><p>Analogous to how rpy2 makes it possible to share data frames between R and Python in Jupyter Notebooks, <a href="https://rstudio.github.io/reticulate/articles/r_markdown.html">reticulate</a> makes the same possible in Quarto. The syntax is different. To retrieve a data frame in an R cell from a previous Python cell, use:</p><pre>{r} <br>library(reticulate) <br>r_df&lt;- py$python_df ```</pre><p>The opposite conversion from R to Python in a Python cell is:</p><pre>{python} <br>python_df = r.r_df ```</pre><p>I find sometimes that loading library(reticulate) is necessary, although Quarto tries to figure this out for you, assuming it is already installed in your R environment.</p><p>Once a Python object is created from an R object it is available in all subsequently executed cells. The same holds true for R objects created from Python.</p><h4>Code display options</h4><p>There is both a source mode and a visual editing mode for Quarto files. Cells in source mode (unlike in visual mode) have a live link along their upper edge to execute them. Unlike Jupyter notebooks, where the output of cell computations is included in-line following the cells, the output is shown in a separate pane when running cells in .qmd files. Running an individual R cell injects the cell&#39;s code into a terminal running R. In Quarto, Python cell output can be displayed in various ways, sometimes in a separate pane called &quot;Interactive&quot;, or in a pop-up window.</p><p>When rendering a document, Quarto gives one freedom whether the code or its output is shown or both. By default, both the code and output are included in the document. To not include the output, create a cell like this:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/313/0*w5-zAwvT9Lljp3nU.png" /></figure><p>Similarly, the comment to not show the code in the document is #| echo: false. This works in both R and Python cells. To enforce this globally, these comments can also be included as yaml comments in the .qmd file header as shown here. For instance, here&#39;s another comment to make the code visible in a drop down labeled &quot;The code&quot;:</p><p>See the Quarto documentation on <a href="https://quarto.org/docs/reference/cells/cells-jupyter.html">Code Cells: Jupyter</a> for all cell comment options.</p><h3>Bottom line: Which environment to use for which purpose?</h3><p>This post is a brief survey of the existing R &lt;-&gt; Python integrations available for the <em>VS Code</em> development environment. Here are my thoughts about when they come in useful.</p><ol><li>Running R code in <em>VS Code</em> is a work-alike to running <em>RStudio</em>. <em>RStudio</em> is a dedicated R development environment with some recent additions to accommodate Python. Conversely, <em>VS Code</em> competes as a multi-language project development environment, with extensions for notebook, Python, and R programming. I find <em>RStudio</em> more intuitive and R-ish, and having to suffer multiple ways to accomplish similar tasks in <em>VS Code</em> adds needless complication to the interface. But if you are familiar with <em>VS Code</em> and don’t want to learn another tool, <em>VS Code</em> runs R code just fine.</li><li>When I need some R code capabilities not available in Python (yes, R still has better statistics and graphics libraries), I use rpy2 to extend my Jupyter notebooks. This way I can still use the extensive features of <em>VS Code</em> such as evolving my (pure) notebooks into Python modules. On the other hand just running R notebooks in <em>VS Code</em> has no advantages over running Quarto documents.</li><li>I’m learning Quarto not only for its “polyglot” language features but also for its extensive and varied document creation possibilities. This blog is actually created as a .qmd document! It remains to be seen if <em>VS Code</em> will equal the ease of Quarto development in <em>RStudio</em> as I get more proficient with it.</li></ol><h3>About the author</h3><p><em>Recently, after departing his Data Scientist position at Microsoft, John Mark Agosta began teaching the “Math Foundations” course for the SJSU online Master’s program and founded Fondata, LLC to pursue his interest in probabilistic graphical models and related Bayesian modeling methods.</em></p><p><em>Originally published at </em><a href="https://rworks.dev/posts/r-python-vs-code/"><em>https://rworks.dev</em></a><em> on May 8, 2025.</em></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=9107cf6fa7b6" width="1" height="1" alt=""><hr><p><a href="https://medium.com/data-science-collective/a-data-scientists-view-of-running-r-in-visual-studio-code-9107cf6fa7b6">A Data Scientist’s View of Running R in Visual Studio Code</a> was originally published in <a href="https://medium.com/data-science-collective">Data Science Collective</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Decision Intelligence, the evolution of AI/ML]]></title>
            <link>https://medium.com/decision-analysis/decision-intelligence-the-evolution-of-ai-ml-d9e14b99a42d?source=rss-fdb3e1d9846e------2</link>
            <guid isPermaLink="false">https://medium.com/p/d9e14b99a42d</guid>
            <category><![CDATA[decision-analysis]]></category>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[business-consulting]]></category>
            <category><![CDATA[decision-intelligence]]></category>
            <dc:creator><![CDATA[John Mark Agosta]]></dc:creator>
            <pubDate>Tue, 03 Oct 2023 18:58:20 GMT</pubDate>
            <atom:updated>2023-10-03T18:58:20.608Z</atom:updated>
            <content:encoded><![CDATA[<h4>In March 2022 twelve forward-looking battle-tested practitioners from the related fields of Decision Analysis and Data Science spent a day brainstorming how both fields could benefit from each other. The day generated an agenda of five themes that forge a new approach to Decision Intelligence.</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*KYRh3yj8JxlGf5G21UoaTA.png" /><figcaption>Image created with Stable Diffusion by the author</figcaption></figure><h3>Why this Combination?</h3><p>Decision Analysis and Data Science are complementary and both are in need of a tune-up: Data Science modeling needs to be anchored in decision making, and Decision Analysis needs to become data-driven. To label this synthesis we borrow the term “Decision Intelligence” — -bringing together the data-driven analytics of Data Science within the larger context of decision modeling. The need is to mature the practice of Data Science and to extend the evidence-based reach of Decision Analysis. Fundamental to both fields is the principle of rational choice; meaning taking actions that meet one’s goals. This is obvious, but difficult to achieve in practice and tends to get side-tracked with the current promotion around “AI.”</p><p>By engaging both communities, we are driving an initiative to create a new kind of practice that is anchored in the union of decision-focused consulting and takes advantage of the profusion of software and cloud-based computing to improve and automate decisions. In addition to exploiting Data Science’s predictive and statistical aspects, <em>Decision Intelligence</em> solves the business alignment problem by integrating predictive and value modeling aspects. By “decision” we mean any action, choice, or control that affects an outcome — a transaction, a prescription, an investment, a change in direction, or routing of information, which is valuable only if it affects a decision. We need to be able to “connect the dots” from the decision to the outcome, a problem that recently has been recognized as “misalignment” but has a long been understood in the field of Decision Analysis.</p><p><a href="https://youtu.be/Q8tIU5bakqg">Our previous workshop</a> brought together a group of senior practitioners from the Decision Analysis, Data Science, Statistics, and High-Tech communities. Its findings are summarized under five themes, detailed below. We intend to build on these themes to create a new field, taking advantage of the energy and momentum that came out of the day, by hosting a one-day virtual conference, where we invite broader participation as the next step in making <em>Decision Intelligence</em> real.</p><h3>Five Themes for a New Synthesis</h3><p><strong>1. Decision Analysis and Data Science,</strong></p><p><strong>2. A common vocabulary,</strong></p><p><strong>3. Humans at the center,</strong></p><p><strong>4. A new practice, </strong>and</p><p><strong>5. Integrating value and predictive modeling.</strong></p><p>As is true of any practice of an applied science, Decision Intelligence draws on several related disciplines.</p><ol><li><em>Integration of best practices of Decision Analysis and Data Science</em></li></ol><p>Decision Analysis and Data Science bring complementary methods to business problems.</p><p>Data science is “everywhere”, but it’s also “all over the place.” The common view of Data Science as simply starting with a table of data, then optimizing predictive accuracy is oversimplified and inadequate to meet real world business challenges. Conversely Decision Analysis, which has traditionally not been data driven, can work in data-rich areas, using the expertise Data Science brings in aligning data discovery to the problem at hand.</p><p>Both fields share origins in common principles in Bayesian methods, and from early work in Artificial Intelligence and Statistics that we can build upon.</p><p><em>2. The need for common vocabulary</em></p><p>The different fields that Decision Intelligence draws upon use different terms for the same concepts. Just take the term “decision,” which in Decision Analysis implies a commitment of resources — -an individual making a tangible change in the world. A statistician uses the same word for the threshold determined by a p-value. To confuse things further, an organizational analyst might speak of “deciding on one’s value.” Terminology gets even murkier once the topic turns to probability and uncertainty. Can we agree on what are “inference”, “prediction”, “learning”, “information”, “features”, “co-variates”, and so on?</p><p>A common vocabulary begs for a detailed explanation of this new synthesis — -a new teachable canon that can borrow from not only Machine Learning, Statistics and Decision Analysis, but from Bayesian methods — -the graphical methods that causal models draw upon: Influence (e.g., Decision) Diagram notation, and the tenants of Decision Quality.</p><p><em>3. Humans (back) at the center</em></p><p>Models are artifacts for humans to use, not agents in their own right. Ultimately any decision involves an individual. Either the model is making a recommendation to a person or taking an action in their stead. Commitment to taking an action requires both “head and heart.” It is an act of will that brings in emotional, organizational, and moral considerations. Maybe the purely rational choice is not best? Behavioral psychology, specifically Prospect Theory tells us how individuals’ perceptions play into poorly made choices. Similarly organizational theory of “satisficing” explains why organizations fail.</p><p>How best do we engage the user’s involvement? For instance, recent Design Thinking methods help in listening to the customer when formulating their problem. And a user’s acceptance of model results relies on having an explanation that can best be done by explaining the model in causal terms.</p><p><em>4. The practice of Decision Intelligence</em></p><p>How does the field translate into practice? What will the tools and techniques of the profession look like? How does this manifest itself as a business opportunity?</p><p>Any modelling project that adds value will change the business process to which it is applied. This is true at operational, product design or strategic direction levels. Decision models have a comprehensive scope, including longer term goals and tradeoffs. Consequently, how such analysis changes an organization — -how peoples’ jobs will change — -is central to the analysis.</p><p>To model the entire decision process means that the practitioner must be involved in the effort from the beginning, when the customer’s issues are first translated into a model formulation. As the name implies, the practice of formulation makes the decision explicit. The formulation would start modeling an initial decision on whether to pursue the project or not and then be expanded to focus on subsequent, operational choices.</p><p><em>5. Integrating value and predictive modeling</em></p><p>Combining value and predictive modeling raises novel algorithmic questions. A model, when fully formulated is an integration of decision, value, and uncertain — -that is predictive — -functions. Models that currently are optimized against an accuracy objective will have a different optimization objective when optimized against value. For example, ROC curve analysis is an illustrative case of using a prediction to make value tradeoffs.</p><p>Under some conditions the predictive model and value model can be factored so the components can be learnt separately and then combined. Or one may take a comprehensive approach as in Reinforcement Learning, which attempts to solve stochastic, dynamic optimization problems for a given value model, while learning the predictive model simultaneously by combining “exploration” with “exploitation.” But Reinforcement Learning is brittle and data-inefficient, and its success depends on running a huge number of online trials — a luxury that few domains can tolerate.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=d9e14b99a42d" width="1" height="1" alt=""><hr><p><a href="https://medium.com/decision-analysis/decision-intelligence-the-evolution-of-ai-ml-d9e14b99a42d">Decision Intelligence, the evolution of AI/ML</a> was originally published in <a href="https://medium.com/decision-analysis">Decision Analysis</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Artificial Intelligence Is Dead, Long Live Common Sense]]></title>
            <link>https://medium.com/decision-analysis/artificial-intelligence-is-dead-long-live-common-sense-ef6f127dbe14?source=rss-fdb3e1d9846e------2</link>
            <guid isPermaLink="false">https://medium.com/p/ef6f127dbe14</guid>
            <category><![CDATA[john-mccarthy]]></category>
            <category><![CDATA[iris-murdoch]]></category>
            <category><![CDATA[ai-generative-models]]></category>
            <category><![CDATA[common-sense]]></category>
            <category><![CDATA[ai]]></category>
            <dc:creator><![CDATA[John Mark Agosta]]></dc:creator>
            <pubDate>Wed, 12 Jul 2023 15:29:35 GMT</pubDate>
            <atom:updated>2026-02-10T16:38:40.223Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*ngqMYdWKVrtHoM9gV855SQ.png" /><figcaption>Image generated by Stable Diffusion</figcaption></figure><h4><strong>At long last we have machines with common sense.</strong></h4><p>We’ve achieved Artificial Intelligence’s (AI) foundational inspiration, to create a machine with common sense. Hence the current generative language models, (GLMs) the technology behind <em>chatGPT</em>. But it’s not what we all imagined it would be. True intelligence aspires to be rational, and in this way generative language models fall short.</p><p>Not only rationality, but true creativity cannot be derived from merely compiling what has been said in the past. This article analyzes GLMs’ capabilities in terms of common sense, rationality and creativity.</p><h3><strong>The origins of automated common sense</strong></h3><p>The origins of AI date back to John McCarthy, who was in the Mathematics department at Stanford, and is attributed as the father of AI. In 1958 he wrote the seminal paper in the field, <a href="http://jmc.stanford.edu/articles/mcc59.html"><em>Programs with Common Sense</em></a>. The history of his research program reveals much about how we got to where we are today. McCarthy was a logician and had a peculiar computational view of intelligence. It is worth looking back at it because it touches on many of the same questions we are discussing today. In the paper, his proposal for his “advice taker” program, he offers this definition:</p><blockquote>“A program has common sense if it automatically deduces for itself a sufficiently wide class of immediate consequences of anything it is told and what it already knows. (p.78)</blockquote><p>This succinct language proposes several criteria for an intelligent machine.</p><p><strong>1. Common sense implies a wide range of general knowledge from which conclusions can be drawn.</strong></p><p>With current models that are trained on the “entire web”, general knowledge is possible. At the time when computer memory was measured in kilobytes (one millionth of today’s terabyte), storage of knowledge was a major concern that is no longer a limiting factor.</p><p><strong>2. A common sense system could interact with a person as if it were human.</strong></p><p>This is eerily reminiscent of current GLM-based chat programs. The idea is that it interacts with a person in “some simple way”,</p><blockquote>“…a system which can be told to make a specific improvement in its behaviour with no more knowledge of its internal structure (e.g. the person doesn’t need to know how it is programmed) or previous knowledge than is required in order to instruct a human. (p.79)</blockquote><p><strong>3. A common sense machine would be an “advice taker” and learn from its interactions.</strong></p><p>This is one area where the current AI and McCarthy’s dream diverge.</p><p>We’ve discovered an AI machine needs to be carefully taught, so our current crop of generative language models are constrained by extensive guardrails, trained by human judgment. The danger that malicious users may corrupt current models by poisoning the public data on which they are trained makes it prudent <em>not</em> to update the models on the most current sources scraped from the internet.</p><p><strong>4. Common sense is derived from logical deduction.</strong></p><p>In the opening words of the paper</p><blockquote>“The Advice Taker is a proposed program for solving problems by manipulating sentences in formal languages.</blockquote><p><strong>This is where things really take different directions; it turns out it did not work to derive general intelligence from logical deduction. </strong>“Formal languages” are how logic is made computable; for instance, “All persons are mortal”, could be expressed formally (excuse the pseudocode) as person(X) and implies(X, mortal(X)). <a href="https://www.nytimes.com/2023/07/02/science/ai-mathematics-machine-learning.html?searchResultPosition=1">Computational logic has notable successes</a>, but this isn’t how current “AI” works.</p><p>McCarthy and others spent the last 50 years working with versions of logics — predicate logics, sorted logics, circumspection logics, probabilistic logics and so on — in an attempt to create “AI”. The result, not surprisingly plowed new ground with novel reasoning tools, useful as automated theorem provers, for formal program synthesis, and for other forms of inference. But none of them resembled common sense reasoning. This isn’t how people think.</p><blockquote>People really don’t make logical deductions when they think</blockquote><h3><strong>How people think</strong></h3><p>You may notice day to day that people really don’t make logical deductions when they think. So what is common sense thinking? The phrase has several overlapping meanings, as <em>sound practical judgment</em>, or as <em>generally accepted belief,</em> based on a wide range of general knowledge and experience. The latter aspect — generally accepted belief — a common ability among humans, is arguably now shared with machines.</p><p>Psychologists make a distinction between —</p><ul><li>“Fast Thinking” that happens automatically, involuntarily, and with little effort. Responses by learned associations and patterns. For example the ability to read falls into this category.</li><li>The conscious, reasoning self that has beliefs and can focus attention. “Slow Thinking” takes over when choices are difficult, and it has the last word.</li></ul><p>Daniel Kahneman details these in his book <a href="https://en.wikipedia.org/wiki/Thinking,_Fast_and_Slow"><em>Thinking fast and slow</em></a>. Thinking slow includes rational thought — aligning current actions with future goals. Rational thought is hard, less frequent than one would imagine, and never perfect. In contrast most all the time our actions follow learned, established patterns. We draw upon a vast repertoire of rules, habits, and shared precepts that come from our moral training, experience, and social situation. To apply them we don’t have to reason about them, we react to a situation in what we perceive to be an expedient way.</p><p><a href="https://en.wikipedia.org/wiki/Gary_A._Klein">Gary Klein</a> a critic of the claimed prevalence of rational thought, gives examples of people who have mastered a game or sport, say chess for example, who can immediately perceive the right move. They don’t have to “think” about it. Obviously this has survival value when danger arises. Consider the opposite extreme, personified by Chidi in “The Good Place” sitcom, who was paralyzed by his desire to always do exactly the right thing.</p><p><strong>Thinking fast is common sense in action.</strong> Or common sense is a kind of fast thinking. When one refers to a role model, asking “what would they do” in this situation, one is relying on a commonly accepted behavior. When one asks, “what do people expect” in this situation, one is looking for the common belief about it.</p><h3><strong>Artificial common sense</strong></h3><blockquote><strong>Common sense — now you have it!</strong></blockquote><p>With any invention touted as AI, there comes a flurry of amazement at its tantalizing ability to solve new problems at human or better levels. Chess, self-driving cars, translation,… and now question answering that total up to make a claim of “strikingly close to human-level performance” (How high a bar is that, really?) The recent paper <a href="https://arxiv.org/pdf/2303.12712.pdf"><em>Sparks of Artificial General Intelligence </em></a>showcases successes and failures, and is recommended reading to understand GLM’s power and limitations.</p><p>Current GLMs, the engines that run chatbots depend on the compilation and distillation of pretty much every written source of information that has been made available to them. With their global view of human knowledge, they have literally, in both senses of the word, digested every available digital text there is, and since they have the capacity to memorize everything they’ve been exposed to, it’s plausible that they can regurgitate appropriately cogent text with uncanny ability. Certainly the breadth of their recall is more than any one human can comprehend. Their responses are driven by the trained values of billions of variables that together precisely rank the text most suitable to be adjoined to the current text, given deep context of the situation. Then fine tuning with human input — the process of domesticating this wild animal — by tuning it to mimic generally held beliefs (There’s a discussion begging to be had about “whose generally held beliefs”) is applied to the compilation of what they’ve digested. Hence the tendency for chatbot responses to adopt middle-of-the road “milk-toast” opinions.</p><p>Common sense AI is alluring, perhaps because it actually imitates the way human speech generation works. Pretty mind-boggling if you think about it. But the speech so generated is not sound; it is not grounded in the truth. And what is commonly believed is often tainted by commonly held biases.</p><p>Here’s an example of OpenAI’s chat model’s “common sense” responses. Regretfully it can generate multiple, inconsistent descriptions for the same issue.</p><p>As an example of something every school child commonly “knows,” the English nation was formed in the year 1066 by William the Conqueror. When asked about the origin of the current English nation, the model says exactly that:</p><blockquote>“The seminal event to which the origin of the current nation of England can be traced is the Norman Conquest of 1066. This event marked the invasion and successful conquest of England by William the Conqueror,…resulting in the emergence of a distinct English identity that laid the foundation for the nation-state we recognize as England today.</blockquote><p>But given a request for a short paragraph on the origins of the English nation state, it tactfully replies with a different opinion:</p><blockquote>“The origins of the English nation as a nation-state can be traced back to the early medieval period. (where it goes on to describe the Angles and Saxons, and the process of unification under Alfred the Great)… The Norman Conquest in 1066 brought about further changes … Thus, the English nation gradually evolved into a cohesive nation-state with its own distinct political, cultural, and linguistic characteristics.</blockquote><p>The latter might have been suggested by the more academic term “nation state”, but it is a distinctly different opinion than the previous. In its fount of knowledge we hear echoes of an ongoing debate, to some extent echoing what we want to hear, given a suggestive prompt. Which is correct? <strong>The chat model can’t resolve that, because it can’t decide. Decision making isn’t in its DNA.</strong></p><h3><strong>Why common sense is not intelligence</strong></h3><p>As conceived in The Enlightenment, <strong>intelligence requires rational thought.</strong> Descartes made the distinction clear when deriding “bon sens” (common sense) as that thing everyone believes they possess fully — -to which he contrasts his famous method of thought for determining the truth. He has a point. The first glimmers of scientific method are appearing about this time. As Ian Hacking argues in <a href="https://www.cambridge.org/core/books/emergence-of-probability/9852017A380C63DA30886D25B80336A7"><em>The Emergence of Probability</em></a> , that the term probability was used up until that time to mean <em>knowledge generally held</em>. “The primary sense of the word <em>probabilitas</em> (from the Latin) is not evidential support but support from respected people” [p. 23]. With the rise of the scientific method probability took on the technical meaning used today that distinguished it from common sense. Surely probability is a foundational concept in rational choice.</p><p>Let’s also not forget <a href="https://plato.stanford.edu/entries/chinese-room/">Searle’s “Chinese Room</a>” argument against computer intelligence (although today it would go by a more politically correct name.) In short, if one could write out the instructions in complete detail for converting any one string of symbols (a natural language question) into a correct response (an answer) would one credit this set of instructions with having comprehended the question? No, it is just a case of following instructions that, were it a person doing the same, would have no need to understand any of the symbols they were manipulating. In its day Searle’s argument was criticized as requiring an impractical combinatorial number of operations — -a rule book of exponential complexity, but given today’s deep neural networks with hundreds of billions of parameters the complexity demands of his Chinese Room one could argue are well within capacity of today’s models. <strong>Today’s actual “Chinese Room” AI models fail the test of intelligence by the same argument.</strong></p><p>There are well understood specific cases of intelligence that “general intelligence” does not comprise. To illustrate what might distinguish these cases, here’s a problem that might occur on a test that calls for higher than average intelligence: A farmer can plant wheat or corn, that profit <em>Pwheat, Pcorn </em>per acre respectively. The requirements of each crop are <em>Ywheat</em>, <em>Ycorn</em> acre-feet of water per acre that costs <em>$Y</em> per acre-foot, and <em>Fwheat</em>, <em>Fcorn</em> lbs of fertilizer — that costs <em>$Z </em>per lb, of which only <em>K</em> lbs are available. Roughly you solve this by producing the most profitable crop until you run up against the availability constraint. Let us presume current AI can solve this “above average intelligence” problem (When I tried it, GPT4 wrote out the program formulation.), for which I will grant it an IQ over 100.</p><p>This problem can be solved with some effort by intuition, and in general, is easily solved as a linear program, by methods invented in the middle of the last century (e.g. before AI was conceived). In fact linear programs routinely solve problems of huge scale. Instead of 2 products and 2 constraints, one could easily imagine 200 products and 200 constraints. The computational complexity of linear programming is polynomial in theory, but in practice results vary widely.¹ Let’s make the conservative assumption that its difficulty scales just linearly with the sum of products and constraints — an increase in difficulty by a factor of 10,000. So the pre-AI methods that can solve this problem demonstrate an increase in intelligence by a factor of 10,000!</p><p>Thinking of intelligence in these terms then, we have existing “intelligent” tools that on computational problems vastly out-rank any measure of general intelligence.</p><h3><strong>So what good is Common Sense?</strong></h3><p>I’m not dismissing current “AI” tools. Just because they don’t use a rational process for generating a response does not mean that their results are irrational. Instead think of these tools as a “translator” in a very general sense. Since they write English well, one thing they do well is re-write stuff in English. Converting tables to descriptions and vice versa, or modifying the style or tone of a passage is their forte. <em>“ChatGPT: Express the following passage in the style acceptable to a high school English teacher…”</em> This is becoming a significant up-leveler for non-native speakers who can have GLMs rewrite their stuff in style indistinguishable from native speakers.</p><p>If generative language models are good style translators, they are bad at reading between the lines. They don’t understand why they fail. Asking them to correct code they generate is a losing proposition. Tellingly they are bad at creating and understanding jokes. So was my high school English teacher.</p><h3><strong>Why intelligence is not enough (to make us human)</strong></h3><blockquote>There is a place for common sense. But we also have rationality and insight to make us human.</blockquote><p>So not having achieved rationality, what else is missing? Amidst much fanfare, if current generative language models appear human — if it is not “intelligence” itself, what else is it that makes us human? Why did McCarthy not explicitly set out to create “artificial humanity”, or was it just a confusion between what it means to be intelligent versus what it means to be human?</p><h4><strong>Stroking our fears</strong></h4><p>Automatic text generation is a good and bad news story, since it’s also the source of a threat to humanity. On the one hand generative AI’s disruptive power could empower marginalized groups, giving access to ideas and skills previously available only to a few. On the other hand our ability to be fooled, and our innate inability to differentiate the artificial from the natural means we can’t avoid being deceived by generative AI output. This enables social manipulation, and obscures our perception of reality. We risk getting lost in a hall of mirrors of past imprecisions and will fail to notice what is actually in front of us.</p><p>We are such creatures of habit and convention. What if our future is that GLMs become so alluring that every PowerPoint presentation, TV episode, web content, social media post, everything we see and hear, is merely an amalgam of those things most popular in the past? Wouldn’t such a world of delusion mean the end of all human creative progress?</p><h4><strong>Insights that make us human</strong></h4><figure><img alt="Stable Diffusion’s rendition of dancing angels" src="https://cdn-images-1.medium.com/max/381/1*oWEiQXewcEDUJKn_lgstQA.png" /><figcaption>Image created by Stable Diffusion</figcaption></figure><p>Accordingly given their reliance only on the past, generative AI cannot genuinely be creative. What of the apparent creativity it exhibits; is it on a par with humans? Deep question! Time to dive into the philosophy, for which we borrow some incidental comments on Plato from the recent, late British philosopher, Iris Murdoch’s book <a href="https://openlibrary.org/works/OL34017161W/Sovereignty_of_Good?edition=key%3A/books/OL46019215M">Sovereignty of Good.</a> It starts with Plato and the idea of Beauty, a particularly human ability to appreciate the pleasure of experiencing something as the thing in itself. Beauty is instinctual; a source of pleasure without any need to consume the object. A meal may smell good; that’s (just) desire. But a flower can smell “beautiful” — something not reducible to animal desires. ( <a href="https://plato.stanford.edu/entries/aristotle-aesthetics/">This example </a>is taken from Stanford Encyclopedia of Philosophy.) Our appreciation of Beauty is a particularly human quality to see in something its intrinsic merit. The argument goes on to make the same point about a person’s appreciation of truth, or of a moral action — a disinterested appreciation of its “beauty” just from the good sense it affords. Why is this so? What is the source of this good sense? The intrinsic source is its ability to bring us closer to reality by giving us an insight we otherwise don’t appreciate. Thanks to the Greeks, we can say that Beauty is about insight, and insight is revelatory, improving our perception of what is real.</p><p>Beauty is something realized when the artist brings it to light. Quoting Murdoch, “(We attribute) to art, to stretch the imagination, enlarge the vision and strengthen the judgment.” In this sense, it’s just a small leap to see that in the appreciation of art we “use our imagination not to escape the world but to join it.” And this is one with “…our ability to forget the self, to be realistic, to perceive justly.”</p><p>Creative insight originates in this clarity of perception. So where does such a perceptive ability arise? Perhaps we have to look to the mystical traditions that strive to clear the mind and sharpen perception.</p><p>Granted creatives can use generative AI to spawn alternatives, probably faster than they can dream them up themselves, and this accelerates the creative process. But the difficult insight of winnowing them down to find the one “true” alternative remains something uniquely human. I concede that the creative human spark that underlies appreciation of beauty, truth and the good is a kind of fast thinking, not reducible to rationality.</p><h3>AI for good</h3><p>I’ve distinguished three kinds of human cognitive ability; <strong>common sense</strong>, <strong>rational intelligence</strong>, and — if I daresay —<strong> moral insight</strong>. Understanding these distinctions clarifies how GLMs are best applied.</p><p><strong>Common sense</strong> is what conversational GLMs convey. But “common sense” that comprises a swath of human knowledge inconceivable by any one person. How do we best use it? Great pains have been taken with commercial GLMs to constrain them to avoid mis-behaving. First thing to note is that doing the right thing is not just not doing the wrong thing. It is not sufficient to simply apply known, approved guidelines. First, guidelines can not be complete since malicious actors will find new unanticipated exploits. But more importantly just a set of injunctions about what not to say doesn’t help say what is closer to the truth.</p><p>What’s to be done about the dangers of deception? One thing a GLM should know for sure is what it’s said. It should be possible for it to recognize its own output. It takes one to know one. What if GLM products were required to have a digital twin that could recognize if a text was something they generated? Fight fire with fire. One could ask the twin “Is this output yours?” “Did you do this?” It could be a requirement that GLM services provide such a twin for any GLM. Knowing the source of a proposition is one way to discern its truth.</p><p>I’ve touched on the value of GLMs’ ability to imitate styles. Some may say <a href="https://www.goodreads.com/quotes/150224-style-is-the-answer-to-everything-a-fresh-way-to">“Style is the answer to everything.”</a> There is something deep going on about how style is key to contextualizing and conveying meaning. With different styles come different viewpoints. “Generative” implies variety and that brings bias. Much like a skilled sophist can generate an argument to support differing viewpoints, a GLM has value to explore multiple, conflicting opinions.</p><p><strong>As for rationality, </strong>GLM’s ability to surface knowledge makes them an information accelerator, useful when analyzing a decision. A good decision derives in part from consideration of a rich set of alternatives and relevant information, for which GLMs are effusive assistants. Interestingly one could also conceive of building a chat model for eliciting one’s values; how would require going into some detail. In the end however, with alternatives, information, and values specified, the GLM is not actually making the decision. So much of past AI has targeted automated decision making. That is the realm of rational intelligence, not the domain of GLMs.</p><p>Creative occupations already recognize the multiplier effect of using GLMs to generate numerous candidates as part of the design process. This has broad applicability: not only for generating multiple images, this applies also to designs, labels, plans, phrases (think slogans) — anything over a range of styles. The bottom line though is that the final step is the hard choice of choosing which candidate is most suitable — what is best or right. More philosophy: This is purely the domain of <strong>human insight. In the end only a human can be the arbiter of what is beautiful, truthful or moral.</strong></p><p>[1] N. Megiddo (1986). “On the complexity of linear programming”, IBM. ch.6</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=ef6f127dbe14" width="1" height="1" alt=""><hr><p><a href="https://medium.com/decision-analysis/artificial-intelligence-is-dead-long-live-common-sense-ef6f127dbe14">Artificial Intelligence Is Dead, Long Live Common Sense</a> was originally published in <a href="https://medium.com/decision-analysis">Decision Analysis</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[The True Meaning of Alignment: Decision Modeling in Data Science]]></title>
            <link>https://medium.com/decision-analysis/the-true-meaning-of-alignment-decision-modeling-in-data-science-4acdbafc04bf?source=rss-fdb3e1d9846e------2</link>
            <guid isPermaLink="false">https://medium.com/p/4acdbafc04bf</guid>
            <category><![CDATA[rationality]]></category>
            <category><![CDATA[responsible-ai]]></category>
            <category><![CDATA[probability]]></category>
            <category><![CDATA[data-science]]></category>
            <dc:creator><![CDATA[John Mark Agosta]]></dc:creator>
            <pubDate>Sun, 13 Mar 2022 19:06:03 GMT</pubDate>
            <atom:updated>2022-03-20T21:58:32.278Z</atom:updated>
            <content:encoded><![CDATA[<h4>Combining prediction with value modeling</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/869/1*sENRXnf6VNlPSGiEnf1ZQw.png" /><figcaption>Decision modeling raw material</figcaption></figure><h3>I. What does “Decision Modeling” mean?</h3><p><strong>Decision modeling is part of the data science process — actually it should be part of it, and overlooking it has populated the graveyard of data science failures.</strong> It is a mistake to limit the modeling scope just to prediction, as suggested by the term “Predictive Analytics”, and this leads to fundamental confusion about how to model a situation where decisions are involved — and invariably they are involved. This should be obvious to any student who has studied organizational behavior or decision-making theory in economics, and is a cautionary tale to anyone wanting to become a competent data scientist.</p><blockquote>It is a mistake to limit the modeling scope just to prediction</blockquote><p>This article explains how to formulate a decision model from a predictive machine learning (ML) model. This is illustrated with a kind of causal diagram known as a Bayes network. A decision model needs to model — that is to <em>align </em>— decisions, uncertainties, and outcomes. Data Scientists will (or should) find modeling uncertainties with probabilities familiar. The corresponding value model over outcomes is equally important, supported by an equally rich theory, but unfortunately it is not given the same attention. Often one is advised to “be sure to solve the business problem”, and it’s left at that. I’ll make the case for value modeling, then show how, when decision variables are identified, the two models combine into one optimization model. Owning the decision model implies yet another competency required of a data scientist.</p><h3>II. Kinds of variables: Decisions, Uncertainties, and Values</h3><p>The decision modeling process begins by setting the problem scope. This makes it possible to ascertain which variables — quantities susceptible to measurement — to include. We partition variables into 3 types:</p><ul><li>choice variables that make up a decision,</li><li>uncertainties that describe the world, and</li><li>values that quantify outcomes.</li></ul><p>These three variable classes comprise the extent of a decision model. Modeling is the task of defining these variables and integrating them into one consistent model, as I will discuss. First let’s define these three classes.</p><h4>Decisions: actions, choices, and alternatives</h4><p>By “decision” I mean a set of actions to chose from, identified by their tangible effects. The term gets bandied about confusingly; one may speak of “deciding on one’s values”, but that’s not what I mean. I mean a commitment that affects money, people, and things of that sort. In the model, <em>decisions refer to the variable or variables that are under your control</em>. Identifying them is the first step to scoping the model — only those outcomes and values affected by one’s decisions are relevant, and conversely, any alternatives that do not affect outcomes can be left out of the model, since they will have no effect.</p><p>Decision variables are inputs to the predictive model: Thinking that a model should “predict a decision” is a fundamental confusion.</p><blockquote>Thinking that a model should “predict a decision” is a fundamental confusion.</blockquote><p>Since the decision is a variable under one’s control, it is an input, <strong><em>not an output of the model.</em></strong> Once I’ve mapped out how the parts of a decision model come together, it will become clear exactly how decision variables fit in the complete model, and how this model, with prediction included, leads to recommended decisions — the <em>policy.</em> The important point is that one should start model formulation by asking “What are the variables under one’s control?” and then move on to what can be modeled that is relevant.</p><h4>Uncertainties and outcomes</h4><p>The predictive model takes current known quantities and computes an estimate of outcomes. It is by the model’s outcomes that values can be assigned, and the relative merits of decision alternatives imputed.</p><p>The danger is that the set of outcomes included does not include all the considerations by which decisions need to be judged. Invariably there are tradeoffs to consider — as noted by the current attention paid to responsible and ethical AI. In businesses, at first glance, it may seem that we can just borrow the current operation’s “Objectives and Key Results” (OKR) as outcome variables. However, the value model needs to be comprehensive so that, for one, it captures concerns not apparent in an OKR. An OKR’s purpose is simply to track progress toward a goal — and it is from the goal that the model outcomes are derived.</p><blockquote>…the value model needs to be comprehensive so that … it captures concerns not apparent in an OKR</blockquote><p>An outcome variables’ uncertainty needs to be modeled as part of the prediction. It is the rare case where a prediction can be made with such accuracy that its implication is obvious, and any inaccuracy can be ignored. These cases do arise in modeling perception, of objects in images, or words in speech, but in business processes, the <em>uncertain</em> nature of the world must be taken into account. This is recognized in the widespread adoption of statistical methods in machine learning.</p><p><strong>Model errors</strong></p><blockquote>It is only by knowing probabilities of the predictions together with values of outcomes that errors can be managed.</blockquote><p>Models are <strong><em>never</em></strong> clairvoyant. Machine learning models are statistical tools, and, as a result, their predictions vary around the “<strong><em>true value</em></strong>.” But also, some processes are inherently uncertain — take for example network packet traffic, or customers arriving at a queue.</p><p>Consider making predictions for the “Newsvendor” problem — the <em>single-period</em> case of how to stock up on supplies before knowing the demand. For instance, as the name suggests, one needs to assess the daily number of newspapers to order in anticipation of how many may will actually be sold. A “newsvendor” predictor of the rate of demand may be entirely accurate, despite there being significant variation in the actual daily events modeled. To simply take the model’s advice and purchase the most likely number of newspapers is rarely the best solution. One should hedge and stock up on a few more. Exactly how many depends on knowing the variation around the model prediction, and the relative value of over or under selling.</p><blockquote>It is only by knowing probabilities of the predictions together with values of outcomes that errors can be managed.</blockquote><p>Since all models have errors, the fundamental design question is how to model both the extent and severity of the errors to come up with the best solution. It is only by knowing probabilities of the predictions together with values of outcomes that errors can be managed.</p><h4>Probability</h4><p>Probability is the language of uncertainty, and offers a rich and complete toolset for manipulating and quantifying uncertainty. Uncertainty in values are expressed by probabilities of outcomes. Incidentally, these are not to be confused with “Confidences” as that term is used by statisticians. The field that makes probability a first-class concept is Bayesian statistics. There is a formal axiomatization of the probability calculus that enforces strict consistency among probabilities, both those generated statistically and those that express a person’s judgment.</p><blockquote>Any ML method that does not associate a probability distribution over random variables to represent its error is incomplete</blockquote><p>Any predicted value needs to come with a probability distribution: Nothing actual is for certain. Any ML method that does not associate a probability distribution over random variables to represent its error is incomplete. In decision making, uncertainty about outcomes as a basis for a decision plays a primary role. <em>This is a challenge in machine learning, since many widely used tools lack probabilistic outputs.</em></p><h3>III. The “Alignment Problem”</h3><blockquote>“Alignment” needs to be considered among all three corresponding variable classes.</blockquote><p>The so-called alignment problem, recently popularized by Brian Christian¹ arises when poorly specified goals for AI system lead to unintended consequences. The apocryphal example (attributed to Nick Bostrom²) is an all-powerful AI system tasked to, say, make paper clips. Sounds silly, but the point is that such a system left to its own devices would cause rampant damage in other areas just to maximize its goal. Remember the story of the Sorcerer’s Apprentice?</p><p>Such a system needs to incorporate human values. The alignment problem has been cast as an existential threat of AI. Theatrics aside, the grain of truth is that an autonomous AI system cannot succeed unless the objective it is maximizing is correct. <em>This comes down to having a correct value model — -even if the prediction model would be completely accurate, the values it applies still need to be aligned.</em> You get exactly what you ask for, which is not necessarily what you want.</p><p>I propose that “alignment” needs to be considered among all three corresponding variable classes — predictions, alternatives, and outcome variables. Models for each need to be aligned with the others for the decision model to work.</p><p>Now it’s time to turn to the third set of variables — -and how values are measured so they can be modeled.</p><h3>IV. The value model</h3><p>We define the value model as a function <em>f(s)</em> that maps outcomes into quantifiable values, using a common unit among all outcome variables. Trivially, if we know the price of different items, we know their value, and can choose among them. However if the consequences of the choices are in the future, depend on a host of factors, and involve intangibles, valuing these choices, and thus making a rational choice is hard.</p><p>Familiar examples of value models are the economic models used in finance. Domains (e.g. “verticals”) each have different models — -in healthcare the value model may measure quality of life, for example. Despite the differences in domains, values modeling consists of a set of common principles that apply uniformly across them all.</p><h4>Why is it necessary for Machine Learning?</h4><p>In simple terms, a machine learning (ML) model predicts an event (Did the customer churn?) or a quantity (What will be the final score?). <em>The model prediction is not the modeled value; these terms need to be distinguished carefully.</em> As described the prediction of the model is an uncertain variable. The value model associates preferences over predicted outcomes — -what is desirable — - and hence needs to be combined with the prediction. So there’s both an ML model, and a value or utility model that are combined in the overall decision model.</p><h4>What does the value function have to do with how decisions are modeled?</h4><p>Clarifying model’s value outcomes also helps clarify which decision alternatives to include in the model. An example may make this clear. The recent fad in Sales Opportunity Scoring demonstrated that it’s possible to predict with high accuracy the probability that a sale will close or not. From the salesperson&#39;s point-of-view this is a “so what?” Does a high probability mean the salesperson should put more effort into it, or that it can be left to itself and effort is best applied to lower probability opportunities? The fact that it will eventually close is not her concern, it is what is best to do given the likely outcome. The model is useful only if it comprises actions that available to the decision maker. Once it is clear how the range of actions affect the value of different outcomes, we see how the value model needs to be a function of both outcomes and actions, whose purpose is to match the best action to each outcome.</p><p>Overall, by identifying who makes the decision, we enforce consistency between the value function and the decision. Whose values are to be included in the model? If, for instance, it happens that a root causing analysis tool is more accurate, but increases the time and effort by the troubleshooting engineer, have we truly addressed the engineer’s needs?</p><h4>So how does value modeling combine with modeling uncertainty?</h4><p>So there is this inescapable duality between probability and value. As mentioned, every predicted outcome has two aspects, a predicted probability of occurrence and a value, which could be either positive or negative. Together they form a theory of measurement or estimation, although the terms are sometimes confused — “value theory” being also a term for the combination.<br> <br>How does one make a choice under uncertainty? For making choices we need a way where a distribution of values can be reduced to an equivalent single number. So we need a concept of <em>expected value,</em> or a <em>certain equivalent.</em> We borrow the notion of <em>expectation</em> from probability theory — -which is simply the product of probability times value. If all outcomes were equally probable, then is reduces to just the average value.<br> <br>There is a perversity in supervised learning to confuse this by trying to predict a value to be obtained i.e. predicting expected values. This muddled thinking leads to confusion in the model formulation. Here’s a fairly contrived example that makes the point — -predicting water consumption. Future uncertainty depends on weather conditions. But variation due to policy choices — -who get water for what — -are values not determined by the weather.</p><h4>Creating the value model</h4><p>Value modeling is it’s own field. How do you assign a number to an outcome? An everyday example of reducing seemingly immeasurables to a certain value occurs whenever we set a price. In one instance, a negotiation for purchase results in a process where two parties come to agreement on a certain number. Another case is when prices are set in a competitive market. There no one person affects the price, but by the magic of the process of market clearing, it generates a single price.</p><p>One method is to elicit values by a structured interview technique applied to the consequences of the model’s decision. One posits hypothetically the two most extreme outcomes, then calibrates the value of intermediate outcomes by eliciting the probability from an expert that makes them equivalent in value to the extremes. This is especially useful with “intangible” outcomes — -those for which there are no objective measures.</p><p>An example of eliciting the value of an intangible outcome could be evaluating the cost of spurious alerts — -the cost of “alert fatigue” by, say, comparing it to other work nuisances that are more tangible: “If you had to stay late because of a spurious alert, what is the cost of a spurious alert in equivalent minutes of staying late?” One can imagine creative approaches depending on the circumstances for finding a common unit of value among different outcomes.</p><p>What if there are multiple outcomes, each valued differently? For example, I can keep sick patients in the hospital longer to assure their cure, but at the risk of running out of hospital beds if hospital admissions increase. Perhaps <strong>the hardest part of building a value model is the necessity to weigh the tradeoff between competing outcomes, </strong>such as by coming up with a weighting that reduces multiple values to a common scale. In summary, elicitation interview techniques can resolve this.</p><h4>Pairwise comparisons to map out preferences</h4><p>These elicitation methods can be extended by posing pairs of hypothetical outcomes for the decision-maker to rank. This is a versatile tool that can be used to</p><ul><li>order possible outcomes to create a single function from them,</li><li>come up with a relative weighting for otherwise incommensurable quantities,</li><li>quantify the value of intangible quantities,</li><li>compare the relative value based on the time when an outcome occurs,</li><li>determine how the level of risk affects value.</li></ul><p>For example, imagine valuing sales growth in different geographic areas, or for different new product introductions. One could assess the relative value of future growth rates versus current revenue. What is it worth to “buy” market share by pricing aggressively, compared to the loss of current revenue? The key point is that it is more important to include all factors that determine value, including “intangibles” that can not be measured with high accuracy and require judgment. Better a model that is inclusive instead of a model that avoids important factors presuming they are too hard or subjective to measure. Doug Hubbard’s book argues this point.³</p><h4>Errors in judgment</h4><p>Since people don’t always say, or even know what their true preferences are, the human aspect of how values and probability are perceived plays an important role in elicitation techniques. Methods to de-bias responses have been studied in a behavioral psychology under the term <em>prospect theory</em>, most notably by Kahneman and Tversky. This is worth its own article.</p><h4>Utility models of time and risk preference</h4><p>We use “utility model” specifically to express how time and risk preference affect values. If you’re familiar with the economic literature the terms “utility model and “value model” tend to be used interchangeably, however we use utility model <strong><em>U()</em></strong> specifically as an additional function to which the value is passed to express time and risk preferences. Hence we can write <strong><em>V(sᵢ) = U(f(sᵢ))</em></strong> to distinguish the parts of the model that express risk tolerance and time preference. Risk tolerance and time preference modify the value computed by the value model, expressed as a discount factor. Even though both contribute to a discount factor, risk and time preferences are different phenomena. It suffices to say both can be elicited with the same techniques used for other preferences; the details deserve a separate article.</p><h4>Can the value model be learned?</h4><p>In principle, by observing the choices a person makes, they reveal their value model. Arguably revealed value models avoid biases that arise when asking people for their values. This is still an area of active research, often identified with reverse reinforcement learning and the challenges that such a theory that implies.</p><p>In summary by extending the economic concept of value by the methods of elicitation just mentioned we have a general approach when conventional models economic value, e.g. when dollars versus dollars are not adequate.</p><h3>V. Computing expected value from a value function.</h3><p>Once the value model is in hand we can borrow the computation of “expected value” from Economics that combines a value function, <strong><em>V(sᵢ)</em></strong> and it’s probability distribution, <strong><em>P(sᵢ)</em></strong> over the set of outcomes <strong><em>sᵢ</em></strong>. This is a way to reduce an uncertain value to a single number that can be used in comparisons.</p><p>The expected value for a discrete set of outcomes, <strong><em>sᵢ</em></strong> each with predicted probability <strong><em>P(sᵢ),</em></strong> is</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/702/1*U9NPCDH5ZK7hCHIFZWBecg.png" /></figure><p>To choose between two sets of outcomes, one described by one distribution <strong><em>P(sᵢ)</em></strong>, the other by <strong><em>P’(sᵢ),</em></strong> we should prefer the one whose predicted probability has a greater expected value. As probabilities not just any percentages or fractions will do; this is why we need “precise” probabilities from our model — -the term is “calibrated.” But we can go a step further: If both probabilities and values of a set of outcomes differ then their expected value reveals the comparison. How might they differ? As a function of the <em>decision</em>, of course.</p><h4>Making value tradeoffs is an optimization problem</h4><p>To pull these various themes together, the combination of predictive and value models generates an expected value, by which each alternative can be compared so that all aspects that need to be considered are included in making a choice; in other words, in a <em>decision model</em>. The various components of the decision model are interpretable, so the effects of values, their tradeoffs, uncertainties, risk, and time preference are explainable.</p><p>A graphic illustration of a decision model in the case of a binary outcome is the application of ROC curves to find the optimal tradeoff that balances true positive and false positive rates. The ROC curve is the probability model. The value model is the linear tangent line whose slope represents the tradeoff, and its intersection with the ROC curve sets the threshold that determines the decision policy that maximizes the value.</p><p>The ROC example can be generalized mathematically to the optimization of any decision model, where there may be multi-valued or continuous outcomes. These are solvable optimization problems constructed from two component models — -of probability and value.</p><h4>A Decision Model as a Bayes Network (or two)</h4><p>Mathematically the model when making a decision against an uncertainty can be written as an optimization (here written as maximization),</p><figure><img alt="formula to maximize expected value" src="https://cdn-images-1.medium.com/max/695/1*IXhxB8Ol32qTdI8zL8idfQ.png" /></figure><p>or equivalently, as this Bayes network. This may make it obvious if the math isn’t. Decision nodes are shown as rectangles, probabilities (random variables) as ovals, and value nodes as diamonds. Arrows show dependencies. With discrete-valued variables, the math here is simple; just enumerate the expected value for all alternatives and chose the best.</p><figure><img alt="Causal Bayes network of a decision with an uncertain state" src="https://cdn-images-1.medium.com/max/390/1*_ot22GiwGfUhL2oQrqcGtA.png" /></figure><p>This case is not very interesting: Where does the predictive model come in, you ask? A predictive model inputs a list of features, some of which may be known when making the decision. <strong>Thus we designate a subset of features as the information <em>i</em> known when the decision is made.</strong></p><p>Introducing the predictive model, we condition the prediction and the decision on the information features:</p><figure><img alt="Formula to maximize expected value given partial information" src="https://cdn-images-1.medium.com/max/695/1*WGQXOsDbYmH45Mpmulh5qg.png" /></figure><p>where<strong><em> V(d,s) </em></strong>is the value function, <strong><em>d(i)</em></strong> is the decision policy as a function of the observed features (think a lookup table) and <strong><em>P(s | i)</em></strong> is the model prediction of outcome <strong><em>s</em></strong> depending on the features <strong><em>i.</em></strong> <em>The machine learning model is embedded in the decision model.</em> This equation applies generally for any data science application.</p><p>To show this in a Bayes network we add the observed features as a node that conditions the random variable on which the value depends, as shown here:</p><figure><img alt="Causal Bayes network diagram of a decision under uncertainty with partial information." src="https://cdn-images-1.medium.com/max/388/1*M2D_TBQYrguDa_r2KELq5g.png" /></figure><p>“Solving” the decision model means finding the best choices — -the policy function <strong><em>d(i)</em></strong> — -that maximizes this expression, given <strong><em>P(s | i)</em></strong> and <strong><em>V(d,s):</em></strong> the prediction and value models. Written out, the equation says to take expectation over the predictive distribution of the ML model, conditional on the observed features, then for each combination of features make the choice with the highest expected value. In the fortunate case where the set of alternatives and observables are small enough to count on one hand, one can enumerate over all values to fill in entries in the policy table. <br>Similarly the just mentioned ROC curve example works when <strong><em>d</em></strong> is binary and <strong><em>s</em></strong> can be partitioned in two, which just reduces it to the discrete case.</p><h4>Extensions: Economic Models</h4><p>The equation for the decision model is for just a single-person decision-maker with one decision. What about multiple decisions and multiple decision-makers? Decision models are not limited to a single decision. They could be embellished with secondary decisions, to form a sequence, to collect more information, include other alternatives, and so on. The optimization and Economics literature are replete with more extensive models that extend this basic case — -</p><ul><li>There are a sequence of decisions, where earlier decisions may affect the information at subsequent ones.</li><li>The decision-maker must work within resource constraints — — “budgets” — -that create dependencies among decisions, in addition to those created by the value function.</li><li>There are multiple decision-makers, either with a common value function, or with competing values.</li><li>Or some combination of the three.</li></ul><h3>VI. What can one hope to achieve by doing this?</h3><ul><li>The Data Scientist (should we say “Decision Scientist”) needs to own the model formulation. The reason many Data Science models fail is just because their decision model is wrong.</li><li>Data Science ethical concerns and model fairness fall entirely within value modeling. Conversely, some conundrums in ethics are best resolved by making the valuing of the tradeoffs explicit. Fairness is something that can be modeled.</li><li>Despite the universality of decision modeling, there are widely pursued areas of Data Science where the value analysis, and the optimization problem are already well codified and implicit. Any area where A/B testing is widespread, think product recommendations or advertising, one assumes that the decision problem is settled, and improvements can best be found simply in reducing the uncertainty around current predictions. However, when A/B testing fails to deliver results, it’s likely that the decision model is wrong.</li></ul><h3>Additional thoughts</h3><blockquote>Decision modeling reduces rational choice to practice.</blockquote><ul><li>Rational choice as a founding principle — -that one’s actions should be driven by desired outcomes — -has been a long-standing theme in AI. Decision modeling reduces rational choice to practice. But for this to have effect, one also needs to drive adoption and this means generating trust in the model. The visibility into the components of the model improve explainability that helps in this.</li><li>These ideas are neither novel nor obscure. They have been around for half a century, in the economics, management science, and the behavioral psychology literature. Bad imitations of them reappear with depressing regularity in the popular business literature.</li><li>What about using AI to learn the value model, similarly to how the predictive model is learned? This comes under the topic of “reverse reinforcement learning”, and is the basis of a proposal to humanize AI, in Stuart Russell’s well-regarded book on the future of AI: “Human Compatible”.⁴</li><li>Decision modeling implies the data scientist needs to get involved in operational aspects of the business — yet another competency demanded of the data scientist! These “transformative” opportunities go beyond just looking for cost savings that can be obtained by introducing new tools and software, to disruptive (in a good way) value opportunities that affect the business model, but require intensive investments in analysis over much longer time-spans that a typical ML modeling project.</li></ul><p>[1] Brian Christian <em>The Alignment Problem</em> (2020) Norton Publishing. <br>[2] This example is attributed to Nick Bostrom, <em>Superintelligence </em>(2014) Oxford University Press.<br>[3] Douglas Hubbard, <em>How to Measure Anything: Finding the Value of Intangibles in Business</em> (2014) Wiley.<br>[4] Stuart Russell, <em>Human Compatible: Artificial Intelligence and the Problem of Control</em> (2010) Cambridge.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=4acdbafc04bf" width="1" height="1" alt=""><hr><p><a href="https://medium.com/decision-analysis/the-true-meaning-of-alignment-decision-modeling-in-data-science-4acdbafc04bf">The True Meaning of Alignment: Decision Modeling in Data Science</a> was originally published in <a href="https://medium.com/decision-analysis">Decision Analysis</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Simply, Just Python in the Cloud]]></title>
            <link>https://medium.com/data-science/simply-just-python-in-the-cloud-78274389ab00?source=rss-fdb3e1d9846e------2</link>
            <guid isPermaLink="false">https://medium.com/p/78274389ab00</guid>
            <category><![CDATA[azure-machine-learning]]></category>
            <category><![CDATA[vscode]]></category>
            <category><![CDATA[python]]></category>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[blob-storage]]></category>
            <dc:creator><![CDATA[John Mark Agosta]]></dc:creator>
            <pubDate>Wed, 28 Jul 2021 21:47:38 GMT</pubDate>
            <atom:updated>2021-07-30T06:00:24.798Z</atom:updated>
            <content:encoded><![CDATA[<h4>So you just want to run python at scale?</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*AfEmZqkexs5L_ZFo-tI6bQ.png" /><figcaption>Image by Author</figcaption></figure><blockquote><strong>tl;dr</strong></blockquote><blockquote>This is a “lean” tutorial of basics of running your code in Azure. With your data residing in storage alongside a VM in the cloud, without exploring the labyrinthine complexity of Azure, and using the newly-released VS-Code “Azure Machine Learning — Remote” extension,<strong> programming on the VM is as simple as developing code on your local machine</strong>, but with the scaling benefits of the Cloud. It’s the first step to exploiting what cloud computing with Azure Machine Learning(AML) has to offer.</blockquote><p>Cloud services offer many options, Azure more than others; but with such variety it’s hard to know where to start. Let’s assume you’ve already <a href="https://azure.microsoft.com/en-us/">subscribed to Azure</a>, and have started to learn your way around Azure’s web portal, portal.azure.com,---its universal graphical interface for setting up and managing web resources. The general management term for Cloud components, services, and products is &quot;resources.&quot; If this is not familiar to you, search for an online course in Azure Fundamentals that covers things like subscriptions, resource groups, regions, etc. <a href="https://datascience.harvard.edu/azure-tutorials-2020">Harvard&#39;s DS intro</a> so that you feel comfortable using the Azure web portal. Keep in mind most on-line training is targeted at enterprise users, with more detail than you&#39;ll need to know about to start. Assumedly you&#39;re interested in fast results at reasonable cost, and the enterprise user&#39;s overriding concerns also include availability, reliability and other &quot;ilities&quot; are not your immediate concern---you&#39;ll get there eventually. In this tutorial you will create a few Azure ML resources interactively in the Azure Portal, then do development by a combination of either VS-Code or command line tools. I assume you are facile with programming and working at the command line. I&#39;ll target specifically our task of moving your Python application to run in the Cloud, infested with my strong personal biases to avoid common pitfalls, based on my experience.</p><p>Why would you want to move your code to a machine in the Cloud? For scale. Azure has some huge machines, vast network bandwidth, and almost infinite memory. Surprisingly, applications with large computational loads may be faster on a single machine than on a cluster, and easier to set up. Of course if your aim is to run petabytes of data, or train billion-node neural networks there are suitable VM-cluster solutions. Either via cluster or large VM, it’s a great boon for any large scale problem, not only conventional machine learning applications, but for those large scale MCMC tasks, or training on GPUs.</p><h3>Cloud resources</h3><p>Cloud services consist essentially of three kinds of resources: <em>compute, storage, and networking.</em> The collection of your application resources are designated a “resource group” — -a logical construct for creating, tracking and tearing down a cloud application. Your resource group needs to include a <em>Machine Learning Workspace</em>, a convenient collection of resources that itself is a kind of resource. Find it here in the Azure’s graphical interface, portal.azure.com:</p><figure><img alt="Creating a Machine Learning Workspace in the Azure Portal" src="https://cdn-images-1.medium.com/max/626/1*gpgG0w-99WEQFE7u1dUAHw.png" /><figcaption>Image by Author</figcaption></figure><p>Go ahead and create one. The Portal prompts are kind enough to offer to create a new resource group and new component resources for it’s Storage account and others. Take the defaults and go ahead and create new component resources. Just remember the Workspace name, and use the same resource group name for additional resources you create in your application. If you make a mistake it’s easy to delete the whole resource group and start again.</p><h4>Compute</h4><p>In simple terms, <em>compute</em> consists of virtual machines (VMs), from tiny VM images to VMs with enormous RAM and GPUs, bigger than anything you would ever have at your desk. The Workspace doesn’t come with any default compute resources, since it gives you a choice of creating individual VMs, VM compute clusters, or using your existing compute resources. For this tutorial you need to create a “Compute instance” — -a single VM — -by choosing a size that meets your need. These Ubuntu VMs created in Azure Machine Learning Studio (AML) are managed VMs — -with most all software needed already installed and with updates managed for you. This takes the work of setting up and configuring the VM off your shoulders. Alternatively Azure still offers legacy “Data Science VMs” that come with extensive pre-installed ML tools, but except for some special purposes, (such as running Windows OS) AML Compute Instances can take their place.</p><h4>Storage</h4><p>Storage also is available at scale. But unlike with your desktop, primary storage is not a file system running on a disk local to the VM, but a separate component with an access API. One reason is that storage is permanent, but VMs come and go; start them when you need work done, then tear them down to save money. Storage consists of “blobs” with strong guarantees of persistence and redundancy. A key point in this tutorial is demonstrating how to integrate existing Azure Storage, that might come already loaded with your data.</p><h4>Networking</h4><p>Networking services connect <em>compute</em> and <em>storage.</em> Moreover networking in the Cloud consists of software-defined virtual networks that connect to your local “on-premise” equipment and your cloud resources. With cloud resources, Networking glues components together with combinations of local subnets, security services (firewalls), and public-facing IP addresses. For the simple case of a VM with connected storage, the associated network looks like your home network, with a public IP address, and NAT-ed addresses on a subnet. You won’t need to know much about this, since it is set up for you when you create a VM. We’ll get to that; but the place to start is storage, and getting your data onto it.</p><h4>Setting up your development environment</h4><p>The tools you’ll need to run your code in the cloud are:</p><ul><li>VS-Code — Microsoft’s open source IDE</li><li>git: Open source version control</li></ul><p>And two Azure resources:</p><ul><li>Azure Storage</li><li>An AML Workspace with a Compute Instance.</li></ul><p>The title illustration shows how these are cobbled together. An AML Workspace brings together an integrated collection of resources: You’ll need just two: the managed VM, and attached storage where it keeps the file system with your code. Additionally you’ll set up an Azure Storage account, or use an existing one for your data. This isn’t truly necessary, but learning to manage storage is good experience. Resource setup and management can be done entirely in the Portal. As for code, you move that to AML version it with git. Then you run your edit-run-debug cycle with <em>two</em> copies of VS-Code running locally. <em>The magic is that one copy is attached to the AML compute instance making it appear as if it&#39;s just another local instance.</em></p><p>My focus on using VS-Code for code development, along with two of its Extensions, the Azure ML Extension, and the Azure ML — Remote Extension. Install Extensions from the VS-Code “Activity Bar”, found in a column on far left, that look like this:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/308/1*6E9xP0UgCFSW37Zm8Lt0UA.png" /><figcaption>Image by Author</figcaption></figure><p>Use the Extensions icon to install the two VS-Code Extensions.</p><p>The Azure icon will be used to launch remote sessions.</p><p>As an open-source project, the momentum to improve and expand VS-Code is worth noting, as told by <a href="https://devblogs.microsoft.com/python/python-in-visual-studio-code-july-2021-release/">its dedicated blog.</a> Hard-core Python developers may have a preference for other dedicated Python tools such as <em>PyCharm </em>or <em>Spyder</em>, however in this tutorial I’m using Microsoft’s <a href="https://code.visualstudio.com/download">VS-Code</a> for its elegant project management, symbolic debugging, and Extensions, which integrate nicely with Azure. For these reasons I prefer it to Microsoft’s legacy Visual Studio IDE Tools for Python.</p><p><em>So install VS-Code if you don’t have it already.</em></p><p>Besides your Azure subscription, git, and VS-Code you&#39;ll need the az command line package, known as the <a href="https://docs.microsoft.com/en-us/cli/azure/">Azure command-line interface (Azure CLI)</a>, and its extension for Azure Machine Learning which work equivalently on either Windows or Unix shells. There are installers for it on the webpage.</p><p>The az package manages every Azure resource, and its <a href="https://docs.microsoft.com/en-us/cli/azure/reference-index?view=azure-cli-latest">az reference page</a> is a complete catalog of Azure. It&#39;s implemented as a wrapper for Azure&#39;s RESTful management API. You rarely need to refer to it directly since VS-Code Extensions will call into az for you, and offer to install it if it&#39;s missing. And the Portal exposes the same functionality. But each of these are syntactic sugar over command line tools, and you should be aware of both.</p><p>In the long run, the other reason to know about az is for writing scripts to automate management steps. Every time you create a resource in the Portal, it gives you the option to download a &quot;template&quot;. A template is just a description in JSON of the resource you&#39;ve created. Using az you can invoke ARM (the Azure Resource Manager) to automate creating it. ARM is a declarative language for resource management that reduces resource creation to a single command. Look into the az deployment group create... command for details.</p><p>Commands specific to Azure Machine Learning (AML) in the az extension need to be installed separately. az will prompt you to do this the first time you use az ml.</p><p>Don’t confuse the az CLI with the Azure python SDK that has overlapping functionality. Below is an example using the SDK to read from Storage programmatically. Most things you can do with the az CLI are also exposed as an SDK in the python azureml package.</p><p>Finally, to check that az is working try to authenticate your local machine your Cloud subscription. Follow the directions when you run</p><pre>$ az login<br></pre><p>You do this once at the start. Subsequent az commands will not need authentication.</p><h3>The Application</h3><p>Ok, you’ve got your code and data, and installed tools. Let’s begin! Assume you’ve gone ahead and got a new AML Workspace, the steps are</p><ol><li>Setup Storage,</li><li>Prepare your code,</li><li>Create an AML compute instance, and</li><li>Launch a remote VS-Code session.</li></ol><p>One option when you are done with your code is to deploy it by wrapping it in a “RESTful” API call as an <em>Azure Function Application.</em> This exposes your application as an “endpoint”, callable from the Web. Building an Azure Function could be the subject of another tutorial.</p><p><strong><em>First thing — Prepare your data. It pays to keep your data in the Cloud.</em> </strong>Storage, such as Azure blob storage is cheap, fast and seemingly infinite in capacity; it costs about two cents a gigabyte per month, with minor charges for moving data around. For both cost and network bandwidth reasons you are better off moving your data one time to the cloud and not running your code in the Cloud against your data stored locally. In short, you’ll create a storage account, then create a <em>container</em> — -the equivalent of a hierarchical file system — -of “Azure Data Lake Generation 2” (ADL2) blob storage. There are numerous alternatives to blob storage, and even more premium options to run pretty much any database software on Azure you can imagine, but for the batch computations you intend to run, the place to start is with ADL2 blob storage.</p><p>Blob storage appears as files organized in a hierarchical directory structure. As the name implies, it treats data as just a chunk of bytes, oblivious to file format. Use csv files if you want, but I suggest you use a binary format like parquet for its compression, speed and comprehension of defined data-types.</p><p>Perhaps your data is already in the Cloud. If not the easiest way to create storage is using the Portal — -”create a resource”, then follow the prompts. It doesn’t need to be in the same resource group as your Workspace, the choice is yours. The one critical choice you need to make is to select “Data Lake Storage Gen2”:</p><figure><img alt="This option converts blob storage into adl2 blob storage." src="https://cdn-images-1.medium.com/max/725/1*YvzQ3n_BMoCExuW0JHy9YQ.png" /><figcaption>Image by Author</figcaption></figure><p>Essentially this is blob storage that supersedes the original blob storage that was built on a flat file system. Note that “Azure Data Lake Gen 1” storage is <em>not</em> blob storage for reasons not worth going into and I will ignore it.</p><p>You can upload data interactively using this Portal page. Here’s an example for a container named “data-od.” It takes a few clicks from the top level storage account page to navigate to the storage container and path to get here. It contains file and directory manipulation commands much like the desktop “File Explorer.”</p><figure><img alt="Browsing the contents of storage in the Azure Portal" src="https://cdn-images-1.medium.com/max/714/1*QQDH5L-GVcubhaQxrivQDw.png" /><figcaption>Image by Author</figcaption></figure><p>Later I will show how to programmatically read and write data to storage.</p><p>In the lean style of this tutorial, I’ve included only the absolute minimum of the Byzantine collection of features that make up Azure ML. Azure ML has it’s own data storage, or more accurately, thinly wraps existing Azure Storage components as “DataSets,” adding metadata to your tables, and authenticating to Storage for you, which is nice. However, the underlying storage resource is accessible if you bother to look. This tutorial — -an end run around these advanced features — -is good practice even if only done once, to understand how Azure ML storage is constituted.</p><p><strong><em>Prepare your code</em></strong></p><p>Going against the common wisdom, I recommend you should convert your notebook code to Python modules once you get past initial exploratory analysis. If you are wed to using notebooks, VMs work just fine: Both Azure ML workspaces and Spark clusters let you run notebooks, but you will be limited going forward in managing, debugging, testing, and deploying your code. Notebooks are great for exploring your data, but when developing code for subsequent use one get’s tired of their limitations:</p><ul><li>They don’t play well with git. If there are merge conflicts between versions, it&#39;s a real headache to diff them and edit the conflicts.</li><li>They get huge once filled data and graphics. Either strip them of output before you commit them, to keep them manageable. Or convert them to html as a sharable, self-documenting document.</li><li>As your code grows, all the goodness of modular, debuggable object-oriented code is missing. Honestly the symbolic debugger in VS-Code is worth the effort of conversion, not to mention the manageability of keeping code organized by function in separate modules.</li><li>Python modules encourage you to build tests, a necessary practice for reliable extension, reuse, sharing and, productionizing code.</li></ul><p>So relegate notebooks to run-once experiments, and promote blocks of code to separate files that can be tested and debugged easily.</p><p>Jupyter generates a .py file from a notebook for you at the command line with</p><pre>$ jupyter nbconvert --to-py &lt;your notebook&gt;.ipynb</pre><p><strong><em>Rely on </em></strong><strong><em>git</em></strong></p><p>git deserves true homage. As your projects scale and you collaborate with others ---the things that make you valuable to your organization--- git is the comfortable home for your work. As any SW Engineer knows there&#39;s no greater solace and respite from the wrath of your colleagues if you&#39;ve &quot;broken the build&quot; to be able to revert to a working state with just a few commands. Having an documented history is also a gift you give to posterity. Arguably the complexity of git dwarfs the complexity of a hierarchical file system, but there&#39;s no shame of relying on Stack Overflow each time you step out of your git comfort zone. And believe me, there is a git command for any conceivable task.</p><p>Assume your code is in a git repository of your choice. git will be your conveyance to move code back and forth from cloud to local filesystems.</p><p><strong><em>Setting up cloud resources — an Azure Machine Learning (AML) Workspace</em></strong></p><p>I assume you’ve created an AML Workspace. To launch the web version there’s this button once you navigate to the Workspace:</p><figure><img alt="launch studio button" src="https://cdn-images-1.medium.com/max/137/1*K8zyF30Qs4U5oFG7r50wkw.png" /><figcaption>Image by Author</figcaption></figure><p><strong><em>A tour of the Workspace</em></strong></p><p>The Workspace collects a universe of machine learning-related functions in one place. I’ll just touch on a few that this tutorial uses. However the beauty of using VS-Code is that you don’t need to go here: Everything you need to do in the Studio can be done from the Azure Remote Extension.</p><ul><li><strong>compute instances</strong> Once inside the Workspace pane, on its left-side menu, there’s a “compute” menu item, which brings you to a panel to create a new compute instance. Create one. You have a wide choice of sizes — can you say 64 cores w/128G RAM? Or with 4 NVIDA Teslas? Otherwise it’s fine to just accept the defaults. You can start and stop instances to save money when they are not being used.</li><li><strong>code editing</strong> Under “notebooks” (a misnomer) there’s a view of the file system that’s attached to the Workspace. It is actually a web-based IDE for editing and running code in a pinch. It can include notebooks, plain python files, local git repositories and so on. This file system is hidden in the workspace’s associated AML storage, so it persists even when compute instances are deleted. Any new compute instance you create will share this file system.</li><li><strong>AML storage</strong> The Studio does not reveal its connected AML Storage Account, so consider it a black box where the remote file system and other artifacts are stored. For now work with external Azure storage — why? So you see how to integrate with other Azure services via storage. The brave among you can use the Portal to browse the default storage that comes with your workspace. The storage account can be found by its name that starts with a munged version of the workspace name followed by a random string.</li></ul><p>Remember the nice thing is that all this Workspace functionality you need is exposed at your desktop by VS-Code Extensions.</p><h3>The development cycle</h3><p>Let’s get to the fun part — -how these components work together and give a better experience. Notably the local edit-run-debug cycle for VS-Code is almost indistinguishable when using cloud resources remotely. Additionally both local and remote VS-Code instances are running simultaneously on your desktop, a much smoother experience than running a remote desktop server (“rdp”), or trying to work remotely entirely in the browser. How cool is that!</p><p><em>Let’s go through the steps.</em></p><h4>Running locally</h4><p>Authentication to the cloud is managed by the Azure ML extension to VS-Code; once you are up and running and connected to your Workspace you will not need to authenticate again.</p><ul><li>Start VS-Code in the local directory in your git repository where VS-Code&#39;s .vscode folder resides. (The launch.json file there configures run-debug sessions for each file. Learn to add configurations to it.)</li><li>Once you are satisfied the code runs locally, use git to commit and push it. If you commit notebooks, it&#39;s better to &quot;clean&quot; them of output first, to keep them manageably-sized.</li><li>Start the Azure ML compute instance from the Azure ML extension. First connect to your Workspace using the Azure Icon (see above) in the VS-Code Activity menu. <strong>You can spawn a remote-attached instance of VS-Code from the local VS-Code instance from the well-hidden right-click menu item “Connect to Compute Instance”:</strong></li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/385/1*KVDX7Km7BaVdTGOfSx9l-A.png" /><figcaption>Image by Author</figcaption></figure><p>This will both start the VM, and spawn a second VS-Code instance on your desktop connected to your remote Workspace file system!</p><h4>Running remotely</h4><p>Note that the terminal instance that appears in the remotely-connected VS-Code terminal window is running on your VM! No need for setting up a public key for “putty” or ssh to bring up a remote shell. Similarly the file system visible in VS-Code is the Workspace file system. How cool is that!</p><ul><li>First thing of course is to bring the repository in the Workspace file system current by using git to pull your recent changes. Then it&#39;s just edit-run-debug as normal, but with all compute and file access in the cloud.</li><li>With both VS-Code instances running on your local machine you can work back and forth between local and remote versions simultaneously, transferring code changes with git.</li><li>In the remote instance, the Azure ML Extension menus are disabled, naturally. It doesn’t make sense to spawn a “remote” to a “remote”.</li><li>When you are done spin-down the VM to save money. Your remote file system is preserved in Storage without you needing to do anything.</li></ul><p>If you are sloppy about moving files with git (you forgot something, or you need to move a secrets file), note that the remote-attached instance of VS-Code can view either the cloud or local file systems making it a &quot;back door&quot; to move files between them. Advisedly git is used to maintain consistency between code in the two file systems, so use this back door sparingly.</p><p>Alternately the remote session can be invoked in the Portal. Try opening a remote VS-Code session on your local machine from the AML Studio notebooks pane.</p><figure><img alt="An alternate way to launch VS-Code in a Workspace." src="https://cdn-images-1.medium.com/max/955/1*09VI_BHDvpDNnuwH54-viA.png" /><figcaption>Image by Author</figcaption></figure><p>If your compute instance is running, you can launch a <em>local</em> instance of VS-Code attached remotely to the Workspace’s <em>cloud</em> file system (not just the file that’s open in the Portal editor) and a shell running on the compute instance. But I think you’ll find it’s easier to spawn your remote session directly from your local VS-Code instance, than having to go to the Studio pane in Azure ML page in the Portal to launch it.</p><h3>Programmatic adl2 file access</h3><p>One change you’ll need to run your code in the cloud is to replace local file IO. The primary change to your code is to replace local file system IO calls with calls to the python SDK for Blob Storage. This works both when the code runs locally, or in the VM as you switch your application to work on data in the cloud.</p><p>There are several ways to access files for the VM. Here’s one way with the Azure python SDK that works with minimal dependencies, albeit it is a lot of code to write. Wrap this in a module, then put your efforts into the computational task at hand. The python SDK defines numerous classes in different packages. This class, DataLakeServiceClient is specific to adl2 blob storage. Locally you&#39;ll need to get these packages,</p><pre>pip install azure-storage-file-datalake <br>pip install pyarrow</pre><p>“pyarrow” is needed by pandas. Apache project “arrow” is a in-memory data format that speaks many languages.</p><p>In this code snippet, the first function returns a connection to storage. The second function locates the file, retrieves file contents as a byte string, then converts that to a file object that can be read as a Pandas DataFrame. This method works the same whether you reference the default storage that comes with the Azure ML workspace you created, or some pre-existing storage in other parts of the Azure cloud. For details see <a href="https://docs.microsoft.com/en-us/azure/storage/blobs/data-lake-storage-directory-file-acl-python">https://docs.microsoft.com/en-us/azure/storage/blobs/data-lake-storage-directory-file-acl-python</a></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*NLAECirR-f7P7oCLChIguA.png" /></figure><p>You can use the __name__ == &#39;__main__&#39; section to embed test code. It doesn&#39;t get run when you import the file. But as you develop other scripts that use this one, you can invoke this file with the path to a known blob to test it hasn&#39;t broken.</p><p>The DataLakeServiceClient authenticates to Azure Storage using an &quot;account key&quot; string that you recover from the Portal, in the Storage Account &quot;Access keys&quot; pane. An account key is an 88 character random string that works as a &quot;password&quot; for your storage account. Copy it from this pane:</p><figure><img alt="Where to find a storage account access key." src="https://cdn-images-1.medium.com/max/1005/1*bvu9ODfpnBp7Tlm-m6IYUg.png" /><figcaption>Image by Author</figcaption></figure><p>Since its confidential it shouldn’t get embedded in the code, and remains hidden in another file,secrets.py, in the local directory that contains only global variable assignments, in this case AZURE_STORAGE_ACCOUNT, STORAGE_ACCOUNT_KEY, and CONTAINER.</p><p>Place the file name in your .gitignore file, since it&#39;s something you don&#39;t want to share. This is a simple way to enforce security, and there are several better, but more involved alternatives. You can keep secrets in environment variables, or even better used Azure &quot;key vault&quot; to store secrets, or just leave secrets management up to Azure ML. Granted account keys are a simple solution, if you manage them carefully when individually developing code, but you&#39;ll want to button-down authentication when building code for enterprise systems.</p><p>Similarly there are more involved, “code-lite” ways to retrieve blobs. As mentioned Azure ML has it’s own thin wrapper and a default storage account created for the Workspace where it’s possible to store your data. But as I’ve argued, you’ll want to know how to connect to any Azure data, not just AML-managed data. To put it politely, AML’s tendency to “simplify” fundamental cloud services for data science doesn’t always make things simpler, and can hide what’s under the hood that makes for confusion when the “simple” services don’t do what you want.</p><h3>Other services, embellishments, and next steps</h3><p>For computational tasks that benefit from even more processing power, the cloud offers clusters for distributed computing. But before jumping to a cluster for a distributed processing solution (e.g. <a href="https://dask.org/">dask</a>), consider python’s multiprocessing module that offers a simple way to take advantage of multiple core machines, and gets around python’s single threaded architecture. For an embarrassingly parallel task, use the module to run the same code on multiple datasets, so each task runs in it’s own process and returns a result that the parent process receives. The module import is</p><p>from multiprocessing import Process, Manager</p><p>The Process class spawns an new OS process for each instance and the Manager class collects the instance&#39;s results.</p><p>As for going beyond one machine, the sky’s the limit. Apache Spark clusters seamlessly integrate coding in Python, R, SQL and Scala, to work interactively with huge datasets, with tools well-tailored for data science. Spark on Azure comes in several flavors, including third party DataBricks, and database-centric integration of Spark and MSSQL clusters branded “Synapse.”</p><h4>More machine learning features</h4><p>The point of this article has been the scaling opportunities available in Azure ML, but did I mention the ML features of Azure ML? In addition to the fun you’ll have up-leveling your development to scale, you’ve started up the learning curve to using Azure’s tools for data science. For tracking and comparing results of multiple runs AzureML integrates with MLFlow. See <a href="https://medium.com/swlh/how-to-setup-mlflow-on-azure-5ba67c178e7d">this article.</a> For managing data, AML &quot;DataSets&quot; embellish datafiles with metadata. There is even an <a href="https://github.com/Azure/azureml-sdk-for-r">SDK for R</a> similar to the python SDK. There are objects for experiments, runs, environments, and model management. &quot;AutoML&quot; automates supervised learning model selection and tuning. That&#39;s just a couple examples of how Azure ML repackages services to be a &quot;one stop&quot; shop for machine learning tasks.</p><h3>Azure Function deployment and beyond</h3><p>Azure can be used not only for running experiments at scale, but for building an application around your code. Cloud applications consist of collections of networked services, running on separate VMs. For convenience one avoids working at the “infrastructure” level and moves to “managed services” where the VMs are not exposed, just the interfaces are, as we did with managed compute instances. You can do the same thing with your code by publishing it to the cloud as a RESTful http endpoint. To your client it looks like a URL except instead of returning a web page it returns the result of your computation. A basic way to build this is by wrapping your code in an Azure Function. An <a href="https://docs.microsoft.com/en-us/azure/azure-functions/create-first-function-vs-code-python">Azure Function</a> typically runs in a few seconds, and doesn’t save state from one call to the next. (Saving state is where blob storage comes in, remember?) There are several more featureful alternatives for implementing http endpoints, based on Docker containers for instance, of course some available from the Azure ML Workspace, but all work as managed services, making implementation independent of the platform “infrastructure” it’s built on. How to use AFs deserves a separate tutorial, but to get you started, load the Azure Function extension into VS-Code and browse it’s features. Remember when I said that local management for everything comes with az? Well I wasn&#39;t telling you the whole truth. For AF you&#39;ll need to install separately the <a href="https://github.com/Azure/azure-functions-core-tools">Azure Functions Core tools</a> for commands that start with az func. VS-Code will prompt you for this install when you install the VS-Code Azure Functions extension.</p><p>Since you’re not put-off by writing code, you can proceed to integrate with the full gamut of cloud services: data bases, web hosting, advanced AI, and security — -much as your favorite on-line retail site is likely to have done when building an enterprise Azure application.</p><h4>The VS-Code remote extension source</h4><p><a href="https://github.com/microsoft/vscode-tools-for-ai">https://github.com/microsoft/vscode-tools-for-ai</a><br><a href="https://github.com/MicrosoftDocs/azure-docs/blob/master/articles/machine-learning/how-to-manage-resources-vscode.md">https://github.com/MicrosoftDocs/azure-docs/blob/master/articles/machine-learning/how-to-manage-resources-vscode.md</a></p><h4>Python SDK reference</h4><p><a href="https://azuresdkdocs.blob.core.windows.net/$web/python/azure-storage-file-datalake/12.0.0b7/azure.storage.filedatalake.html?highlight=datalakefileclient#azure.storage.filedatalake.DataLakeFileClient">https://azuresdkdocs.blob.core.windows.net/$web/python/azure-storage-file-datalake/12.0.0b7/azure.storage.filedatalake.html?highlight=datalakefileclient#azure.storage.filedatalake.DataLakeFileClient</a></p><p><em>Originally published at </em><a href="https://techcommunity.microsoft.com/t5/ai-customer-engineering-team/simply-just-python-in-the-cloud/ba-p/2591940">https://techcommunity.microsoft.com/t5/ai-customer-engineering-team/simply-just-python-in-the-cloud/ba-p/2591940</a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=78274389ab00" width="1" height="1" alt=""><hr><p><a href="https://medium.com/data-science/simply-just-python-in-the-cloud-78274389ab00">Simply, Just Python in the Cloud</a> was originally published in <a href="https://medium.com/data-science">TDS Archive</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[The need for rights to protect public facts against baseless claims]]></title>
            <link>https://johnmark54.medium.com/the-need-for-rights-to-protect-public-facts-against-baseless-claims-504ab7e240c5?source=rss-fdb3e1d9846e------2</link>
            <guid isPermaLink="false">https://medium.com/p/504ab7e240c5</guid>
            <category><![CDATA[propaganda]]></category>
            <category><![CDATA[public-health]]></category>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[first-amendment]]></category>
            <category><![CDATA[statistics]]></category>
            <dc:creator><![CDATA[John Mark Agosta]]></dc:creator>
            <pubDate>Sun, 07 Feb 2021 21:27:12 GMT</pubDate>
            <atom:updated>2021-02-10T01:10:30.612Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/626/1*uOIUjAWcH4c8vvQYp8-OPA.jpeg" /><figcaption>courtesy of FreePIk</figcaption></figure><p>The current NY Times article on <a href="https://www.nytimes.com/2021/02/06/business/media/conservative-media-defamation-lawsuits.html?action=click&amp;module=Top%20Stories&amp;pgtype=Homepage">“Lawsuits Take the Lead in Fight Against Disinformation”</a> makes me think its about time to enforce protections for certain kinds of speech in our democratic society. The article in question covers the legal fallout against abusive, anti-democratic claims of voter fraud, in a spate of cases by firms that were harmed by such accusations. But current legal remedies are limited. Why should we have to wait until after the damage is done for false information to be acted upon?</p><p>Besides the political debacle we just experienced around the past presidential election, this strikes a professional chord for me, when doing statistical analysis. I approach quantitative statistical analysis as a subtle art, to tease truth that will stand up to scrutiny out of raw, ragged facts. It’s hard to get right. There needs to be a better remedy to protect such individuals and facts they are duty-bound to reveal in the face of factions that will trample truth with baseless propaganda.</p><p>My heart goes out, in the recent case of Rebekah Jones, a data scientist who defied the Florida Department of Public Health’s blatant bullying to doctor COVID statistics. Her defiance in the name of public safety has l<a href="https://en.wikipedia.org/wiki/Rebekah_Jones">ed to her arrest</a>.</p><p>But what about preserving free speech you may ask? How can this be done without tarnishing one of the cornerstones of democracy? The place to start is with the fundamentals. Publishing information “with reckless disregard for the truth” isn’t a First Amendment right. The difficult question is what can we do about it without collateral damage to our current legal framework. Specifically the government avoids being an arbiter of truth, but leaves this instead to be threshed out in the marketplace of ideas. To quote Grynbaum in today’s Times, in discussing Roberta Kaplan’s defense of Mary Trump:</p><blockquote>Some First Amendment lawyers say that an axiom — the best antidote to bad speech is more speech — may no longer apply in a media landscape where misinformation can flood public discourse via countless channels…</blockquote><p>The current remedies against false information are tightly defined, as well they should be. There exists a legal case if one can show actual damage, as the current case being brought by Smartmatic Voting Machines against right wing media, for harming its business. The problem is that litigation as a remedy is slow, and after the fact. Of course there are also injunctions about clear and present danger, about statements that incite violence. Again until the clamor on social media clearly reached this level, social platforms didn’t feel they were on firm legal ground to restrict content. The bar should be lowered to restrict clearly baseless claims with such predictable consequences.</p><p>Those who propagandize with false information should not be give the presumptive right that their claims have value. The point of propaganda is to accumulate political power. Attempting to co-opt the media should not be confused with “participating in the marketplace of ideas.”</p><p>Similarly media should have better protected rights to not be abused as forums for false information, and <em>public officials whose efforts would be harmed should have rights to label dissident claims as baseless</em>. Anyone would still be free to say or publish what they want, but they would have no right to insist that spewing nonsense would be given a forum. This is just an incremental change, from a retrospective view, to one where “facts have rights” before obvious damages occur with public consequences.</p><p>The novelty would be to give teeth to calling out “baselessness” — a counterclaim that has no facts to support it. Current examples of baseless claims are legion. That there was widespread voter fraud. That common sanitary and public health precautions against spread of disease are unnecessary or harmful. That projections about epidemic spread are false. That human activity does not cause climate change. And so on —the abuse of information which is necessary for the future of a safe society. The hard question is who gets to say a claim is obviously false information? I wouldn’t rely on the courts. We already have numerous non-political fact-checking groups who tally these. The change I propose would be to empower public (non-political, scientific) officials, presumably by certain panels or bodies — those who the public rely on to be informed — to make this determination —<em> of no credible evidence to the counterclaim.</em> Such presumption of “clear and future danger” would carry weight similar to the “clear and present danger” test of allowed speech. Our science today is good enough to predict future danger. Those media that have policies to not carry baseless claims would have rights — similar to the current “Section 230” safe harbor provisions, to remove them without the presumption of violating free speech. Those media that decided to carry such claims would have to deal with the presumption that they are trafficking in harm.</p><p>Our legal legacy of relying after the fact, on remedies to damages is a good thing, <strong>but what if the “damages” are the destruction of the society that should enforce them? What will be the remedy then?</strong></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=504ab7e240c5" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[The PCA Trick with Time-Series]]></title>
            <link>https://medium.com/data-science/the-pca-trick-with-time-series-d40d48c69d28?source=rss-fdb3e1d9846e------2</link>
            <guid isPermaLink="false">https://medium.com/p/d40d48c69d28</guid>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[anomaly-detection]]></category>
            <category><![CDATA[timeseries]]></category>
            <category><![CDATA[data-networking]]></category>
            <category><![CDATA[making-sense-of-big-data]]></category>
            <dc:creator><![CDATA[John Mark Agosta]]></dc:creator>
            <pubDate>Mon, 21 Dec 2020 05:16:27 GMT</pubDate>
            <atom:updated>2020-12-29T17:57:27.617Z</atom:updated>
            <content:encoded><![CDATA[<h4><a href="https://towardsdatascience.com/tagged/making-sense-of-big-data">Making Sense of Big Data</a></h4><h4>PCA can be used to reject cyclic time-series behavior, and this works for anomaly detection.</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/742/1*Q5BQh6uH5fSulXAeZoAlVA.png" /><figcaption>Top: Noisy periodic signals, Bottom: Reduced anomaly signal. (All images by author).</figcaption></figure><p>Detecting an anomaly typically means thresholding a signal, to alarm when the signal is out-of-range. For something like an assembly line, where tolerances are precise, the difference between normal and abnormal is clear. But network traffic has a noisome characteristic that makes this hard. It varies with large daily cycles as customers’ activity peaks and wanes. One could try to tenderly tease out this daily variation by modelling it. However this trick using Principal Component Analysis (PCA) avoids that hard work.</p><p><strong>The periodic components embedded in a set of concurrent time-series can be isolated by Principal Component Analysis (PCA), to uncover any abnormal activity hidden in them.¹</strong> This is putting the same math commonly used to reduce feature sets to a different purpose. PCA and similar dimension reduction methods may be part of your every-day data science toolkit, but I bet you never thought of using PCA for this.</p><h4>Here’s the problem it solves.</h4><p>Say we’ve got a cluster of networked machines running a distributed web service. Customer traffic flows through various machines, depending on their function, and each records a set of performance variables, such as memory and cpu usage. Because of the common external origin of the traffic — -to wit, the customer load — -these variables share periodic fluctuations on top of whatever innate source of variation one may seek to detect. That variation could be due to software or hardware faults among the machines — -the stuff that you’re monitoring for.</p><p>Any stationary time-series can be expressed as sums of sines and cosine functions, in what is called a Fourier expansion. These periodic functions are a natural way to analyze stationary time-series in a fundamental way that will become clear. My first foray to solve the problem was to extract the Fourier expansion explicitly, much as a tool like <a href="https://github.com/facebook/prophet">Facebook’s Prophet forecaster</a> does. With the extracted components in hand, I then attempted to weave together a set responsible for the external traffic, as a predictor of normal traffic, deviations from which could be classified as abnormal. But, all this effort turns out to be unnecessary.</p><h4>The linear algebra of PCA.</h4><p>Recall from linear algebra that one may construct a <em>basis </em>for any vector space, meaning a set of independent vectors that span the space, of which any other vector in the space is a unique linear combination. All bases for the space have the same size: This size defines the <em>dimension </em>of the space. PCA discovers a basis with two desirable properties. First, among the many possible arbitrary bases, the eigenvalues of the subspace of the first <em>d </em>PCA components minimize the reconstruction error of the original signals. More precisely, for any <em>d </em>less than the full dimension, this subspace <em>d </em>gives the best <em>d</em>-dimensional approximation of the feature vectors. As a bonus, since the covariance matrix whose eigenvalues are recovered is symmetric, the PCA eigenvectors are orthogonal. This property is well known, previously as the Karhunen-Loeve expansion, originally published in the 1940s.</p><p>Secondly, note the Fourier expansion also forms an orthonormal basis of eigenvectors invariant to translation. This means that the components don’t change as the time-index is shifted. This is important because, besides removing the concern about how the starting point of the time-series affects the results (it doesn’t) it means the Fourier components are the eigenvectors of stationary time-series. In fact, any basis of a stationary time-series can arguably be expressed as a combination of Fourier components.</p><p>A bit of math shows how the translation invariance arises. Any exponential function has this desired property</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/189/1*srCNh02o2nssqSvOaXOZgw.png" /></figure><p>where <em>x </em>can be a complex number, and <em>h</em> is a translation. In words, this states the exponential function is the same before and after displacement by <em>h</em>. Of course, for our purposes, this property carries over to the familiar pair of trigonometric functions because</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/247/1*3zOljZN6LnW6OTyFtaNmZw.png" /></figure><p>Thus trigonometric functions are eigenvectors in translation, meaning any periodic signal has the same eigen-decomposition if shifted in time. Consequently — -and one might say, magically, if there is a common periodic component to the set of time-series variables, PCA will find it and the Fourier components will appear in the PCA results.</p><h4><em>How this works.</em></h4><p>In our case, we are interested in the PCA maximum variation subspace as a way to identify the components of the periodic signal.<strong> In short, instead of throwing away the least variation basis vectors among the <em>d</em> as we do for dimension reduction, we will ignore the largest (in the sense of variance) basis vectors until we are left with those that are normally “quiet”, where the anomalies appear.</strong></p><p>Our original signals are considered as vectors, one for each time-series, with length equal to <em>n </em>the number of time-steps. This <em>n </em>is not to be confused with the dimension of the set of signal features — -think of the signals matrix as the table of <em>p</em> features of <em>n </em>samples.</p><p>Here’s the <strong><em>R</em></strong> code to convert thesignals time-series matrix into the pca_timeseries components. prcomp() is part of the built-in <em>mva</em> <strong><em>R</em></strong> package.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/08c9c084465bee64ffbc6aaf9bbb6a01/href">https://medium.com/media/08c9c084465bee64ffbc6aaf9bbb6a01/href</a></iframe><h4>Let’s have a look.</h4><p>We can see how PCA can reject periodic noise with a simulation. We create a combination of few sine and cosine features, with a pinch of noise, and run it through a PCA decomposition. The result are another set of orthogonal time-series that pick up the periodicities of the input set in the first two components, leaving the residual in the rest. Here are five noisy features with different phases, and their five PCA components:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/865/1*6_EixQ8l-fN8UgQ5yhU1dQ.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/77/1*4Y9LvxeWBTiDF5Zx6KODTw.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/865/1*FfWwJb05-sAC7ClBnJxVDQ.png" /><figcaption>PCA isolates the common Fourier component among timeseries.</figcaption></figure><p>Now we corrupt the features with a step function that is largely invisible in the original features. Looking at the basis vectors returned by PCA, we see the periodic parts expressed in the first two components . Once PCA has captured these, what’s left is a residual where the step function is obvious.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/865/1*tH0inTDxgt6UM3IxGYNO1w.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/77/1*4Y9LvxeWBTiDF5Zx6KODTw.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/865/1*8c3_7ONlbRwpiWuCIn17Qw.png" /><figcaption>An anomaly introduced in the cyclic features is evident in the PCA components.</figcaption></figure><p>This works because there’s a common cause to the periodic features. They are driven by the external customer cycles. Most prominent is a daily cycle, but weekly and hourly cycles can appear also — any period that is a fraction of the time-series length.</p><h4>Back to the application.</h4><p>I took data of several performance measures from one cluster, for which we know an significant anomaly occurred over the course of a day within an observation period of two weeks. See the illustration at the top of the article with the title. We can see in the top graph the daily cycles in the different features, in different degrees before and after the anomaly. The point is that the PCA component — shown in black in the bottom graph — preserves the anomaly while rejecting the various levels of daily cycles.</p><p>Once the PCA model is trained on “normal” traffic we can take the last PCA component that we’ve kept and threshold it. In comparison to thresholding the individual measures directly the result is a significant reduction is spurious detections, so that we can tighten the threshold to increase sensitivity.</p><h4>Runs on a real dataset.</h4><p>Let’s look at an example where this method works. I ran the anomaly detector on suspect traces for 400 machine clusters, a few of which I knew had obvious anomalies. To cross-validate, the detector was trained on the normal, but cyclic traffic before the known anomalous event, then the detection rate was measured on the day when the anomaly occurred, and separately on the week after, once operations returned to normal. I found that the PCA-based anomaly detector clearly picked up the anomaly in known cases, but more importantly was almost silent during traffic in normal times. Here’s an example clearly identified as anomalous:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Hluhbfh3IazIS6YteQHlcA.png" /><figcaption>Example of Anomalous Memory (left) and Cpu (right) features</figcaption></figure><p>Here’s a case where the data has strong cyclic, noisy features, but no anomaly is detected:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*YN6_-Z58nbBReBkoUF0aKg.png" /><figcaption>Memory (left) and Cpu (right) features not classified as anomalous</figcaption></figure><p>More interestingly, on examination of previously unseen cases, many are anomalous but were missed entirely by existing detectors. This can be put in statistical terms. The lack of spurious detection when no anomaly occurs means the false positive rate is small, in this case actually below measurable levels for the one-week testing trace I ran. The other impressive news is that the detection rate for anomalies, the true positive rate, was higher than current detection methods.</p><h4>tl;dr</h4><ol><li>The PCA-Trick is practical for revealing anomalous common effects while rejecting strongly cyclic daily customer traffic.<br>2. Applying “train and test” for anomaly detection analysis on a known event demonstrated substantially improved hourly anomaly detection rates at zero measurable false positive rates. <br>3. Similar analysis of 400 clusters for the same episode discovered numerous anomalous clusters not caught by current tools.</li></ol><blockquote>¹ Lakhina, A., Crovella, M., &amp; Diot, C. (2004). Diagnosing network-wide traffic anomalies. ACM SIGCOMM computer communication review, 34(4), 219–230.</blockquote><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=d40d48c69d28" width="1" height="1" alt=""><hr><p><a href="https://medium.com/data-science/the-pca-trick-with-time-series-d40d48c69d28">The PCA Trick with Time-Series</a> was originally published in <a href="https://medium.com/data-science">TDS Archive</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Decision Quality for Data Scientists: Lessons from Decision Professionals]]></title>
            <link>https://medium.com/decision-analysis/decision-quality-for-data-scientists-lessons-from-decision-professionals-5a52100a7b41?source=rss-fdb3e1d9846e------2</link>
            <guid isPermaLink="false">https://medium.com/p/5a52100a7b41</guid>
            <category><![CDATA[automation]]></category>
            <category><![CDATA[bounded-rationality]]></category>
            <category><![CDATA[decision-making]]></category>
            <category><![CDATA[data-science]]></category>
            <dc:creator><![CDATA[John Mark Agosta]]></dc:creator>
            <pubDate>Mon, 04 May 2020 00:11:42 GMT</pubDate>
            <atom:updated>2020-05-04T00:23:44.926Z</atom:updated>
            <cc:license>https://creativecommons.org/licenses/by-nc-sa/4.0/</cc:license>
            <content:encoded><![CDATA[<h4>Six principles that organize what I’ve learned over the years building models.</h4><figure><img alt="Source: Climate Change Risk Analysis for Projects in Kenya and Nepal  Van der Vat, M.P., J.E. Hunink, D. Stuparu" src="https://cdn-images-1.medium.com/max/1024/1*3HQ4d2EQcrt0OW76MlNllA.png" /><figcaption>Photo credit: Climate Change Risk Analysis for Projects in Kenya and Nepal</figcaption></figure><blockquote>Decision Quality systematizes the best practices for modeling decisions, which are practices that the field of Data Science sorely needs to adopt.¹</blockquote><h3>Introduction</h3><p>Let’s say you are a Data Scientist at your first meeting with your client. Or you are a software engineering or program manager thinking of starting a project where you can apply what you know of Machine Learning (ML) or Artificial Intelligence (AI). Perhaps analysts on your team suggested a project based on their preliminary data analysis and the promise of almost unlimited data. Now people are looking to you to know where to start, how to model the situation, and how to apply automation that achieves business value.</p><p>Where do you start? What questions do you ask? Bringing together analysis of business decision making and Data Science, this article draws on principles that have been shown to improve Decision Quality in practice. <em>Decision Quality</em> codifies the experience of decision professionals who advise on decisions that place companies at risk, span millions of dollars, and span multiple years. It’s based on principles derived from Decision Theory, of techniques to quantify and improve decision making. This article extends the Decision Quality framework to Data Science practice by focusing on the decisions to be automated.</p><blockquote>Data Science should start with identifying decisions, as one of five aspects, as implemented by the Decision Quality framework.</blockquote><p>Current Data Science has demonstrated notable successes driven by data-rich online services for recommendation, search, and the like. And this is just the beginning. But extending Data Science to new applications is hampered by an incomplete theory. As the name of the field implies, the focus has been to chase after sources of data with the presumption that past practice forms an adequate precedent for new applications. The common myth that the Data Science process starts with the data is a non-starter. It starts instead with identifying the decision, as one of five aspects, as implemented by the Decision Quality framework.</p><p>There are two ways in which decision-making principles can be applied to Data Science. The first is to decide on a project to undertake an automated application. “What is the automated application to build?” The other is to incorporate Decision Quality principles within an automated software system. Automated decisions occur numerous times — -at “internet” speed — -and that’s the theme of this article— and what Data Science should really be about.</p><h3>Beginning with The Essence of a Decision</h3><p>So back to day-zero when you’re standing in front of a blank whiteboard with your client.</p><figure><img alt="Source: https://en.wikipedia.org/wiki/Decision_tree" src="https://cdn-images-1.medium.com/max/220/0*Z6iMWtr0gbJvp-U9.jpg" /><figcaption>Example of group facilitation for framing decisions.</figcaption></figure><p>One may have a presumption that there’s a data-rich problem to be automated, but before one starts working on a data set or even searches out available data, framing the problem should begin by examining the set of choices or possible actions comprising the “essence of the decision.² ” A clearly specified decision leads to a course of action that drives model design and implementation.</p><h4>What is a decision?</h4><blockquote>A decision implies</blockquote><blockquote>1. A tangible change — something moved or modified; a resource committed,</blockquote><blockquote>2. At a point in time,</blockquote><blockquote>3. By a decision-maker, typically identified with a person,</blockquote><blockquote>4. That can be defined by a set of uncertainties and consequences.</blockquote><p>A decision’s choices form an “either-or,” and when resources are allocated, reversing a decision would incur some cost. Starting analysis with the decision, as opposed to say the prediction, clarifies the analysis, as we shall see. We concentrate on what Economists call “single-person decision theory,” which implies trade-offs need to be made based on resource constraints, uncertain outcomes, and values. We don’t consider the extension when a decision-maker faces an adversary where it becomes instead an application of game theory.</p><p>Decisions often gain attention when there’s a fear that something is going — -or is about to — -go wrong. But better yet, considering circumstances creatively may suggest alternatives before a crisis arises. The framework present here can be applied to any decision, but it only makes sense to invest time and effort in analysis when the consequences justify the automation effort.</p><h4>Aspects of formulating a decision and what pitfalls they avoid</h4><p>Succinctly, <em>Decision Quality </em>means applying these principles:</p><p>1. <strong>Solve the right problem.</strong> Frame a scope that encompasses the problem fully. Don’t confuse analysis of the data-set with analysis of the actual situation.</p><p>2. <strong>Consider a true range of alternatives.</strong> For instance, a model intended just to advocate for certain outcomes will overlook valid alternatives and be subject to bias.</p><p>3. <strong>Search out curated and relevant information,</strong> not only as data, but including the proper use of judgment combined with data.</p><p>4. <strong>Value consequences of decision outcomes,</strong> making trade-offs explicit, bounded by ethical constraints. Incomplete optimization of trade-offs can lead to models with perverse, unintentional consequences.</p><p>5. <strong>Apply solid reasoning</strong>. Use methods that de-bias information. Similarly, be able to measure “signal” versus “noise” so that consequences can be evaluated based on probabilities of outcomes. Models that optimize predictive accuracy and ignore probability of error can lead to wrong choices.</p><p>6. <strong>Implement what has been decided.</strong> Entering into analysis without the means and commitment to follow through doesn’t achieve anything.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/552/1*WjLzmSaR-KCQrrlNZimWcQ.png" /><figcaption>From C. Spetzer (2016), <em>Decision Quality.</em></figcaption></figure><h3>How things go wrong</h3><p>Of course, all of this sounds obvious! How could one deny any one of these six principles? For the longest time, I have applied them implicitly, only to be greeted by the uncomprehending looks of my fellow team members. The benefit of a shared framework is to foster a common language and get everyone on the same track.</p><p>The principles of <em>Decision Quality</em>, by bringing decision theory to practice, reveal flaws in decision making in the large and the small. The pressures that impede successful software automation are the same that underlie failures in major organizational decisions. We can learn from how well-intentioned, “rational³ ” persons managed to make bad decisions on a global scale — take the case of the Cuban Missile Crisis. It began when the U.S. discovered intermediate-range ballistic missile sites in Cuba just off the U.S. coast. Both the young, inexperienced U.S. President and a Soviet military bureaucracy blundered along, from one bad decision to another, each fearing the other would start an unintentional nuclear war. The Soviets had wanted to address an imbalance they perceived in the “missile gap.” The U.S. response to Cuba obtaining armaments that threatened the U.S was to run a naval blockade. The Soviets saw the blockade justifiably as an act of war.</p><blockquote>Organizational theory explains why the Cuban Missile Crisis evolved in ways that were not rational.</blockquote><p>Why did both sides act in ways that were obviously inconsistent with their ultimate desires to contain the conflict? Both sides’ leadership suffered from the military organizations operating “by the book” that avoided short-term organizational risks but raised the risk of the situation spinning out of control. Organizational theory explains why the Crisis evolved in ways that were not rational. Russian Intelligence, typically expert at covert action, unwittingly delegated construction of missile installations, resulting in sites that were visible and easily identifiable from overflights. Subsequently, the U.S. Navy ran the blockade on the high seas rather than in coastal waters, preserving its tactical advantage, but aggravating the risk of inciting escalating military confrontation that the U.S. leadership desperately wanted to avoid. In both cases, there was a complete disconnect between desired outcomes and actions, poor use of intelligence, poor consideration of viable alternatives, and implementation of inappropriate policies.</p><p>For thirteen days both sides remained at a standoff, at the same time desperately hoping to avoid escalation, until, in short, both sides made concessions and backtracked from their course of action. The Crisis is studied as a textbook case of organizational decision making. For those curious, a detailed analysis is offered by <em>The</em> <em>Essence of Decision </em>[Allison, 1999], which uses the Crisis to illustrate different models of decision making. Without going into too much detail here, Allison starts with the same <em>rational </em>model, borrowing from decision theory — whose origins are identical to the basis for Decision Quality. Allison develops next an <em>organizational theory</em> model which explores the limitations of organizations as they “satisfice³ ” by reacting with a standard repertoire of operating procedures to circumstances. The bounds on rationality⁴ that cause organizations to fail are in principle the same as occur in complex automated systems built into software.</p><h4>Automating Decisions with Software</h4><blockquote>The question is how to make the automated application more rational — to incorporate the Principles into automation itself.</blockquote><p>We can draw on the lessons of organizational decision-making to reveal strong parallels with the challenges of automating decisions that plague Data Science. At one level the implementation of Data Science projects can be analyzed as just one more example of an organizational process: do I incorporate such-and-such an automated application into the business or not? This implies applying Decision Quality to business choices as it is applied in conventional areas. At a different level, the more interesting question for Data Science is to approach intelligent software as a problem of how to automate decisions. “Intelligence”, specifically in the sense of Artificial Intelligence, has the goal of acting rationally⁵. Hence better Decision Quality guides one how to create more intelligent applications. Fortunately for the sake of this analysis, decision-making and its foibles are universal. We can make some initial observations of what organizational failures imply about decision making failures in general, and how they might apply to decisions made by software systems.</p><p>1. Just as decisions made in organizations are subject to time and resource constraints, software systems are constrained by inadequate alternatives, information, and lack of consideration of decision consequences.</p><p>2. Systems, similar to individuals, fall into patterns of re-using techniques (think off-the-shelf software algorithms) whose properties are well known rather than exploring the problem fully for an appropriate solution.</p><p>3. The consequence of bounded rationality is that systems are designed to down-play uncertainty, limit search, and sub-optimize toward immediate objectives.</p><p>In short, “premature optimization⁶ ” and improper risk avoidance are endemic also in Data Science applications. Such failings are not new to system design; however, the opportunities Data Science creates for more intelligent automation opens a new field of application.</p><h3>Object Lessons from “The Wild”</h3><p>The following case studies of data science projects applied to business problems I’ve experienced in my professional life illustrate where questions of Decision Quality arose, and had they been raised, would have made a difference.</p><h4>Object Lesson 1: “On Time and In Full.” Is that the right problem?</h4><p>As a Data Science team, we were engaged by a C-level executive at a goods manufacturing firm. He had raised concerns about one of his remote manufacturing plant’s ability to deliver against orders <em>on-time and in full (OTIF), </em>as indicated by data collected in the plant’s MRP system — or so we thought. His request — the model framing we adopted — was to see how well we could use state-of-the-art ML tools to predict OTIF based on the orders received. Investigation revealed that the accuracy of the OTIF numbers predicted by the model stood at a modest 60%. In part this was due to the negligible variation in the dependent OTIF variable; there was hardly any effect to predict. This also was because the prediction was made on features available only up to the time the order was received, and much of the variability in observed OTIF was due to variation in the manufacturing process <em>after</em> receipt of the order.</p><p>Further investigation showed that OTIF variation was more often negative than positive; that is, goods often were delivered <em>before</em> the promised order delivery date! As a consequence of manufacturing’s desire to avoid the risk of missing deadlines, manufacturing runs were scheduled immediately on receipt of the order, then goods kept in inventory until needed. Consequently, inventories grew out of control, taxing storage capacity. Put into perspective, this overemphasis on OTIF prediction while overlooking inventory holding costs obscured the actual decision trade-off. Fortunately once revealed, this finding was welcomed by the client.</p><p>This case highlights the lack of application of several Decision Quality principles. When swimming through large quantities of data it’s important to “reach bottom” and discover how the data is grounded in actual physical, business processes. Had we scrutinized the entire manufacturing process initially, we might have begun instead by <em>(1) solving the right problem.</em> This would have been apparent had we canvassed the organization to understand the <em>(4) values attached to consequences</em> of the manufacturing process. Then we could have traced consequences back to the relevant decisions: In principle outcomes point to the decisions that drive those outcomes and hence must be included in the model. In this case, the <em>(2) true decision alternatives</em> are when to schedule production, which determines the trade-off between holding inventory and meeting OTIF deadlines.</p><p>It is likely that had we instead learned a model that determined the trade-off between inventory-holding and OTIF , we could have used inventory holding times and quantities — the <em>(3) relevant information given this decision</em> — to produce both a more accurate and relevant model.</p><blockquote>Any problem worth analysis at its core involves evaluating trade-offs, so —</blockquote><p><em>(1) solving the right problem</em> means including the consequences of trade-offs. Furthermore since inaccuracy is part of the nature of any statistical prediction, quantification of risks due to inaccuracies determines what decisions should be made. The risks determine the trade-offs. A solitary focus on accuracy is not an application of <em>(5) apply sound reasoning,</em> since it overlooks the probability of outcomes that will be inevitably inaccurately predicted.</p><p>A <em>confusion matrix</em> tabulates the probabilities of predicted versus actual outcomes. Cases where predicted and actual correspond fall on the diagonal and the count of these entries indicates the model accuracy. Off-diagonal entries indicate errors. The confusion matrix for a binary prediction has four outcomes, like this:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*fC-Wo5LE9u-P_wMjKlk-XA.png" /><figcaption>A Confusion Matrix shows the four outcomes of a binary prediction</figcaption></figure><p>This simple comparison shows the difference between accuracy and a risk-based value function as an objective. Consider evaluating a binary classifier’s confusion matrix’s four entries, “true positive” <em>(TP),</em> “true negative” <em>(TN),</em> “false positive” <em>(FP),</em> and “false negative” <em>(FN).</em> As mentioned, accuracy counts only the true elements and ignores the errors. Assuming correct predictions incur zero cost, properly one should evaluate value, <em>v, </em>by assigning costs specific to the two error terms:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/314/1*Je-tfGEpoYAfx0uCOZIHvA.png" /></figure><p>where<em> FP</em> is the count of spurious predictions — false positives, <em>FN</em> is the count of missing predictions — false negatives.<em> cs</em> is the unit cost of a spurious prediction, and <em>cm</em> is the unit cost of a missed prediction. Note in the OTIF case how costs differ, of missing one order delivery and having to store extra stock to assure the delivery is met.</p><p>This example also highlights the distinction between an operational Key Performance Indicator (KPI), in this case, OTIF, and the eventual business goals that a model needs to incorporate. So pursuing OTIF as the dependent variable was “barking up the wrong tree.”</p><p>In conclusion a thorough business analysis — direct discussions with parties involved — could have set a proper scope before commencing on predictive approaches. With the consolation that the client gained valuable insights from the analysis, had all the facts been available at the outset of the work and had the cited Decision Quality principles been applied at that time, the work might have resulted in a better model.</p><h4>Object Lesson 2: Sales Opportunity Propensities: What to do next?</h4><p>“Opportunity Scoring” applies an ML classifier (think logistic regression) that assigns a score to predict closing a sales opportunity. It promiscuously incorporates features of the deal(dollar amount, product, etc.), the customer, the sales team, and the business climate, and anything else that improves accuracy. These opportunities are characteristically large deals for enterprise customers, involving the efforts of the sales team over the course of months to sometimes years. In short, a score near one implies an opportunity likely to win compared to a low score, near-zero, implying a lost sale.</p><p>I’ve been a member of Data Science projects attempting quarter-to-quarter sales Opportunity Scoring that achieve remarkable accuracy across sales portfolios of large multinational companies . The data covers thousands of opportunities using hundreds of features. <strong>Despite the claims of accuracy and comprehensive approach, the usefulness of these models is suspect, since no attention was paid to how the prediction applies to the actions of the sales team.</strong> Properly, an appeal to Decision Quality starts by asking what actions — <em>(2) consider a valid range of alternatives</em> — should be modeled.</p><p>In one case, almost all the “opportunities” considered were routine subscription renewals that are most often extended by the customer with little effort by the sales team. The high overall accuracy of renewal predictions compared to new sales obscures the predictive accuracy on the smaller number of new accounts, which attracts the majority of sales team efforts, and where most value of the model is. By not breaking out new sales separately most of the model’s usefulness is lost.</p><p>But speaking of usefulness, more importantly, the idea of Opportunity Scoring fundamentally confounds prediction of the outcomes modeled, in this case the binary sales outcome “succeeds or fails” with informing available alternatives. A useful model’s predictions should indicate which choice leads to the best-expected outcome, rather than just predicting the “highest scoring” outcome based on recent history. Is it best that the sales team contact the client, in person, or by email at this time? Should they engage their legal or IT resources to advance the deal? And as for <em>(1) proper framing of the problem</em>, this implies also modeling temporal aspects. For example depending on the status of a sale, it may be better to postpone efforts to close the deal until next quarter, to not eat one’s seed corn, and instead to focus on those opportunities near completion. Predictions not predicated on conditions specific to the sale are pointless: If the model would predict a high opportunity score, does that imply less effort is necessary, since the sale would close anyway, or does it imply the need for more effort, to “make the prediction come true”?</p><p>Considering the right alternatives makes it possible to identify what qualifies as <em>(3) curated and relevant information </em>i.e. model features for these alternatives. With the huge number of available features that one can conceivably add to one’s data set, one should be cautious about including those that would empirically improve prediction, but which no argument of relevance can be made. The hazard is that a prediction’s accuracy may be driven by a spurious correlation that is not guaranteed to persist when the model is put into use. This is roughly what happened with the Sales Opportunity model: Scores were best predicted by the features of the sales team’s makeup assigned to the deal. When model predictions failed subsequently in later quarters, it was evident that the sales team variable was a surrogate for the strategy behind sales quota setting for that team. The efficacy of teams depended on how they set their sales quotas. That did not appear in the model, but caused quarter to quarter discrepancies and the model failed to generalize.</p><h4>A Final word about Implementation</h4><p>We haven’t said much about commitment to making the right choice: <em>(6) Implement what has been decided.</em> This is where automated decisions differ from human-implemented decisions since, clearly an automated decision is not in a person’s hands. Automation raises engineering challenges that often require orders of magnitude more effort than the Data Science modeling they implement. Building an “end-to-end” system is where most work occurs. The determinants of engineering success are out of scope for this article, but caveats apply. Ethical concerns arise when a decision is automated that should take into account human judgment, and this is covered by <em>(4) properly valuing consequences.</em></p><h3><strong>Do we need a new term for “Data Science”?</strong></h3><p>Armed with this new set of principles we can carve out a new approach to the design and formulation of data-driven automation. Borrowing from the terms “Artificial Intelligence” and “Decision Analysis” suggests <em>Decision Intelligence.⁷ </em>I’m using the term specifically to mean incorporating Decision Theory in the practice of Data Science practice, to create intelligent data-driven systems. Just as the decision to automate is subject to Decision Analysis, the same fundamental principles apply to decisions incorporated in data-driven automation. The arguments I’ve given here will sound old-hat to my Decision Analysis colleagues, albeit dressed up in new terminology. In the end, the result is a comprehensive method based on Decision Quality for Data Science to advance, mature, and broaden as a field.</p><h4>Endnotes</h4><p>[1] Thanks to discussions at DAAG 2019, and various teams at Microsoft: Particular acknowledgments are due to Dave Matheson, Brad Powley, Somik Raha, Tomas Singliar, and Carl Spetzler, for contributions, valuable comments, and guidance. The inspiration for this article comes from C. Spetzler, H. Winter, J. Meyer. (2016). <em>Decision Quality: Value Creation from Better Business Decisions, </em>NJ:<em> </em>Wiley.</p><p><a href="#_ftnref1">[2]</a> We borrow the term from the similarly-titled book, G. Allison, P. Z. (1999). <em>Essence of Decision 2nd Ed.</em> NY: Pearson, on theories of decision making in public policy. Some may find it far-fetched to link this to our current automation concerns, however this shows how universal these principles are.</p><p><a href="#_ftnref2">[3]</a> Bounded rationality results in “satisficing.” The term <em>satisficing</em>, invented by H. Simon implies finding a satisfactory, if non-optimal solution of a decision maker with limited resources and adversity to risk, subject to the constraints of a bureaucratic hierarchy. Satisficing is a consequence of bounded rationality. He introduced the term in the ’50s. As for how it applies to AI, I recommend looking at this recently re-issued version of his seminal text: H. A. Simon, (2019). <em>The Sciences of the Artificial.</em> MA: MIT Press.</p><p>[4] To quote H. Simon, <em>[Bounded] Rationality denotes a style of behavior that is appropriate to the achievement of given goals, within the limits imposed by given conditions and constraints. </em>The author goes on to explain why bounded rationality applies equally to behavior of individuals and of organizations.See H. Simon (1972) <a href="http://innovbfa.viabloga.com/files/Herbert_Simon___theories_of_bounded_rationality___1972.pdf">“Theories of Bounded Rationality”,</a> Ch 8 in C.B. McGuire &amp; R. Radner Eds., <em>Decision and Organization, </em>North Holland Publishing Co.</p><p><a href="#_ftnref3">[5]</a> Fortunately there is a contingent in the AI community that advances this approach. See Russell, S., &amp; Welfald, E. (1991). <em>Do The Right Thing</em>, MA: MIT.</p><p><a href="#_ftnref4">[6]</a> Attributed to D. Knuth: “Premature optimization is the root of all evil.” The sense of the aphorism is that one should avoid detailed performance improvements until those that best contribute to the entire system operation become clear.</p><p><a href="#_ftnref7">[7]</a> A nod to <a href="https://medium.com/@kozyrkov">Cassie Kozyrkov</a> for <a href="https://towardsdatascience.com/introduction-to-decision-intelligence-5d147ddab767">proposing the term</a> “Decision Intelligence.” We could riff off the term “Data Science” and call it “Decision Science” but the term is not new: It has been used to refer to a kind of policy analysis, as in government response to climate change. See for example: R. J. Lempert, (2002) “A new decision sciences for complex systems” <em>Proc Nat. Aca Sci, </em>99(suppl 3) 7309–7313. <a href="https://doi.org/10.1073/pnas.082081699">https://doi.org/10.1073/pnas.082081699</a>.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=5a52100a7b41" width="1" height="1" alt=""><hr><p><a href="https://medium.com/decision-analysis/decision-quality-for-data-scientists-lessons-from-decision-professionals-5a52100a7b41">Decision Quality for Data Scientists: Lessons from Decision Professionals</a> was originally published in <a href="https://medium.com/decision-analysis">Decision Analysis</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
    </channel>
</rss>