<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Patrick Hall on Medium]]></title>
        <description><![CDATA[Stories by Patrick Hall on Medium]]></description>
        <link>https://medium.com/@jphall_22520?source=rss-45328ad3cf8d------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/2*dYkGaf6_1W6cO162WLBqcw.jpeg</url>
            <title>Stories by Patrick Hall on Medium</title>
            <link>https://medium.com/@jphall_22520?source=rss-45328ad3cf8d------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Fri, 20 Nov 2026 10:25:55 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@jphall_22520/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[From GLM to GBM (Part 2)]]></title>
            <link>https://medium.com/data-science/from-glm-to-gbm-part-2-7045e3fd52a2?source=rss-45328ad3cf8d------2</link>
            <guid isPermaLink="false">https://medium.com/p/7045e3fd52a2</guid>
            <category><![CDATA[credit]]></category>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[python]]></category>
            <category><![CDATA[fairness]]></category>
            <dc:creator><![CDATA[Patrick Hall]]></dc:creator>
            <pubDate>Tue, 11 Aug 2020 18:21:00 GMT</pubDate>
            <atom:updated>2020-08-11T18:21:00.187Z</atom:updated>
            <content:encoded><![CDATA[<h4>The Business Value of a Better Model</h4><p>Authors: Patrick Hall and Michael Proksch, Ph.D.</p><h3>TL; DR</h3><ul><li>This post gives a technical overview of transitioning from trusted generalized linear models (GLMs) to newer gradient boosting machines (GBMs) while actually considering known risks, compliance requirements, and business impact.</li><li>It comes with <a href="https://nbviewer.jupyter.org/github/jphall663/interpretable_machine_learning_with_python/blob/master/glm_mgbm_gbm.ipynb">code</a>.</li><li>Check out Michael Proksch’s Part 1 <a href="https://towardsdatascience.com/from-glm-to-gbm-5ff7dbdd7e2f">post</a> and the H2O.ai <a href="https://www.h2o.ai/blog/from-glm-to-gbm-part-2/">post</a>.</li></ul><h3>Introduction</h3><p>In <a href="https://towardsdatascience.com/from-glm-to-gbm-5ff7dbdd7e2f">Part 1</a>, we proposed better revenue and managing regulatory requirements with machine learning (ML). We made the first part of the argument by showing how gradient boosting machines (GBMs), a type of ML, can match exactly, then exceed, both the technical merits and the business value of popular generalized linear models (GLMs) using a straightforward insurance example.</p><p>Part 2 of this blog uses a more realistic and detailed credit card default scenario to show how monotonicity constraints, Shapley values and other post-hoc explanations, and discrimination testing can enable practitioners to create direct comparisons between GLM and GBM models. Such comparisons can then enable practitioners to build from GLM to more complex GBM models in a step-by-step manner, while retaining model transparency and the ability to test for discrimination. In our credit use case, we show that a GBM can lead to better accuracy, more revenue, and that the GBM is also likely to fulfill model documentation, adverse action notice, and discrimination testing requirements.</p><p>Some bankers have recently voiced skepticism about artificial intelligence (AI) in lending — and rightly so. (See: <a href="https://www.americanbanker.com/opinion/ai-models-could-struggle-to-handle-the-market-downturn">https://www.americanbanker.com/opinion/ai-models-could-struggle-to-handle-the-market-downturn</a> or <a href="https://www.americanbanker.com/opinion/dont-let-ai-trigger-a-fair-lending-violation">https://www.americanbanker.com/opinion/dont-let-ai-trigger-a-fair-lending-violation</a>.) To be clear, we’re <em>not</em> advocating for brain-dead AI hype. We hope to present a judicious, testable and step-by-step method to transition from GLM to GBM in Part 2 of this post. Perhaps obviously, we feel that GBMs can model the real-world and credit risk better than GLMs. We also think ML can be done while preserving extremely high-levels of transparency and while keeping algorithmic discrimination at bay.</p><p>Part 1 of this article has already shown that GBMs might not only be more accurate predictors, but when combined with Shapley values, they can also be more accurate for explanations and attributions of causal impact to predictors. To build off Part 1, we now want to showcase the potential transparency and business benefits presented by a transition from GLM to GBM in lending. For full transparency (and hopefully reproducibility) we use the <a href="https://archive.ics.uci.edu/ml/datasets/default+of+credit+card+clients">UCI credit card data</a>, which is freely available from the UCI machine learning dataset repository, and <a href="https://nbviewer.jupyter.org/github/jphall663/interpretable_machine_learning_with_python/blob/master/glm_mgbm_gbm.ipynb">open source h2o-3 code</a>. The credit card dataset contains information about 30,000 credit card customer’s demographic characteristics and payment and billing information. The dependent variable to predict is payment delinquency. Our use case will compare a GLM to several different GBMs from a technical and business perspective.</p><h3>Machine Learning Can Drive Business Impact</h3><p>In order to show the business value of GBMs compared to GLMs, we trained several different algorithms (GLM, Monotonic GBM (MGBM), GBM, and a Hybrid GBM). Here’s a short summary of those models:</p><ul><li><strong>GLM:</strong> Elastic net penalized logistic GLM.</li><li><strong>MGBM:</strong> 46 trees, max. depth 3, all variables monotonically constrained.</li><li><strong>GBM:</strong> 44 trees, max. depth 5, no monotonic constraints.</li><li><strong>Hybrid:</strong> 50 trees, max. depth 14, learned monotonic constraints for some variables.</li></ul><p>As you’ll see below, there’s a large difference in projected revenue across these models in our tests. Why? The models judge risk differently. A GLM is not able to reflect non-linear relationships and interactions between variables without manual adjustments. GBMs do inherently model non-linear relationships and interactions between variables. In the past, GLMs were considered highly transparent, and as such, preferred for business purposes. GBMs were considered to be black-boxes in the past. But using monotonicity constraints can give GBMs much of the same inherent interpretability as GLMs, and without sacrificing accuracy. Table 1 provides an overview of the different test models and their capabilities.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/909/1*zXpIbfOyPo0alk3Su2GzYA.png" /><figcaption><strong>Table 1: Overview of models and their capabilities. “+” indicates the presence of a capability. “-” indicates the absence of a capability. Capabilities are measured relative to the GLM.</strong></figcaption></figure><p>All models were selected by grid search and evaluated using validation AUC. The outcome can be seen in Table 2. The GLM model has the lowest AUC score with 0.73, while the best GBMs reach an AUC of 0.79. Great … but what does that mean for a business?</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/911/1*se3Qoe-0jgghaVGg9cF6CA.png" /><figcaption><strong>Table 2: Model simulation outcomes. Arrows indicate the direction of improvement for each measurement. Currency is $NT.</strong></figcaption></figure><p>To assess the business impact of each model, we make a few basic assumptions as summarized in Table 3. A credit card customer, which we accurately classify as a delinquent customer, is neutral to costs. An accurately classified customer that is not delinquent will bring $NT 20,000 of lifetime value, classifying a customer incorrectly as delinquent will cost $NT 20,000 in lifetime value, and incorrectly extending credit to a delinquent customer will lead to write-offs of $NT 100,000.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/909/1*3_HM7z1U5IYZOCuKM7XieA.png" /><figcaption><strong>Table 3: Assumptions used to estimate business impact in Table 2. Currency is $NT.</strong></figcaption></figure><p>Based on those assumptions, the outcome of the GLM model shows the lowest revenue of $NT 7.9M. The model with the highest impact is the Hybrid GBM, with a business value of $NT 19.22M, almost 2.5 times of the GLM model!</p><p>How is this possible? The Hybrid GBM model is much better at avoiding those costly false negative predictions. This simple example shows that the transition from GLM to GBM models can significantly increase business value and reduce risk. Now, let’s have a look at how to compare GLMs to GBMs from a technical perspective, so you can decide how and when to transition from GLMs to GBMs.</p><h3>Better Model, Same Transparency</h3><p>It is true that GLMs have exceptional interpretability. They are so interpretable because of their additive monotonic form and their low-degree of interacting variables. Believe it or not, GBMs can also have extremely high interpretability. By the judicious application of monotonicity and interaction constraints, GBM users can now bring domain knowledge to bear on a modeling task, increase interpretability, avoid overfitting to development data, and make judicious decisions about when to use what kind of model. (E.g.: <a href="https://xgboost.readthedocs.io/en/latest/tutorials/monotonic.html">https://xgboost.readthedocs.io/en/latest/tutorials/monotonic.html</a>, <a href="https://xgboost.readthedocs.io/en/latest/tutorials/feature_interaction_constraint.html">https://xgboost.readthedocs.io/en/latest/tutorials/feature_interaction_constraint.html</a>.)</p><h4><strong>Monotonicity Constraints</strong></h4><p>Users can often specify the same monotonicity of variables found in GLMs in a GBM. In Figure 1 (right)<strong>,</strong>the GLM-modeled behavior of the feature PAY_0, or a customer’s most recent repayment status, is monotonic increasing. As PAY_0 values become larger, probability of default also becomes larger. In Figure 1 (right), the MGBM models the same behavior using a positive monotonic constraint — just like the GLM! As PAY_0 increases under the GBM, probability of default also increases.</p><p>What’s different is the functional form of the GLM versus the functional form of the GBM. The GLM is restricted to a logistic curve in this case, whereas the GBM can take on an arbitrarily complex stair-step form that also obeys the user-supplied monotonicity constraint. Additionally, Figure 1 shows how the GBM and Hybrid models behave for PAY_0 without monotonicity constraints. Which looks more realistic to you? The histogram and mean behavior of PAY_0 in Figure 1 (left) can help you determine which model fits the data best. The goal is to match the red mean target line as weighted by the histogram. The good news is, no matter which functional form you prefer, you can now use it in an interpretable way that reflects business domain knowledge.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*wRM7jfCcSPTnULqytiZymg.png" /><figcaption><strong>Figure 1: Histogram of PAY_0 with the mean value of the target, DEFAULT_NEXT_MONTH, by PAY_0 in red (left). Partial dependence and ICE estimated behavior of each model (right). Weighing the mean behavior by the frequency of data in the left figure helps gauge the fit of the models on the right. This figure also allows for a meaningful comparison of a simple model to more complex models.</strong></figcaption></figure><h4><strong>Post-hoc Explanation</strong></h4><p>How are we able to see the way that the GBM modeled PAY_0? Through what’s known as partial dependence and individual conditional expectation (ICE) plots [1], [2]. In Figure 1 (right), partial dependence (red) displays the estimated average behavior of the model. The other ICE curves in the plot show how certain individuals behave under the model. By combining the two approaches, we get a chart that displays the overall and individual behavior of our model. Partial dependence and ICE are just one example of post-hoc explanation techniques, or processes we can run on a model after it’s trained to get a better understanding of how it works.</p><p>Shapley values, a Nobel-laureate technique from game theory, provide additional and crucial insights for the GBM models [3]. When applied to tree-based models like GBM, Shapley values are a highly accurate measurement of how variables contribute to a model’s predictions, both overall and for any individual customer. This is incredibly helpful for interpretability as it enables:</p><ul><li>Analysis of overall business drivers of model predictions (Figure 2).</li><li>Descriptions of predictions for individual customers, and potentially generation of adverse action notices (Figure 3).</li><li>Comparison of GLM coefficients and contributions to GBM Shapley values (Figures 2 and 3).</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/634/1*3p38hOqrTo2TaIzmw9ZSBA.png" /><figcaption><strong>Figure 2: Absolute Pearson correlation to DEFAULT_NEXT_MONTH and average feature contributions for each model. GLM and GBM contributions can be interpreted in the log odds space. This figure allows for a meaningful overall comparison of simple models to more complex models.</strong></figcaption></figure><p>Just like Figure 1 shows how the treatment of PAY_0 changes from a GLM to different GBMs, a direct comparison between the overall (Figure 2) and per-customer (Figure 3) variable contributions for GLM and GBM is now possible. In Figure 2, we can see how each model treats variables from an overall perspective, and compare simple models (e.g., Pearson correlation, GLM) to the more complex models. In Figure 3, we can see how each model arrived at its prediction for three individual customers. All this enables a direct comparison of GLM and GBM treatment of variables, so you can both adequately document GBMs and make decisions about the transition to GBM with confidence! Moreover, the per-customer information displayed in Figure 3 could also provide the raw data needed for adverse action notices, a serious regulatory consideration in credit lending.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*DPN-7-XWIAyxEFyE4vzvGQ.png" /><figcaption><strong>Figure 3: Variable contributions to predictions for individuals at selected percentiles of GLM model predictions. This type of analysis allows for meaningful comparisons of how simpler and more complex models treat individual customers, and potentially for the generation of adverse action notices.</strong></figcaption></figure><p>Another crucial aspect of model diagnostics that must be conducted under several federal and local regulations in the US is discrimination testing. Of course, if certain types of discrimination are found, then they must be fixed too. The transparency of GLMs is often helpful in this context. However, the constraints and post-hoc explanation steps outlined above, make finding and fixing discrimination in GBM models much easier than it used to be. Moreover, the concept of the multiplicity of good models in ML — where a single dataset can generate many accurate candidate models — presents a number of options for fixing discrimination that were often not available for GLMs. (See: <a href="https://www.youtube.com/watch?v=rToFuhI6Nlw">https://www.youtube.com/watch?v=rToFuhI6Nlw</a>.)</p><h4>Discrimination Testing</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/909/1*79gBcMXKXXM2JQOi9TGfog.png" /><figcaption><strong>Table 4. Female (protected) vs. male (control) discrimination measures for the tested models. This table allows for the comparison of simple and more complex models.</strong></figcaption></figure><p>In our credit card example, the GBM is tested for discrimination using measures with long-standing legal and regulatory precedent: adverse impact ratio (AIR), marginal effect (ME), and standardized mean difference (SMD). Those results are available in Table 4. Luckily, between men and women, there is little evidence of discrimination for any of our models. However, if discrimination was found, GBMs may actually present more options for remediation than GLMs.</p><p>Basically, GBMs just have more knobs to turn than GLMs, leaving more wiggle room to find an accurate and non-discriminatory model. In addition to variable and hyperparameter selection, researchers also have put forward potentially compliant adversarial approaches for training non-discriminatory ML models [4], and GBMs now offer users monotonicity and interaction constraints that can help fix discriminatory model outcomes. Likewise, the post-hoc explanation techniques described above can also be used to understand drivers of algorithmic discrimination and to validate their removal from GBM models.</p><h3>Machine Learning and Compliance</h3><p>Many government agencies have telegraphed likely future ML regulation, or outside of the US, started to implement such regulations. It’s important to note that US government watchdogs are not saying ML is forbidden. Generally speaking, they are saying make sure your ML is documented, explainable, managed, monitored, and minimally discriminatory. Arguably, the steps outlined in Part 2 provide a blueprint for explainability and discrimination testing with GBM, which should in turn help with aspects of model documentation. Moreover, most large financial institutions already have model governance and monitoring processes in place for their traditional predictive models. These could potentially be adapted to ML models.</p><p>Of course, it’s really not the place of non-regulators to opine on what is, and what is not, compliant with regulations. So, have a look for yourself to see what some US government agencies are thinking:</p><ul><li><a href="https://www.finra.org/sites/default/files/2020-06/ai-report-061020.pdf">Artificial Intelligence (AI) in the Securities Industry</a></li><li><a href="https://www.cftc.gov/media/2846/LabCFTC_PrimerArtificialIntelligence102119/download">A Primer on Artificial Intelligence in Securities Markets</a></li><li><a href="https://www.whitehouse.gov/wp-content/uploads/2020/01/Draft-OMB-Memo-on-Regulation-of-AI-1-7-19.pdf">Office of Management and Budget Draft Guidance for Regulation of Artificial Intelligence Applications</a></li><li><a href="https://www.ftc.gov/news-events/blogs/business-blog/2020/04/using-artificial-intelligence-algorithms">Using Artificial Intelligence and Algorithms</a></li><li><a href="https://www.consumerfinance.gov/about-us/blog/innovation-spotlight-providing-adverse-action-notices-when-using-ai-ml-models/">Innovation spotlight: Providing adverse action notices when using AI/ML models</a></li></ul><h3>Momentum for Machine Learning</h3><p>Outside of government, some financial services organizations are already claiming to use machine learning in regulated dealings and researchers are publishing on GBM and Shapley values for credit lending applications. For instance, in 2018 Equifax <a href="https://www.prnewswire.com/news-releases/new-patent-pending-technology-from-equifax-enables-configurable-ai-models-300701153.html">announced</a> their Neurodecision system, “a patent-pending machine learning technology for regulatory-compliant, advanced neural network modeling in credit scoring.” Since 2018, Wells Fargo has also introduced several machine learning techniques for model validation, including LIME-SUP [5], explainable neural networks [6], and a number of additional model debugging <a href="https://ww2.amstat.org/meetings/jsm/2019/onlineprogram/AbstractDetails.cfm?abstractid=303053">methods</a>.</p><p>In 2019, Bracke et al. at The Bank of England published an explainable AI use case for credit risk featuring GBM and Shapley values [7]. Later the same year, Bussman et al. published a similar piece, introducing a GBM and Shapley value example in the journal <em>Credit Risk Management </em>[8]. In March of 2020, Gill et al. published a mortgage lending workflow based on monotonically constrained GBM, explainable neural networks, and Shapley values, that gave careful consideration to US adverse action notice and anti-discrimination requirements [9].</p><h3>Conclusion</h3><p>It now appears possible to take cautious steps away from trusted GLMs to more sophisticated GBMs. The use of constraints, post-hoc explanation, and discrimination testing enables you to compare GBMs to GLMs. These techniques may very well enable compliance with adverse action notice, discrimination, and documentation requirements too. And with a little luck, GBMs could lead to better financial outcomes for consumers, insurers, and lenders. As all this momentum and hype mounts for machine learning in regulated financial services, we hope that Parts 1 and 2 of this post will be helpful for those looking to responsibly transition from GLM to GBM.</p><h3>References</h3><ol><li>Friedman, J., Hastie, T., &amp; Tibshirani, R. (2001). <em>The Elements of Statistical Learning</em> (Vol. 1, №10). New York: Springer series in statistics. URL: <a href="https://web.stanford.edu/~hastie/ElemStatLearn/">https://web.stanford.edu/~hastie/ElemStatLearn/</a>.</li><li>Goldstein, A., Kapelner, A., Bleich, J., &amp; Pitkin, E. (2015). Peeking inside the black box: Visualizing statistical learning with plots of individual conditional expectation. <em>Journal of Computational and Graphical Statistics</em>, <em>24</em>(1), 44–65. URL: <a href="https://arxiv.org/pdf/1309.6392.pdf">https://arxiv.org/pdf/1309.6392.pdf</a>.</li><li>Lundberg, S. M., &amp; Lee, S. I. (2017). A unified approach to interpreting model predictions. In <em>Advances in neural information processing systems</em> (pp. 4765–4774). URL: <a href="http://papers.nips.cc/paper/7062-a-unified-approach-to-interpreting-model-predictions.pdf">http://papers.nips.cc/paper/7062-a-unified-approach-to-interpreting-model-predictions.pdf</a>.</li><li>Zhang, B. H., Lemoine, B., &amp; Mitchell, M. (2018, December). Mitigating unwanted biases with adversarial learning. In Proceedings of the 2018 AAAI/ACM Conference on AI, Ethics, and Society (pp. 335–340). URL: <a href="https://arxiv.org/pdf/1801.07593.pdf">https://arxiv.org/pdf/1801.07593.pdf</a>.</li><li>Hu, L., Chen, J., Nair, V. N., &amp; Sudjianto, A. (2018). Locally interpretable models and effects based on supervised partitioning (LIME-SUP). <em>arXiv preprint arXiv:1806.00663</em>. URL: <a href="https://arxiv.org/pdf/1806.00663.pdf">https://arxiv.org/pdf/1806.00663.pdf</a>.</li><li>Vaughan, J., Sudjianto, A., Brahimi, E., Chen, J., &amp; Nair, V. N. (2018). Explainable neural networks based on additive index models. <em>arXiv preprint arXiv:1806.01933</em>. URL: <a href="https://arxiv.org/pdf/1806.01933.pdf">https://arxiv.org/pdf/1806.01933.pdf</a>.</li><li>Bracke, P., Datta, A., Jung, C., &amp; Sen, S. (2019). Machine learning explainability in finance: an application to default risk analysis. URL: <a href="https://www.bankofengland.co.uk/-/media/boe/files/working-paper/2019/machine-learning-explainability-in-finance-an-application-to-default-risk-analysis.pdf">https://www.bankofengland.co.uk/-/media/boe/files/working-paper/2019/machine-learning-explainability-in-finance-an-application-to-default-risk-analysis.pdf.</a></li><li>Bussman, N., Giudici, P., Marinelli, D., &amp; Papenbrock, J. (2019). Explainable AI in credit risk management. <em>credit risk management (December 18, 2019)</em>. URL: <a href="https://papers.ssrn.com/sol3/papers.cfm?abstract_id=3506274">https://papers.ssrn.com/sol3/papers.cfm?abstract_id=3506274</a>.</li><li>Gill, N., Hall, P., Montgomery, K., &amp; Schmidt, N. (2020). A Responsible Machine Learning Workflow with Focus on Interpretable Models, Post-hoc Explanation, and Discrimination Testing. <em>Information</em>, <em>11</em>(3), 137. URL: <a href="https://www.mdpi.com/2078-2489/11/3/137">https://www.mdpi.com/2078-2489/11/3/137</a>.</li></ol><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=7045e3fd52a2" width="1" height="1" alt=""><hr><p><a href="https://medium.com/data-science/from-glm-to-gbm-part-2-7045e3fd52a2">From GLM to GBM (Part 2)</a> was originally published in <a href="https://medium.com/data-science">TDS Archive</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Strategies for Model Debugging]]></title>
            <link>https://medium.com/data-science/strategies-for-model-debugging-aa822f1097ce?source=rss-45328ad3cf8d------2</link>
            <guid isPermaLink="false">https://medium.com/p/aa822f1097ce</guid>
            <category><![CDATA[privacy]]></category>
            <category><![CDATA[fairness]]></category>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[security]]></category>
            <category><![CDATA[machine-learning]]></category>
            <dc:creator><![CDATA[Patrick Hall]]></dc:creator>
            <pubDate>Fri, 08 Nov 2019 21:45:15 GMT</pubDate>
            <atom:updated>2020-03-29T21:19:03.345Z</atom:updated>
            <content:encoded><![CDATA[<h3>Real-World Strategies for Model Debugging</h3><h4>Systematic ways to test and fix machine learning models</h4><p><strong>Contents:</strong></p><ol><li><strong>Introduction<br></strong>1.1<strong> </strong>A Quick Note on Trust and Understanding<br>1.2 Example Problem and Dataset</li><li><strong>Detection Strategies<br></strong>2.1 Sensitivity Analysis<br>2.2 Residual Analysis<br>2.3 Benchmark Models<br>2.4 Security Audits for ML Attacks</li><li><strong>Remediation Strategies<br></strong>3.1 Data Augmentation<br>3.2 Noise Injection and Strong Regularization<br>3.3 Model Editing<br>3.4 Model Assertions<br>3.5 Unwanted Sociological Bias Remediation<br>3.6 Model Management and Monitoring<br>3.7 Anomaly Detection</li><li><strong>Further Reading and Conclusion</strong></li><li><strong>References</strong></li></ol><p>—</p><h3>1. Introduction</h3><p>So, you’ve trained a machine learning (ML) model. You did everything right: cross-validation, early stopping, grid search, monotonicity constraints, and regularization. It’s accurate and stable in out-of-time test data, and better than the linear model it’s replacing. You even wrapped your model in a Docker container with all its dependencies for your data engineering and information technology (IT) pals. Time to deploy? Not so fast. Current best practices for ML model training and assessment just don’t tell us how to find and fix all the nasty things that can blow up in a real-world ML system. If only these systems could be tested and debugged like regular applications … enter: <em>model debugging</em>. Model debugging is an emergent discipline focused on finding and fixing problems in ML systems. Model debugging attempts to test ML models like code (because they are usually code) and to probe complex ML response functions and decision boundaries to systematically detect and correct accuracy, fairness, and security problems in ML systems.</p><h4>1.1 A Quick Note on Trust <em>and</em> Understanding</h4><p>In ML, trust and understanding are similar but not the same. Unpacking that difference helps me think through model debugging and how it relates to other parts of a ML workflow. As shown in Figure 1, there are lots of tools available today to promote human trust and understanding of ML. Some techniques, such as model debugging and social bias testing and remediation, help us make ML more accurate, fair, and secure without telling us very much about how the model works. These help us trust a model, more than they help us understand it. Other techniques, like interpretable ML models and post-hoc explanations, increase our understanding directly by revealing model mechanisms or by summarizing model decisions. These techniques only increase trust as a side-effect, when we like the model or explanations.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*rIry2kFeffmRahQCzww7lA.png" /><figcaption>Figure 1: Many tools exist today to increase trust and understanding of ML systems. Some promote trust directly, while others promote understanding directly. Figure courtesy of Patrick Hall and H2O.ai.</figcaption></figure><p>This post will focus on mathy aspects of debugging ML and promoting trust in ML. However, it’s crucial to consider these two additional aspects of a responsible ML workflow:</p><ul><li>Enabling understanding of ML</li><li>Testing and hardening of IT systems that surface ML models</li></ul><p>Even if you value trust, and in particular accuracy, more than fairness, interpretability, or security, making ML systems understandable is needed to enable logical appeal or operator override of wrong ML decisions. It’s very hard to argue against a black-box.[1] You won’t like it when your children are kept out of their preferred college by an accurate, but inscrutable, ML system. Nor would you enjoy being denied healthcare by the same kind of black-box system. You’d like the opportunity to appeal wrong ML decisions that affect your life, and that’s why business- or life-critical ML systems should always be understandable. Figure 2 proposes a workflow that includes steps for increasing both trust and understanding in ML systems. Model debugging likely works best when used along with the other techniques proposed in Figure 2.</p><p>Many of you likely know more than me about testing and hardening systems that surface ML models to consumers. I won’t say much more than that kind of work is still necessary. Just because a system houses a ML model doesn’t exempt it from testing. Moreover, Google (and probably others) have already put forward some workable thoughts and frameworks on the subject.[2]</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*3f7R7gxtbVd076QoYh0Bvw.png" /><figcaption>Figure 2: A ML workflow that can enhance trust and understanding. Figure courtesy of Patrick Hall and H2O.ai.</figcaption></figure><p>Now, let’s get into how to debug the guts of ML models. First we’ll discuss the example problem and dataset used in this post, then how to detect bugs, and finally, how to squash them.</p><h4>1. 2 Example Problem and Dataset</h4><p>Some examples below are based on the well-known Taiwanese credit card dataset from the University of California Irvine (UCI) ML repository.[3] In this dataset, we try to predict if someone will pay, DEFAULT_NEXT_MONTH = 0, or default, DEFAULT_NEXT_MONTH = 1, on their next credit card payment. Variables about payments are used to generate probabilities of default, or p_DEFAULT_NEXT_MONTH. I’ll use one of my favorite types of models, an interpretable monotonically constrained gradient boosting machine (M-GBM), to make these predictions. In the M-GBM, p_DEFAULT_NEXT_MONTH must only increase, or only decrease, as a certain input variable increases. This makes explaining and debugging the model much, much easier and does not really affect the overall accuracy of the model for this dataset. The M-GBM is trained on payment variables including PAY_0 — PAY_6, or a customer’s most recent through six months prior repayment statuses (higher values are late payments), PAY_AMT1 — PAY_AMT6, or a customer’s most recent through six months prior payment amounts, and BILL_AMT1 — BILL_AMT6, or a customer’s most recent through six months prior bill amounts. All monetary values are reported in Taiwanese Dollars (NT$).</p><p>Some of the example results also contain the variables LIMIT_BAL and r_DEFAULT_NEXT_MONTH. LIMIT_BAL is a customer’s credit limit. r_DEFAULT_NEXT_MONTH is a logloss residual value, or a numeric measure of how far off the M-GBM prediction, p_DEFAULT_NEXT_MONTH, is from the known correct answer, DEFAULT_NEXT_MONTH. I’ll also use demographic variables in the dataset, like SEX, to test for unwanted sociological bias. For the most part, this post treats the example credit lending problem as a general predictive modeling exercise, and does not consider applicable regulations. (Of course, noncompliance penalties probably make model debugging more commercially appealing.)</p><h3>2. Detection Strategies</h3><p>How do we find math bugs in ML models? I can think of at least four major ways: sensitivity analysis, residual analysis, benchmark models, and ML security audits. You can probably think of others too.</p><h4>2.1 Sensitivity (“What-if”) Analysis</h4><p>Sensitivity analysis, sometimes called “What-if” analysis, is a simple and powerful idea. Just simulate data that represents interesting scenarios, then see what kind of predictions your model makes in those scenarios. Because it’s almost impossible to know how a nonlinear ML model will react to data it did not see during training, it’s really important to conduct sensitivity analysis on all of our important ML models.[4] You might have some ideas about what kinds of scenarios to test. If so, that’s great and just do it. Maybe you just want to play with your model too? Also great. For those cases, definitely checkout the <a href="https://pair-code.github.io/what-if-tool/index.html">What-If Tool</a>, which provides an interactive sandbox experience for certain types of ML models.</p><p>It can also help to have a more structured approach to sensitivity analysis. This subsection will present three strategies for structured sensitivity analysis:</p><ul><li>Partial dependence, individual conditional expectation (ICE), and accumulated local effect plots (ALE)</li><li>Adversarial example searches</li><li>Random attacks</li></ul><p>Before we jump into those, it’s good to know which variables are most important to your model. I always focus my testing efforts on those variables first. Figure 3 is an accurate and consistent variable importance plot calculated with Shapley values using <a href="https://github.com/dmlc/xgboost">XGBoost</a>.[5] (Shapley variable importance measurements are also available natively in <a href="https://github.com/h2oai/h2o-3">H2O</a> and <a href="https://github.com/microsoft/LightGBM">LightGBM</a>.) Figure 3 shows us that PAY_0 is very important, and probably too important as we will see later.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*LUuqqNXR4SOkvrZ_PFYmLg.png" /><figcaption>Figure 3: An accurate and consistent variable importance chart for the M-GBM model and the credit card dataset. Figure courtesy of Patrick Hall and H2O.ai.</figcaption></figure><h4>2.1.1 Partial dependence, ICE, and ALE</h4><p>Partial dependence sets all the values of a column of interest, say PAY_0, in a dataset of interest, say the example validation dataset, to a value of interest, say missing or NaN, or any other reasonable value. This new dataset is run back through the trained model, creating a prediction for every row. Taking the average of all those predictions is the estimated partial dependence for that variable, for that dataset, for that value, and for that model. Now, we repeat that process over several different values that we’re interested in to create a partial dependence curve to plot. In Figure 4, that plot shows us the rough average behavior of PAY_0 in the M-GBM model.</p><p>Though it’s pretty easy to understand, partial dependence is far from perfect. It’s known to show untrustworthy results when there are strong interactions or correlations in a dataset. We have at least two options to improve partial dependence, ALE and ICE. Sounds fun right? ALE is almost a direct replacement for partial dependence. It’s more efficient to calculate and typically more accurate. ALE is available in R packages like: <a href="https://cran.r-project.org/web/packages/ALEPlot/index.html">ALEPlot</a>, <a href="https://cran.r-project.org/web/packages/DALEX/index.html">DALEX</a>, and <a href="https://cran.r-project.org/web/packages/iml/index.html">iml</a>.</p><p>ICE is often used along with partial dependence. ICE is a very similar calculation to partial dependence. Follow the same steps as described above, except the dataset of interest is just one row. That, and we usually calculate ICE curves for many different rows in a dataset. When the ICE curves, representing many different real or simulated individuals, follow along with the overall average behavior represented by partial dependence, this is a good sign that partial dependence is accurate enough. When ICE curves diverge from partial dependence, this potentially indicates the presence of interactions in the model (that are likely averaged-out of the partial dependence). In short, ICE can tell us about the behavior of interesting real or simulated individuals under our model, if partial dependence is trustworthy, and if we should be on the lookout for any strong interactions in our model. Combinations and variants of partial dependence and ICE are available in several of open source packages like <a href="https://github.com/SauceCat/PDPbox">PDPbox</a>, <a href="https://github.com/AustinRochford/PyCEbox">PyCEbox</a>, <a href="https://cran.r-project.org/web/packages/ICEbox/index.html">ICEbox</a>, and <a href="https://bgreenwell.github.io/pdp/index.html">pdp</a>.</p><p>Figure 4 combines partial dependence, ICE, and a histogram to provide a lot of insight about the most important variable, PAY_0, in the M-GBM model. First we can see the training data is really sparse for PAY_0 &gt; 1. This is usually a bad sign. ML models need lots of data to learn. This model has almost no data to learn about people who are more than 1 month late on their most recent repayment. Looking at partial dependence we can see a few other potential problems. The model gives the lowest average probability of default for missing, i.e. NaN, data. Does this make business sense? I don’t see how. Also, it’s kind of scary from a security perspective. If I want a good score from this model, I might only need to hack an adversarial example with a missing value for PAY_0 into the M-GBM’s production scoring queue. Also, we can see a huge swing in predictions from PAY_0 = 1 to PAY_0 = 2. Does this make sense from a business perspective? Maybe so and that’s fine, but it’s another thing to be aware of from a security and data drift perspective. If I want to conduct a denial of service attack for a customer of this model, I just need to change their PAY_0 value to larger than 1. For this model, we might want to let our IT pals know to monitor for adversarial example attacks involving PAY_0 = NaN and PAY_0 &gt; 1. Also, if our market changes toward recession conditions, and customers have more late bills, it’s important to remember how sensitive this M-GBM is to values of PAY_0 &gt; 1.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/881/1*D6qdpeyzTSHvWifPpTfQIg.png" /><figcaption>Figure 4: Histogram, partial dependence, and ICE for PAY_0. Figure courtesy of Patrick Hall and H2O.ai.</figcaption></figure><p>On the more reassuring side, the partial dependence and ICE curves show the monotonicity constraint for PAY_0 held on average and for many individuals. The monotonicity constraints luckily helped us deal with the data sparsity issue too. Due to the monotonicity constraints, the model held the probability of default from PAY_0 = 2, where there is some training data, all the way out to PAY_0 &gt; 8, where there is no training data. Without the constraint, model predictions in this domain of PAY_0 would probably just be random noise. Finally, because ICE and partial dependence are mostly aligned, we can see that the partial dependence curve for PAY_0, this dataset, and the M-GBM model is probably pretty trustworthy. Now, you just need to perform this analysis for the rest of your most important variables. For brevity’s sake, I’ll move onto the next sensitivity analysis debugging strategy, adversarial example searches.</p><h4>2.1.2 Adversarial example searches</h4><p>Adversarial examples are rows of data that make a model produce unexpected results. Looking for adversarial examples is a great debugging technique. The search process allows us to see how our model performs in many different scenarios. Finding and understanding actual adversarial examples can direct us toward ways to make our models more robust and tell us about anomalies to look for when the model moves into a production setting. If you’re working in the Python deep learning space, look into <a href="https://github.com/tensorflow/cleverhans">cleverhans</a> and <a href="https://github.com/bethgelab/foolbox">foolbox</a> for finding adversarial examples. For structured data, there’s less freely available software to help us out, but I have a little heuristic search method you can try or modify. Figure 5 shows the results of that heuristic search.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*n48NFyLtHlUoPd5b0N-ERQ.png" /><figcaption>Figure 5: Maximum predicted probability of default for adversarial example search across several important variables. Figure courtesy of Patrick Hall and H2O.ai.</figcaption></figure><p>For the M-GBM model and data, the heuristic search starts with PAY_0, calculating ICE like I did in Figure 4, and finding the ICE curve with the largest swing in predictions. For this dataset and model, that is the ICE curve at the 90th percentile of p_DEFAULT_NEXT_MONTH. Then the row of data at the 90th percentile of p_DEFAULT_NEXT_MONTH is perturbed 10,000 times, through ten different values for four important variables: PAY_0, PAY_3, PAY_AMT1, and PAY_AMT2. (These variables were selected due to their wide range of Shapley values, not directly from the variable importance plot in Figure 3.) This dataset of 10,000 perturbed instances of the row at the 90th percentile of p_DEFAULT_NEXT_MONTH was then rescored by the M-GBM model, and the results are plotted in Figure 5.</p><p>Figure 5 shows us several interesting things about the M-GBM model. First, we can see the monotonicity constraints held even for combinations of several variables. Second, we may have discovered a logical flaw in the M-GBM model. It appears that no matter how large someone’s most recent or second most recent payments are, the model will generate high probabilities of default if their most recent payment is more than one month late. This means the M-GBM model may not be able to account for prepayment, or for someone making large payments to make up for late payments, without issuing default predictions. If I did want to account for these conditions in my M-GBM model or in a ML-based credit lending system based on the M-GBM model, I could consider editing the M-GBM model or using model assertions to enable the credit lending system to handle these more complex scenarios. (See Subsections 3.3 and 3.4.) Third, this search did turn up at least one adversarial example. Extremely low values of PAY_AMT1 and PAY_AMT2, when combined with other values in the row used to initialize the search, will cause the M-GBM model to generate surprisingly high probabilities of default. These values are something for which to monitor when the M-GBM model is moved into production. They could indicate the model is under adversarial attack.</p><p>If you think the proposed adversarial search method seems useful, give it a try. The heuristic search process can be summarized as follows.</p><p><strong>For each important variable:</strong></p><ol><li>Calculate ICE curves at each decile of model predictions.</li><li>Find the ICE curve with the largest swing in predictions.</li><li>Isolate the row of data associated with this ICE curve.</li><li><strong>For this row of data:</strong></li></ol><ul><li>Perturb 1–3 additional important variables in the row. (It’s hard to plot results for more than 1–3 variables.)</li><li>Rescore the perturbed row.</li><li><strong>Continue</strong> <strong>until</strong> each additional important variable has cycled through its domain in the training data, and through missing or other interesting out-of-range values.</li></ul><p>5. Plot and analyze the results.</p><h4>2.1.3 Random attacks</h4><p>Random attacks are conducted by exposing a model to all kinds of random data. Think: double-byte character sets, datasets with one row and one column, datasets with 10 million columns and one row, etc. etc. etc. Random attacks can find both conventional IT bugs and math bugs. Consider that upon being exposed to a dataset with 10 million columns and 1 row, your API misbehaves by coughing up a stacktrace with too much private or internal information. Or maybe it just fails in a really ugly, service-crashing way? Maybe your API and model treat double-byte characters like missing values, and always assign records containing them a low probability of default? Who knows? Not you … if you don’t test for this kind of stuff. Also, if you’re totally clueless about where to start with your model debugging efforts, start with a random attack. I bet you will find some very interesting things.</p><h4>2.2 Residual Analysis</h4><p>Residual analysis has long been a cornerstone of linear model diagnostics, and that should remain true in an age of ML. Residuals refer to the difference between the known true outcome and what a model predicted that outcome to be. There are lots of ways to calculate residuals, but usually a large residual value means a model was wrong, and small residual value means a model was right. Residual plots put all your input data and predictions into a two-dimensional visualization where influential outliers and other types of math bugs can become plainly visible. The only draw-back of residual analysis is that to calculate residuals, we need true outcomes. So sometimes we can’t work with residuals in real-time if we’re making predictions where the true outcome won’t be available for a period of time. (Say in mortgage lending.)</p><p>Figure 6 displays the logloss residuals of the M-GBM model plotted by the levels of the important variable, PAY_0. Magenta residuals are from customers who actually defaulted. Blue residuals are from customers who did not default. Sadly, Figure 6 paints a damning portrait of my M-GBM model.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*D62heTzz84oBF-k3tkNi7A.png" /><figcaption>Figure 6: Logloss residuals plotted by PAY_0. Figure courtesy of Patrick Hall and H2O.ai.</figcaption></figure><p>In Figure 6, we can see that for desirable values of PAY_0 &lt; 1, i.e. NO CONSUMPTION (-2), PAID DULY (-1), or USE OF REVOLVING CREDIT (0), there are a lot of large magenta residuals. This means that the model basically fails to predict default when someone made their most recent payment. For PAY_0 &gt; 1, model errors are driven by the large blue residuals, meaning that when a customer has an undesirable value for PAY_0, i.e. several months late, the M-GBM model cannot predict on-time payment. Combining this information with the variable importance plot in Figure 3 shows that the M-GBM is pathologically over-dependent on PAY_0. I could likely deploy a single business rule: IF PAY_0 &gt; 1 THEN DEFAULT_NEXT_MONTH = 1, and have the same accuracy as the M-GBM, or almost any GBM. That way I would only be unleashing one brittle rule into the wild, instead of thousands or millions of broken, potentially biased and hackable rules.</p><p>We can try to fix this dangerous bug with data augmentation, strong regularization, model editing, or model assertions, all of which are discussed in the remediation section of this post. But one thing is clear, this model is broken, untrustworthy, and unfit for use in the real-world. This problem does not show up in fit statistics, lift, or AUC plots, and I never would have seen it so clearly it without looking at the residuals. In fact, it can be shocking what residual plots show us about seemingly healthy models.[6] Hopefully I’ve convinced you plotting residuals is a high-impact debugging technique. It’s usually easy to do yourself with any language you like and a decent plotting library, but packages like <a href="https://cran.r-project.org/web/packages/DALEX/index.html">DALEX</a>, <a href="https://github.com/cosmicBboy/themis-ml">themis-ml</a>, and <a href="https://github.com/ModelOriented/auditor">auditor</a> provide this functionality right out-of-the-box.</p><h4>2.2.1 Disparate impact, accuracy, and error analysis</h4><p>Disparate impact (DI) roughly refers to unintentional discrimination in decision-making systems. DI testing methods are a well-known way to find some types of unwanted sociological bias in training data and predictive modelling results. Are they perfect? No. Are they the least you can do to stop your ML model from perpetuating or exacerbating unwanted sociological bias? Probably. There are also several open source packages that can help you do your DI testing, including <a href="https://github.com/dssg/aequitas">aequitas</a>, <a href="http://aif360.mybluemix.net/">AIF360</a>, and <a href="https://github.com/LASER-UMASS/Themis">Themis</a>. Basic DI testing methods look at accuracy and error rates across demographic variables. Ideally we want these accuracy and error rates to be roughly equal across different demographic groups. If they’re not, this a strong indication your model is perpetuating or exacerbating unwanted sociological bias. In Figure 7, we can see with respect to the variable SEX, accuracy and error rates look fairly similar for men and women. This is a good sign, but it does not mean our model is free of unwanted sociological bias, even with respect to SEX.</p><p>All models have the ability to treat similar people differently based small changes in their input data, potentially resulting in instances of local bias, or a lack of <em>individual fairness</em>. An example of local bias could be granting a credit extension to a young woman with a good payment history and an income of NT$100,000, but denying credit to a very similar young woman with an income NT$99,999. We know that the NT$1 difference in income makes no real difference, but a ML model can arbitrarily place these two similar individuals on different sides of a nonlinear decision boundary. Worse still, standard DI testing usually can’t catch local bias problems. How do we ensure fairness on an individual level? That’s still a somewhat open question as of today, that many stellar researchers are trying to answer.[7] One practical suggestion I can give is to take a hard look at individuals closest to your model’s decision boundary or probability cutoff. In most cases, very similar individuals should not be on different sides of that boundary. Now, before moving onto discuss disparate accuracy and error rates as a general bug detection tool, it’s important to say fairness in ML goes far beyond the small discussion here. If you want to learn more about it, checkout the Fairness, Accountability, and Transparency in ML (FATML) conference and associated resources and try out some of those packages linked above. [8]</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*vx0ZN4n4DsdBe9ODfc0vAQ.png" /><figcaption>Figure 7: Accuracy and different types of error for PAY_0 and SEX across levels of these categorical variables. Figure courtesy of Patrick Hall and H2O.ai.</figcaption></figure><p>Traditional DI testing methods can also be applied to categorical variables in general, and I find this to be an excellent bug detection method. Figure 7 displays numerous accuracy and error metrics across the different categorical levels of the important variable, PAY_0. Here we can see the dramatic difference between the M-GBM’s performance for PAY &gt; 1, likely driven by sparsity of training data in that domain. This table does a good job of presenting just how brittle the model performance is in this domain, and how different the model performance is for PAY &gt; 1. This bug detection technique can also be applied to numeric variables, simply by binning them.</p><h4>2.2.2 Explanation of residuals</h4><p>The last few years have brought us numerous techniques to explain ML model predictions. These techniques can also be used to improve our residual analysis. It’s possible to create local interpretable model-agnostic explanations (LIMEs), partial dependence, or individual conditional expectation plots of residuals. Recent additions to the <a href="https://github.com/slundberg/shap">shap</a> package, make it possible to calculate Shapley contributions to residuals, meaning you can get an accurate picture of which variables are driving errors, both locally, i.e. for single rows, and globally, i.e. over an entire dataset. Another good option for explaining residuals is to fit a model to them. Figure 8 is a decision tree fit to the M-GBM residuals when DEFAULT_NEXT_MONTH = 1. Figure 8 shows a model of why the M-GBM model missed future defaults, that would likely cause write-offs in the real-world.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*_u4H6QNVQbW5MXhSIYupMQ.png" /><figcaption>Figure 8: Decision tree model of residuals for DEFAULT_NEXT_MONTH = 1. Figure courtesy of Patrick Hall and H2O.ai.</figcaption></figure><p>The decision tree in Figure 8 has a cross-validated R-square of 0.89 and mean absolute percentage error of about 17% for DEFAULT_NEXT_MONTH = 1 residuals. So it’s fairly accurate, and this means there are strong patterns in the M-GBM’s wrong guesses. Moreover, the decision tree can help us see what those patterns might be. The largest residuals appear for PAY_0 &lt; 0.5 AND PAY_AMT2 &lt; NT$ 2802.5 AND PAY_4 &lt; 1 AND PAY_AMT2 ≥ NT$ 1312.5, or for customers with good most recent and fourth most recent repayment statuses, and smaller second most recent payments between NT$1300 and NT$2800. Under these conditions, we know the M-GBM model often fails to predict future defaults. This decision policy for residuals echoes earlier results that point to an over-emphasis of PAY_0 by the M-GBM, but adds an additional hint to look further into customers with second most recent payments between NT$1300 and NT$2800. This extracted decision policy for high-residual customers could also be used to create a model assertion, or a post-prediction business rule. (See Section 3.4.) The model assertion could send customers with these characteristics for processing by human case workers or to more specific models. It could possibly be used to correct wrong predictions in real-time too!</p><h4>2.3 Benchmark Models</h4><p>Benchmark models are stable, trusted, transparent models, typically something like a linear model, single decision tree, simple rule-based model, or a previously existing and well-understood ML model. It’s always a good idea to check that your new ML model does actually out-perform a known benchmark in out-of-time test data. If your new ML model doesn’t out-perform a more traditional or transparent model, please don’t use it.</p><p>Once you’ve established your ML model is at least more accurate than a simpler or pre-existing benchmark, that benchmark model can be a solid debugging tool. I use benchmark models to ask questions like: who did my ML model get wrong that my benchmark model got right? If you can isolate the erroneous ML model behavior, you can consider blending benchmark model predictions with ML model predictions to make better predictions for these cases. Also, you can probably reason through why a transparent model is behaving better for some subset of data, and develop potential remediation strategies. For instance, when comparing misbehaving ML models to linear models, one probable cause of ML model inaccuracy is overemphasis of non-robust interactions in the ML model. Additionally, benchmark models can be used to spot anomalies in real-time. In most low signal-to-noise ratio, human-oriented ML problems, simple model and ML model predictions should not be extremely different. Comparing benchmark model and ML model predictions in real-time can help you catch some accuracy, fairness, or security anomalies as they are happening.</p><h4>2.4 Security Audits for ML Attacks</h4><p>There are several known attacks against ML models that can lead to altered, harmful model outcomes or exposure of sensitive training data.[9][10] Figure 9 outlines some of the most well-known ML attacks. Unfortunately, traditional model assessment measures don’t tell us much about whether a model is secure. In addition to other debugging steps, it may be prudent to add some or all of the known ML attacks into any white-hat hacking exercises or red-team audits your organization is already conducting.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*2c2YdnUzc_9cRKF2rkosqA.png" /><figcaption>Figure 9: A cheat sheet that describes several different types of known attacks against ML. A full size version of this cheat sheet is available: <a href="https://github.com/jphall663/secure_ML_ideas">https://github.com/jphall663/secure_ML_ideas</a>. Figure courtesy of Patrick Hall and H2O.ai.</figcaption></figure><h3>3. Remediation Strategies</h3><p>Now that we’ve gone over some systematic ways to find accuracy, fairness, and security problems in ML-based systems, let’s consider some remediation strategies to fix any detected problems.</p><h4>3.1 Data Augmentation</h4><p>If your model debugging exposes sparse places in your training data or logical errors in your model related to a lack of data, then you probably need to get more data. You may be able to simulate the needed data, add it back into your training data, retrain your model, and re-test your model. More likely, you’ll have to go back to the whiteboard, rethink how your training data is being collected, and wait until more complete data becomes available. To help avoid these kinds of problems in the future, consider experimental design techniques.[11] In the examples here, collecting information about debt-to-income ratio or employment status may have helped to de-emphasize PAY_0 in the M-GBM model.</p><p>Data augmentation can be a remediation strategy for unwanted sociological bias in ML models as well. One major source of sociological bias in ML is demographically unbalanced training data. If your model is going to be used on all kinds of people, it’s best to ensure your training data has a representative distribution of all kinds of people too. The Gender Shades line of research is both a cautionary tale and a success story about, among other important things, the necessity of demographically balanced training data.[12]</p><h4>3.2 Noise Injection and Strong Regularization</h4><p>Most of us use L1 and L2 regularization in our ML models today. We should probably continue to do so. Unfortunately standard types of regularization may not overcome strong and pathological biases, correlations, or dependencies in our training data. This is the case with PAY_0 in the discussed examples. A potential fix is to crank up the L1 and L2 regularization. If that remedy is not strong enough, you might consider additional measures like L∞ regularization, dropout, weight-clipping, or noise-injection techniques.</p><p>Going to such extremes to fix your training data may also be indicative of a data collection problem. If one variable is driving your entire model, you’re probably missing at least one confounding input variable. In this case, consider data augmentation as another remediation strategy.</p><h4>3.3 Model Editing</h4><p>Some ML models are designed to be interpretable, so you can understand how they work directly. Some of these models, like variants of decision trees or <a href="https://github.com/microsoft/interpret">GA2M</a> (a.k.a. explainable boosting machines, EBM), can be directly editable by human users. If you see something you don’t like in the inner workings of a GA2M model, it’s not very hard to change the model equation to get rid of what you don’t like. Other models might not be as easy to edit as GA2M or decision trees, but if they generate human-readable scoring code, they can be edited. If there are a lot of wrong rules in your M-GBM scoring code, maybe you can fix them or maybe you can remove them. Definitely, try out GA2M, but also consider editing for other models as a viable bug-fixing strategy.</p><p>One thing to consider about model editing is it will probably make your model look worse in training or validation data. Whatever was in your model was there because it made the training and validation error better. So, if you edit a model, you need to have solid reasoning backing up your decision.</p><h4>3.4 Model Assertions</h4><p>Model assertions, a.k.a. post-prediction business rules, are assertion statements on model predictions that can help correct erroneous or problematic model predictions.[13] In my examples, I saw my M-GBM model cannot account for pre-paying or over-paying after a customer is more than one month late on their most recent payment. Before issuing a default decision, it might be wise to check if the customer has prepaid or overpaid on their last payment. A model assertion for the M-GBM model in production could say: IF ((PAY_0 &gt; 1) AND (PAY_AMT1 &gt; LIMIT_BAL)) THEN DEFAULT_NEXT_MONTH = 0</p><p>(A facetious, ripped-from-the-headlines model assertion might be more along the lines of: check to ensure a customer is not a minor before sending pregnancy-related coupons to their family home.[14])</p><h4>3.5 Unwanted Sociological Bias Remediation</h4><p>There are a lot of human and technical ways to fix unwanted sociological bias in ML models today. Many human fixes involve promoting diversity of opinion and experience on data science teams, and ensuring diverse intellects are involved in all stages of model building. Technical remediation methods fall into three basic buckets:</p><p><strong>Data preprocessing:</strong></p><ol><li>Judicious feature selection, e.g. using time-on-file as an input variable as opposed to a bankruptcy flag.[15]</li><li>Sampling and reweighing rows in training data to minimize unwanted sociological bias in training data, e.g. reweighing in <a href="http://aif360.mybluemix.net/">AIF360</a>.</li></ol><p><strong>Model training and selection:</strong></p><ol><li>Considering fairness metrics when selecting hyperparameters and cutoff thresholds.</li><li>Training fair models directly:</li></ol><ul><li>Learning fair representations (LFR) and adversarial debiasing in <a href="http://aif360.mybluemix.net/">AIF360</a>.</li><li>Using dual objective functions that consider both accuracy and fairness metrics.</li></ul><p><strong>Prediction post-processing: </strong>Changing model predictions after training,<strong> </strong>like reject-option classification in <a href="http://aif360.mybluemix.net/">AIF360</a> or <a href="https://github.com/cosmicBboy/themis-ml">themis-ml</a>.</p><p>(Of course, giving two paragraphs to the subject of fixing unwanted sociological bias in ML models in 2019 borders on the absurd and offensive. Many techniques exist to fix unwanted sociological bias today. Learn more about them. There’s really no excuse to deploy a racist model today, but it keeps happening, even in high-stakes life-critical scenarios.[16])</p><h4>3.6 Model Management and Monitoring</h4><p>Know how many models you have, who trained them and when. Document them like other software assets. Also, monitor important deployed ML model inputs and predictions. Watch for concerning trends and anomalies. Don’t focus solely on accuracy. Think about fairness and security problems too.</p><p>Most data scientists now understand models are typically trained on data that is a snapshot of reality. Over time, reality changes. New data drifts away from that snapshot, and model accuracy can degrade on that new data. This kind of model drift would likely be evident in statistical trends in the model’s inputs and predictions. Can this kind of drift affect the fairness characteristics of a model as well? Probably. So consider real-time tests for DI in addition to any input or prediction monitoring you might be doing today.</p><p>Finally, when an older model is taken down to be replaced, the old model should be decommissioned, meaning stored carefully for any future diagnostic, forensic, or litigation needs. Important models should not just be deleted.</p><h4>3.7 Anomaly Detection</h4><p>Anomalous inputs and predictions are always worrisome, and can be indicative of an adversary attacking your model. In the example M-GBM model, we saw that it’s particularly sensitive to missing values for PAY_0, high values for PAY_0, and extremely low values of PAY_AMT1 and PAY_AMT3. Missing and other illogical values should probably not even be allowed into the M-GBM’s production scoring queue in this case. Real-time monitoring for new data with high values for PAY_0 or with extremely low values of PAY_AMT1 and PAY_AMT3 could potentially divert those customers to processing by human case workers or to even more transparent models.</p><p>To catch anomalous predictions in real-time, think about traditional statistical process control methodologies, comparing ML predictions to stable and transparent benchmark model predictions, or monitoring how new data flow through your model. (The latter is sometimes referred to as <em>activation analysis</em>.) For benchmark models, compare ML model predictions to benchmark model predictions. If they’re very different, take a closer look before issuing a prediction or just use the benchmark model predictions for these data. For activation analysis, new data should likely not commonly flow through model mechanisms (e.g. decision tree nodes or neural network hidden units) that were not activated frequently during model training. If that’s happening a lot, it’s probably safest to investigate why.</p><h3>4. Further Reading and Conclusion</h3><p>This post presents several debugging strategies for ML models, mostly focusing on structured data and standard commercial data mining use cases. If you’re interested, you can dig deeper into the code used to create my examples on GitHub.[17] To get a flavor for what’s going on with model debugging in the ML research community, checkout the International Conference on Learning Representations (ICLR) Debugging Machine Learning Models workshop proceedings.[18]</p><p>While practitioner and research buzz is building around model debugging, the discipline is likely still in its infancy. Where it goes from here is largely up to us. Whenever ML is adopted for business- or life-critical decisions, I personally hope to see a lot more trustworthy ML and model debugging.</p><p>—</p><h3>5. References</h3><p>[1] See: <a href="https://www.nytimes.com/2017/06/13/opinion/how-computers-are-harming-criminal-justice.html"><em>When a Computer Keeps You in Jail</em></a></p><p>[2] E.g., <a href="https://developers.google.com/machine-learning/testing-debugging">Testing and Debugging Machine Learning Models</a>, <a href="https://ai.google/research/pubs/pub43146"><em>Machine Learning: The High Interest Credit Card of Technical Debt</em></a>, or <a href="https://github.com/tensorflow/model-analysis">Model analysis tools for TensorFlow</a></p><p>[3] See: <a href="https://archive.ics.uci.edu/ml/datasets/default+of+credit+card+clients">Default of credit card clients dataset</a></p><p>[4] See: <a href="http://www.vias.org/tmdatanaleng/cc_ann_extrapolation.html"><em>Teach me Data Analysis</em></a></p><p>[5] See: <a href="https://christophm.github.io/interpretable-ml-book/shapley.html"><em>Interpretable Machine Learning</em></a>, Section 5.9</p><p>[6] See: <a href="https://www4.stat.ncsu.edu/~stefanski/NSF_Supported/Hidden_Images/Residual_Surrealism_TAS_2007.pdf"><em>Residual (Sur)Realism</em></a></p><p>[7] For instance, <a href="http://www.cs.yale.edu/homes/jf/Dwork.pdf"><em>Fairness Through Awareness</em></a> and <a href="https://scholar.google.com/scholar?cites=15887350027958465759&amp;as_sdt=20005&amp;sciodt=0,9&amp;hl=en">related work</a></p><p>[8] <a href="https://www.fatml.org/">Fairness, Accountability, and Transparency in Machine Learning</a></p><p>[9] <a href="https://www.oreilly.com/ideas/proposals-for-model-vulnerability-and-security"><em>Proposals for Model Vulnerability and Security</em></a></p><p>[10] <a href="https://fpf.org/wp-content/uploads/2019/09/FPF_WarningSigns_Report.pdf"><em>Warning Signs: The Future of Security and Privacy in an Age of Machine Learning</em></a></p><p>[11] <a href="https://machinelearningmastery.com/controlled-experiments-in-machine-learning/">Controlled Experiments in Machine Learning</a></p><p>[12] <a href="http://gendershades.org/">How well do IBM, Microsoft, and Face++ AI services guess the gender of a face?</a></p><p>[13] <a href="https://cs.stanford.edu/~matei/papers/2018/mlsys_model_assertions.pdf"><em>Model Assertions for Debugging Machine Learning</em></a></p><p>[14] <a href="https://www.forbes.com/sites/kashmirhill/2012/02/16/how-target-figured-out-a-teen-girl-was-pregnant-before-her-father-did/"><em>How Target Figured Out A Teen Girl Was Pregnant Before Her Father Did</em></a></p><p>[15] <a href="https://www.youtube.com/watch?v=rToFuhI6Nlw">Responsible Data Science: Identifying and Fixing Biased AI</a></p><p>[16] <a href="https://www.nature.com/articles/d41586-019-03228-6"><em>Millions of Black People Affected by Racial Bias in Health-care Algorithms</em></a></p><p>[17] <a href="https://github.com/jphall663/interpretable_machine_learning_with_python">Interpretable Machine Learning with Python</a></p><p>[18] <a href="https://debug-ml-iclr2019.github.io/">Debugging Machine Learning Models</a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=aa822f1097ce" width="1" height="1" alt=""><hr><p><a href="https://medium.com/data-science/strategies-for-model-debugging-aa822f1097ce">Strategies for Model Debugging</a> was originally published in <a href="https://medium.com/data-science">TDS Archive</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
    </channel>
</rss>