<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Engine.Finance ( Pavel Mironchyk ) on Medium]]></title>
        <description><![CDATA[Stories by Engine.Finance ( Pavel Mironchyk ) on Medium]]></description>
        <link>https://medium.com/@EngineFinance?source=rss-f20082e7b2a6------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/1*7t1DHMkNFUzsI-h3NIT95A.jpeg</url>
            <title>Stories by Engine.Finance ( Pavel Mironchyk ) on Medium</title>
            <link>https://medium.com/@EngineFinance?source=rss-f20082e7b2a6------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Fri, 20 Nov 2026 03:55:47 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@EngineFinance/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[On future of risk modelling]]></title>
            <link>https://medium.com/@EngineFinance/on-future-of-risk-modelling-950637371ef1?source=rss-f20082e7b2a6------2</link>
            <guid isPermaLink="false">https://medium.com/p/950637371ef1</guid>
            <category><![CDATA[banking]]></category>
            <category><![CDATA[finance]]></category>
            <dc:creator><![CDATA[Engine.Finance ( Pavel Mironchyk )]]></dc:creator>
            <pubDate>Mon, 23 Mar 2026 22:51:12 GMT</pubDate>
            <atom:updated>2026-03-23T22:51:12.188Z</atom:updated>
            <content:encoded><![CDATA[<p>Around 2016 when I got a job in the credit risk modelling department of a bank, I honestly had no idea about the field. I guess, I was inspired by stories of quantitative hedge funds, but their golden age was already over by mid-2010s. Well, the reality of being an analyst turned out to be completely different from I had imagine. Sure, I enjoyed plenty of intellectual challenges and some fun along the way! But I developed a strong unsettling perception of the industry is facing in the existential crisis. Now the year 2026 feels like living in a juncture, with developing impact of AI (LLMs) on banking.</p><h3>Epochs</h3><p>Through the 2010–2025 years I subjectively experienced subjectively two epochs:</p><ul><li><strong>Quants</strong>: Rise and decline of quantitative modelling in 2010s</li><li><strong>Clerks</strong>: Focus on compliancy and harmonization in 2020s</li></ul><p><strong>Quants (2010s).</strong> The influx of a talent with fundamental science backgrounds resulted in proliferation of modelling approaches applied to financial data, driven by goals contradicting those of risk managers. The failure of this movement was imminent. Noisy financial data are very specific to each institution’s circumstance and are subject to continuous, hidden changes in the external economical and sociopolitical context. It is clearly not that insightful. Talented people were obsessed with technical aspects of fitting models to data rather than with broadening understanding of the mechanics of risk. This epoch resulted in the internal model approach to be reduced to trivial econometric models that reinforced expert beliefs. Further, quants failed to anticipate the subprime 2008 crisis.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/275/1*lGX3RbxU3mKjf_3cChoQOA.jpeg" /></figure><p><strong>Clerks (2020s)</strong>. The introduction of Basel IV End Gamemarked the end of quant epoch. Overregulation and increasing supervisory pressure changed hiring priorities. The financial sector opened to professional clerks from governmental supervisory agencies. These are lured by higher salaries in private sector. With them came a major cultural shift toward prudence. Not surprising they view their former government employers as role models. The standardization of models, the introduction of output floors, and extensive regulation has created a situation in which most effort is devoted to compliance with a conservative interpretation of the rules. The standardization harmonized the risk modelling, but it suffocated the exploration and search for opportunities that was happening in the quant epoch.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/266/1*9ZvX-c5DjsBb_GzlbROBJQ.jpeg" /></figure><p>The biggest worry now is that younger generations of clients will turn away from banks to less prudent service providers that use aggressive opportunistic targeting. As of today, the banks have lost the SME lending market.</p><p>So, what epoch comes next? Am I ready to ride the next wave? Where to draw the inspiration for my next move? Well, I have few loose thoughts on this.</p><h3>Asimov’s Psychohistory and Cliodynamics</h3><p>Isaac Asimov (1920-1992) was a science-fiction and popular-science writer. He is mostly famous for formulation of the<a href="https://en.wikipedia.org/wiki/Three_Laws_of_Robotics"> Three Laws of Robotics</a>. He also wrote an enduring <em>Foundation </em>series of sci-fi novels, which I read in my childhood. The Apple TV+ series <a href="https://en.wikipedia.org/wiki/Foundation_(TV_series)"><em>Foundation</em></a><em> </em>(premiered 24 September 2021) is loosely based on Isaac Asimov’s Foundation novels.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/525/1*RNxttuLIS_1Fl7Qxri5nEA.jpeg" /></figure><p>In Foundation, Azimov imagined <em>psychohistory </em>is as a blend of science and mathematics that uses probability and historical analysis to predict trends and patterns in large social groups. While it could not predict the actions of an individual, it could predict the behavior of large societal group. The inspiration for this science clearly comes from thermodynamics.</p><p>The idea to treat the history as dynamic process certainly should not sound crazy to specialist in econometry or sociometry. This idea has materialized in actual research. The most prominent link is Peter Turchin’s Cliodynamics. He transitioned from biology (studying predator-prey cycles) to history, arguing that human societies follow predictable patterns of instability. He worked on a database to translate historical records to quantitative data. He claims to develop structural-demographic theory, basically a model with three variables — the economical state, the population and the elites. He claims that his model was able to predict the election of conservatives (Trump) to power in US in 2010.</p><p>To me is fascinating idea, why not to look broader to the risk, the risk is never local to any large organization, it is always systemic. Risk is neither static, it is always a dynamical system with second order “chaos” — people reading prediction are affected by it and effectively breaking prediction.</p><p>Why not to model the socio-politico-economy dynamical system of the globe to predict the systemic risk. Given modern developments in LLM and accessible computation power, extracting quantitative data from historical records and journalist essays is quite possible.</p><p>It is probably not a bad idea to see the world of risk as a dynamical system, seemingly chaotic but, not random and very sensitive to conditions. The financial institution irrespective of size is always connected to the system.</p><h3>From Dataset to Knowledge base Shift</h3><p>It is wrong to view large language models as mere information‑extraction devices. LLMs grew out of research on models that predict the next token in a sequence. The Transformer architecture delivered a real breakthrough: it is based on the attention mechanism, which lets each token attend to the entire sequence both before and after it. When trained on massive amounts of text from diverse domains, a large LLM begins to exhibit emergent behavioral properties — it can act, imitate, and follow prompts — still a surprising development for many researchers. Thank to these properties it can fuse different domain knowledge together. If you train LLM on texts about risk regulation and programming language domains, it can fuse these domains to synthesize regulatory compliance models.</p><p>Large language models (LLMs) are driving a fundamental shift in how structured datasets are collected and generated. Traditional data‑collection methods risk becoming obsolete. LLMs can synthesize datasets on demand and augment information across a wide range of knowledge domains.</p><p>Traditionally, the process began with collecting documents and publications from external sources, mechanically extracting specific fields, and recording them in a database. Such processing was cost‑conscious: sources were chosen for ease of access or because they were already available within the organization. Over time the connection between sources and database fields was often lost, and historical remediation of these datasets required tremendous effort. Consequently, the datasets became highly institution‑specific.</p><p>The new way is to index and feed all available sources to a pre‑trained, general LLM, which then becomes a knowledge base capable of relating and fusing information and distilling it into quantitative insights.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*4IZ7wInzFxwDFZpZznifRw.png" /></figure><p>Generally available large language models (LLMs) already serve as sources of data. They are trained on vast amounts of publications, books, and publicly available online data. A good example of this capability is the so‑called process of LLM distillation, in which a larger LLM is prompted to create a more focused training dataset for a smaller model. LLM distillation has delivered a breakthrough in the creation of small, effective reasoning LLMs (see DeepSeek papers).</p><p>The idea that a subscription to a general LLM might be all you need to obtain data for research is mind‑blowing.</p><h3><strong>Conclusion</strong></h3><p>I am certain the next epoch will be the epoch of creative people that are will be focusing on identification of opportunities withing the risk view constraints. For now my TODO list includes:</p><ul><li><strong>Prototype a retrieval‑augmented knowledge base</strong>: index regulatory texts, historical datasets, and news to test signal extraction.</li><li><strong>Design a small cliodynamics‑inspired pilot model</strong>: combine economic, demographic, and historical indicators to explore systemic risk signals.</li></ul><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=950637371ef1" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Let’s ditch regressions in econometrics]]></title>
            <link>https://medium.com/@EngineFinance/lets-ditch-regressions-in-econometrics-94a89cb8c8bc?source=rss-f20082e7b2a6------2</link>
            <guid isPermaLink="false">https://medium.com/p/94a89cb8c8bc</guid>
            <category><![CDATA[machine-learning-ai]]></category>
            <category><![CDATA[predictive-analytics]]></category>
            <category><![CDATA[statistics]]></category>
            <category><![CDATA[economics]]></category>
            <category><![CDATA[machine-learning]]></category>
            <dc:creator><![CDATA[Engine.Finance ( Pavel Mironchyk )]]></dc:creator>
            <pubDate>Mon, 09 Jun 2025 11:29:05 GMT</pubDate>
            <atom:updated>2025-06-09T11:35:56.525Z</atom:updated>
            <content:encoded><![CDATA[<p>Some years ago (2024) there was a sudden splash of publications on so-called KAN (Kolmogorov-Arnold networks), which is nothing more than just reincarnation of GAM (generalized additive models), popularized by Hastie T.J. and Tibshirani R.J. in 80–90s.</p><p>The generalized additive models are built on the idea that a model can be derived from regressors using smooth functions :</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*JFCUPAcw3Gw-Fg7y.png" /></figure><p>where smooth functions f_n(x) have some kind of parametric forms. If smooth function are strictly linear (f_n(x)= ax+b) the GAN reduces to the linear model.</p><p>Originally the creators of GAM popularized splines as smooth function forms. And this is where it gets specially interesting. I think splines or piecewise polynomials are a very underutilized tools in building explainable models. Splines offer versatility comparable to neural nets, allowing to create complex models and maintain computation stability unlike high order polynomial approximators.</p><p>In this article I would like to demonstrate an intuitive approach to building spline models. I hope that this article would help adoption of splines (and GANs) in econometrics.</p><p><strong>Piecewise approximation</strong></p><p>The central idea behind spline is piecewise approximation. Instead of finding a complex nonlinear function that maps X to Y, one can use set of simpler functions that maps subsets of X to Y. In other words instead of complex approximation/interpolation function that covers the whole range of inputs, we split input space on subranges, for each subrange we find a simple function that approximates data. The typical choice of approximation function is second or third order degree polynomial.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*VYGWQx0CCYnCWVYI.png" /></figure><p>Suppose we have the following knots on x in points 0, 1, 2, 3 (epsilons). See below the example.</p><pre>import numpy as np<br>import matplotlib.pyplot as plt<br><br>knot0, knot1, knot2, knot3 = 0.0, 1, 2, 3<br><br>segment1 = np.linspace(knot0, knot1, 10)<br>segment2 = np.linspace(knot1, knot2, 10)<br>segment3 = np.linspace(knot2, knot3, 10)<br><br>spline1 = lambda x: -5 * x**2 + 1 * x + 0.1<br>spline2 = lambda x: 10*x**2 + 2 * x - 40<br>spline3 = lambda x: -10*x**2 - x + 80<br><br><br># Create a basic plot<br>plt.plot(segment1, spline1(segment1), label=&#39;Segment 1&#39;, color=&#39;blue&#39;)<br>plt.plot(segment2, spline2(segment2), label=&#39;Segment 2&#39;, color=&#39;red&#39;)<br>plt.plot(segment3, spline3(segment3), label=&#39;Segment 2&#39;, color=&#39;red&#39;)<br><br>plt.axvline(x=1, color=&#39;r&#39;, linestyle=&#39;--&#39;, linewidth=0.5)<br><br># Add a vertical line at x = 2<br>plt.axvline(x=2, color=&#39;r&#39;, linestyle=&#39;--&#39;, linewidth=0.5)<br><br><br>plt.legend()<br>plt.show()</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/554/1*G8IYnkcJ08zAwdGRaSoptg.png" /></figure><p><strong>Approach I: Right-sided functions</strong></p><p>In order to fit polynomials to data, we need to define an approach to “stich” them together in “smooth” and continious approximation. The straightforwad way is to enforce a set of costraints on knots:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/288/0*ZbamjOazgKG1mhO3.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/290/0*8z9DxpIEbM_5bkN9.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/proxy/0*M16gJJI2T292aY2k.png" /></figure><p>And then fit functions that satisfy these constraints. In real life applications this turns to be impractical. Every quadratic spline has three degrees of freedom initially, and with two constraints, it is left with one degree of freedom. It make sense to rewrite the approximation in the following way:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*UWQW5G3bj9SuCbAn.png" /></figure><p>Basically we just add additional function for every segment:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/558/0*tXNA6RVkGPXUzT1i.png" /></figure><p>Obviously this shape of this function ensures smoothness as f_i(\xi_i)=0 and f’_i(\xi_i) = 0.</p><p>The demo below demonstrates how such approximation approach works.</p><pre>import numpy as np<br>import panel as pn<br>from matplotlib.figure import Figure<br><br><br>knot0, knot1, knot2, knot3 = 0.0, 1, 2, 3<br><br>segment1 = np.linspace(knot0, knot1, 10)<br>segment2 = np.linspace(knot1, knot2, 10)<br>segment3 = np.linspace(knot2, knot3, 10)<br><br>spline1 = lambda a,b,c,x: a * x**2 + b * 0 * x + c<br>spline2 = lambda x: (x-knot1)**2<br>spline3 = lambda x: (x-knot2)**2<br><br><br>pn.extension()<br><br>fig = Figure()<br>ax = fig.subplots()<br>#line1, = ax.plot(p1[0].get_points()[0], p1[0].get_points()[1])<br>ax.axvline(x=1, color=&#39;r&#39;, linestyle=&#39;--&#39;, linewidth=0.5)<br>ax.axvline(x=2, color=&#39;r&#39;, linestyle=&#39;--&#39;, linewidth=0.5)<br><br>slider = pn.widgets.FloatSlider(start=-100, end=100, value=1, name=&#39;a&#39; )<br>slider1 = pn.widgets.FloatSlider(start=-100, end=100, value=1, name=&#39;b&#39; )<br>slider2 = pn.widgets.FloatSlider(start=-100, end=100, value=1, name=&#39;c&#39; )<br>slider3 = pn.widgets.FloatSlider(start=-100, end=100, value=1, name=&#39;a1&#39; )<br>slider4 = pn.widgets.FloatSlider(start=-100, end=100, value=1, name=&#39;a2&#39; )<br><br>line1, = ax.plot(segment1, spline1(slider.value, <br>                                   slider1.value,<br>                                   slider2.value, segment1), label=&#39;Segment 1&#39;, color=&#39;blue&#39;)<br>line2, = ax.plot(segment2, spline1(slider.value, <br>                                   slider1.value,<br>                                   slider2.value, segment2) + slider3.value*spline2(segment2), label=&#39;Segment 2&#39;, color=&#39;red&#39;)<br>line3, = ax.plot(segment3, spline1(slider.value, <br>                                   slider1.value,<br>                                   slider2.value, segment3) + slider3.value*spline2(segment3) + slider4.value*spline3(segment3), label=&#39;Segment 2&#39;, color=&#39;red&#39;)<br><br>plot = pn.pane.Matplotlib(fig, dpi=96)<br><br>@pn.depends(slider, watch=True)<br>@pn.depends(slider1, watch=True)<br>@pn.depends(slider2, watch=True)<br>@pn.depends(slider3, watch=True)<br>@pn.depends(slider4, watch=True)<br>def update(val):<br>    line1.set_data(segment1, spline1(slider.value, <br>                                   slider1.value,<br>                                   slider2.value, segment1))<br>    line2.set_data(segment2, spline1(slider.value, <br>                                   slider1.value,<br>                                   slider2.value, segment2) + slider3.value*spline2(segment2))<br>    line3.set_data(segment3, spline1(slider.value, <br>                                   slider1.value,<br>                                   slider2.value, segment3) + slider3.value*spline2(segment3) + slider4.value*spline3(segment3))<br><br>    plot.param.trigger(&quot;object&quot;)<br>    <br>pn.Row(pn.Column(slider2, slider, slider3, slider4), plot).servable()</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*pwPaIczbbBWsXkG3Ejqhzw.png" /></figure><p>Run this app in browser using this link:</p><p><a href="https://urry.nl/pyodide/interval-splines.html">Panel Application</a></p><p><strong>Approach II: De Boor’s algorithm</strong></p><p>However such approch is less computationally stable. The best alternative is so called de Boor’s algorithm. The idea behind algorithm is rather simple, we construct a basis of polynomial functions, each function is smooth in every knot point, then we just sum these functions up with “control” coefficients/points. As functions smooth and continuous by design in every control point, resulting functions is also smooth. Let’s this demonstrate on some example. Let’s first show Cox de Boor formula for basic functions:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/838/1*Euf10vkikRW0FcCxxDst9Q.png" /></figure><pre>import numpy as np<br>import panel as pn<br>from matplotlib.figure import Figure<br><br>order_spline = 2<br>n_unique_knots = 4<br>knot0, knot1, knot2, knot3 = 0.0, 1, 2, 3<br>total_knots = n_unique_knots + max(2 * (order_spline-1), 0)<br>print(total_knots)<br><br>knot_m3 = knot0<br>knot_m2 = knot0<br>knot_m1 = knot0<br>knot4 = knot3<br>knot5 = knot3<br>knot6 = knot3<br>data = np.linspace(knot0-0.0001, knot3, 200)<br><br>BSpline = np.zeros((7,4), dtype=object)<br><br># 0 order splines, 4 knots, 3 params<br>Bm3_0 = lambda x: 0.0*x<br>Bm2_0 = lambda x: 0.0*x<br>Bm1_0 = lambda x: 0.0*x <br>B0_0 = lambda x: np.where((x &gt;= knot0) &amp; (x &lt; knot1), 1, 0)<br>B1_0 = lambda x: np.where((x &gt;= knot1) &amp; (x &lt; knot2), 1, 0)<br>B2_0 = lambda x: np.where((x &gt;= knot2) &amp; (x &lt; knot3), 1, 0)<br><br>BSpline[0, 0] = Bm3_0<br>BSpline[1, 0] = Bm2_0<br>BSpline[2, 0] = Bm1_0<br>BSpline[3, 0] = B0_0<br>BSpline[4, 0] = B1_0<br>BSpline[5, 0] = B2_0<br><br><br># first order, 4 knots, 4 params<br>Bm3_1 = lambda x: 0.0*x<br>Bm2_1 = lambda x: 0.0*x<br>Bm1_1 = lambda x:                                           (knot1 - x) * B0_0(x) / (knot1 - knot0)<br>B0_1 = lambda x:  (x - knot0) * B0_0(x) / (knot1 - knot0) + (knot2 - x) * B1_0(x) / (knot2 - knot1)<br>B1_1 = lambda x:  (x - knot1) * B1_0(x) / (knot2 - knot1) + (knot3 - x) * B2_0(x) / (knot3 - knot2)<br>B2_1 = lambda x:  (x - knot2) * B2_0(x) / (knot3 - knot2) <br><br><br>BSpline[0, 1] = Bm3_1<br>BSpline[1, 1] = Bm2_1<br>BSpline[2, 1] = Bm1_1<br>BSpline[3, 1] = B0_1<br>BSpline[4, 1] = B1_1<br>BSpline[5, 1] = B2_1<br><br><br># second order, 6 knots, 5 params<br>Bm3_2 = lambda x: 0.0*x<br>Bm2_2 = lambda x:                                                (knot1 - x) * Bm1_1(x) / (knot1 - knot_m1)<br>Bm1_2 = lambda x: (x - knot_m1) * Bm1_1(x) / (knot1 - knot_m1) + (knot2 - x) * B0_1(x) / (knot2 - knot0)<br>B0_2 = lambda x:  (x - knot0) * B0_1(x) / (knot2 - knot0)      + (knot3 - x) * B1_1(x) / (knot3 - knot1)<br>B1_2 = lambda x:  (x - knot1) * B1_1(x) / (knot3 - knot1)      + (knot4 - x) * B2_1(x) / (knot4 - knot2)<br>B2_2 = lambda x:  (x - knot2) * B2_1(x) / (knot4 - knot2) <br><br><br>BSpline[0, 2] = Bm3_2<br>BSpline[1, 2] = Bm2_2<br>BSpline[2, 2] = Bm1_2<br>BSpline[3, 2] = B0_2<br>BSpline[4, 2] = B1_2<br>BSpline[5, 2] = B2_2<br><br># third order, 8 knots, 6 params<br>Bm3_3 = lambda x:                                                (knot1 - x) * Bm2_2(x) / (knot1 - knot_m2) <br>Bm2_3 = lambda x: (x - knot_m2) * Bm2_2(x) / (knot1 - knot_m2) + (knot2 - x) * Bm1_2(x) / (knot2 - knot_m1)<br>Bm1_3 = lambda x: (x - knot_m1) * Bm1_2(x) / (knot2 - knot_m1) + (knot3 - x) * B0_2(x) / (knot3 - knot0)<br>B0_3 = lambda x:  (x - knot0) * B0_2(x) / (knot3 - knot0)      + (knot4 - x) * B1_2(x) / (knot4 - knot1)<br>B1_3 = lambda x:  (x - knot1) * B1_2(x) / (knot4 - knot1)      + (knot5 - x) * B2_2(x) / (knot5 - knot2)<br>B2_3 = lambda x:  (x - knot2) * B2_2(x) / (knot5 - knot2)       <br><br>BSpline[0, 3] = Bm3_3<br>BSpline[1, 3] = Bm2_3<br>BSpline[2, 3] = Bm1_3<br>BSpline[3, 3] = B0_3<br>BSpline[4, 3] = B1_3<br>BSpline[5, 3] = B2_3<br><br><br>pn.extension()<br><br>fig = Figure()<br>ax = fig.subplots()<br>ax.axvline(x=1, color=&#39;r&#39;, linestyle=&#39;--&#39;, linewidth=0.5)<br>ax.axvline(x=2, color=&#39;r&#39;, linestyle=&#39;--&#39;, linewidth=0.5)<br><br>oder_select = pn.widgets.Select(name=&#39;Spline Order&#39;, options=[0, 1, 2, 3], value=2)<br>slider = pn.widgets.FloatSlider(start=-100, end=100, value=1, name=&#39;control[-3]&#39; )<br>slider1 = pn.widgets.FloatSlider(start=-100, end=100, value=1, name=&#39;control[-2]&#39;)<br>slider2 = pn.widgets.FloatSlider(start=-100, end=100, value=1, name=&#39;control[-1]&#39;)<br>slider3 = pn.widgets.FloatSlider(start=-100, end=100, value=1, name=&#39;control[0]&#39;)<br>slider4 = pn.widgets.FloatSlider(start=-100, end=100, value=1, name=&#39;control[1]&#39;)<br>slider5 = pn.widgets.FloatSlider(start=-100, end=100, value=1, name=&#39;control[2]&#39;)<br><br><br>line, = ax.plot(data, slider.value * BSpline[0,2](data) + \<br>                    slider1.value * BSpline[1,2](data) + \<br>                    slider2.value * BSpline[2,2](data) + \<br>                    slider3.value * BSpline[3,2](data) + \<br>                    slider4.value * BSpline[4,2](data) + \<br>                    slider5.value * BSpline[5,2](data), linewidth=4)<br><br>line2, = ax.plot(data, slider.value * BSpline[0,2](data))<br>line3, = ax.plot(data, slider1.value * BSpline[1,2](data))<br>line4, = ax.plot(data, slider2.value * BSpline[2,2](data))<br>line5, = ax.plot(data, slider3.value * BSpline[3,2](data))<br>line6, = ax.plot(data, slider4.value * BSpline[4,2](data))<br>line7, = ax.plot(data, slider5.value * BSpline[5,2](data))<br><br>plot = pn.pane.Matplotlib(fig, dpi=96)<br><br>@pn.depends(slider, watch=True)<br>@pn.depends(slider1, watch=True)<br>@pn.depends(slider2, watch=True)<br>@pn.depends(slider3, watch=True)<br>@pn.depends(slider4, watch=True)<br>@pn.depends(slider5, watch=True)<br>@pn.depends(oder_select, watch=True)<br>def update(val):<br>    o = oder_select.value<br>    line.set_data(data, slider.value * BSpline[0,o](data) + \<br>                    slider1.value * BSpline[1,o](data) + \<br>                    slider2.value * BSpline[2,o](data) + \<br>                    slider3.value * BSpline[3,o](data) + \<br>                    slider4.value * BSpline[4,o](data) + \<br>                    slider5.value * BSpline[5,o](data))<br>    <br>    line2.set_data(data, slider.value * BSpline[0,o](data))<br>    line3.set_data(data, slider1.value * BSpline[1,o](data))<br>    line4.set_data(data, slider2.value * BSpline[2,o](data))<br>    line5.set_data(data, slider3.value * BSpline[3,o](data))<br>    line6.set_data(data, slider4.value * BSpline[4,o](data))<br>    line7.set_data(data, slider5.value * BSpline[5,o](data))<br><br>    plot.param.trigger(&quot;object&quot;)<br><br>column = pn.Column(oder_select, <br>                 slider, <br>                 slider1, <br>                 slider2, <br>                 slider3, <br>                 slider4, <br>                 slider5)<br><br># first order,  4 knots, 4 params<br># second order, 6 knots, 5 params<br># third order,  8 knots, 6 params<br># generalization knots = n_unique_knots + max(2 * (order_spline-1), 0)<br># generalization params = order_spline + n_unique_knots - 1<br><br>@pn.depends(oder_select, watch=True)<br>def active_params(val):<br>    match val:<br>        case 0:<br>            slider.visible = False<br>            slider1.visible = False<br>            slider2.visible = False<br>        case 1:<br>            slider.visible = False<br>            slider1.visible = False<br>        case 2:<br>            slider.visible = False<br>        case _:<br>            slider.visible = True<br>            slider1.visible = True<br>            slider2.visible = True<br><br>active_params(oder_select.value)<br>pn.Row(column, plot).servable()</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*laAu7-eD1vFAhLiZOupwlw.png" /></figure><p>Try this app here:</p><p><a href="https://urry.nl/pyodide/deboor-splines.html">Panel Application</a></p><h3><strong>Real-life Example</strong></h3><p>Now let’s try to apply the theory of GAM to semi-realistic data. In the example below I will try to build an LGD model (Loss Given Default).</p><p>I use data provided with the book Credit Risk Analytics by H. Scheule, D. Roesch, B. Baesens</p><p><a href="http://www.creditriskanalytics.net/datasets.html">http://www.creditriskanalytics.net/datasets.html</a></p><p>The LGD dataset has been kindly provided by a European bank and has been slightly modified and anonymized. It includes 2,545 observations on loans. Key variables are:</p><ul><li>LTV: Loan-to-value ratio, in %</li><li>Recovery_rate: Recovery rate, in %</li><li>lgd_time: Loss rate given default (LGD), in %</li><li>y_logistic: Logistic transformation of the LGD</li><li>lnrr: Natural logarithm of the recovery rate</li><li>Y_probit: Probit transformation of the LGD</li><li>purpose1: Indicator variable for the purpose of the loan; 1 = renting purpose, 0 = other</li><li>event: Indicator variable for a default or cure event; 1 = event, 0 = no event</li></ul><p>A small explaratory sample from the dataset is below:</p><pre>import pandas as pd<br>dataset = pd.read_csv(&#39;lgd.csv&#39;, dtype=np.float32)<br>dataset.rename(columns={&#39;lgd_time&#39;: &#39;Loss&#39;, &#39;purpose1&#39;: &#39;BuyToLet&#39;}, inplace=True);<br>dataset.head()</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1000/1*ZeOibD_oqbE1GQZ7uALE9A.png" /></figure><p>If we inspect the histograms of LTV and Observered losss, we could come to the realization that that underlaying loans for this dataset are highly collaterised. This is quite normal for a portfolio of mortgage loans.</p><pre>figs, axs = plt.subplots(1,2, figsize=(10,4))<br><br>axs[0].hist(dataset.LTV, bins=20);<br>axs[0].set_xlabel(&#39;LTV&#39;);<br>axs[0].set_ylabel(&#39;Observations&#39;);<br><br>axs[1].hist(dataset.Loss, bins=20);<br>axs[1].set_xlabel(&#39;Observed Loss&#39;);<br>axs[1].set_ylabel(&#39;Observations&#39;);<br>bins = np.linspace(0, 2.0, 20)<br>heights, _ = np.histogram(dataset.LTV, bins=bins)</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/850/1*OAVpUT5fo4ow-vz_x0v4Og.png" /></figure><p>I was struggling to understand the contents of variable “event” in this dataset, it seems it indicates some observations that have never defaulted, either fully repayed of refinanced. Notable that the higher the value of LTV the higher the probability of default/cure event.</p><pre>dataset[&#39;LTV_bin&#39;] = np.digitize(dataset.LTV, bins=bins, right=False)<br>fig, ax = plt.subplots(1,1)<br>response = dataset.groupby(&#39;LTV_bin&#39;).agg({&#39;LTV&#39;: &#39;count&#39;, &#39;Loss&#39;: &#39;mean&#39;, &#39;event&#39;: &#39;mean&#39;}).reset_index()<br>response[&#39;mid_LTV&#39;] = (bins[response.LTV_bin-1] + bins[response.LTV_bin])/2.0<br><br>response[&#39;LogOdds&#39;] = np.log(response[&#39;event&#39;] / (1 - response[&#39;event&#39;]))<br>ax.bar(response.mid_LTV, response.LTV, width=2.0/21.0)<br>ax2 = ax.twinx()<br>ax2.plot(response.mid_LTV, response.Loss, &#39;red&#39;)<br>ax2.plot(response.mid_LTV, response.event, &#39;green&#39;)<br><br>ax2.set_ylabel(&#39;Loss&#39;)<br>ax.set_ylabel(&#39;Counts&#39;)<br>ax.set_xlabel(&#39;LTV&#39;);<br>fig.legend(labels=[&#39;Counts&#39;, &#39;Loss&#39;, &#39;Event&#39;])<br>factor = dataset.groupby(&#39;BuyToLet&#39;).agg({&#39;BuyToLet&#39;: &#39;count&#39;, &#39;Loss&#39;: &#39;mean&#39;, &#39;event&#39;: &#39;mean&#39;}).rename(columns={&#39;BuyToLet&#39;: &#39;Count&#39;}).reset_index()<br>factor[&#39;LogOdds&#39;] = np.log(factor[&#39;event&#39;] / (1 - factor[&#39;event&#39;]))<br><br><br>fig, ax = plt.subplots(1, 2, figsize=(10,4))<br>ax[0].bar(factor.BuyToLet, factor.Count)<br>ax12 = ax[0].twinx()<br>ax12.plot(factor.BuyToLet, factor.Loss, &#39;vr&#39;)<br><br>ax[1].bar(factor.BuyToLet, factor.Count)<br>ax22 = ax[1].twinx()<br>ax22.plot(factor.BuyToLet, factor.LogOdds, &#39;vy&#39;)<br><br>fig.subplots_adjust(wspace=0.5, hspace=0.2)<br><br>ax[0].set_xlabel(&#39;Buy To Let&#39;);<br>ax[1].set_xlabel(&#39;Buy To Let&#39;);<br>ax[0].set_xticks([0, 1])<br>ax[1].set_xticks([0, 1])<br>ax[0].set_ylabel(&#39;Count&#39;)<br>ax12.set_ylabel(&#39;Loss&#39;)<br>ax22.set_ylabel(&#39;LogOdds&#39;)<br><br>fig.suptitle(&#39;Buy To Let Factor on Loss and Event Prob&#39;);</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/628/1*pxZbTK_AayB7CFdjd6MeSA.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/910/1*iUbc1YKuhx6bg44jkQy8Ew.png" /></figure><p>Let’s split dataset on training/test part. We will use the following variables — LTV and Buy to Let as predictors of loss.</p><pre># 20 % of the data for training and 80% for testing<br>split = np.random.uniform(size=dataset.shape[0])<br>train_mask = np.ones(dataset.shape[0])<br>train_mask[split &lt; 0.2] = 0<br><br>train_ds = dataset.loc[train_mask == 1, :]<br>test_ds = dataset.loc[train_mask == 0, :]<br><br># reshuffle and batch by 32 samples <br>tf_train_ds = tf.data.Dataset.from_tensor_slices((train_ds[[&#39;LTV&#39;, &#39;BuyToLet&#39;]], <br>                                                  train_ds[[&#39;Loss&#39;]])).shuffle(1000).batch(32)<br>tf_test_ds = tf.data.Dataset.from_tensor_slices((test_ds[[&#39;LTV&#39;,&#39;BuyToLet&#39;]], <br>                                                 test_ds[[&#39;Loss&#39;]])).batch(100)</pre><p>Let’s build a model using this dataset and KAN/GAN theory. Or model will be of the following form:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/622/0*Bb5D46goL7pFW8ts.png" /></figure><p>The code is implementation of de Boor’s basis using Tensorflow framework.</p><pre>@tf.function<br>def bfunc(x, i, m, tau):<br>    if m == 0:<br>        mask = (tau[i] &lt;= x) &amp; (x &lt; tau[i+1])<br>        return tf.where(mask, tf.ones(x.shape, dtype=np.float32), tf.zeros(x.shape, dtype=np.float32))<br>    b1 = bfunc(x, i, m-1, tau)<br>    b2 = bfunc(x, i+1, m-1, tau)<br>    val = tf.where(b1 !=0, b1 * (((x - tau[i]) / (tau[i+m] - tau[i]))), 0) + \<br>            tf.where(b2 != 0, b2 * (((tau[i+m+1]-x) / (tau[i+m+1] - tau[i+1]))), 0) <br>    return val <br><br>class SplineLayer(tf.keras.layers.Layer):<br>    def __init__(self, order, knots): <br>        super(SplineLayer, self).__init__()<br>        self.order = order<br>        self.knots = knots<br><br>    def build(self, input_shape):<br>        self._knots = self.knots.copy()<br>        for _ in range(max(self.order, 0)):<br>            self._knots = tf.concat([[self.knots[0]],self._knots], 0)<br>            self._knots = tf.concat([self._knots, [self.knots[-1]]], 0)<br><br>        self.nparams = len(self.knots) + self.order - 1<br>        self.coeffs = self.add_weight(name=&quot;b-spline&quot;,<br>                            shape=[int(input_shape[-1]),<br>                                    self.nparams],<br>                            initializer=tf.keras.initializers.Ones())   <br>        <br>        generator = lambda ii: (lambda x: bfunc(x, ii, self.order, self._knots))<br>        basis_funcs = [generator(ii) for ii in range(self.nparams)]<br>        # create basis of functions<br>        self.basis = lambda x : tf.stack([tf.map_fn(func, x) for func in basis_funcs], axis=2)<br><br>    @tf.function<br>    def call(self, inputs):   <br>        return tf.reduce_sum(tf.multiply(self.basis(inputs), self.coeffs), axis=-1)<br><br>import timeit<br><br>class LGDModel(tf.keras.Model):<br>  def __init__(self):<br>    super().__init__()<br>    self.LTV_tranf = SplineLayer(3, [0.25,  1.0,  1.75])<br>    self.BuyToLet_tranf = SplineLayer(0, [-0.01, 0.5, 1.01])<br><br>  def build(self, input_shape):<br>    self.LTV_tranf.build(input_shape)<br>    self.BuyToLet_tranf.build(input_shape)<br><br>  @tf.function<br>  def call(self, data):<br>    LTVs = tf.reshape(data[:, 0],(-1,1))<br>    BTLs = tf.reshape(data[:, 1],(-1,1))<br>    return self.LTV_tranf(LTVs) + self.BuyToLet_tranf(BTLs) <br><br>optimizer = tf.keras.optimizers.Adam()<br><br>train_loss = tf.keras.metrics.Mean(name=&#39;train_loss&#39;)<br>train_accuracy = tf.keras.metrics.SparseCategoricalAccuracy(name=&#39;train_accuracy&#39;)<br><br>test_loss = tf.keras.metrics.Mean(name=&#39;test_loss&#39;)<br>test_accuracy = tf.keras.metrics.SparseCategoricalAccuracy(name=&#39;test_accuracy&#39;)<br><br>@tf.function<br>def objective_func(targets, predictions):<br>   return tf.reduce_mean((targets-predictions)**2)<br><br># optimization objective function<br>def train_step(data, targets):<br>   with tf.GradientTape() as tape:<br>      # training=True is only needed if there are layers with different<br>      # behavior during training versus inference (e.g. Dropout).<br>      predictions = model(data, training=True)<br>      loss = objective_func(targets, predictions)<br><br>   gradients = tape.gradient(loss, model.trainable_variables)<br>   optimizer.apply_gradients(zip(gradients, model.trainable_variables))<br><br>   train_loss(loss)<br>   #train_accuracy(targets, predictions)<br><br>def test_step(data, targets):<br>   # training=False is only needed if there are layers with different<br>   # behavior during training versus inference (e.g. Dropout).<br>   predictions = model(data, training=False)<br>   loss = objective_func(targets, predictions)<br><br>   test_loss(loss)<br>   #test_accuracy(labels, predictions)<br><br>EPOCHS = 100<br>for epoch in range(EPOCHS):<br>   # Reset the metrics at the start of the next epoch<br>   train_loss.reset_state()<br>   train_accuracy.reset_state()<br>   test_loss.reset_state()<br>   test_accuracy.reset_state()<br><br>   for data, target in tf_train_ds:<br>      train_step(data, target)<br><br>   for data, target in tf_test_ds:<br>      test_step(data, target)<br><br>   if (epoch % 10 == 0) or (epoch == EPOCHS -1) :<br>      print(<br>         f&#39;Epoch {epoch + 1}, &#39;<br>         f&#39;Loss: {train_loss.result():0.6f}, &#39;<br>         #f&#39;Accuracy: {train_accuracy.result() * 100:0.6f}, &#39;<br>         f&#39;Test Loss: {test_loss.result():0.6f}, &#39;<br>         # f&#39;Test Accuracy: {test_accuracy.result() * 100:0.2f}&#39;<br>         )<br><br>fig, ax = plt.subplots(1,2, figsize=(15,4))<br><br>samples = 1000<br>LTVs = np.linspace(0, 1.999999, samples, dtype=np.float32)<br>ax[0].plot(LTVs, model.LTV_tranf(tf.reshape(LTVs, (-1, 1))), label=&#39;LTV&#39;)<br><br>BTLs = np.linspace(0, 1, 100, dtype=np.float32)<br>ax[1].plot(BTLs, model.BuyToLet_tranf(tf.reshape(BTLs, (-1, 1))), label=&#39;BuyToLet&#39;)<br>ax[0].set_xlabel(&#39;LTV&#39;)<br>ax[0].set_title(&#39;LTV Spline&#39;)<br>ax[1].set_xlabel(&#39;BuyToLet&#39;)<br>ax[1].set_title(&#39;BuyToLet Spline&#39;)<br>ax[0].set_ylabel(&#39;Loss&#39;)<br>ax[1].set_ylabel(&#39;Loss&#39;)<br><br>fig.suptitle(&#39;Fitted Model&#39;)</pre><p>The final fitted model:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*P3Eud2fWmG6Eg75pDchxGA.png" /></figure><p>The model is slightly overfit, which can be easily cured, but enforcing monotonicity constraints. The advantage of the model is that it is easilty interpretable. It is easy to communicate it to client as the model is basically represented by set of graphs.</p><p>Let’s check calibration accuracy of the created model as a last step.</p><pre><br><br># visualize the results<br>#print(model(dataset[[&#39;LTV&#39;]].values, training=False))<br>dataset[&#39;PredictedLoss&#39;] = model(dataset[[&#39;LTV&#39;, &#39;BuyToLet&#39;]].values, training=False)<br><br>fig, ax = plt.subplots(1,2, figsize=(15,4))<br>response = dataset[dataset.BuyToLet == 0].groupby(&#39;LTV_bin&#39;).agg({&#39;LTV&#39;: &#39;count&#39;, &#39;PredictedLoss&#39;: &#39;mean&#39;, &#39;Loss&#39;: &#39;mean&#39;}).reset_index()<br>response[&#39;mid_LTV&#39;] = (bins[:-1] + bins[1:])/2.0<br>ax[0].bar(response.mid_LTV, response.LTV, width=2.0/21.0)<br>ax2 = ax[0].twinx()<br>ax2.plot(response.mid_LTV, response.Loss, &#39;red&#39;)<br>ax2.plot(response.mid_LTV, response.PredictedLoss, &#39;green&#39;)<br><br>#ax2.set_ylabel(&#39;Loss&#39;)<br>ax[0].set_ylabel(&#39;Counts&#39;)<br>ax[0].set_xlabel(&#39;LTV&#39;);<br>#ax[0].legend(labels=[&#39;Counts&#39;, &#39;Loss&#39;, &#39;Predicted Loss&#39;]) <br>ax2.legend(labels=[&#39;Loss&#39;, &#39;Predicted Loss&#39;])<br>ax2.set_title(&#39;Buy To Let = 0&#39;)<br><br>response = dataset[dataset.BuyToLet == 1].groupby(&#39;LTV_bin&#39;).agg({&#39;LTV&#39;: &#39;count&#39;, &#39;PredictedLoss&#39;: &#39;mean&#39;, &#39;Loss&#39;: &#39;mean&#39;}).reset_index()<br>response[&#39;mid_LTV&#39;] = (bins[:-1] + bins[1:])/2.0<br>ax[1].bar(response.mid_LTV, response.LTV, width=2.0/21.0)<br>ax2 = ax[1].twinx()<br>ax2.plot(response.mid_LTV, response.Loss, &#39;red&#39;)<br>ax2.plot(response.mid_LTV, response.PredictedLoss, &#39;green&#39;)<br><br>ax2.set_ylabel(&#39;Loss&#39;)<br>#ax[1].set_ylabel(&#39;Counts&#39;)<br>ax[1].set_xlabel(&#39;LTV&#39;);<br>#ax[0].legend(labels=[&#39;Counts&#39;, &#39;Loss&#39;, &#39;Predicted Loss&#39;]) <br>ax2.legend(labels=[&#39;Loss&#39;, &#39;Predicted Loss&#39;])<br>ax2.set_title(&#39;Buy To Let = 1&#39;);</pre><p>Final test results, segmented with Buy To Let indicator:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*ieikKbdt6872EkK4sWkv8w.png" /></figure><p>GAM with Boor’s basis splines are very easy to implement, the resulting models are easy to understand. They are safe to apply in model risk sensitive applications. I think it is a very powerful tool.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=94a89cb8c8bc" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Credit Risk Drivers]]></title>
            <link>https://medium.com/@EngineFinance/credit-risk-drivers-911a737c2b59?source=rss-f20082e7b2a6------2</link>
            <guid isPermaLink="false">https://medium.com/p/911a737c2b59</guid>
            <category><![CDATA[business]]></category>
            <category><![CDATA[banking]]></category>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[finance]]></category>
            <category><![CDATA[money]]></category>
            <dc:creator><![CDATA[Engine.Finance ( Pavel Mironchyk )]]></dc:creator>
            <pubDate>Sun, 31 Mar 2024 11:00:14 GMT</pubDate>
            <atom:updated>2024-03-31T11:00:14.174Z</atom:updated>
            <content:encoded><![CDATA[<p><em>What are the typical risk drivers of credit? Why the search for risk sometimes goes very far from reasonable? Are the risk drivers for corporates and individuals that fundamentally different? Is it possible to somehow categorise and systematise approaches to risk driver determination?</em></p><p>In lending the ability to assess the risk systematically and precisely is a competitive advantage. At the same time it is important that the risk data gathering process is easy and frictionless for the borrower and for the lender.</p><p>In order to understand what drives the risk and what data to collect, let’s define the risk in clear intuitive terms.</p><p>A typical approach would be to define risk drivers of corporates separately from private individuals. However on the foundation there is a lot of commonalities between these two worlds. For me it would be interesting to draw parallels between corporate and personal lending.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/550/1*6ZG04TRS-uwI6oOZmuQv4Q.png" /><figcaption>A lender stereotype</figcaption></figure><blockquote><em>The credit risk is measured as </em><strong><em>a potential loss</em></strong><em> that lender would incur if borrower fails to fulfil its payment obligations.</em></blockquote><p>The loss manifests in two dimensions <strong>1)</strong> <em>missed interest</em> or inability of a borrower to service timely the loan, and <strong>2)</strong> <em>inability of a lender to recover the principal</em> of the loan.</p><p><strong>On the first point</strong> — fundamentally, a borrower should be always be able to generate enough positive cashflow to pay interest and principal of the loan. A failure to generate enough cashflow constitutes probability of default.</p><p><strong>On the second point</strong> — in order to protect itself from the loss in case of default, lenders may ask for a security (as a protection against loss) in forms of pledged assets, material collaterals (like real estate), and guarantees. By claiming the right and then selling a security a lender may recover the part or full principal. The fancy term for this is the<em> credit risk mitigation</em>.</p><p>Let’s define some pseudo model to explain credit risk:</p><p><strong><em>Lender Risk =NPV (expected loan payments) — NPV (borrower income cashflows) — FV (pledged assets, insurances, guarantees etc)</em></strong></p><p><em>Net present value (expected loan payments) </em>— present value (discounted on timing) of all the cashflows expected to received from borrower</p><p><em>Net preset value (borrower income cashflows) </em>— preset view on cashflows that borrower’s business will generate for lifetime of loan.</p><p><em>Fair value (pledged assets, insurances, guarantees etc if any) </em>— fair market value for assets that are pledged by borrower for lender in case of default. The determination of of fair value has a lot to do with demand on the market for a specific assets. Or whether insurer or guarantor is ready to pay the claim in full. We will not focus on this in the article much.</p><p>If at any point of time the Lender Risk is above zero, the borrower is more likely to generate a loss as a result of a contact.</p><h3>Direct drivers of risk</h3><p>The obvious resultion from the formula — the credit risk can be precisely determined if we know amounts of <em>current and future cashflows</em>.</p><p>Both for individuals and corporates this is difficult to determine. The future cashflows are merely projections and they could change depending on circumstances and operational decisions of corporates. The life of individual can take also rather unexpected turn. Neither corporate nor individual can precisely formulate their exact income over long period or time, neither will stick to a commitment of keeping their expenses at some predefined levels for all time. Plainly, they might not be willing to do so at all.</p><p>Ideally it is impossible to specify precisely all income and all costs for some lengthy period of time ahead.</p><h3>Indirect risk driver</h3><p>The only choice left is to use some assumptions, projections and some signs of whether there is a sufficient level of a discipline, quality of planning and an ability of executing plans by an individual or an organization that would allow repayment of a loan, i.e. generate sufficient cashflows.</p><p>The table below summarises the indirect drivers of risk.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*MBChkRV97Q0yx2A2l2qsUg.png" /></figure><p><a href="https://docs.google.com/spreadsheets/d/e/2PACX-1vS0oJ8zXWaKcS6rgaXi31jdBKiS5hidUNiC_fvUzoUV4qzr1kS5Ms_H6lYHvCLm8GopRwi20e4Iv0yp/pubhtml">Risk Drivers - Google Drive</a></p><p>The indirect risk drivers are not clearly connected to the risk. They are merely a hints. The exact relation to risk needs to be established through an analysis of data.</p><p>In most of the cases there is nonlinear correlation between risk driver, linearly driver and risk might not be correlated at all.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/200/1*t7M7YIlcmTddOdJfrB9vrA.png" /></figure><p>Further in the analysis of dependency it is important to check whether there is a casual relation. In the pursuit for drivers of risk, it is not uncommon to select a driver that coincidently indicate the risk, i.e. in a new sample the relation would not generalize as in old data sample. Another situation to check for is when a driver is filled with awareness of default of after default materialized (time and causality).</p><p>One senior colleague told me when he worked in some eastern european bank they used a proof of car ownership and brand of car (a plastic card everyone carries with a wallet) as a risk driver when offering an instant consumer loan in big electronic stores. Asking a customer to provide many documents on the spot is not very convenient, but a fact on car possession in combination with a car brand was effective mean to filter out some very risky category of customers. An explanation — car price and maintenance costs was a good proxy variable lined to consumer disposable income.</p><h3>Conclusion</h3><p>The credit risk can be very precisely determined only in ideal world. The indirect risk drivers are proxies to measure ability of an individual or an organization to generate sufficient cashflows over the lifetime of a loan.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=911a737c2b59" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Geometric intuition for Artificial Neural Nets / Deep Learning]]></title>
            <link>https://medium.com/@EngineFinance/geometric-intuition-for-artificial-neural-nets-deep-learning-7329bbd7b8da?source=rss-f20082e7b2a6------2</link>
            <guid isPermaLink="false">https://medium.com/p/7329bbd7b8da</guid>
            <category><![CDATA[neural-networks]]></category>
            <category><![CDATA[deep-learning]]></category>
            <category><![CDATA[python]]></category>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[ai]]></category>
            <dc:creator><![CDATA[Engine.Finance ( Pavel Mironchyk )]]></dc:creator>
            <pubDate>Wed, 30 Nov 2022 21:30:12 GMT</pubDate>
            <atom:updated>2022-11-30T21:30:12.155Z</atom:updated>
            <content:encoded><![CDATA[<p>The artificial neural networks are considered to be a black boxes of model inference from data. The majority of tutorial books and introductory texts focus on abilities to approximate and interpolate from multidimensional data, covering backpropagation, warning about overfitting issue and etc. An intuitive ways to comprehend the basic idea on how networks actually work are not discussed to a sufficient level in my humble opinion.</p><p>Building up intuition is important in order to work comfortably with technique in industrial applications.</p><p>And what is intuition anyway? Once I was a student studying theoretical physics, struggling to understand many concepts intuitively, and my supervisor taught me that mathematics and physics has special department for intuition and it is called geometry, and I should look always for geometrical interpretation of anything I studied to master it.</p><p>In this article I argue that feed-forward NNs with hidden layers are intuitive to some extent (with limitation on dimensionality), if you approach them geometrically.</p><p>Further below humblingly I reveal my example to view geometry of NNs as “fields” scores separated by hyperplanes. Scores arise because I primarily focus on classification tasks.</p><p>In the next section a toy dataset will be introduced, a ANNs model manually assembled using only geometrical intuition and then this model is compared with model learned from dataset by gradient descent method.</p><pre># necessary imports<br>import matplotlib.pyplot as plt<br>import numpy as np<br>import pandas as pd<br>import tensorflow as tf</pre><h3>Problem</h3><p>Let’s set up a toy dataset to illustrate inner workings on ANNs. This dataset should represent a simple enought problem to allow to focus on the basics. I propose we work on some artificial classification problem, where the dataset consists of just two features — “x” and “y”, and binary label (0 or 1). If x and y belong to some region in 2D space we assign to label 1 otherwise 0. Again for simplicity let’s use a circle centered a x=0, y=0 as a region shape.</p><pre>model_ds = lambda x,y: ((x*x + y*y) &lt; 1)<br><br># some trivial visualization<br>x,y = np.meshgrid(np.linspace(-2,2,100), np.linspace(-2,2, 100))<br><br>fig = plt.figure(figsize=(10,10))<br>ax = fig.add_subplot(221)<br><br>im = ax.pcolormesh(x,y, model_ds(x,y), shading=&#39;auto&#39;)<br>fig.colorbar(im, ax=ax)<br>ax.set_title(&quot;Unit circle classification problem&quot;)<br><br>ax.set_aspect(1.0);</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/412/1*PH6g4Q42wLzHMrjrL27rOg.png" /></figure><p>On the picture above the yellow circle is class 1, and the rest of x and y are class 0. So our problem is classification whether x and y belong some region in space.</p><p><strong>Neutron with RELU</strong></p><p>Let’s have a close look at the fundamental unit of ANN machinery, a single neuron (or perceptron). The typical mathematical notation for neuron i in layer j:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/592/1*o4-VOhyCzxn0ylWYi_T6XQ.png" /></figure><p>The activation function is essential, if you omit activation function the whole net would collapse to a just a linear regression. There are several activation functions actively used — logistic, RELU. If you look from perspective of classification, an activation function is ultimately responsible for the separation of scores between two classes of inputs.</p><p>Further let’s focus on RELU activation (rectified linear unit), to the current date it is the most popular activation function, reported in in many modern state of the art models. The typical notation for RELU:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/692/1*r2fp16H-x-rEAuZjbk1K6w.png" /></figure><p>What is actually neuron is doing from geometric point of view? From analytical geometry we know the dot product of two vectors represents projection of vector on vector. If we define a vector <strong>W</strong> of neuron weights as an orientation of some plane, we can also factor bias <strong>b</strong>=<strong>W dot O</strong> , so <strong>O</strong> can be seen as a anchor point for a plane. So a artificial neuron can be rewritten as a plane in a space of R of n dimensions:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/645/1*xyeHJeabFxsrtdCnZAuvcw.png" /></figure><p>If point X lies above plane (with origin in <strong>O</strong>, in direction of <strong>W</strong>) the expression above will result in “score” bigger than 0, point below will “score” a negative number. The RELU activation will “cut” output scores only to positive values.</p><p>It turns out that a neuron with RELU activation is some sort of “vector machine” that “separates” space of input values with linear plane in two areas of scores, one area get score 0, on another area all points are scored positively. As a point is farther away from separating plane, the assigned score gets higher.</p><p>To illustrate let’s use our 2D eucledian space as example, there a plane becomes a line.</p><pre># Neuron &quot; vector machine&quot;<br>vmachine = lambda v, p, x: np.maximum(0, np.dot(v/np.linalg.norm(v), (x.T - p).T))<br><br># 2D space <br>x,y = np.meshgrid(np.linspace(-2,2,100), np.linspace(-2,2, 100))<br><br><br>input_space = np.array([x.reshape(-1), y.reshape(-1)])<br>w = np.array([1,2])<br>o = np.array([0,-1])<br><br>fig = plt.figure()<br>ax = fig.add_subplot(111)<br>ax.pcolormesh(x,y, vmachine(w, o, input_space).reshape((100,-1)), shading=&#39;auto&#39;)<br>ax.plot([-2, 2], [0, -2])<br>ax.quiver(0, -1, 1, 2, color=&#39;b&#39;, units=&#39;xy&#39;, scale=2)  <br>ax.set_aspect(1.0)<br>fig.colorbar(im, ax=ax);<br>ax.set_title(&quot;Example of neuron (plane) with $y = RELU(W \cdot X - b)$&quot;);</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/507/1*6Hpf7QW3ZewpVgT9v-WFNQ.png" /></figure><p>The picture below represent the scoring of input space by neuron with weights [1,2] and origin [0, 0] (bias).</p><h3><strong>Graphical derivation of model with one hidden layers</strong></h3><p>Now we have established some intuition around a neuron (with RELU) — it is some sort of scoring machine that split space on area of 0 scores and area with higher scores as they further away from separating plane. Using such knowledge I can construct a model to perform classication for the dataset I made earlier. See below sampling for dataset.</p><pre># create dataset <br>N = 100000<br>ranges = {&#39;low&#39;: -2, <br>          &#39;high&#39;:2}<br>ds = pd.DataFrame()<br>ds[&#39;x&#39;] = np.random.uniform(size=N, **ranges)<br>ds[&#39;y&#39;] = np.random.uniform(size=N, **ranges)<br>ds[&#39;depvar&#39;] = model_ds(ds.x, ds.y).astype(float)<br><br>ds.describe()<br><br>mask = ds.depvar == 1<br>fig = plt.figure()<br>ax = fig.add_subplot(111)<br>ax.scatter(ds.loc[mask, &#39;x&#39;], ds.loc[mask, &#39;y&#39;], s=0.01)<br>ax.scatter(ds.loc[~mask, &#39;x&#39;], ds.loc[~mask, &#39;y&#39;], s=0.01)<br>ax.legend([&quot;yellow 0 score&quot;, &quot;blue 1 score&quot;])<br>ax.set_aspect(1.0)<br>ax.set_title(&quot;Sample point set for toy problem&quot;);PY</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/432/1*8G6EbWKnVsHoZdK2vy9J2g.png" /></figure><p>At first sight an NN with just one hidden layer should do the job of classification fine. The arbitrary chosen design is NN with 2x6x1 (2 inputs 6 neuron in hidden and 1 output).</p><p>Let’s do the following — let’s create 6 planes (neurons) around the [0,0], with vectors W “radiating”, each pointint in different direction. Then combine the scores of six neurons into one with negate sign and see what happens.</p><pre># neuron<br>neuron = lambda v, p, x: np.maximum(0, np.dot(v/np.linalg.norm(v), (x.T - p).T))<br>vect = np.array<br>relu = lambda x : np.maximum(0, x);<br><br>n = 6 # number of neurons in hidden layer<br>x,y = np.meshgrid(np.linspace(-2,2,100), np.linspace(-2,2, 100))<br><br>t = np.linspace(0, np.pi*2.0, n, endpoint=False)<br># coordinates for W vectos<br>u, v = np.cos(t), np.sin(t)<br><br>def model2x6x1(x,y):<br>     z = np.zeros(x.shape)<br>     for i in range(u.shape[0]):<br>          z += neuron(vect([u[i],v[i]]),  # W vector <br>                      vect([0,0]),        # O vector<br>                      vect([x.reshape((-1)),y.reshape((-1))])).reshape(x.shape)<br>     z = 2 - relu(z)<br>     return z<br><br>z = model2x6x1(x,y)<br>fig = plt.figure()<br>ax = fig.add_subplot(111)<br>ax.pcolormesh(x,y,tf.nn.sigmoid(z).numpy().reshape(100,-1), vmin=0, vmax=1)<br>ax.quiver(np.zeros(n), np.zeros(n), u, v, color=&#39;b&#39;, units=&#39;xy&#39;, scale=1);<br>angles = np.linspace(np.pi / 6, 2.*np.pi + np.pi / 6, 7)<br>ax.plot(np.cos(angles), np.sin(angles))<br>ax.set_xlabel(&quot;x&quot;)<br>ax.set_ylabel(&quot;y&quot;)<br><br>fig.colorbar(im, ax=ax);<br>ax.set_aspect(1.0);<br>ax.set_title(&quot;Manually assembled 2x6x1 ANN model&quot;)</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/527/1*80a_-0Uz__Hs0gaWUiGkkA.png" /></figure><p>The picture above shows the scores from some our artificial model. vector of weights from six neurons. The last ouput-layer neuron is :</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/374/1*fAAEISEhpNvagaeIGoeqRg.png" /></figure><p>Why bias is 2? I just visualy chose it to match close area of zero scores with borders of circle. The sigmoid function is used to map from space of relative scores to probabilities. This is needed as I would like to compute AUC score for my model.</p><pre>m = tf.keras.metrics.AUC(num_thresholds=3)<br>m.update_state(ds[&#39;depvar&#39;], tf.nn.sigmoid(model2x6x1(ds.x.to_numpy(), ds.y.to_numpy())).numpy().ravel())<br>auc = m.result().numpy()<br>auc, 2*auc-1</pre><p>(0.9874465, 0.9748929738998413)</p><p>This model gives me 97% GINI precision, which is quite good. However if you inspect carefully the model outputs you would find that is does not separate the borders of unit cirle sharply, the border is rather smooth. Further there rays of relatively higher scores radiating from the center that contributing to classification error.</p><p>Let’s improve this model.</p><h3><strong>Assembly of model with two hidden layers</strong></h3><p>With geometrical intuition established for first layer neurons, is it possible to improve model with introduction of one extra hidden layer? What is our intuation for deeper layer neuron?</p><p>After quite of consideration I concluded that the most intuitive way for me to percieve higher order neuron is just imagine it as weighted superposition of “scoring fields” supplied by previous layers. The bias is just a calibration that allows to level up the suposition to some thereshold.</p><p>It still possible to employ the “plane” intuitve abstraction we describe earlier. However this become difficult to imagine as basis for the space becomes non-euclidean — the orientation vectors from previous layer act as basis axes. So superposition of fields of scores works for me at least.</p><p>To proceed let’s build a model with additional hidden layer (2x6x2x1). Our aim is to impove model from above aven further, by sharpening the border near edge of unit circle. A straightforward way to do this is to create field of scores behind the unit circle and substruct it from the filed we made in previous model.</p><pre>sigmoid = lambda x : tf.nn.sigmoid(x).numpy().ravel()<br><br>layer2_1 = neuron(vect([u[0],v[0]]), <br>                  vect([u[0],v[0]]), <br>                  vect([x.reshape((-1)),y.reshape((-1))]))<br>for i in range(1, u.shape[0]):<br>     layer2_1 += neuron(vect([u[i],v[i]]), <br>                        vect([u[i],v[i]]), <br>                        vect([x.reshape((-1)),y.reshape((-1))]))<br>layer2_1 = relu(layer2_1)<br><br><br>layer2_2 = neuron(vect([u[0],v[0]]), <br>            vect([0,0]), <br>            vect([x.reshape((-1)),y.reshape((-1))]))<br>for i in range(1, u.shape[0]):<br>     layer2_2 += neuron(vect([u[i],v[i]]), <br>                          vect([0,0]), <br>                          vect([x.reshape((-1)),y.reshape((-1))]))<br>layer2_2 = relu(layer2_2) <br><br>layer3 = np.dot(vect([-1, -2]), vect([layer2_2, layer2_1])) + 2 <br>z = sigmoid(layer3)<br><br>fig = plt.figure(figsize=(8, 8))<br>ax = fig.add_subplot(221)<br>ax.pcolormesh(x,y, sigmoid(layer2_1).reshape(100,-1))<br>ax.set_aspect(1.0)<br>ax.set_title(&quot;Layer 2 Node 1 Scores&quot;)<br><br>ax = fig.add_subplot(222)<br>ax.pcolormesh(x,y, sigmoid(layer2_2).reshape(100,-1) + 2)<br>ax.set_aspect(1.0)<br>ax.set_title(&quot;Layer 2 Node 2 Scores&quot;)<br><br><br>ax = fig.add_subplot(223)<br>ax.pcolormesh(x,y, z.reshape(100,-1))<br>angles = np.linspace(0, 2.*np.pi, 100)<br>ax.plot(np.cos(angles), np.sin(angles))<br>ax.set_aspect(1.0)<br>fig.colorbar(im, ax=ax);<br>ax.set_title(&quot;Model (Layer22 - Layer21) Scores&quot;)<br><br><br>m = tf.keras.metrics.AUC(num_thresholds=3)<br>m.update_state(model_ds(x.ravel(),y.ravel()), z)<br>auc = m.result().numpy()<br>auc, 2*auc-1</pre><p>(0.9871159, 0.9742318391799927)</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/684/1*Xz5UnL0f8ayaw94AxVAlaA.png" /></figure><p>Qua discriminatory power this second model does not outperform the latter model. However practically it is more closer to ideal dataset and it is much more useful. Also scores are more consistent (star like rays are eliminated).</p><h3><strong>Training model from data</strong></h3><p>Just to challenge models assembled before let’s train a model directly from data. I will use <em>tenfsorflow </em>framework for that.</p><p>In order to apply the same intuition driven visualization (colormaps of scores/probabilities) I will replace standard Dense layer with “planes”, where bias is replace with dot product b=W dot O. This will increase number of parameters in the model, but it should work for our problem dataset.</p><pre># The layer<br>class GeometricLayer(tf.keras.layers.Layer):<br>    def __init__(self, num_outputs, relu=True):<br>        super(GeometricLayer, self).__init__()<br>        self.num_outputs = num_outputs<br>        self.relu = relu<br>    <br>    def build(self, input_shape):<br>        self.normal = self.add_weight(&quot;normal&quot;, <br>                    shape=[int(input_shape[-1]),<br>                            self.num_outputs])<br>                            #initializer=tf.keras.initializers.RandomNormal(stddev=1))<br>        self.offset = self.add_weight(&quot;offset&quot;, <br>                    shape=[int(input_shape[-1]),<br>                            self.num_outputs],<br>                            initializer=tf.keras.initializers.Zeros())<br><br><br>    def call(self, inputs):<br>        n = inputs.shape[-1]<br>        y = tf.multiply(tf.reshape(inputs, (-1, n, 1)) - self.offset, self.normal) <br>        y = tf.reduce_sum(y, axis=-2)<br>        if self.relu:<br>            return tf.nn.relu(y)<br>        return y<br><br>model = tf.keras.Sequential([<br>    tf.keras.Input(shape=(2,)),<br>    GeometricLayer(6),<br>    GeometricLayer(2),<br>    GeometricLayer(1, relu=False), # sigmoid will applied later, training will be done in score space<br>])<br><br>model.compile(optimizer=&#39;adam&#39;, <br>  loss=tf.keras.losses.BinaryCrossentropy(from_logits=True, <br>                                           label_smoothing=0.1), <br>    metrics=[&#39;AUC&#39;, &#39;accuracy&#39;])<br><br>h = model.fit(ds[[&#39;x&#39;,&#39;y&#39;]], ds[&#39;depvar&#39;], batch_size=100, <br>    epochs=10)</pre><p><em>Epoch 1/10 1000/1000 [==============================] — 8s 6ms/step — loss: 0.3673 — auc: 0.5540 — accuracy: 0.8048 <br>Epoch 2/10 1000/1000 [==============================] — 4s 4ms/step — loss: 0.1155 — auc: 0.9255 — accuracy: 0.9429 <br>…<br>Epoch 10/10 1000/1000 [==============================] — 4s 4ms/step — loss: 0.0173 — auc: 0.9902 — accuracy: 0.9921</em></p><pre># first layer<br>U1, V1 = model.variables[0].numpy()<br>X1, Y1 = model.variables[1].numpy()<br><br># second layer<br>U2, V2 = model.variables[2].numpy().T<br>X2, Y2 = model.variables[3].numpy().T<br><br># third output<br>U3, V3 = model.variables[4].numpy()<br>X3, Y3 = model.variables[5].numpy()<br><br># Layer 1<br>L1 = []<br>for i in range(6):<br>    L1.append(vmachine(np.array([U1[i],V1[i]]), <br>                           np.array([X1[i],Y1[i]]), <br>                           np.array([x.reshape((-1)),y.reshape((-1))])).reshape(100,-1))<br><br><br># Layer 2<br>z1 = np.dot(U2, (np.array(L1).reshape((6, 10000)).T - X2).T)<br>z1 = np.maximum(0, z1)<br><br>z2 = np.dot(V2, (np.array(L1).reshape((6, 10000)).T - Y2).T)<br>z2 = np.maximum(0, z2)<br><br># Layer 3<br>Z = sigmoid(U3 * (z1 - X3) + V3 * (z2 - Y3))<br><br># Visualization<br>fig = plt.figure(figsize=(8, 8))<br>ax = fig.add_subplot(221)<br>ax.pcolormesh(x,y,sigmoid(z1).reshape((100,100)), shading=&#39;auto&#39;)<br>ax.set_aspect(1.0)<br>ax.set_title(&#39;Score field Layer 2 N 1&#39;)<br><br>ax = fig.add_subplot(222)<br>ax.pcolormesh(x,y,sigmoid(z2).reshape((100,100)), shading=&#39;auto&#39;)<br>ax.set_aspect(1.0)<br>ax.set_title(&#39;Score field Layer 2 N 2&#39;)<br><br><br>ax = fig.add_subplot(223)<br>ax.quiver(X1, Y1, U1, V1, color=&#39;b&#39;, units=&#39;xy&#39;, scale=2)  <br>ax.set_xlim([-2,2])<br>ax.set_ylim([-2,2]);<br>ax.set_aspect(1.0)<br>ax.set_title(&#39;Layer 1 Orientations&#39;)<br><br><br>ax = fig.add_subplot(224)<br>ax.pcolormesh(x,y,Z.reshape((100,100)), shading=&#39;auto&#39;)<br>ax.set_aspect(1.0)<br>fig.colorbar(im, ax=ax);<br>ax.set_title(&#39;Trained model&#39;)<br><br><br>result = model.predict(np.array([x.reshape((-1)),y.reshape((-1))]).T)<br>p = tf.nn.sigmoid(result).numpy().ravel()<br>p.mean()<br><br>m = tf.keras.metrics.AUC(num_thresholds=3)<br>m.update_state(model_ds(x.ravel(),y.ravel()), p)<br>auc = m.result().numpy()<br>auc, 2*auc-1</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/685/1*QRCACGHak9xrq99IMSXqnA.png" /></figure><p>As we can see for our ideal dataset this particular model is far from ideal. Actually there are infinite number of ways to derive a decent model with ~98–99% accuracy for classification purposes on our dataset.</p><p>While writing this article I have tried many times to rebuild a model (with different initial state and various learning / loss parameters), few times of was really surprised by final arrangement of “score fields” and how “visually suboptimal” it is.</p><p>This proves a point that analytical (visually clear) model built on deep understanding of underlying process potentially can be better than data driven model, and more explainable.</p><h3><strong>Conclusion</strong></h3><p>This article is written mainly for infotainment. However there are some useful ideas that I wanted to convey along.</p><p>So it is possible to apply some geometrical intuition to convolutional / dense networks.</p><p>With that intuition you can <em>influence design of NN</em> in many ways — make it more explainable to final user, limit the model risk, exclude specific interactions from learning, etc. This can be implemented though introducing of constraints on <strong>W </strong>and <strong>O </strong>parameters of neuron, i.e. by limiting direction of <strong>W </strong>you can choose what interactions between input variable to allow into model and etc.</p><p>Hope you would this idea useful for your problem.</p><h3><strong>Appendix</strong></h3><p>Here I will output few alternative models I got from runinng over and over the code from above as weights are initialized differently.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/685/1*88cRNob9lDZa4sgNtSPwVA.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/685/1*_d-L4fkjqzGr85LKNx1G3Q.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/685/1*FfNKjXQuHH2Qqq3Gzk_6vQ.png" /></figure><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=7329bbd7b8da" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Agile manifesto for banks]]></title>
            <link>https://medium.com/@EngineFinance/agile-manifesto-for-banks-a00b006b6db6?source=rss-f20082e7b2a6------2</link>
            <guid isPermaLink="false">https://medium.com/p/a00b006b6db6</guid>
            <category><![CDATA[banking]]></category>
            <category><![CDATA[agile]]></category>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[finance]]></category>
            <category><![CDATA[it]]></category>
            <dc:creator><![CDATA[Engine.Finance ( Pavel Mironchyk )]]></dc:creator>
            <pubDate>Fri, 04 Nov 2022 12:26:51 GMT</pubDate>
            <atom:updated>2022-11-07T08:48:56.557Z</atom:updated>
            <content:encoded><![CDATA[<p>In this article I would like to apply or translate the agile manifesto to traditional financial institutions, banks. The manifesto originated in software engineering goes much deeper than just ditching waterfall planning in favor of short iterations, it is also a model how whole organization can function.</p><p><em>In order to apply agile principles I will introduce concept of </em><strong><em>the Product,</em></strong><em> that should be developed, a right understanding of what is actually the product of the bank.</em></p><p>Debatably these are the features of a relevant and healthy financial organization:</p><ul><li>continuously undergoing transformation</li><li>adequately and timely reacts to market demand</li><li>embracing regulatory changes</li></ul><p>Let’s see how agile manifesto actually facilitates all of this if we have right definition of banking product.</p><p><strong>Regulation and Innovation</strong></p><p>Before diving into the manifesto I would like to focus on the last bullet point, because the compliance is perceived typically somewhat negatively. Adhering to regulation is not just matter of costs and avoiding even more costlier fines, but actually it is about being on the ahead of competition and being innovative.</p><p>Banks, large corporate lenders (at least those lenders and insurers that have reached a substantial size on the market) cannot be that easily compared to IT companies. Where IT companies are focusing entirely on satisfaction of customer’s needs and being ahead of the competition, the systemic financial institution have very important societal role. Policymaker using financial regulation to ensure long term stability and prosperity of society, and even to reform society — think about energy transition, resisting climate change, fighting inequality and discrimination etc.</p><p>Being strongly supervised is what makes bank so complicated on internal governance and decision making in comparison with IT companies. Despite that the banks and lenders are very much market driven companies that try to innovate to be ahead of competition, paradoxically <strong><em>it is the strive for the regulatory compliance what often drives the innovation transformation</em></strong> in them.</p><p>If there is way to offer products faster and cheaper, banks will surely invest in such innovation, as it drives sales and helps to increase market share. However things like economic downturn risk management, due diligence, open banking APIs, mandatory sharing/reporting economical data with society are seen as burden and costs caused by regulation. In reality the society and banks benefits a lot from these.</p><p>And funny enough the most notable developments that happened in the banking and that made them so fundamental for modern economical systems in the last decennials are triggered by regulation. For instance, the standardization with Basel rules as response to crisis of 2008 helped to develop a much more healthier view on creditworthiness of clients to their benefits, and in general increase stability of sovereign financial systems, that were able to survive through nightmare of COVID-19. Notable that those organizations that were able to become compliant faster than competitors, were able to win more trust and new clients.</p><p>The financial services are backbone of any sovereign country, it is worth to introduce some costly reforms which would benefit everyone at the end. Despite the rise of DEFI and crypto, it is still banks that people trust their life savings, and it is still banks that hold countries and governments from falling apart.</p><h3>Organization</h3><p>In order to translate the Agile manifesto, let’s see how typical bank is organized.</p><p>These is a typical high level divisions in a bank:</p><ul><li><strong>Front desks</strong> (retail, corporate lending desks)<br>These fellows are business owners, client facing managers. They also take risks and produce data. They face customers mainly.</li><li><strong>IT Middleware</strong><br>These are IT people, responsible for the IT infrastructure, accounting software, databases.</li><li><strong>Back-offices</strong><br>These are the risk managers, compliance, legal, accounting, reporting officers. These fellows have also the external customer - supervisor, government agencies</li><li><strong>Management Board </strong><br>These fellows take strategic decisions, set the risk appetites.</li></ul><p>In reality in a large bank you can find all kind of departments with various names, however you can always classify any department into one of the architype above.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/825/1*RkAiaTxF-l-bm-S62YbHYA.png" /></figure><p>It would like to introduce the concept of the bank backbone — all infrastructure that connects front, backend through middleware and allows flowing of data, analytics, decisions through the organization to external clients and regulators.</p><p>This backbone further on should be called as <strong>the Product.</strong></p><p>There is big confusion in banks what is the core product. It is certainty not a financial contract that bank issues on its own— loans, guarantees. It is the whole system that services and supports it, and makes it compliant.</p><p><em>The typical mistake is to try to apply agile principles on the the level other than </em><strong><em>the Product</em></strong>, on some more shallow level, on department level, and define product as some little service that this department is doing for the whole chain. Because of the complexity this becomes just another management clog of interdependencies to address and to align, and benefits of agile approach becomes nihil.</p><h3>Agile manifesto</h3><p>Below I will formulate a bank-applied translation of fundamental agile postulates, with the <strong>Product</strong> in mind.</p><ul><li><strong>Direct interaction of personnel from front desks, middleware and back-offices on the Product</strong> over formal boards and delegations<br><em>Original: Individuals and interactions over processes and tools</em></li><li><strong>the Product actually serves the needs organization today</strong>, over comprehensive design for the future<br><em>Original: Working software over comprehensive documentation</em></li><li><strong>Front, middle and back collaborate in single squads to address actual customer need</strong>s over formal requirements writing<br><em>Original: Customer collaboration over contract negotiation</em></li><li><strong>Short (weekly to month) iterative updates to the Product to respond to market and regulatory environment changes</strong> over long term planning and big updates<br><em>Original: Responding to change over following a plan</em></li></ul><p>So it does make sense now, doesn’t? The agile postulates applied to the Product level facilitate a more customer focused requirements definition, more precise measurement of budgets, reduce risk of not validating customer expectation.</p><p>Whether there is desire to implement this in reality, is another topic.</p><p>What is the thought that I am trying to convey here?</p><p><em>In order to to make bank agile, interdisciplinary and wholistic approach should be taken to development of the whole chain that connects customers, regulators and internal offices of the banks.</em></p><p><strong>Appendix</strong></p><p>Below there are the agile basic principles translated to to concept of Product and bank.</p><p>Our highest priority is to satisfy the customer<br>through early and continuous delivery<br>of the valuable <em>Product.</em></p><p>Welcome changing requirements, even late in<br>development. Agile processes harness change for<br>the customer’s competitive advantage.</p><p>Deliver working <em>the Product</em> frequently, from a<br>couple of weeks to a couple of months, with a<br>preference to the shorter timescale.</p><p><em>Front, middle and back office</em> must work<br>together daily throughout the project.</p><p>Build projects around motivated individuals.<br>Give them the environment and support they need,<br>and trust them to get the job done.</p><p>The most efficient and effective method of<br>conveying information to and within a development<br>team is face-to-face conversation.</p><p><em>the Product</em> is the primary measure of progress.</p><p>Agile processes promote sustainable development.<br><em>The front, middle and back people</em> should be able<br>to maintain a constant pace indefinitely.</p><p>Continuous attention to technical excellence<br>and good design enhances agility.</p><p>Simplicity — the art of maximizing the amount<br>of work not done — is essential.</p><p>The best architectures, requirements, and designs<br>emerge from self-organizing teams.</p><p>At regular intervals, the team reflects on how<br>to become more effective, then tunes and adjusts<br>its behavior accordingly.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=a00b006b6db6" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Reducing variance in socio-economical models]]></title>
            <link>https://medium.com/@EngineFinance/reducing-variance-in-socio-economical-models-aa7a8a1afbf?source=rss-f20082e7b2a6------2</link>
            <guid isPermaLink="false">https://medium.com/p/aa7a8a1afbf</guid>
            <category><![CDATA[economics]]></category>
            <category><![CDATA[econometrics]]></category>
            <category><![CDATA[analytics]]></category>
            <category><![CDATA[modelling]]></category>
            <category><![CDATA[machine-learning]]></category>
            <dc:creator><![CDATA[Engine.Finance ( Pavel Mironchyk )]]></dc:creator>
            <pubDate>Tue, 07 Dec 2021 22:43:45 GMT</pubDate>
            <atom:updated>2021-12-07T23:02:00.968Z</atom:updated>
            <content:encoded><![CDATA[<p>In this article I would like to touch upon a typical fallacy of modelling in business — to improve model forecasting you need to increase complexity of the model. The context I have in mind is the credit risk scoring, however the main idea I am discussing here is applicable to any<em> socio-economical modelling</em>, where the scoring of individuals or groups occurs for the the economic purposes.</p><p><strong>Problem</strong></p><p>Let’s imagine a typical organization that scores customers for risk, adjusts dynamically pricing based on data. As organization matures, more and more investment made into retrieving insights from the existing data. More and more investments made into <em>sophisticated analytical tools</em>, attempts made to adopt more <em>complicated modelling methodology</em>. However only some <em>marginal successes</em> are booked so far, no real breakthroughs established. Recognizable? I thought so.</p><p><strong>Cause</strong></p><p>Let’s make a step back and review in simple words what it means to score systematically some individual or an organization of individuals (which behavior is likely determined by few personalities) for risk.</p><p>A scoring model is a system to forecast behavior. Learning it from data means that under specific conditions some specific type of decisions observed in the past, and it is fair to anticipate that under similar conditions the same decision to be repeated by a similar individual.</p><p>Should I start with that in the real world individuals do not always act rationally? Our behavior is partly rational it is influenced by our personal nudges, aspirations. And, what is very important by external factors — cultural, economical, political. We are part of very big system called society with not very deterministic rules. In taking economic decision there so many factors involved that we cannot even define, because we are even unaware of it. The <strong>external factors constantly evolve</strong>, forcing changes in how we perceive conditions and it may not lead to taking same decision response as it was in the past.</p><p>A more scientific explanation for this situation —learning a model for the system where <em>all variables cannot be observed</em>, and afterthere is <strong>hidden regime change occurring</strong> (hidden variables change their value), a model performance degrades.</p><p>After a model performance worsens, a typical response is to employ more deeper and complex models with more recent data included. So the focus is always <strong>on trying to get more out of existing data</strong>!</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/684/1*UXmLBVnYCznRKD1YmxNfKg.png" /></figure><p><strong>Solution</strong></p><p>What I am advocating here, is a change of paradigm — shift focus from modelling toward collection of risk drivers, to be explained below. Ridiculous focus on modelling techniques leads to a confusion for business - the <strong>intuition gets replaced with black boxes</strong>, unwarranted trust given to<strong> non-causal relations</strong> learned from the data.</p><p>We have to understand that building models for social-economical systems is different from other type of models like for example in computer vision, or voice recognition. We cannot conduct experiment easily to collect more complete data, we cannot even verifiably reproduce results from recent past as economy and society constantly changes — the system is out of control and we cannot revert it back. Unlike in computer science, we cannot conduct experiment with very same conditions to see whether we can collect more features or more richer dataset.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/667/1*cR5ShqfsTUaQHQIFS2hc8g.png" /></figure><p>So the approach should be different — disentangle and linearize relations, focus on unexplainable variance in your results, look for direct variables or reasonable proxies that can reduce variance.</p><p>The following steps give the “blueprint” of the idea:</p><ol><li>Simplify and linearize models<br>The changes in unobserved hidden variables cannot be always easily seen. Also by nature, the social-economical data is never complete, verification of result is not fully possible. It is likely that non-causal ( non-dependent) relation is learned, full understanding of relation is required for verification, hence striving for simpler linear models is important.</li><li>Quantify unexplained variance<br>The errors between prediction an observation in linearized model give good ground for research. It becomes possible to factorize errors and ideate on what actually can drive differentiation of the risk.</li><li>Capture unexplained variance<br>Collect promising risk drivers as early as possible, costs of collecting and storing data is becoming lower. Where there is an obstacle for collecting risk drivers (privacy, regulation or costs reasons), look for proxies. Correlate with error residuals as early as possible to recognize benefits of drivers.</li></ol><p><strong>Summary</strong></p><p>Building applied social-economical models is very different as data does not capture fully macro and micro economic environment state and other context (cultural and political).</p><p>The key to success is establishing a process that leads to continuing effort of identification proper risk drivers and inclusion of them into input data stream.</p><p>Premature focus on sophisticating modelling methodology for the purpose to get most of the data could lead to devaluation of modelling efforts.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=aa7a8a1afbf" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Credit risk challenger models, a road towards innovation]]></title>
            <link>https://medium.com/@EngineFinance/credit-risk-challenger-models-a-road-towards-innovation-2de2c885feb4?source=rss-f20082e7b2a6------2</link>
            <guid isPermaLink="false">https://medium.com/p/2de2c885feb4</guid>
            <category><![CDATA[banking]]></category>
            <category><![CDATA[neural-networks]]></category>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[risk-management]]></category>
            <category><![CDATA[credit]]></category>
            <dc:creator><![CDATA[Engine.Finance ( Pavel Mironchyk )]]></dc:creator>
            <pubDate>Tue, 27 Nov 2018 15:44:50 GMT</pubDate>
            <atom:updated>2018-11-27T15:44:50.455Z</atom:updated>
            <content:encoded><![CDATA[<p><strong>What is this post about?<br></strong>This post is about innovation in credit risk modelling in large lending organizations. The content might be interesting for ambitious professionals in old fashion commercial banking. This is about attempt of integrating machine learning in regulated environment, transparency, accuracy and overfitting. In the end of article I will come to conclusions that should be considered as invitation for discussion.</p><p><strong><em>A little disclaimer<br></em></strong><em>This post is my personal opinion and it does not represent opinion of my current employer.</em></p><h3>Credit risk models</h3><p>In a typical financial company offering retail or corporate loans the management of credit risk is at the core of business. The loans are priced proportionally to credit risk level, the risk appetite is determined with credit risk acceptance levels. The success and profitability of a company depends on the quality of credit risk models.</p><p>The large lenders with banking licence are also subjects of heavy regulatory supervision. The regulatory supervision aims at maintaining stability of financial system and it put strict requirements for capital buffers that banks needs to maintain in order to survive through unexpected bad times. In Europe the foundation is <a href="http://www.eba.europa.eu/regulation-and-policy/single-rulebook/interactive-single-rulebook/-/interactive-single-rulebook/toc/2">Credit Requirement Regulation </a>, refined and clarified by EBA and ECB into numerous guidelines and standards. This is a massive regulatory directive package that goes as far as specifying modelling approaches in general terms that banks need to follow till governance of credit risks and even embedding of credit risk models in IT and business processes. Banks are limited to thee approaches when it comes to credit risk modelling — standardized, foundation internal-rating-based and advanced internal-rating-based approaches. Only the latter two approaches — F-IRB and A-IRB mean more or less a development of an actual mathematical model, with former one meaning no statistical model but just an accounting exercise.</p><p>For banks the development of F/A-IRB model is usually an expensive project with a lot of personnel from different departments involved — new model often means that organisation has to adapt a series of internal transformations in IT, front office and so on. And supervisors are invited to review and approve the model at a very final stage before bank can actually start using it.</p><p>Obviously this means the banks want be more on safe side when it comes to modelling innovation. The developed models are meant to be approved in single iteration, with maybe just few minor changes following after review. This also means development of simple, linear and methodologically conservative models, explainable to regulator and to business people. By explainable I mean the model is transparent that its outcomes can be related to inputs in intuitively understandable ways.</p><p><strong>Non-innovative approach</strong><br>Therefore a typical way to development of model looks like the following:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/232/1*ywfYvh10g_uehw25l0JEog.png" /><figcaption>A typical approach to model development</figcaption></figure><p>In step (1) the existing regulation, internal policies and expert opinions are distilled into strict methodology guide that modelers should follow to produce models in step (2). Such internal guide restricts models to be of specific types — for instance logistic/linear regressions, survival models or time series etc; with specified diagnostic criteria and development steps.</p><p>This approach does not foster much innovation. In such environment where everything turns about cost efficiency, nobody would even try to check, out of curiosity, how deep learning or gradient boosted trees methods perform on the task. As the result of this self restricting the bank may be seen as less innovative and attractive place for young talent. And this is sad. Although the simpler explainable models in finance does not mean model has much worse performance or discriminatory power, on opposite it could be that simpler explainable model is better to use on noisy financial data with a lot of hidden regime changes. But anyway for innovation we need to maintain a healthy level of exploration to see what is possible with available data in this connected world.</p><p><strong>Approach of Innovation</strong><br>Banks do not have to change much in existing processes and give up on explainability and business. Banks simply have to challenge themselves with more innovation at processes in order to become more attractive and interesting places to work. My proposal of approach to development of the new model in banking is the following:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/405/1*DWMFQOQ-97D2akSw1avGxQ.png" /><figcaption>The approach for credit risk modelling</figcaption></figure><p>As a first step (1), go wild and develop model using the most innovative techniques, focus on maximum performance and less of business applicability of it. In the next step (2), linearize and simplify the model for the purpose of business applicability and regulatory approval. This way we gain insights into what state of the art is possible and what sacrifices (technically) we need to make to satisfy stakeholders. The methodology guide now will not be limiting innovation but it will be helping making innovation more business applicable.</p><h3>Illustrative example</h3><p>This is the second part of this article, where I would like to demonstrate approach of innovation to build credit risk models. However I will not build credit risk model nor use real financial data although I was tempted to do so. There are number of reasons — at first, it is not easy to find publicly available decent dataset for credit risk that is well researched, at second I just want to make article more popular, more understandable for everybody.</p><p>So instead of building model to recognize default/loss risk level of customer file with hundreds of parameters, I will be building model to recognize handwritten digit from picture with hundreds of pixels.</p><p><strong>Dataset</strong><br>The dataset I am going to use consists of 28x28 images grayscale images with a digit class class assigned. Think of each image as of file with 784 parameters and a digits as of class we need to predict. See example below.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/635/1*MtYSM2DfmGD4-s6s1x4REQ.png" /><figcaption>Two examples from MNIST dataset</figcaption></figure><p><strong>Step1. Build a challenger model<br></strong>The first step is to build a “state of the art” challenger model that is not bounded by policies. I need to use innovative approaches and techniques. Since my problem is machine vision problem I used deep learning techniques that are reported to perform very well on such class of problems. Alternatively I could have used gradient boosted decision tree methods. Below there is python notebook with implementation of DNN model using Keras framework. This challenger I borrowed from repository of Everett Robinson.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/e5d39e0a26e0fe7c14dc2ec2f6603d03/href">https://medium.com/media/e5d39e0a26e0fe7c14dc2ec2f6603d03/href</a></iframe><p>The implemented model gives precision of around <strong>99%</strong> on validation dataset. This is very impressive, also because it took just one-two minutes of training on my GPU. This model is complex enough and probably overfitted. With 9 neural layers it is hard to explain how individual parameters/pixels contribute to final decision. Using such a complicated model can be seen as risk so we we try to linearize the model.</p><p><strong>Step 2. Linearize challenger model<br></strong>Now we are going to check if simpler, more linear approach can deliver us reasonable results. To begin I would start training a multinomial Logistic Regression with L1 regularizer, a workhorse for credit risk and generally any classifier tasks. For the input of logistic regression I will use 784 pixel intensities normalized — from 0 (fully dark) to 1(fully lit). I would set some small L1 punishment value, determined on several trials, just to prevent from overfitting.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/4f4f73afa22ea9c22bc249e074455666/href">https://medium.com/media/4f4f73afa22ea9c22bc249e074455666/href</a></iframe><p>This model gave me precision of <strong>91–92%</strong>.<strong> </strong>This is quite good however there is 8 percent precision gap with challenger model.</p><p>The initial convolutions and max-pooling layers of DNN model are believed to help model to recognize also digits of the same class under affine transformations like rotation and shear. Let’s try to make our linear model to pickup similar transformations, for that I need to introduce second order terms — product of pixel on other pixel. To do that I just added array from kronecker product of image on image itself, note that I scaled images down in size just reduce number of extra features.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/293/1*zddY9pixwwF_ua43j-M2Jw.png" /></figure><p>These extra features helped to improve precision of the model on validation dataset to <strong>96%</strong>, see notebook above. This is quite close to performance of DNN, but the model is much simpler.</p><p>I can add more features to my multinomial logistic regression and eventually reach exact same level of performance as DNN. But for this example I would like to stop and compare this model to DNN.</p><p><strong>Compare Challenger and Linearized model<br></strong>DNN model I have, might be overfitted to the data. The deeper the DNN is, the more “informational capacity” it has to fit to peculiarities of dataset. Let’s “attack” DNN challenger model by generating adversary dataset that leads to severe mispredictions.</p><p>For that I used a <a href="https://github.com/tensorflow/cleverhans">“cleverhans” framefork</a> built on tensorflow. Using Fast Gradient Sign Method (FGSM) algorithm and playing with parameters, I was able to generate slightly modified digits (adversary data), that visually are not that different from original image, however are not wel recognized by DNN. See examples of adversary images below:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/707/1*6h8TEMkY8mqZrA2VS9PXAA.png" /><figcaption>Generate adversary dataset</figcaption></figure><p>DNN demonstrated only <strong>9%</strong> precision on this adversary dataset. But the linearized model s robust enough to recognize <strong>93%</strong> of adversary dataset correctly.</p><p><strong>So what just happened</strong>: <br><strong>1.</strong> The challenger model DNN model was built<br><strong>2.</strong> Then the linearized model was built, its performance was brought to level comparable to the challenger model, using insights coming from challenger model.<br><strong>3.</strong> I have used the challenger model to produce <strong>extra data </strong>to validate that simpler linearized model turned out to be more robust for the task.</p><p><strong>Conclusion</strong><br>It would be beneficial for banks to change the approach to model development methodology. Instead of designing workflow where the model development is targeted from the beginning till the end at producing a simple explainable model that would certainly pass regulatory review, and thus limiting innovation in modelling, banks could introduce a workflow were innovative and not so explainable models are developed from the beginning and then linearized and simplified till the level of regulatory acceptance and business use. This would allow bank to be more innovative and develop more competence in modern data modelling methods.</p><p><strong>Extra Disclaimer<br></strong>The illustrative example is just illustrative, there are number of technical aspects of this article that were not properly addressed for sake of clarity of this approach advocated by this article.</p><p>P.S. A special thank you to Viktor Tchistiakov (Rabobank).</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=2de2c885feb4" width="1" height="1" alt="">]]></content:encoded>
        </item>
    </channel>
</rss>