<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Ann Vitiuk on Medium]]></title>
        <description><![CDATA[Stories by Ann Vitiuk on Medium]]></description>
        <link>https://medium.com/@annvityck?source=rss-d72057642e80------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/0*kzP0DbJ2RljmLwPw.</url>
            <title>Stories by Ann Vitiuk on Medium</title>
            <link>https://medium.com/@annvityck?source=rss-d72057642e80------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Sat, 21 Nov 2026 18:04:25 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@annvityck/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[SHAPMIThe importance of Feature Importance]]></title>
            <link>https://medium.com/@annvityck/shapmithe-importance-of-feature-importance-113e58e7a0e5?source=rss-d72057642e80------2</link>
            <guid isPermaLink="false">https://medium.com/p/113e58e7a0e5</guid>
            <category><![CDATA[feature-selection]]></category>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[feature-engineering]]></category>
            <category><![CDATA[feature-importance]]></category>
            <category><![CDATA[data-science]]></category>
            <dc:creator><![CDATA[Ann Vitiuk]]></dc:creator>
            <pubDate>Thu, 24 Mar 2022 14:51:43 GMT</pubDate>
            <atom:updated>2022-03-24T14:53:03.285Z</atom:updated>
            <content:encoded><![CDATA[<h3>The importance of Feature Importance</h3><p>Any experienced data scientist knows — the model makes a minor contribution to its score, the major role plays in the quality of the data, and especially, the relevance of the feature space (X) to its target (y). Especially, this is true for time-series and stochastic processes problems, which are said to be strongly prone to overfitting.</p><p>Overfitting happens when a model does not generalize well. Instead, it learns the detail and noise in the training data to the extent that it negatively impacts the performance of the model on new data. Given noise or stochastic data — you’re resulting with a very unstable, overfit model. Some real-world problems deal with quite noisy, redundant data. So the main problem becomes — how to avoid overfitting and build a stable model. One of those techniques is to stack more layers in a deep learning model, which is costly, performance-heavy, and not so efficient. The other one is — performing strong feature engineering and feature selection tasks, which is much less performance heavy (if made correctly), much cheaper, and lets us use much lighter models with comparable scores to the deep neural nets.</p><p><em>You can also check my </em><a href="https://www.youtube.com/watch?v=eEx0HgicmRU&amp;t=1538s"><em>recent webinar</em></a><em> on this topic (2022.02.19)</em></p><pre><a href="https://github.com/Infatum/Feature-Importance/blob/main/Features%20Selection.ipynb">Feature Selection Repository</a></pre><blockquote><strong>Let’s start with generating synthetic data for the classification problem (X,y) composed of the three kinds of features:</strong></blockquote><ul><li><strong>Informative</strong>: <em>These are features that are used to determine the label.</em></li><li><strong>Redundant</strong>: <em>These are random linear combinations of the informative features. They will cause substitution effects.</em></li><li><strong>Noise</strong>: <em>These are features that have no bearing on determining the observation’s label.</em></li></ul><pre><strong>def</strong> get_test_data(<br>    n_features<strong>=</strong>40, n_informative<strong>=</strong>10, n_redundant<strong>=</strong>10, n_samples<strong>=</strong>10000, n_classes<strong>=</strong>2, weights<strong>=</strong>[0.5, 0.5], shuffle<strong>=True</strong><br>):<br>    <em># generate a random dataset for a classification problem</em><br>    <strong>from</strong> sklearn.datasets <strong>import</strong> make_classification<br><br>    trnsX, y <strong>=</strong> make_classification(<br>        n_samples<strong>=</strong>n_samples,<br>        n_features<strong>=</strong>n_features,<br>        n_informative<strong>=</strong>n_informative,<br>        n_redundant<strong>=</strong>n_redundant,<br>        shuffle<strong>=</strong>shuffle,<br>        n_classes<strong>=</strong>n_classes,<br>        weights<strong>=</strong>weights,<br>    )<br>    df <strong>=</strong> pd<strong>.</strong>date_range(periods<strong>=</strong>n_samples, end<strong>=</strong>pd<strong>.</strong>datetime<strong>.</strong>today(), freq<strong>=</strong>FREQ)<br>    df0 <strong>=</strong> pd<strong>.</strong>DatetimeIndex(df)<br>    trnsX, y <strong>=</strong> pd<strong>.</strong>DataFrame(trnsX, index<strong>=</strong>df0), pd<strong>.</strong>Series(y, index<strong>=</strong>df0)<br>    df0 <strong>=</strong> [&quot;I_&quot; <strong>+</strong> str(i) <strong>for</strong> i <strong>in</strong> range(n_informative)] <strong>+</strong> [&quot;R_&quot; <strong>+</strong> str(i) <strong>for</strong> i <strong>in</strong> range(n_redundant)]<br>    df0 <strong>+=</strong> [&quot;N_&quot; <strong>+</strong> str(i) <strong>for</strong> i <strong>in</strong> range(n_features <strong>-</strong> len(df0))]<br>    trnsX<strong>.</strong>columns <strong>=</strong> df0<br><br>    <strong>return</strong> trnsX, y</pre><blockquote>I highly recommend to use tree-based models (like Random Forest or Boostings), because they have a trade-off between training computational speed and the ability to capture non-linear, high dimensional relationships. For this tutorial we are going to use <a href="https://lightgbm.readthedocs.io/en/latest/">LightGBM</a> model.</blockquote><h3>Gini Impurity/Mean decrease impurity (MDI)</h3><blockquote><em>MDI is a fast, explanatory-importance (in-sample, IS) method specific to tree-based classifiers, like RF. MDI counts the times a feature is used to split a node, weighted by the number of samples it splits:</em></blockquote><ul><li>Gini Importance or Mean Decrease in Impurity (MDI) calculates each feature importance as the sum over the number of splits (across all tress) that include the feature, proportionally to the number of samples it splits.</li><li>At each split in each tree, the improvement in the split-criterion is the importance measure attributed to the splitting variable, and is accumulated over all the trees in the forest separately for each variable.</li></ul><blockquote>(LightGBM) importance_type (string, optional (default=”split”)) — How the importance is calculated. If “split”, result contains numbers of times the feature is used in a model. If “gain”, result contains total gains of splits which use the feature.</blockquote><ul><li>split measure is split-based and is very similar with the one given for Gini Importance. But it doesn’t take the number of samples into account.</li><li>gain measure is gain-based. It’s basically the same as the Gini Importance implemented in scikit-learn with Gini impurity replaced by the objective used by the gradient boosting model.</li></ul><pre>feature_imp <strong>=</strong> pd<strong>.</strong>DataFrame(sorted(zip(clf<strong>.</strong>feature_importances_, X<strong>.</strong>columns)), columns<strong>=</strong>[&#39;Value&#39;,&#39;Feature&#39;])<br><br>plt<strong>.</strong>figure(figsize<strong>=</strong>(20, 10))<br>sns<strong>.</strong>barplot(x<strong>=</strong>&quot;Value&quot;, y<strong>=</strong>&quot;Feature&quot;, data<strong>=</strong>feature_imp<strong>.</strong>sort_values(by<strong>=</strong>&quot;Value&quot;, ascending<strong>=False</strong>))<br>plt<strong>.</strong>title(&#39;LightGBM Features (avg over folds)&#39;)<br>plt<strong>.</strong>tight_layout()<br>plt<strong>.</strong>show()<br>plt<strong>.</strong>savefig(&#39;lgbm_importances.png&#39;)</pre><h3>Permutation Importance/Mean decrease accuracy (MDA)</h3><blockquote><em>Mean decrease accuracy (MDA) is a slow, predictive-importance (out-of-sample, OOS) method.</em></blockquote><h4>Pros:</h4><ul><li>First, it fits a classifier;</li><li>Second, it derives its performance OOS according to some performance score (f1, accuracy, negative log-loss, etc.);</li><li>Third, it permutates each column of the features matrix (X), one column at a time, deriving the performance OOS after each column’s permutation.</li></ul><h4><strong>The importance of a feature is a function of the loss in performance caused by its column’s permutation.</strong></h4><h4>Cons:</h4><ul><li>When two important features share information, shuffling one may not result in a material reduction in model performance:</li><li>MDA may wrongly dismiss one or both as uninformed</li><li>Typically MDA is more robust to substitution effects than p-values (see right plot), however this is a potential vulnerability</li><li>One possibility is to shuffle together all features with mutual information</li></ul><pre><em>def </em>feature_importance_MDA(<br>clf, X: pd.DataFrame, y: pd.Series, cv: _BaseKFold, scoring: Callable = accuracy_score<br>):<br>    scr0, scr1 = pd.Series(), pd.DataFrame(columns=X.columns)<br>    <em>for </em>i, (train, test) <em>in </em>enumerate(cv.split(X=X, y=y)):<br>        X0, y0 = X.iloc[train, :], y.iloc[train]<br>        X1, y1 = X.iloc[test, :], y.iloc[test]<br>        fit = clf.fit(X=X0, y=y0)<br>        <em>if </em>scoring == log_loss:<br>            prob = fit.predict_proba(X1)<br>            scr0.loc[i] = scoring(y1, prob, labels=clf.classes_)<br>        <em>else</em>:<br>            pred = fit.predict(X1)<br>            scr0.loc[i] = scoring(y1, pred)<br>        <em>for </em>j <em>in </em>X.columns:<br>            X1_ = X1.copy(deep=<em>True</em>)<br>            np.random.shuffle(X1_[j].values)  <em># permutation of a single column<br>            if </em>scoring == log_loss:<br>                prob = fit.predict_proba(X1_)<br>                scr1.loc[i, j] = scoring(y1, prob, labels=clf.classes_)<br>            <em>else</em>:<br>                pred = fit.predict(X1_)<br>                scr1.loc[i, j] = scoring(y1, pred)<br>    imp = (-scr1).add(scr0, axis=0)<br>    <em>if </em>scoring == &quot;neg_log_loss&quot;:<br>        imp = imp / -scr1<br>    <em>else</em>:<br>        imp = imp / (1.0 - scr1)<br>    imp = pd.concat({&quot;mean&quot;: imp.mean(), &quot;std&quot;: imp.std() * imp.shape[0] ** -0.5}, axis=1)<br>    <em>return </em>imp, scr0.mean()</pre><h3>Single Feature Importance (SFI)</h3><blockquote><em>Single feature importance (SFI) is a cross-section predictive-importance (out-of- sample) method. It computes the OOS performance score of each feature in isolation.</em></blockquote><ul><li>This method can be applied to any classifier, not only tree-based classifiers.</li><li>SFI is not limited to accuracy as the sole performance score.</li><li>Unlike MDI and MDA, no substitution effects take place, since only one feature is taken into consideration at a time.</li><li>Like MDA, it can conclude that all features are unimportant because performance is evaluated via OOS CV.</li></ul><pre><em>def </em>single_feature_importance(<br>features_names, clf, X, y, scoring, cv<br>):<br>    imp = pd.DataFrame(columns=[&quot;mean&quot;, &quot;std&quot;])<br>    <em>for </em>feature_name <em>in </em>features_names:<br>        df0 = cv_score(clf, X=X[[feature_name]], y=y, scoring=scoring, cv=cv)<br>        imp.loc[feature_name, &quot;mean&quot;] = df0.mean()<br>        imp.loc[feature_name, &quot;std&quot;] = df0.std() * df0.shape[0] ** -0.5<br>    <em>return </em>imp</pre><h3>Clustered Feature Importance (CFI)</h3><ul><li>Apply a single-linkage agglomerative clustering algorithm on a distance matrix:</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*CmKvIhNQPw2e4kIIb4sQmw.png" /></figure><ul><li>Base the distance on correlation, because the substitution effect is linear</li><li>Instead of shuffling each variable individually, shuffle together all variables within a cluster</li><li>Clustered MDA gives the right answer:</li></ul><p><em>All informed and redundant features are important, redundant cause multicollinearity.</em></p><p><em>Noise features have zero contribution to the model’s performance</em></p><pre><em>def </em>feature_importance_CFI_MDA(<br>    X,<br>    y,<br>    clf,<br>    cv,<br>    scoring=accuracy_score,<br>    C=2,<br>    epsilon=1e-5,<br>):<br>    <em>def </em>correlDist(corr):<br>        <em># A distance matrix based on correlation, where 0&lt;=d[i,j]&lt;=1<br>        # This is a proper distance metric<br>        </em>dist = ((1 - corr) / 2.0) ** 0.5  <em># distance matrix<br>        return </em>dist<br><br>    <em>def </em>cluster_features():<br>        dist_matrix = correlDist(X.corr())<br>        dist_matrix = dist_matrix.fillna(epsilon)<br>        link = linkage(dist_matrix, method=&quot;ward&quot;)<br>        clusters = fcluster(link, t=C, criterion=&quot;maxclust&quot;)<br>        <em>return </em>clusters<br><br>    <em>def </em>feat_imp_MDA_clusterized(clf, X, y, cv, scoring, cluster_subsets):<br>        <em># feat importance based on OOS score reduction<br>        </em>scr0, scr1 = pd.Series(), pd.DataFrame(columns=X.columns)<br><br>        <em>for </em>i, (train, test) <em>in enumerate</em>(cv.split(X=X, y=y)):<br>            X0, y0 = X.iloc[train, :], y.iloc[train]<br>            X1, y1 = X.iloc[test, :], y.iloc[test]<br>            fit = clf.fit(X=X0, y=y0)<br>            <em>if </em>scoring == log_loss:<br>                pred = fit.predict_proba(X1)<br>                scr0.loc[i] = scoring(y1, pred)<br>            <em>else</em>:<br>                pred = fit.predict(X1)<br>                scr0.loc[i] = scoring(y1, pred)<br>            <em># for j in X.columns:<br>            for </em>j <em>in </em>cluster_subsets:<br><br>                X1_ = X1.copy(deep=<em>True</em>)<br><br>                <em>for </em>ji <em>in </em>j:<br>                    np.random.shuffle(X1_[ji].values)  <em># permutation of a single column<br>                if </em>scoring == log_loss:<br>                    pred = fit.predict_proba(X1_)<br>                    scr1.loc[i, j] = scoring(y1, pred)<br>                <em>else</em>:<br>                    pred = fit.predict(X1_)<br>                    scr1.loc[i, j] = scoring(y1, pred)<br><br>        imp = (-scr1).add(scr0, axis=0)<br>        imp = imp / (1.0 - scr1)<br>        imp = pd.concat({&quot;mean&quot;: imp.mean(), &quot;std&quot;: imp.std() * imp.shape[0] ** -0.5}, axis=1)<br>        <em>return </em>imp, scr0.mean()<br><br>    clusters = cluster_features()<br>    cluster_subsets = [[f <em>for </em>c, f <em>in zip</em>(clusters, X.columns) <em>if </em>c == ci] <em>for </em>ci <em>in range</em>(1, C + 1)]<br><br>    fit = clf.fit(X=X, y=y)<br>    imp, oos = feat_imp_MDA_clusterized(clf, X, y, cv, scoring, cluster_subsets)<br>    <em>return </em>imp, oos</pre><h3>Mutual Information/Informatio Gain (MI)</h3><blockquote><em>Information Gain, or IG for short, measures the reduction in entropy or surprise by splitting a dataset according to a given value of a random variable.</em></blockquote><p><em>A larger information gain suggests a lower entropy group or groups of samples, and hence less surprise. You might recall that information quantifies how surprising an event is in bits. Lower probability events have more information, higher probability events have less information. Entropy quantifies how much information there is in a random variable, or more specifically its probability distribution. A skewed distribution has a low entropy, whereas a distribution where events have equal probability has a larger entropy.</em></p><p>In information theory, we like to describe the “surprise” of an event. Low probability events are more surprising therefore have a larger amount of information. Whereas probability distributions where the events are equally likely are more surprising and have larger entropy.</p><ul><li>Skewed Probability Distribution (unsurprising): Low entropy.</li><li>Balanced Probability Distribution (surprising): High entropy.</li></ul><p>Now, let’s combine all discussed methods into one:</p><pre><em>def </em>feature_importance(X, y, cv, scoring=accuracy_score, method=&quot;SFI&quot;, clf=RandomForestClassifier()):<br>    fit = clf.fit(X=X, y=y)<br>    <em>if </em>method == &quot;MDI&quot;:<br>        imp = feature_importance_MDI(fit, features_names=X.columns)<br>        oos = cv_score(clf, X, y, cv=cv, scoring=scoring).mean()<br>    <em>elif </em>method == &quot;MDA&quot;:<br>        imp, oos = feature_importance_MDA(clf, X, y, cv=cv, scoring=scoring)<br>    <em>elif </em>method == &quot;SFI&quot;:<br>        oos = cv_score(clf, X, y, cv=cv, scoring=scoring).mean()<br>        imp = single_feature_importance(X.columns, clf, X, y, scoring=scoring, cv=cv)<br>    <em>elif </em>method == &quot;CFI&quot;:<br>        C = 3<br>        imp, oos = feature_importance_CFI_MDA(X, y, clf, cv, scoring, C)<br>    <em>elif </em>method == &quot;SHAP&quot;:<br>        imp = SHAP_importance(clf, X)<br>        oos = <em>None<br>    elif </em>method == &quot;MI&quot;:<br>        mutual_imp = mutual_info_classif(X, y)<br>        mutual_imp = pd.Series(mutual_imp, index=<em>list</em>(X.columns))<br>        oos = <em>None<br>        </em>imp = pd.DataFrame(index=<em>list</em>(X.columns))<br>        imp[&quot;mean&quot;] = mutual_imp<br>        imp[&quot;std&quot;] = mutual_imp<br>    <em>return </em>imp, oos</pre><p>And perform feature importance computation:</p><pre>X, y <strong>=</strong> get_test_data(12, 2, 3, n_samples<strong>=</strong>1000)<br>clf <strong>=</strong> LGBMClassifier()<br>cv <strong>=</strong> StratifiedKFold(n_splits<strong>=</strong>10)<br>scores <strong>=</strong> cv_score(clf, X, y, f1_score, cv)</pre><pre>df, imp <strong>=</strong> compute_feature_importance(<br>        X,<br>        y,<br>        cv,<br>        f1_score,<br>        clf<strong>=</strong>clf,<br>        methods<strong>=</strong>[&quot;MDA&quot;, &quot;SFI&quot;, &quot;MDI&quot;, &quot;MI&quot;, &quot;CFI&quot;, &quot;SHAP&quot;]<br>)</pre><p>And plot the results:</p><pre><strong>for</strong> name_i, importance_i <strong>in</strong> zip(df[&#39;method&#39;], imp):<br>    print(name_i)<br>    importance_i<strong>.</strong>sort_values(&#39;mean&#39;)<strong>.</strong>plot<strong>.</strong>barh(figsize<strong>=</strong>(20, 16))</pre><figure><img alt="MDA (Mean Decrease Accuracy)" src="https://cdn-images-1.medium.com/max/1024/1*orHvYGOVPAzEkV62eOcrcw.png" /></figure><figure><img alt="SFI (Single Feature Importance)" src="https://cdn-images-1.medium.com/max/1024/1*qn4bE2ZeBWkuMac4MyoXkA.png" /></figure><figure><img alt="MDI (Mean Decrease Impurity)" src="https://cdn-images-1.medium.com/max/1024/1*z8PV3fYHGtWjaWZ4sqw8hw.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*BYTUfDEVAobU13X1z0-A6A.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*yS2u3YfsdPm_i0Wfz4vuug.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*DMUeSsQ1K4At0VPcD4bsgA.png" /></figure><p>Thanks for your attention, next tutorial in the series will be a novel, state-of-the-art Shapley Values Feature Importance technique.</p><p><em>To be continued…</em></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=113e58e7a0e5" width="1" height="1" alt="">]]></content:encoded>
        </item>
    </channel>
</rss>