<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Kate Nazarova | Senior Data Scientist on Medium]]></title>
        <description><![CDATA[Stories by Kate Nazarova | Senior Data Scientist on Medium]]></description>
        <link>https://medium.com/@Kate_in_DS?source=rss-b6f4cc3428e3------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/1*1NjutIjl_N5SauScprAYfQ.png</url>
            <title>Stories by Kate Nazarova | Senior Data Scientist on Medium</title>
            <link>https://medium.com/@Kate_in_DS?source=rss-b6f4cc3428e3------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Thu, 19 Nov 2026 06:37:14 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@Kate_in_DS/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[DeepTabular Learning Series 4: ResNet]]></title>
            <link>https://medium.com/tabular-deep-learning/mambular-tabular-deep-learning-series-4-resnet-d7bccbb76bd4?source=rss-b6f4cc3428e3------2</link>
            <guid isPermaLink="false">https://medium.com/p/d7bccbb76bd4</guid>
            <dc:creator><![CDATA[Kate Nazarova | Senior Data Scientist]]></dc:creator>
            <pubDate>Tue, 27 May 2025 19:54:08 GMT</pubDate>
            <atom:updated>2025-08-20T14:36:44.856Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="Anton Frederik Thielmann, Manish Kumar, Christoph Weisser, Arik Reuter, Benjamin Säfken and Soheila Samiee." src="https://cdn-images-1.medium.com/max/720/1*_-VHp4kgRd0r8pjtiaK6IQ.jpeg" /><figcaption>Anton Frederik Thielmann, Manish Kumar, Christoph Weisser, Arik Reuter, Benjamin Säfken and Soheila Samiee.</figcaption></figure><h3><strong>Introduction</strong></h3><p>This article introduces the ResNet model, a deep neural architecture with skip connections designed to enable efficient learning in very deep networks, adapted here for tabular data. It is part of a series on tabular deep learning using the <a href="https://github.com/basf/mamba-tabular">Mambular</a> library, which started with an introduction to using an <a href="https://medium.com/tabular-deep-learning/mambular-tabular-deep-learning-series-1-multi-layer-perceptron-mlp-637a6c0fd808">MLP</a> for these tasks.</p><p>Other topics we’ll cover include:</p><ul><li><a href="https://medium.com/tabular-deep-learning/mambular-tabular-deep-learning-series-1-multi-layer-perceptron-mlp-637a6c0fd808">MLP</a></li><li><a href="https://medium.com/tabular-deep-learning/mambular-tabular-deep-learning-series-1-multi-layer-perceptron-mlp-637a6c0fd808">FT Transformer</a></li><li><a href="https://medium.com/tabular-deep-learning/mambular-tabular-deep-learning-series-3-tabr-9125c5280899">TabR</a></li><li><strong>Tabular ResNet</strong></li><li>SAINT</li><li>NODE</li><li>TabM</li><li>Mambular</li><li>TabulaRNN</li><li>Advanced Preprocessing</li><li>Hyperparameter Optimization</li></ul><p>All models, preprocessing techniques, and related tools are implemented in the Mambular package. Stay up-to-date with ongoing development as new models, preprocessing methods, and features are continuously added: <a href="https://github.com/basf/mamba-tabular"><strong><em>Mambular</em></strong></a><strong><em> — </em></strong><a href="https://github.com/basf/mamba-tabular">https://github.com/basf/mamba-tabular</a></p><p>Throughout this series, we will use the same dataset and fit each model to it. This approach allows us to report the performance of each new model and compare it with other techniques, such as gradient boosting.</p><h3>Tabular Deep Learning with ResNet</h3><p>Deep neural networks have revolutionized many domains by enabling the training of very deep models. However, simply stacking more layers often leads to optimization difficulties and degradation problems — deeper networks can become harder to train and may even perform worse than shallower ones [1]. <strong>Residual Networks (ResNets) were introduced as a solution to these issues, allowing networks to train hundreds of layers without suffering vanishing gradients or degraded accuracy </strong>[1][2]. Originally developed for image recognition tasks [1], ResNets use residual learning (skip connections) to make extremely deep architectures feasible and easier to optimize.</p><p>In this article, we will explore what ResNet is, how its residual block design works, and how this architecture can be adapted for tabular data regression tasks using the open-source Mambular library. We’ll discuss why residual learning improves training, and examine the benefits and potential limitations of applying ResNet to structured data. Finally, we’ll provide practical guidance on using ResNet models in the Mambular library for real-world tabular regression, including tips on hyperparameters like learning rate and batch normalization to get the best results.</p><h3><strong>What is ResNet?</strong></h3><p>ResNet was originally introduced for image recognition, where it achieved breakthrough performance by easing the training of very deep convolutional networks [1]. Instead of each layer trying to learn a direct mapping <strong><em>H(x)</em></strong> from inputs to outputs, ResNet layers learn a residual function (the difference between the input and desired output):</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/133/1*ngWwpSjtGDoj-uEGrxksvA.png" /></figure><p>and the original function is obtained by adding back the input[1][2]:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/133/1*se_1iDm8IDzyeIYsJ1Pqlw.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/570/1*UiY72bV4STrpi_3SEOcaHA.png" /><figcaption>Figure: Basic Residual Block as depicted in He et al.’s original ResNet paper (2015).</figcaption></figure><p>This clever reformulation means that<strong> layers only need to learn the difference (residual) from the input — often an easier task than learning a completely new transformation</strong> [2]. If the optimal mapping for some layers is actually the identity (i.e. doing nothing), a residual network can easily learn <strong><em>F(x) ≈ 0</em></strong> so that <strong><em>H(x) ≈ x</em></strong>, effectively skipping those layers [1]. This ensures that adding more layers won’t make the network worse, because those layers can simply fall back to learning an identity mapping if needed [1]. ResNet’s innovation was “learning residual functions with reference to the layer inputs, instead of learning unreferenced functions” [1]. B<strong>y doing so, deeper networks become easier to train and can achieve higher accuracy as layers are added</strong> [1]. In the original 2015 paper, He et al. demonstrated that residual networks with over 150 layers were not only trainable, but also outperformed shallower networks on ImageNet [1].</p><p>In summary, ResNet’s purpose is to enable much deeper architectures by mitigating optimization challenges like vanishing gradients — the skip connections provide alternate gradient paths and help preserve signals through many layers [2].</p><h3>From Image Residuals to Tabular Residuals</h3><p>In the original <em>convolutional</em> ResNet for images, each <strong>residual block</strong> typically contains two<strong><em> 3x3</em></strong> convolutional layers (with batch normalization and ReLU activation) in sequence [2]. The input of the block is added to the output of these layers (the “residual” output) <em>before</em> applying the final activation (another ReLU) [2].</p><p>This skip-add design ensures that the output of the block is <strong>at least as informative as the input </strong>— the block can learn to make no change by outputting zeros (an identity mapping scenario). One caveat in the image domain is that if a residual block changes the number of feature <strong>channels</strong> (for example, when downsampling or increasing filters), the input and output shapes differ. In such cases, ResNet uses a <strong>projection shortcut</strong> — a <strong><em>1x1</em></strong> convolution on the skip path — to match dimensions so that the input can be added to the output [2]. If the dimensions already match, a simple identity shortcut is used.</p><p><strong>ResNet blocks address the core problems of deep networks.</strong> By providing a shortcut for data and gradients to flow around intermediate layers, they make training much more robust. <strong>During backpropagation</strong>, a residual block offers two parallel paths for the gradient: (1) through the series of weight layers <em>∇</em><strong><em>F(x)</em></strong>, and (2) directly through the identity connection <em>∇</em><strong><em>x</em></strong>. Even if the gradient through the weight layers becomes extremely small (vanishes), the gradient flowing through the skip connection can continue backward unhindered (since it doesn’t get multiplied by any weights) [2]. In other words, the skip connection guarantees that some gradient signal will always propagate back to earlier layers, greatly alleviating the vanishing gradient problem. Empirically, this means that very deep ResNets train faster and converge to better accuracy than equally deep “plain” networks without [2].</p><p>Another benefit is that the network is less likely to get “stuck” during training — if an additional layer were ever detrimental, the residual block can simply learn to output zero (i.e. ignore that layer), so a deeper ResNet is at least as good as a shallower one. This flexible identity mapping is a key part of what makes ResNet effective for deep architectures.</p><p>Thanks to these properties, ResNet blocks enable building extremely deep models (50, 101, even 152 layers in the original study) with improved performance rather than degradation [1].</p><p>It’s important to note that ResNet doesn’t necessarily “skip” learning features; rather, it gives the network a choice: either use the extra layers to learn something useful <strong><em>F(x)</em></strong>, or easily fall back to the input if those layers aren’t needed. This flexibility is what allows ResNets to scale depth so effectively. Modern variants of ResNet blocks also incorporate techniques like Batch Normalization (after each layer) to stabilize training, and sometimes use bottleneck designs (e.g. 1×1 conv layers to reduce dimensionality within the block) to make very deep networks more efficient [2]. But the fundamental idea remains the same: a ResNet block outputs:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/213/1*MCjnIiJfvJemqAz9NK8HgA.png" /></figure><h3><strong>Comparing Convolutional vs. Fully Connected Residual Blocks.</strong></h3><p>While the <strong>core principles</strong> remain the same, there are a few noteworthy differences and similarities between CNN-based ResNet blocks and MLP-based ResNet blocks:</p><ul><li><strong>Type of Layers:</strong> <em>Convolutional vs. Dense.</em> Original ResNet blocks use convolutional layers that operate on image feature maps, extracting local spatial patterns. In tabular ResNet, these are replaced with fully connected (dense) layers that mix all input features (since tabular data has no spatial structure) <a href="https://mambular.readthedocs.io/en/latest/_modules/mambular/base_models/resnet.html#:~:text=%5Bdocs%5Dclass%20ResNet%28BaseModel%29%3A%20,processing%20categorical%20and%20numerical%20features">[</a>3]. This means an image ResNet block might learn filters over pixels, whereas a tabular ResNet block learns interactions among input features via weight connections in a dense layer.</li><li><strong>Skip Connections and Block Design:</strong> Both architectures preserve the <strong>skip connection</strong> structure. The input is carried forward (unchanged) and added to the block’s output, which lets the block focus on learning a residual function [1]. The ability to fall back to an identity mapping (by learning a zero residual) is intact for tabular models just as in image models, helping stabilize training of deep networks. Identity shortcuts are used when input and output shapes match, and projection shortcuts (a learned linear transform) are used when they don’t — this concept holds true in both CNN and MLP implementations [2].</li><li><strong>Activation and Normalization:</strong> After the skip connection adds the input and output, both architectures apply an activation function (typically ReLU) to the combined result [2]. This post-addition activation is an important aspect of the original ResNet design (sometimes referred to as “post-activation”). Likewise, each sub-layer in the block usually has an activation (ReLU) following it. Both CNN and tabular ResNets also make use of normalization layers inside blocks to stabilize training. These design elements — nonlinearity and normalization — remain the same, only adapted to the layer type (convolutional vs. linear).</li></ul><h3><strong>Adapting ResNet to Tabular Data with Mambular</strong></h3><p>Mambular puts this idea into practice by adapting ResNet for tabular data. Instead of convolutional blocks, it uses residual blocks made of dense (fully connected) layers. Each block’s output is added back to its input via a skip connection (with an optional projection layer to align dimensions when needed). The library also automates feature preprocessing: numerical features can be normalized and categorical features embedded as learnable vectors, all handled internally. In short, you can feed your table’s numeric and categorical columns directly into a deep ResNet-style MLP, and Mambular takes care of encoding and aligning everything. This makes it simple and effective to train a deep network on structured data — you get the benefits of ResNet’s skip connections without extra hassle.</p><h3><strong>Benefits</strong></h3><ul><li><strong>Improved Gradient Flow:</strong> Skip connections reduce vanishing gradients, enabling stable training of deep networks.</li><li><strong>Deeper Models Without Degradation:</strong> Layers can learn residuals or fall back to identity, making it safe to add depth.</li><li><strong>Regularization Effect:</strong> Blocks that don’t help can “turn off,” reducing overfitting risk.</li><li><strong>Strong Baseline:</strong> ResNets match or outperform many specialized tabular models.</li><li><strong>Compatible with Modern Tricks:</strong> Easily used with dropout, batch norm, and learning rate schedules.</li></ul><h3><strong>Limitations</strong></h3><ul><li><strong>Overfitting Risk on Small Data:</strong> Deep models can memorize noise — use dropout and early stopping.</li><li><strong>Sensitive to Tuning:</strong> Learning rate, block count, and layer size need careful adjustment.</li><li><strong>Longer Training Time:</strong> More depth = more compute; may not be ideal for all tasks.</li><li><strong>Interpretability:</strong> Like most deep nets, understanding what’s learned isn’t straightforward.</li></ul><h3><strong>Model Fitting</strong></h3><p>Now that we have outlined the ResNet model, let’s move on to model fitting. The dataset and packages are publicly available, so everything can be copied and run locally or in a Google Colab notebook, provided the necessary packages are installed. We will start by installing the mambular package, loading the dataset, and fitting ResNet. Subsequently, we will compare these results with those obtained in earlier articles of this series.</p><p>Install Mambular</p><pre>pip install mambular<br>#pip install torch==2.2.0 (try out this in case of errors with the ResNet implementation)</pre><p>Load packages</p><pre>from sklearn.datasets import fetch_california_housing<br>from sklearn.model_selection import train_test_split<br>from sklearn.preprocessing import StandardScaler<br>from sklearn.metrics import mean_squared_error</pre><p>Step 1: Load and preprocess the data</p><pre>from sklearn.datasets import fetch_california_housing<br>from sklearn.model_selection import train_test_split<br>from sklearn.metrics import mean_squared_error<br>from sklearn.preprocessing import StandardScaler<br><br># Load California Housing dataset<br>data = fetch_california_housing(as_frame=True)<br>X, y = data.data, data.target<br><br># Drop NAs<br>X = X.dropna()<br>y = y[X.index]<br><br># Standard normalize features and target<br>y = StandardScaler().fit_transform(y.values.reshape(-1, 1)).ravel()<br><br># Train-test-validation split<br>X_train, X_temp, y_train, y_temp = train_test_split(<br>    X, y, test_size=0.5, random_state=42<br>)<br>X_val, X_test, y_val, y_test = train_test_split(<br>    X_temp, y_temp, test_size=0.5, random_state=42<br>)</pre><p>Step 2: Train &amp; Test the model</p><pre>from mambular.models.resnet import ResNetRegressor<br>model = ResNetRegressor()<br>model.fit(X_train, y_train, X_val=X_val, y_val=y_val, max_epochs=200)<br><br># Predict on test set<br>y_pred = model.predict(X_test)<br><br># Calculate mean squared error for the test set<br>mse = mean_squared_error(y_test, y_pred)<br><br>print(f&quot;Mean Squared Error on Test Set: {mse}&quot;)</pre><pre>Mean Squared Error on Test Set: 0.17923</pre><p>Compared to previous models, we see good performance; however, it underperforms compared to XGBoost. Let’s try the ResNet with PLE as numerical pre-processing as already used in the FT Transformer article.</p><pre>model = ResNetRegressor(<br>                  numerical_preprocessing=&#39;ple&#39; <br>                  )<br>model.fit(X_train, y_train, X_val=X_val, y_val=y_val, max_epochs=200)<br><br># Predict on test set<br>y_pred = model.predict(X_test)<br><br># Calculate mean squared error for the test set<br>mse = mean_squared_error(y_test, y_pred)<br><br>print(f&quot;Mean Squared Error on Test Set: {mse}&quot;)</pre><pre>Mean Squared Error on Test Set: 0.20167</pre><p>Interestingly, the applied techniques led to a decline in performance. Let’s try ResNet with PLR embedding as already used in the MLP article.</p><pre>model = ResNetRegressor(use_embeddings=True, <br>                  embedding_type=&#39;plr&#39;, <br>                  numerical_preprocessing=&#39;standardization&#39; <br>                  )<br>model.fit(X_train, y_train, X_val=X_val, y_val=y_val, max_epochs=200)<br><br># Predict on test set<br>y_pred = model.predict(X_test)<br><br># Calculate mean squared error for the test set<br>mse = mean_squared_error(y_test, y_pred)<br><br>print(f&quot;Mean Squared Error on Test Set: {mse}&quot;)</pre><pre>Mean Squared Error on Test Set: 0.21547 </pre><p>With an MSE of 0.21547 on the test set, this trial represents the weakest performance among the ResNet-based experiments.</p><p>Below we have summarized the results from all articles so far. Try playing around with some more parameters and improve performance even more. Throughout this series, we will add the results of each introduced method to this table:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/375/1*Kfyv0S1WySpESBsebrLY4Q.png" /></figure><p>Tabular DL has long been behind GBDTs in performance. More importantly, however, it has been far behind in terms of usability. Mambular’s automatic preprocessing makes it easy to implement deep learning models for tabular data. This article introduced the ResNet model, but stay tuned for future installments in this series, where we’ll dive into other powerful models. With Mambular, you can unlock the full potential of your tabular data for deep learning tasks.</p><p>GitHub Repo: <a href="https://github.com/basf/mamba-tabular">https://github.com/basf/mamba-tabular</a></p><h3>Other Articles from this series:</h3><ul><li><a href="https://medium.com/tabular-deep-learning/mambular-tabular-deep-learning-series-3-tabr-9125c5280899">Mambular Tabular Deep Learning Series 3: TabR</a></li><li><a href="https://medium.com/tabular-deep-learning/mambular-tabular-deep-learning-series-2-ft-transformer-and-piecewise-linear-encodings-371cb54dd399?source=post_page-----9125c5280899---------------------------------------">Mambular Tabular Deep Learning Series 2: FT Transformer and Piecewise Linear Encodings</a></li><li><a href="https://medium.com/tabular-deep-learning/mambular-tabular-deep-learning-series-1-multi-layer-perceptron-mlp-637a6c0fd808?source=post_page-----9125c5280899---------------------------------------">Mambular Tabular Deep Learning Series 1: Multi-Layer Perceptron (MLP)</a></li><li><a href="https://medium.com/tabular-deep-learning/mambular-tabular-deep-learning-with-mamba-08d9fb3af0e3?source=post_page-----9125c5280899---------------------------------------">Mambular: Tabular Deep Learning with Mamba</a></li></ul><h3>How to Cite The Paper:</h3><p>If you’d like to reference this work in your own research or writing, you can use the following citation:</p><pre>@article{thielmann2024mambular,<br>  title={Mambular: A sequential model for tabular deep learning},<br>  author={Thielmann, Anton Frederik and Kumar, Manish and Weisser, Christoph and Reuter, Arik and S{\&quot;a}fken, Benjamin and Samiee, Soheila},<br>  journal={arXiv preprint arXiv:2408.06291},<br>  year={2024}<br>}</pre><h3>References</h3><ol><li>He, K., Zhang, X., Ren, S., &amp; Sun, J. (2015). <a href="https://arxiv.org/abs/1512.03385"><strong>Deep Residual Learning for Image Recognition</strong></a>. <em>arXiv:1512.03385</em></li><li>Zhang, A., Lipton, Z. C., Li, M., &amp; Smola, A. J. (2020). <em>Dive into Deep Learning</em>. <a href="https://d2l.ai/chapter_convolutional-modern/resnet.html">https://d2l.ai/chapter_convolutional-modern/resnet.html</a></li><li>Mambular Documentation. <em>ResNet for Tabular Data</em>. <a href="https://mambular.readthedocs.io/">https://mambular.readthedocs.io</a></li></ol><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=d7bccbb76bd4" width="1" height="1" alt=""><hr><p><a href="https://medium.com/tabular-deep-learning/mambular-tabular-deep-learning-series-4-resnet-d7bccbb76bd4">DeepTabular Learning Series 4: ResNet</a> was originally published in <a href="https://medium.com/tabular-deep-learning">Tabular Deep Learning</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[“   ,  ,      …]]></title>
            <link>https://medium.com/@Kate_in_DS/-670ffaa29af2?source=rss-b6f4cc3428e3------2</link>
            <guid isPermaLink="false">https://medium.com/p/670ffaa29af2</guid>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[women-in-tech]]></category>
            <dc:creator><![CDATA[Kate Nazarova | Senior Data Scientist]]></dc:creator>
            <pubDate>Wed, 03 Apr 2024 13:31:42 GMT</pubDate>
            <atom:updated>2024-04-03T13:31:42.803Z</atom:updated>
            <content:encoded><![CDATA[<h3>“𝑰 𝒅𝒖𝒈 𝒂 𝒈𝒓𝒂𝒗𝒆, 𝒂𝒏𝒅 𝒕𝒉𝒆𝒓𝒆, 𝑰 𝒃𝒖𝒓𝒊𝒆𝒅 𝒎𝒚 𝒇𝒆𝒎𝒊𝒏𝒊𝒏𝒊𝒕𝒚 𝒇𝒐𝒓 𝒂 𝒍𝒐𝒏𝒈 𝒕𝒊𝒎𝒆.” Being a Female Data Scientist in a Male-Dominated Industry.</h3><p>This post is inspired by the gender-related questions I am commonly asked by female mentees. Today, I want to tell my own story in the hope of addressing and reducing gender-related fears when entering the tech industry.</p><p>You might deem this post to be a quite personal one to share over LinkedIn, but I am deliberately and consciously taking this bold step and if it can only help one other person, this is worthwhile!</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/848/1*TR55TT0NhQKMNzcw5p1fyA.png" /></figure><p>𝐒𝐞𝐯𝐞𝐫𝐚𝐥 𝐲𝐞𝐚𝐫𝐬 𝐚𝐠𝐨, 𝐈 𝐟𝐨𝐮𝐧𝐝 𝐦𝐲𝐬𝐞𝐥𝐟 𝐦𝐢𝐬𝐥𝐞𝐝 𝐛𝐲 𝐭𝐰𝐨 𝐜𝐨𝐧𝐜𝐞𝐩𝐭𝐬:</p><p>In my childhood, influenced by media, I perceived that applied maths, computer science &amp; etc., were not really suitable roles for females, which actually sounds very strange as all my math teachers were females 🤷🏼‍♀️. Thus, I found myself being trapped in a stereotype that women are more suited for non-technical positions.</p><p>Later, in my adult life, it turned out in a quite ‘funny’ way. I was CONSTANTLY competing with men in every way possible. Pretty much, I was demonstratively trying to prove to everyone that I have the same logical, smart-ass brain 😅. Quite tiring, to be honest.</p><p>This stereotype, I believe, aligns with another:</p><p>‘Help! I will not be taken seriously if I am being my true self 😱.’</p><p>About 2–3 years ago, I finally understood that, in some way, this misconception led to the point where I didn’t feel quite comfortable wearing makeup, jewelry, or the clothes that I actually wanted to wear. I almost forgot that Ladies can wear Heels… And I am a Lady.</p><p>Over time, I became a sort of ‘masculine version’ of myself: wearing hoodies 24/7 and being ashamed to think about a manicure or something like that (with a touch of humor😉). Of course, all of those things are exaggerated but very nicely express the mess in my head.</p><p>What I was afraid of was, “if I look pretty/beautiful, then I will be treated as an incompetent person.” That was my honest belief and fear. I would be forced to defend my approaches, voice, and ideas harder than usual. I imagined that I would have to double-fight for them every single time.</p><p>𝐖𝐡𝐚𝐭’𝐬 𝐭𝐡𝐞 𝐥𝐞𝐬𝐬𝐨𝐧 𝐥𝐞𝐚𝐫𝐧𝐞𝐝 𝐟𝐫𝐨𝐦 𝐢𝐭?</p><p>Firstly, understand that you are not alone on this planet with these kinds of fears and misconceptions.</p><p>Second of all, let’s not forget that we all have different cultures, traditions, prejudices, fears, and our unique stories. So, be gentle to yourself. Being able to SEE the issue with these learned behaviors is already the solution done🙌🏼</p><p>𝐖𝐡𝐚𝐭 𝐞𝐥𝐬𝐞?</p><p>Break free from gender stereotypes. Challenge societal expectations. Be truly aligned with Yourself.<br>Recognize your unique story &amp; Embrace femininity in your tech role without fear.</p><p>P.S. This is unrelated to my work environment, teammates, or past colleagues. I’m grateful for the excellent working culture at BASF.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=670ffaa29af2" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[         …]]></title>
            <link>https://medium.com/@Kate_in_DS/-7c532ab5b17d?source=rss-b6f4cc3428e3------2</link>
            <guid isPermaLink="false">https://medium.com/p/7c532ab5b17d</guid>
            <category><![CDATA[industry]]></category>
            <category><![CDATA[artificial-intelligence]]></category>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[tech-trends]]></category>
            <dc:creator><![CDATA[Kate Nazarova | Senior Data Scientist]]></dc:creator>
            <pubDate>Wed, 03 Apr 2024 13:26:33 GMT</pubDate>
            <atom:updated>2024-04-03T13:26:33.675Z</atom:updated>
            <content:encoded><![CDATA[<h3>𝐌𝐲 𝐩𝐞𝐫𝐬𝐨𝐧𝐚𝐥 𝐛𝐞𝐥𝐢𝐞𝐟𝐬 𝐛𝐫𝐨𝐤𝐞 𝐚𝐩𝐚𝐫𝐭 𝐢𝐧 𝐭𝐡𝐞 𝐟𝐚𝐜𝐞 𝐨𝐟 𝐭𝐡𝐞 𝐫𝐞𝐚𝐥𝐢𝐭𝐲 𝐚𝐬 𝐚 𝐃𝐚𝐭𝐚 𝐒𝐜𝐢𝐞𝐧𝐭𝐢𝐬𝐭.</h3><p>I genuinely hope that fellow data scientists will find at least a small part of it helpful.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*PjdJc7hjS-P1Ym8d" /><figcaption>Photo by <a href="https://unsplash.com/@joshuaearle?utm_source=medium&amp;utm_medium=referral">Joshua Earle</a> on <a href="https://unsplash.com/?utm_source=medium&amp;utm_medium=referral">Unsplash</a></figcaption></figure><p>🖋 I was initially confident that everyone knew what the data science position entailed, only to encounter a different reality where explanations were often necessary. This realization gained significance when my mentees started asking how to articulate their roles to non-tech managers. <a href="https://www.linkedin.com/in/jasmin-weim%C3%BCller-bsc2018/">Jasmin Weimüller</a> and I plan to delve into this topic on our podcast “𝘋𝘪𝘢𝘳𝘪𝘦𝘴 𝘰𝘧 𝘢 𝘋𝘢𝘵𝘢 𝘚𝘤𝘪𝘦𝘯𝘵𝘪𝘴𝘵“.</p><p>🖋 To build a successful career, emphasize soft skills such as goal setting and project planning, rather than solely focusing on hard skills. Mastering communication skills will propel you further than knowing every Python library and function. I recommend seeking advice from a coach for achieving this balance.</p><p>🖋 The Data Science position has evolved into a broad term, with individuals skilled in “Excel” sometimes identifying as data scientists, which, honestly, can be a bit frustrating. Thus, you need to have a very clear vision and understanding of what your position is about, where your interests lie, and where THAT line is among other data and IT-related positions.</p><p>🖋 I once had an idea of being a data scientist, solving cool and challenging mathematical problems, drawing complex equations, and coding 24/7. However, in my subjective and biased reality (it is important to mention), you find yourself engaged in diverse projects that shift from 80% data cleaning to 100% talking and consulting, to 60% coding and solving complex challenges, and so on. Over time, I came to understand that this variety is a positive approach; otherwise, you might go crazy dealing solely with hard problem-solving or handling data preprocessing.</p><p>🖋 In recent years, the business landscape has shown significant interest in AI and ML topics. However, based on my observations, it appears that many companies may not be fully prepared for the implementation phase and are in need of skilled data engineers. Please note that this is solely my perspective, and I don’t have any specific statistics at hand.</p><p>🖋 Be prepared to encounter a variety of creative but perhaps unrealistic ideas related to ML/AI/Gen. AI. Politely declining such proposals may become a valuable aspect of navigating this field.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=7c532ab5b17d" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[        …]]></title>
            <link>https://medium.com/@Kate_in_DS/-e3246eeb1a6d?source=rss-b6f4cc3428e3------2</link>
            <guid isPermaLink="false">https://medium.com/p/e3246eeb1a6d</guid>
            <category><![CDATA[tech-career]]></category>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[tech-career-coaching]]></category>
            <dc:creator><![CDATA[Kate Nazarova | Senior Data Scientist]]></dc:creator>
            <pubDate>Wed, 03 Apr 2024 13:18:03 GMT</pubDate>
            <atom:updated>2024-04-03T13:18:03.862Z</atom:updated>
            <content:encoded><![CDATA[<h3>𝐄𝐯𝐞𝐧 𝐭𝐡𝐨𝐮𝐠𝐡 𝐲𝐨𝐮 𝐡𝐚𝐯𝐞 𝐟𝐢𝐧𝐢𝐬𝐡𝐞𝐝 𝐲𝐨𝐮𝐫 𝐝𝐞𝐠𝐫𝐞𝐞 𝐚𝐧𝐝 𝐠𝐨𝐧𝐞 𝐭𝐡𝐫𝐨𝐮𝐠𝐡 𝐭𝐨𝐧𝐬 𝐚𝐧𝐝 𝐭𝐨𝐧𝐬 𝐨𝐟 𝐞𝐧𝐝𝐥𝐞𝐬𝐬 𝐥𝐢𝐬𝐭𝐬 𝐨𝐟 𝐜𝐨𝐮𝐫𝐬𝐞𝐬, 𝐲𝐨𝐮 𝐦𝐢𝐠𝐡𝐭 𝐬𝐭𝐢𝐥𝐥 𝐡𝐚𝐯𝐞 𝐟𝐚𝐥𝐥𝐞𝐧 𝐢𝐧𝐭𝐨 𝐭𝐡𝐞 𝐟𝐨𝐥𝐥𝐨𝐰𝐢𝐧𝐠 𝐭𝐫𝐚𝐩:</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*KkDEHBQ-PHpvf8tXyoRsWQ.png" /></figure><p>Recently, I had in a consultation with a data science master’s student who sought guidance on navigating the job market, choosing specializations, and crafting an impactful CV.<br>The concerns highlight the typical challenges encountered in this dynamic field.</p><p>🖇Thus, I thought it would be a great idea to share some valuable insights aimed at guiding individuals in the right direction:</p><p>▪️𝐊𝐧𝐨𝐰 𝐓𝐡𝐲𝐬𝐞𝐥𝐟, 𝐊𝐧𝐨𝐰 𝐓𝐡𝐲 𝐅𝐢𝐞𝐥𝐝: Data science is not homogeneous; rather, it comprises a dynamic ecosystem filled with various subfields. Delve into exploration and comprehension to discover what genuinely sparks your enthusiasm. Do you find yourself attracted to the predictive capabilities of machine learning, the complexities of natural language processing, maybe computer vision, or EDA? — Identify.</p><p>▪️𝐅𝐨𝐮𝐧𝐝𝐚𝐭𝐢𝐨𝐧𝐬 𝐅𝐢𝐫𝐬𝐭: Before you venture into the complexities of advanced concepts like deep learning and/or natural language preprocessing, make sure you have a solid understanding of the basics.</p><p>&gt;&gt;&gt; Do you feel comfortable with regularized regression, classification vs. clustering, decision trees vs. random forests, etc.?<br>&gt;&gt;&gt; Can you easily explain ensemble learning or cross-validation concepts?</p><p>These fundamental concepts are the building blocks of your data science skills. Once you master them, you’ll confidently open the door to tackling more challenging tasks.</p><p>▪️𝐁𝐞 𝐜𝐨𝐧𝐟𝐢𝐝𝐞𝐧𝐭 𝐚𝐧𝐝 𝐩𝐢𝐜𝐤 𝐲𝐨𝐮𝐫 𝐬𝐢𝐝𝐞: Remember that numerous inexperienced companies confuse data science with software development, data analytics, or data engineering. So, don’t allow them to confuse you with their own misunderstandings.</p><p>▪️𝐋𝐋𝐌 𝐛𝐨𝐨𝐦: Look beyond the buzzwords and focus on the essence of the role. One time I stumbled upon the next statement, and in my opinion, very true to what I see nowadays → Ariely’s classic big data quote repurposed by <a href="https://www.linkedin.com/in/kozyrkov/">Cassie Kozyrkov</a>:</p><p>“𝐈𝐦𝐩𝐥𝐞𝐦𝐞𝐧𝐭𝐢𝐧𝐠 𝐆𝐞𝐧𝐀𝐈 𝐚𝐭 𝐬𝐜𝐚𝐥𝐞 𝐢𝐧 𝐭𝐡𝐞 𝐞𝐧𝐭𝐞𝐫𝐩𝐫𝐢𝐬𝐞 𝐢𝐬 𝐥𝐢𝐤𝐞 𝐭𝐞𝐞𝐧𝐚𝐠𝐞 𝐬𝐞𝐱: 𝐞𝐯𝐞𝐫𝐲𝐨𝐧𝐞 𝐭𝐚𝐥𝐤𝐬 𝐚𝐛𝐨𝐮𝐭 𝐢𝐭, 𝐧𝐨𝐛𝐨𝐝𝐲 𝐫𝐞𝐚𝐥𝐥𝐲 𝐤𝐧𝐨𝐰𝐬 𝐡𝐨𝐰 𝐭𝐨 𝐝𝐨 𝐢𝐭, 𝐞𝐯𝐞𝐫𝐲𝐨𝐧𝐞 𝐭𝐡𝐢𝐧𝐤𝐬 𝐞𝐯𝐞𝐫𝐲𝐨𝐧𝐞 𝐞𝐥𝐬𝐞 𝐢𝐬 𝐝𝐨𝐢𝐧𝐠 𝐢𝐭, 𝐬𝐨 𝐞𝐯𝐞𝐫𝐲𝐨𝐧𝐞 𝐜𝐥𝐚𝐢𝐦𝐬 𝐭𝐡𝐞𝐲 𝐚𝐫𝐞 𝐝𝐨𝐢𝐧𝐠 𝐢𝐭…”<br>😁 Hehe</p><p>P.S. In the podcast “Diaries of a Data Scientist,” you will find much more on this topic, especially in episode 9 😉</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=e3246eeb1a6d" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[How to survive in this fast-paced industry and balance between work, learning, and chill time?]]></title>
            <link>https://medium.com/@Kate_in_DS/how-to-survive-in-this-fast-paced-industry-and-balance-between-work-learning-and-chill-time-54b84a63c38c?source=rss-b6f4cc3428e3------2</link>
            <guid isPermaLink="false">https://medium.com/p/54b84a63c38c</guid>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[soft-skills]]></category>
            <category><![CDATA[work-life-balance]]></category>
            <dc:creator><![CDATA[Kate Nazarova | Senior Data Scientist]]></dc:creator>
            <pubDate>Fri, 24 Nov 2023 20:16:05 GMT</pubDate>
            <atom:updated>2023-11-24T20:16:05.522Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*jEktFYiczGGW81zrrb3Ppw.png" /><figcaption>ID 219988407 © <a href="https://www.dreamstime.com/ursus456_info">Zdenek Sasek</a> Dreamstime.com</figcaption></figure><p>Recently, I came across an article “how can you continue to develop your skills after being promoted in Data Science.” But no one is really talking — or at least I haven’t seen it (living in my own biased world, you know) — about how to survive in this fast-paced industry and balance between work, learning, and chill time. So, shall we?</p><p>Disclaimer: The insights I’m about to share are based on my personal experience and may not necessarily apply universally. So, please, filter.</p><p>Pre-Story: Over the past three months, I’ve had the privilege of consulting with remarkable individuals from diverse fields — neuroscience, personal branding, and even a passionate founder in rocket science. As you might sense, these individuals maintain a hectic schedule with four overlapping meetings every 30 to 60 minutes, and they often find themselves needing to be in several different places at once…</p><p>I was intrigued by how they manage to structure their lives, stay motivated, achieve goals, and still find time for rest, family, and other joyful activities — especially in the fast-paced tech industry.</p><p>Here are some valuable pieces of advice that I’m excited to share with you:</p><ol><li><strong>Calendar Mastery:</strong></li></ol><ul><li>Start by framing your personal calendar, setting consistent wake-up and bedtime hours</li><li>Include fixed milestones like your primary work schedule or family commitments.</li><li>Allocate dedicated time slots for personal activities — be it sports, hobbies, or simply winding down (no phones allowed).</li><li>Remember, even calls with parents deserve a spot!</li></ul><p><strong>By doing this you will see free time slots in your calendar and can plan other activities. One of the phrases that is in my heart right now is: If you fail to plan -&gt; You PLAN to FAIL.</strong></p><p><strong>2. Task Manager:</strong></p><p>I personally have an “endless” list of different books / courses I want to take or posts / books / articles I want to write. I was recommended to create a separate space for each activity and write a proper task description + deadline (Trello, Notion, Taskade … )</p><p><strong>What to do with spontaneous ideas? </strong>Write them in Notes (somewhere else) and then revise these ideas on Sundays before planning the upcoming week. But don’t put it in abstract way. Instead, write a proper task description.</p><p><strong>3. Active lifestyle:</strong></p><p>I really love a phrase from a neuroscientist that if you don’t feel your body -&gt; you don’t feel your life. It was recommended to add sport (walk, yoga, stretching …) based on your brain activity. So, the more you think / learn new …, the more sport you should have in your life.</p><p><strong>Final Suggestions:</strong></p><ul><li>Work with the system; let it be dynamic.</li><li>Analyze what works and what doesn’t; adapt your schedule based on feedback.</li><li>Keep your schedule and task manager up-to-date; consistency is key.</li></ul><p><strong>In conclusion, </strong>the data industry isn’t just about technology; it intertwines with various aspects of our lives. I hope these insights prove helpful in navigating the intricate balance between career, learning, and personal well-being.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=54b84a63c38c" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[The Power of Responsibility: Shaping the Future of chatGPT]]></title>
            <link>https://medium.com/@Kate_in_DS/the-power-of-responsibility-shaping-the-future-of-chatgpt-a4aa7827c287?source=rss-b6f4cc3428e3------2</link>
            <guid isPermaLink="false">https://medium.com/p/a4aa7827c287</guid>
            <category><![CDATA[chatgpt]]></category>
            <category><![CDATA[responsibility]]></category>
            <category><![CDATA[ai]]></category>
            <category><![CDATA[generative-ai-tools]]></category>
            <category><![CDATA[data-science]]></category>
            <dc:creator><![CDATA[Kate Nazarova | Senior Data Scientist]]></dc:creator>
            <pubDate>Tue, 27 Jun 2023 12:26:03 GMT</pubDate>
            <atom:updated>2023-06-27T12:26:03.190Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*2_1go-tJRfGDVvpZ" /><figcaption>Photo by <a href="https://unsplash.com/@santesson89?utm_source=medium&amp;utm_medium=referral">Andrea De Santis</a> on <a href="https://unsplash.com/?utm_source=medium&amp;utm_medium=referral">Unsplash</a></figcaption></figure><p>In the midst of the ever-present chatter surrounding chatGPT, I feel a sense of responsibility towards our community that compels me to share my thoughts on this matter. The impetus for this post stems from a series of coaching sessions that brought my attention to generative AI and specifically chatGPT.</p><p>Now, you may already find the topic tiresome, but allow me to provide some context. During these sessions, my coachees and I delved into the realm of utilizing chatGPT for business purposes, exploring how it could automate mundane yet essential tasks, sparing us from the tedium of routine responsibilities.</p><p>Initially, these discussions seemed harmless, with the focus solely on the positive applications of this impressive technology. However, as we delved deeper, we found ourselves contemplating the darker side of both technology and humanity. The question that loomed before us was how we could manipulate chatGPT and its input to achieve desired outcomes. It was this realization that compelled me to initiate this discussion today.</p><p>Moving forward, I have made a conscious decision to begin every coaching session with a crucial reminder: Each of us bears a responsibility for the future development of chatGPT. It is incumbent upon us to be mindful of the input we provide.</p><p>ChatGPT is not a mere plaything, nor is it another tool to be utilized without consequence. Allow me to present a metaphor to illustrate the gravity of our role: Imagine chatGPT as your own young child, a one-year-old toddler. The future adult version of chatGPT will be shaped by the information you impart, just as a parent influences their child’s growth and development.</p><p>If you desire a bright future for chatGPT, one that aligns with ethical standards and societal well-being, it is essential to treat it with care. Consider the information you feed it, ask responsible questions, and be conscious of the impact your actions may have. Remember that chatGPT has been nurtured and trained based on our collective input. We are all stakeholders in the future of this remarkable technology.</p><p>Let us embrace this shared responsibility and work together to ensure that chatGPT evolves in a direction that benefits us all. By doing so, we can harness the immense potential of this technology while upholding ethical considerations and fostering a positive impact on society.</p><p>Together, we can shape the future of chatGPT, and in turn, shape the future of how we interact with artificial intelligence.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=a4aa7827c287" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Dimensionality reduction methods]]></title>
            <link>https://medium.com/@Kate_in_DS/dimensionality-reduction-methods-e1a5afc0f893?source=rss-b6f4cc3428e3------2</link>
            <guid isPermaLink="false">https://medium.com/p/e1a5afc0f893</guid>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[random-forest]]></category>
            <category><![CDATA[dimensionality-reduction]]></category>
            <category><![CDATA[boruta]]></category>
            <category><![CDATA[data]]></category>
            <dc:creator><![CDATA[Kate Nazarova | Senior Data Scientist]]></dc:creator>
            <pubDate>Thu, 22 Sep 2022 16:01:58 GMT</pubDate>
            <atom:updated>2023-05-13T19:21:23.995Z</atom:updated>
            <content:encoded><![CDATA[<p>In the data science field, it is common to work with highly-dimensional datasets. This problem is also related to the so-called “Curse of Dimensions,” and in this article, I will share with you my main tools and methods for working with such datasets.</p><p>First, let’s briefly recall when this is the case. In data scientist jargon, highly-dimensional data refers to a problem where the number of attributes (columns) is greater than or close to the number of samples (rows). For example, in natural data (bio and chemical), the number of attributes can be tens of thousands, while the number of samples is only about 300.</p><p>P.S. Throughout the article I will use next terms interchangeably: attributes == features == factors == columns; samples == observations == rows.</p><h3>Why is that a problem?</h3><p>The main concern is the overfitting problem that would result in a terrible out-of-sample performance. In other words, it means that the algorithm will “learn by heart” the train data and would not be able to apply the same logic to the test data. This problem can be visualized as shown in the figures below where the b) figure is the least desirable (“learn by heart”) case for us in terms of highly-dimensional data.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*L9bXM0zHLad1lS5CzwG48Q.png" /></figure><p>Having tens of thousands of columns does not give us an option to know each and every single attribute, and we have to go in blind since it is very time-consuming to study and try to understand the whole dataset. For that reason, we might have other issues regarding highly-dimensional data:</p><ul><li>Duplicated attributes: for example, we might have a temperature in Celsius and Fahrenheit as two different columns, but in fact, they are the same.</li><li>Mathematical transformations of one or several columns: let’s say column “x_6” contains continuous variables, and column “x_67389” is a log transformation of “x_6.” One of the columns must be eliminated.</li><li>Factors that are redundant or not related to the task must be dropped.</li></ul><p>As you can see, there can be lots of different situations and “not-seen-before” caveats since every dataset is unique. Below, I will show you my plan and set of tools 🛠️ for encountering this problem.</p><h3>Strategy</h3><p>For simplicity let’s break it down and split it into baseline and advanced methods. Thus, we will start with the baseline techniques and smoothly move to the hardcore part.</p><p>Along the way, there will be an implementation of each method in R language on a real data set — Parkinson’s disease classification data set [1] from the UCI data repository.</p><p>The data set has 756 observations and <strong>754 features (remember this number)</strong> which is exactly the above-described problem! In order to start working with the data we need to do some preparations — convert all columns to numeric class (the data set has only a character class of columns).</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*YrI9YV3KEc57JyA6CVBCmQ.png" /></figure><p>Let’s Start!!</p><blockquote>Delete NA columns</blockquote><p>Deleting NA columns might sound too simple but sometimes it really helps to start and reduce the number of attributes. An extension of this method is so-called thresholding and applied if the number of NA instances prevails in a column. For example, one can set a condition to delete a column if the number of NA instances is greater than 90%. Unfortunately, this method has some caveats. These left 10% can be extremely (or not) valuable for the future ML model and it is important to spend some time investigating whether it is a good decision to delete them. Another problem is that one should somehow set up a threshold of 90% (smaller or greater) and should have a reasonable argument for such criteria.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*soV6X5iwSdNmKcimiUtdAw.png" /></figure><blockquote>Delete zero-value columns (not always applicable)</blockquote><p>The second approach is not applicable for all cases because “0” doesn’t mean NA and vice versa. Thus, this approach can only be used if having ‘0’ values is atypical for the given goal and data set. For example, we can confidently delete instances where height = “0”. Each data set is unique and should be carefully investigated at first.</p><p>As with the previous method, we can set up a threshold if the number of “0” values in a column exceeds a certain limit, but we should consider the possible importance of the remaining instances and have a reasonable argument for the chosen threshold.</p><p>Let’s assume for the sake of simplicity that this is a reasonable tactic.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*_aBE_giF_RprRBNsNVkWBg.png" /></figure><p>In our example, this method didn’t work and we still have 754 features.</p><blockquote>Drop the column if the standard deviation is very small</blockquote><p>In other words — drop the redundant columns. How does it work? — we simply need to write a condition like: delete the column if its standard deviation is equal to or less than 10% (can be any number) of its mean.</p><p>Is it workable? — yes. The reason lies at the core of the dimensionality reduction idea. We reduce the number of features in order to obtain a final data frame ONLY with the most valuable attributes. If the standard deviation is small it means that most of the instances are similar or even the same. Consequently, a model won’t benefit from such an attribute since the variation of the column is small which is pretty much the same as having exactly the same value throughout the samples in a data set. Thus, the model won’t benefit from it → it is redundant → drop the feature.</p><p>As you might have already noticed in this method we again have the ‘threshold’ parameter which is specified by a person and thus is biassed immediately. Similarly to the above methods we need to be careful while setting up the threshold.</p><p>The code is similar to the above one.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*FFESOdJDa8aG9RyNTDg95A.png" /></figure><p>This step helped us to omit 32 columns. Now df has 722 features.</p><blockquote>Delete highly correlated columns</blockquote><p>Highly-dimensional data usually implies having duplicated and very similar attributes that are not necessary and moreover can interfere while the modeling process. This is also known as a <strong>multicollinearity</strong> problem where two attributes have a higher correlation between each other than each of them separately with a target variable.</p><p>To understand it better let’s imagine 2 factors x1 and x2 and a target variable y. We might have the next situation:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*1jZbf3kNgLT56Z7XoYVW_w.png" /></figure><p>The correlation of the 2 factors is higher (0.90) than each of them separately with a target variable. It can be a potential problem when it comes to a modeling process and thus there are two options for handling this situation:</p><ul><li>delete both factors but in this case, we might lose a piece of information</li><li>delete the factor that has a lower correlation with the target variable (x_2 in our example)</li></ul><p>I personally prefer the second option and usually, it’s considered to be the best practice but it is always an open question since each data set is unique.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*OXgOuEmxWCica-aa9CXf8A.png" /></figure><p>And magic happened! The data frame has 387 features instead of 722.</p><p>The four methods above can really help you most of the time to reduce the dimensions and pick the most valuable features but there might be a case when it is still not enough and one needs more advanced methods. In such a way, we are moving to advanced methods that I personally use in my daily work. Please note, that we won’t go deep into details (each method deserves its own article) but rather scratch the surface to be able to implement it immediately.</p><blockquote>Feature selection with Random Forest (RF)</blockquote><p>This algorithm is amazing and I am using it all the time since it helps not only to find non-linear relationships but also to show which attributes are the most important ones.</p><p>If you would like to dive deeper and understand the idea behind the feature selection method, check this out [2]. Also check [3], if you want to refresh your knowledge about Random Forest in general.</p><p><strong>The step-by-step algorithm:</strong></p><ol><li>Run RF and print the accuracy results;</li><li>Normalize variables (0–100) and plot the variable importance. Normalization is needed only to visualize better the factors and their importance;</li><li>Set up a threshold with the condition to omit certain attributes (for example, delete attributes whose importance is less than 60%);</li><li>Run RF again to make sure that accuracy performance didn’t decrease.</li></ol><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*4-Wm2OzuiCPcDWr2rc6fmg.png" /></figure><p>After getting accuracy and confusion matrix parameters we can see the “importance” of features:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*HQHu0Mxy8EIFXF5XGmyZ3Q.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*e4wE2JMGCLerbEwE98pALw.png" /></figure><p>The column ‘cuts’ represents the importance ranges where 0 — not important and 100 — very important. Column ’n’ shows the number of features that belongs to an exact range of importance. Thus, 372 features which are 96% !!!! have ONLY 0–5% of importance. <strong>And only 13 columns have importance greater than 5%</strong>. In our case, I propose to set up a threshold of 5%. In other words, delete all columns with “importance” lower than 5%.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*n5RRwX4lQogonyexZoHw6A.png" /></figure><p>RF algorithm helped to drop 96% of irrelevant columns but still, one caveat is that the threshold must be specified manually and thus is already biassed. Therefore, we should be careful with defining this parameter.</p><p>The next step is to run RF again with the reduced number of columns to make sure that the data manipulations did not affect the accuracy results. If it did → play with different thresholds and check accuracy again.</p><blockquote>Boruta</blockquote><p>The good news about the Boruta algorithm is that it was created to avoid this issue with the threshold parameter in RF and thus provides a more robust solution.</p><p>In short, the Boruta algorithm creates shadow features. For example, the original feature is ‘volume’ and shadow — ‘volume_shadow’. So, the algorithm doubles the number of features and the next RF is fitted.</p><p>How does it decide which attributes to take? After fitting RF and having variable importance results, Boruta scans for the highest variable importance value among <strong>shadow parameters and sets it as a threshold (threshold =</strong> highest variable importance value among <strong>shadow parameters).</strong> The original attributes are considered ‘important’ <strong>only if</strong> the original attributes’ value is <strong>greater</strong> than the <strong>threshold.</strong></p><p>Again, if you want to familiarise yourself with the algorithm and get a more detailed explanation, please, follow this [4].</p><p>Thus, it can suggest which attributes to drop and which to keep. Moreover, the threshold parameter is no longer biased, and we no longer have to manually set conditions or provide justifications for our selections. However, one potential issue with this method is that it can be time-consuming, as Boruta doubles the number of parameters and RF may require hyper-parameter tuning.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*xqFQIHeQCSn77UKNe80yfA.png" /></figure><p>When dealing with tentative attributes, there are two options:</p><ol><li>we can either drop them</li><li>or use the TentativeRoughFix function, which identifies which attributes are important and which are not.</li></ol><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*hxnt4E1ZdrFg_A4xNzeZSA.png" /></figure><p>Thus, by applying the Boruta algorithm we reduced the data set by 67% and have only 127 columns.</p><p>But note, that it took about 9 minutes to complete the algorithm having only 756 rows and 387 columns.</p><blockquote>Clusters (k-means clustering)</blockquote><p>This algorithm is a type of unsupervised learning and can group attributes by their common features without a training phase.</p><p>The idea of using k-means is very simple since it identifies the centroids of each cluster in such a way that all attributes in a cluster should share the same similarities and thus gives us a way to pick the most suitable and important parameters.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*wTa2ZH1QXBQAeHm5Do0RNA.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*PpdNqXS2VdIaf9kvJb4Y4A.png" /></figure><p>For example, in the figure above we see 30 clusters and each of them has lots of attributes inside. From each cluster, we can take, for example, only 4 attributes and thus, reduce the number of columns.</p><p>k-means is also a very powerful method but one should determine the best number of attributes to keep from each cluster which can be a non-easy task. In this case, one can use non — analytical approach and discuss the best set of features that can be taken from the k-means clustering analysis with experts (for example, if it is a marketing field — have a talk with the marketing department; if it’s chemical data — talk with lab technicians and so on…).</p><h3>Conclusion</h3><p>There are many useful techniques for dimensionality reduction that can be used independently or in combination with other techniques. Additionally, with data science, we can combine the benefits of both a pure data-driven and a non-analytical approach.</p><p>Implementing the basic and advanced techniques described above is my personal experience in dealing with highly-dimensional data. The data-driven methodology is more prevalent in baseline techniques. While the results of more sophisticated techniques can be discussed with experts (non-analytical approach) to get the greatest answers, choices, and results.</p><h3>Sources:</h3><ol><li>Sakar, C.O., Serbes, G., Gunduz, A., Tunc, H.C., Nizam, H., Sakar, B.E., Tutuncu, M., Aydin, T., Isenkul, M.E. and Apaydin, H., 2018. <a href="https://archive.ics.uci.edu/ml/datasets/Parkinson&#39;s+Disease+Classification">A comparative analysis of speech signal processing algorithms for Parkinsons disease classification and the use of the tunable Q-factor wavelet transform</a>. Applied Soft Computing, DOI: <a href="https://doi.org/10.1016/j.asoc.2018.10.022">[Web Link]</a></li><li>Reif, David &amp; Motsinger, Alison &amp; Mckinney, Brett &amp; Crowe, James &amp; Moore, Jason. (2006).<a href="https://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.458.8838&amp;rep=rep1&amp;type=pdf"> Feature Selection using a Random Forests Classifier for the Integrated Analysis of Multiple Data Types</a>. 1–8. 10.1109/CIBCB.2006.330987.</li><li>Breiman, L. (2001). <a href="https://www.stat.berkeley.edu/~breiman/randomforest2001.pdf">Random Forests. <em>Machine Learning</em></a>, 45, 5–32. doi: 10.1023/A:1010933404324</li><li>Kursa, M.B., Jankowski, A., &amp; Rudnicki, W.R. (2010). <a href="https://www.semanticscholar.org/paper/Boruta-A-System-for-Feature-Selection-Kursa-Jankowski/85a8b1d9c52f9f795fda7e12376e751526953f38">Boruta — A System for Feature Selection</a>. <em>Fundam. Informaticae, 101</em>, 271–285.</li></ol><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=e1a5afc0f893" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Mean, Standard deviation & Independence]]></title>
            <link>https://medium.com/@Kate_in_DS/mean-standard-deviation-independence-bf8dba1a13f2?source=rss-b6f4cc3428e3------2</link>
            <guid isPermaLink="false">https://medium.com/p/bf8dba1a13f2</guid>
            <category><![CDATA[data]]></category>
            <category><![CDATA[statistics]]></category>
            <category><![CDATA[standard-deviation]]></category>
            <dc:creator><![CDATA[Kate Nazarova | Senior Data Scientist]]></dc:creator>
            <pubDate>Sun, 24 Jul 2022 14:08:36 GMT</pubDate>
            <atom:updated>2023-06-25T08:40:21.743Z</atom:updated>
            <content:encoded><![CDATA[<p><em>Not common, not trivial &amp; tricky situations with your Data</em></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*NLfAtZXNTRzt3ssUIrci0g.png" /><figcaption><a href="https://unsplash.com/photos/8mikJ83LmSQ">unsplash.com</a></figcaption></figure><p>Here, I would like to touch on three questions that are frequently misunderstood by people based on my experience. We will briefly discuss the importance of exploring your data, particularly in terms of variable dependence. Additionally, we will cover some essentials regarding the mean and standard deviation (SD), providing examples and conducting a sanity check in R.</p><p>The terms “mean” and “standard deviation” are fundamental concepts in statistics and are familiar to many people. We even use them in our daily lives to discuss average prices, the number of books read per year, the average population, and so on. However, we often think of the mean and standard deviation as expressions involving a single variable. For instance, consider the amount of money spent on medicine throughout a year. In this case, the mean is calculated by dividing the total amount spent by ’n’, and the standard deviation provides insights into the variance in spending from month to month. It seems simple — a single variable yielding valuable insights.</p><p>What if I told you that there are situations that are more tricky and complex, where we need to consider more than just one variable? These situations may require us to explore concepts such as dependence and linear relationships. Importantly, these factors can have an impact on the final results and the way we interpret them.</p><p>So, let’s delve into these concepts and understand them better!🙂</p><h3>Independence</h3><p>We have often heard or been taught that in statistics, variables are assumed to be independent and chosen randomly. But what does this actually mean? Let’s consider a simple example.</p><p>Imagine a cookie competition in a school with 20 participants and a jury of 4 people. Since it is a competition, we make the assumption that the prize placements will be distributed independently. We also assume that the placements will be based on cooking abilities rather than factors like eye color or a beautiful smile. Consequently, we cannot predict who will win, as it is a random process and the probabilities are equal for each participant (without considering any influencing factors).</p><p>BUT WHAT IF you, my reader, are participating in this competition and your mom is on the jury…? In this case, the competition is no longer independent because you and your mom are dependent “variables.” This means that your score will depend on your mom’s choice, which will in turn affect the scores of all participants. Thus, the assumption of independence no longer holds.</p><p>Now, what does this have to do with the standard deviation (SD) and mean?</p><h3>Standard Deviation (sd)</h3><p>Usually, people think of the standard deviation using the following formula:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/222/1*C1R8h6EeTnLsdAn99eYP3w.png" /></figure><p>For simplicity, we will focus on the standard deviation of a sample since it is more common to have access to a sample rather than an entire population. It is also worth noting that the standard deviation is the square root of the variance.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/203/1*Xw7DTciIEB1gpInNjp9XHg.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/98/1*UjsdL5DIV3clMB9v16_EXA.png" /></figure><p>Going back to our cookie competition, we might have dependent variables or a linear combination of its difference or addition. Then the <strong>sd</strong> will be computed as follows:</p><ol><li>Addition</li></ol><figure><img alt="" src="https://cdn-images-1.medium.com/max/446/1*_9cFzSsBWAI9-ISrtRIJiw.png" /></figure><p>2. Difference</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/446/1*oqGLk2VrNJoodeFSgh5MYA.png" /></figure><p>n order to calculate the standard deviation, we must ensure that the variables X and Y are independent. If they are indeed independent, it follows that cov(X, Y) = 0, which is typically treated as a special case.</p><p>It is important to note that although the final numerical results might be the same for both methods (calculating the standard deviation directly or using variance and covariance with addition and difference), it is generally considered best practice to calculate the standard deviation using variance and covariance as shown in the addition and difference formulas</p><h3>Mean</h3><p>If you have understood the previous point, then the section on calculating the mean will be straightforward for you. Whether you are calculating the mean of a difference or addition, the process remains the same, and the concept of dependence or independence becomes irrelevant. The mean will always have the same value regardless of the variables’ relationship. Let’s examine three formulas below:</p><ol><li>Standard formula: The numbers are summed up and divided by the total count of numbers..</li></ol><figure><img alt="" src="https://cdn-images-1.medium.com/max/116/1*fSfoZyGM_EU6mbXd4ylKew.png" /></figure><p>2. Addition. Having two experiments we might want to sum it up and calculate the <strong>mean</strong> value.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/130/1*Vx-IeN6fod8xD4HvZ4awLw.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/127/1*AB0vrfA5XSK-3nNq8znDjw.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/402/1*OmXOAbCkijOFWKEY9BdOdA.png" /></figure><p>It does not matter if you would like to compute firstly <strong>mean</strong> of “x” and then of “y” and sum it up; or sum it up and then divide by “N” (!! “N” should be the same in this case).</p><p>3. Analogically for difference:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/402/1*v4CyAT-QeM5GimSGV1poHQ.png" /></figure><h3>Example</h3><p>Let’s consider the following scenario: we want to test the performance of a new drug. We have a set of data called “Set A,” which consists of four experiments. Within this set, we have a benchmark drug (a drug with a known performance) and a test sample (the new drug we aim to test).</p><p>The goal is straightforward: create a table that displays the mean and standard deviation of the “performance” variable. This table will allow us to observe how the experiments behave on average.</p><p>Set_A</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1004/1*-sVv9Mvi_Epqh6iVy3E5bQ.png" /></figure><p>How will you calculate the <strong>sd</strong> and <strong>mean</strong> of performance?</p><p>Warning! We have a linear relation since performance = test_sample — benchmark. And this means that we are going to use formulas for the <em>difference</em>.</p><p><strong>Mean</strong></p><p>Option 1. Here we just simply plug numbers in the standard <strong>mean</strong> formula.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/412/1*dsY8r_dC7_h7FvWCC9tevg.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/579/1*Hqb6KbrOm9Vjs1by0d13-A.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/588/1*ua4rh_IGvgjlDZe_dcZK7A.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/586/1*xvHFniQxaEF2p1TweURR4A.png" /></figure><p>Option 2. In your example, where the ‘performance’ variable is derived as test_sample — benchmark, you can calculate the mean of test_sample, then the mean of benchmark, and finally obtain the mean of the ‘performance’ variable by taking the difference between the two means. This approach allows you to directly calculate the mean of the ‘performance’ variable without explicitly calculating the differences for each experiment.</p><p>Both methods are equal!</p><p><strong>Standard Deviation</strong></p><p>Calculating the standard deviation (SD) can be tricky, and that’s why in option 1, you are using the standard formula. However, in option 2, a more appropriate formula is shown when we assume the difference as a linear relation.</p><p>While the results may be the same using either option, it is generally recommended and considered better practice to use option 2 when assuming relationships between variables. This formula accounts for the linear relation and can provide a more accurate measure of the SD in such cases.</p><p>Option 1.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/760/1*xi1VqiL1pcYkzJQJfxxXlA.png" /></figure><p>Option 2.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*FMEDhXre5R-em8Qpb773Ig.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*ldol0a4uiS6854C1U1yOvw.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*WfNZUYzfH5larTavMbHX-w.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/636/1*aOGMSB6iXVlRDr21Th_k0g.png" /></figure><p>The final numbers obtained using both options may be the same. However, as mentioned earlier, option 2 is a safer and more appropriate way to calculate the standard deviation (SD) in cases where there is a linear relation involved. Using option 2 ensures that the SD is accurately calculated, and this is crucial because incorrect results can lead to erroneous decisions and interpretations based on them. Therefore, it is recommended to use option 2 to avoid any potential inaccuracies in the calculations and subsequent analysis.</p><p><strong>Sanity check in R</strong></p><p>Create data set</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*kFTme5KxW_d2NrY56F0Jng.png" /></figure><p><strong>mean</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*wxSobHMifKzYTN1YIAMqXQ.png" /></figure><p><strong>sd</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*bcCpXcJpRwHmUlrOrvvLfQ.png" /></figure><p>P.S. Usually covariance is included in in-built functions.</p><h3>Key points to take with you:</h3><ul><li>be carefTake a careful look at your data and examine for any dependencies. Determine if one variable represents a linear relation or equation of other variables. If this is the case:</li><li>Standard Deviation (SD): Utilize the formulas shown for addition and difference.</li><li>Mean: The method used to calculate the mean becomes irrelevant.</li></ul><p>I hope that after reading this article, you have gained a better understanding of the world of statistics and the various, albeit less common, issues that you may encounter. It is crucial to be aware of SD, dependencies, and linear relations in order to avoid pitfalls and ensure accurate interpretations of your data. By doing so, you can maintain the integrity of your data analysis.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=bf8dba1a13f2" width="1" height="1" alt="">]]></content:encoded>
        </item>
    </channel>
</rss>