<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Jinhang Jiang on Medium]]></title>
        <description><![CDATA[Stories by Jinhang Jiang on Medium]]></description>
        <link>https://medium.com/@jinhangjiang?source=rss-1c41ed254f6d------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/2*6k2BPiM3EXWsDgjzSG-Ctw.jpeg</url>
            <title>Stories by Jinhang Jiang on Medium</title>
            <link>https://medium.com/@jinhangjiang?source=rss-1c41ed254f6d------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Sun, 22 Nov 2026 14:54:38 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@jinhangjiang/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[Nothing Will Die. Long Live the Transformers.]]></title>
            <link>https://medium.com/data-science-collective/nothing-will-die-long-live-the-transformers-42adf3732059?source=rss-1c41ed254f6d------2</link>
            <guid isPermaLink="false">https://medium.com/p/42adf3732059</guid>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[ai]]></category>
            <category><![CDATA[revolution]]></category>
            <category><![CDATA[llm]]></category>
            <category><![CDATA[artificial-intelligence]]></category>
            <dc:creator><![CDATA[Jinhang Jiang]]></dc:creator>
            <pubDate>Sat, 20 Dec 2025 03:15:20 GMT</pubDate>
            <atom:updated>2025-12-23T12:50:28.896Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*uXJ9eSFUWfrP1jjqsfxf0A.png" /><figcaption>Generated by ChatGPT 5.2</figcaption></figure><p>Every few years, the tech world declares another profession dead.</p><p>Data science is dead.<br>Data analysts are dead.<br>Software engineers are DEAD.</p><p>The pattern is predictable, lazy, and frankly boring.</p><p>These titles are effective clickbait. They create artificial hierarchy, fuels identity anxiety, and gives people a cheap sense of superiority. Yet, they fundamentally misunderstand how fields actually evolve.</p><p>Titles change. Work does not disappear. Impact does not evaporate.</p><p>The real question is not whether AI will replace human jobs. The real question is whether humans are willing to transform.</p><h3>The Problem With “X Is Dead” Narratives</h3><p>Declaring a role dead is an intellectually narrow view of a very complex system.</p><p>Data science was never a job title. It was a convergence of statistics, computing, domain knowledge, and judgment. Machine learning engineering did not replace it. It absorbed part of it. The rest morphed, specialized, and spread across organizations.</p><p>The same thing happened to software engineering, data analytics, operations research, and product management. These are not rigid boxes. They are evolving capabilities.</p><p>What usually dies is not the field. What dies is a frozen skill set combined with an unwillingness to adapt.</p><p>If your professional identity depends on a title rather than the value you bring, then yes, AI is going to be uncomfortable for you.</p><h3>From Vibe Coding to Existential Panic</h3><p>When ChatGPT first appeared, many people dismissed it. The models made mistakes. The code was messy. The outputs were inconsistent. The conclusion was that this thing was cute but not serious.</p><p>Fast forward to the end of 2025, and suddenly the narrative flipped to the other extreme.</p><p>Software engineers will be fully replaced.<br>Data analysts will disappear.<br>Data science is over.</p><p>Swinging from dismissal to panic is not wisdom. It is intellectual whiplash.</p><p>Both extremes miss the point.</p><p>AI did not suddenly become magic. Humans did not suddenly become useless. What changed is leverage.</p><h3>We Are Optimizing the Wrong Things</h3><p>One uncomfortable truth is that we have been spending an absurd amount of human effort on the wrong problems.</p><p>Code style. Boilerplate refactoring. Minor optimizations. Syntax level correctness. These things matter, but they are also the easiest problems for AI to solve today.</p><p>That does not mean quality does not matter. It means humans no longer need to be the primary bottleneck for quality.</p><p>At McKesson, a phrase that keeps coming up this year is Transformation B. Doing things differently, not just doing them better.</p><p>That distinction matters.</p><p>Let LLMs worry about elegance, formatting, and mechanical correctness. Human attention is too expensive for that. Humans should be focused on system design, problem framing, tradeoff reasoning, and impact.</p><p>Efficiency is no longer the ceiling. Direction is.</p><h3>Nothing Will Die. Roles Will Recompose.</h3><p>We are still at the very early stage of the AI revolution. Many things that AI is bad at today will not even exist as tasks three decades from now.</p><p>But new ideas will never die.</p><p>Ideas are not just outputs. They are bedrock. One idea unlocks ten more. Even if models eventually generate novel ideas at scale, humans will still play a critical role in choosing which ideas matter, which ones align with values, and which risks are acceptable.</p><p>Most people will not be doing the same job duties in five years. That is not a threat. That is the default state of progress.</p><p>Obsessing over how to do one thing better forever is a losing strategy. Best practices are temporary by definition. They exist only until better practices replace them.</p><h3>The Only Group That Truly Thrives</h3><p>In this AI wave, only one group consistently thrives.</p><p>Transformers.</p><p>Not the model. The people.</p><p>They are adaptive.<br>They are eager to learn.<br>They are comfortable admitting ignorance.<br>They are willing to try new things before they feel ready.</p><p>Most importantly, they think like designers.</p><p>They operate at a high level of abstraction. They frame problems, define goals, and evaluate outcomes. They let LLMs handle execution details, exploration, and iteration.</p><p>They do not ask, “How do I code this step by step?”<br>They ask, “What system should exist, and why?”</p><p>That mindset scales. Titles do not.</p><h3>Why AI Will Not Fully Replace Humans</h3><p>There is another inconvenient truth that rarely shows up in hype articles.</p><p>Responsibility still matters.</p><p>When things go wrong, we cannot blame a model. Someone has to own decisions, justify tradeoffs, and take accountability. That will not change anytime soon.</p><p>Despite years of research in explainable AI, most modern models remain black boxes. Many XAI techniques are post hoc approximations using simpler interpretable models to mimic black box behavior. They sound plausible. They produce charts. They look scientific.</p><p>That does not make them ethically sufficient or operationally trustworthy.</p><p>Until models can genuinely explain themselves in a way that holds up under scrutiny, humans must remain in charge of decisions that carry real consequences.</p><h3>A Hard Truth to End On</h3><p>Yes, some people will be pushed out.</p><p>Those who cling to narrow definitions of their role.<br>Those who focus only on execution and ignore context.<br>Those who optimize for comfort instead of growth.</p><p>That is not cruelty. That is reality.</p><p>At the same time, transformers will keep thriving. They will redefine roles, invent new ones, and use AI as leverage rather than as a measuring stick for ego.</p><p>Nothing will die.</p><p>But everything will change.</p><p>And honestly, that is exactly how it should be.</p><p>See you in 2026.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=42adf3732059" width="1" height="1" alt=""><hr><p><a href="https://medium.com/data-science-collective/nothing-will-die-long-live-the-transformers-42adf3732059">Nothing Will Die. Long Live the Transformers.</a> was originally published in <a href="https://medium.com/data-science-collective">Data Science Collective</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Introducing TextRegress: Advanced Regression for Long-form Text Data]]></title>
            <link>https://towardsdev.com/introducing-textregress-advanced-regression-for-long-form-text-data-63ab1febcf35?source=rss-1c41ed254f6d------2</link>
            <guid isPermaLink="false">https://medium.com/p/63ab1febcf35</guid>
            <category><![CDATA[ai]]></category>
            <category><![CDATA[nlp]]></category>
            <category><![CDATA[software-development]]></category>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[deep-learning]]></category>
            <dc:creator><![CDATA[Jinhang Jiang]]></dc:creator>
            <pubDate>Wed, 05 Mar 2025 16:29:20 GMT</pubDate>
            <atom:updated>2025-12-24T07:15:49.115Z</atom:updated>
            <content:encoded><![CDATA[<p><em>TextRegress provides a unified framework for conducting predictive modeling of continuous outcomes.</em></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*f8FoLxUUxiVjnzxU" /><figcaption>Image by Author</figcaption></figure><h4><strong>What is TextRegress?</strong></h4><p>TextRegress is an open-source Python package designed for performing advanced regression analysis on long-form text data. Traditional regression models rely on structured numerical inputs, but with TextRegress, you can now leverage unstructured, especially long and large, text data as a predictive feature in your regression tasks. This package encodes text into numerical representations and applies deep learning-based models to predict continuous target values.</p><h4><strong>Why TextRegress?</strong></h4><p>With the rapid growth of AI applications in various fields, researchers and practitioners often encounter text-based data that holds valuable information. However, integrating long-form text into standard regression pipelines remains a challenge. TextRegress bridges this gap by providing an easy-to-use framework for text regression modeling, enabling:</p><ul><li>Seamless integration of text into numerical prediction models.</li><li>A flexible API with deep learning-powered regression.</li><li>The ability to include exogenous (structured) features.</li></ul><p>By lowering the barrier to using advanced algorithms in text regression, TextRegress helps democratize AI, making advanced text modeling techniques accessible across different domains like social sciences, finance, healthcare, and more.</p><h4><strong>Core Framework</strong></h4><p>TextRegress is inspired by the question: “How can long-form and large-scale text documents be used to predict continuous values?” One of the approaches is to use hierarchical embeddings. The process starts by breaking text into manageable chunks, ensuring no information is lost. These chunks are then embedded into numerical representations using pre-trained models like SBERT. To handle sequential dependencies, deep learning models process these embeddings, creating the document embeddings.</p><p>If additional exogenous data is available, it interacts with the global tokens of the text representation through cross-attention, enriching the predictive model. Finally, multi-layer perceptrons and regression techniques combine all learned features to generate accurate continuous predictions.</p><p>This approach unlocks the power of unstructured text for quantitative forecasting. TextRegress simplifies this entire pipeline, providing you an fit and predict API, making it easy to implement and scale. The flowchart below represents the core framework of how TextRegress works.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*kly3jPVDHh9L40vk" /><figcaption>Image by Author</figcaption></figure><h4><strong>Key Features</strong></h4><ol><li>Simple Fit &amp; Predict API: Designed with an sklearn-like API for easy adoption.</li><li>Multiple Encoding Options: Supports TF-IDF, Sentence Transformers, and Hugging Face models for text encoding.</li><li>Deep Learning-Based Modeling: Uses pytorch-lighting as the backbone for fast and massive processing and training.</li><li>Flexible Loss Functions: Supports MAE, MSE, RMSE, SMAPE, MAPE, and WMAPE. And even allow users to bring in their own loss functions.</li><li>Cross-Attention Mechanism: Allows better interaction between text and structured data.</li></ol><h4><strong>Installation</strong></h4><p>TextRegress is available on PyPI. You can install it with (https://pypi.org/project/textregress/):</p><pre>pip install textregress</pre><h4><strong>Getting Started</strong></h4><p><strong>Example 1: Basic Fit &amp; Predict</strong></p><p>Here’s how easy it is to train and predict with TextRegress:</p><pre>import pandas as pd<br>from textregress import TextRegressor<br><br># Sample data<br># must have columns &quot;text&quot; and &quot;y&quot;<br>data = pd.read_csv(&quot;your_data.csv&quot;) <br><br># Initialize and train model<br>model = TextRegressor()<br>model.fit(data)<br><br># Predict on new data<br>predictions = model.predict(data)<br>print(predictions)</pre><p><strong>Example 2: Customizing Model Parameters</strong></p><p>TextRegress allows fine-tuning key parameters:</p><pre>model = TextRegressor(<br>    encoder_model=&quot;sentence-transformers/all-MiniLM-L6-v2&quot;,<br>    rnn_type=&quot;LSTM&quot;,<br>    rnn_layers=3,<br>    hidden_size=256,<br>    loss_function=&quot;smape&quot;,<br>    learning_rate=5e-4,<br>    max_steps=1000,<br>    dropout_rate=0.2<br>)<br>model.fit(data)</pre><h4>Contribute &amp; Improve &amp; Citation</h4><p>TextRegress is still evolving! We welcome contributions, feedback, and ideas for improvement. Feel free to explore the GitHub Repo (<a href="https://github.com/jinhangjiang/textregress">https://github.com/jinhangjiang/textregress</a>) and join the discussion.</p><p>If this package was helpful in your work, feel free to cite it as</p><ul><li>Jiang, Jinhang and Liu, Ben and Peng, Weiyao and Srinivasan, Karthik, TextRegress: A Python Package for Advanced Regression Analysis on Long-Form Text Data (February 23, 2025). Available at SSRN: <a href="https://ssrn.com/abstract=5160696">https://ssrn.com/abstract=5160696</a></li></ul><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=63ab1febcf35" width="1" height="1" alt=""><hr><p><a href="https://towardsdev.com/introducing-textregress-advanced-regression-for-long-form-text-data-63ab1febcf35">Introducing TextRegress: Advanced Regression for Long-form Text Data</a> was originally published in <a href="https://towardsdev.com/">Towards Dev</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[From Scratch to Deep Quantile Forecasting]]></title>
            <link>https://medium.com/data-science/from-scratch-to-deep-quantile-forecasting-366d84b7dd22?source=rss-1c41ed254f6d------2</link>
            <guid isPermaLink="false">https://medium.com/p/366d84b7dd22</guid>
            <category><![CDATA[time-series-forecasting]]></category>
            <category><![CDATA[hands-on-tutorials]]></category>
            <category><![CDATA[ai]]></category>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[quantile-regression]]></category>
            <dc:creator><![CDATA[Jinhang Jiang]]></dc:creator>
            <pubDate>Tue, 16 Jul 2024 01:24:38 GMT</pubDate>
            <atom:updated>2024-07-16T01:24:38.588Z</atom:updated>
            <content:encoded><![CDATA[<h4>An end-2-end empirical sharing of multi-step quantile forecasting with Tensorflow, NeuralForecast, and Zero-shot LLMs.</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*aHVnjYoxqE-cilt28i_dLQ.png" /><figcaption>Image by Author</figcaption></figure><h3>Content</h3><ol><li>Short Introduction</li><li>Data</li><li>Build a Toy Version of Quantile Recurrent Forecaster</li><li>Quantile Forecasting with the State-of-Art Models</li><li>Zero-shot Quantile Forecast with LLMs</li><li>Conclusion</li></ol><h3>Short Introduction</h3><p>Quantile forecasting is a statistical technique used to predict different quantiles (e.g., the median or the 90th percentile) of a response variable’s distribution, providing a more comprehensive view of potential future outcomes. Unlike traditional mean forecasting, which only estimates the average, quantile forecasting allows us to understand the range and likelihood of various possible results.</p><p>Quantile forecasting is essential for decision-making in contexts with asymmetric loss functions or varying risk preferences. In supply chain management, for example, predicting the 90th percentile of demand ensures sufficient stock levels to avoid shortages, while predicting the 10th percentile helps minimize overstock and associated costs. This methodology is particularly advantageous in sectors such as finance, meteorology, and energy, where understanding distribution extremes is as critical as the mean.</p><p>Both quantile forecasting and conformal prediction address uncertainty, yet their methodologies differ significantly. Quantile forecasting directly models specific quantiles of the response variable, providing detailed insights into its distribution. Conversely, conformal prediction is a model-agnostic technique that constructs prediction intervals around forecasts, guaranteeing that the true value falls within the interval with a specified probability. Quantile forecasting yields precise quantile estimates, whereas conformal prediction offers broader interval assurances.</p><p>The implementation of quantile forecasting can markedly enhance decision-making by providing a sophisticated understanding of future uncertainties. This approach allows organizations to tailor strategies to different risk levels, optimize resource allocation, and improve operational efficiency. By capturing a comprehensive range of potential outcomes, quantile forecasting enables organizations to make informed, data-driven decisions, thereby mitigating risks and enhancing overall performance.</p><h3>Data</h3><p>To demonstrate the work, I chose to use the data from the M4 competition as an example. The data is under <a href="https://creativecommons.org/publicdomain/zero/1.0/">CC0: Public Domain</a> license which can be accessed <a href="https://www.kaggle.com/datasets/yogesh94/m4-forecasting-competition-dataset">here</a>. The data can also be loaded through datasetsforecast package:</p><pre># Install the package<br>pip install datasetsforecast<br># Load Data<br>df, *_ = M4.load(&#39;./data&#39;, group=&#39;Weekly&#39;)<br># Randomly select three items<br>df = df[df[&#39;unique_id&#39;].isin([&#39;W96&#39;, &#39;W100&#39;, &#39;W99&#39;])]<br># Define the start date (for example, &quot;1970-01-04&quot;)<br>start_date = pd.to_datetime(&quot;1970-01-04&quot;)<br># Convert &#39;ds&#39; to actual week dates<br>df[&#39;ds&#39;] = start_date + pd.to_timedelta(df[&#39;ds&#39;] - 1, unit=&#39;W&#39;)<br># Display the DataFrame<br>df.head()</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/580/1*xdCZ_HvCMFFnet73jWRw6g.png" /><figcaption>Image by Author</figcaption></figure><p>The original data contains over 300 unique time series. To demonstrate, I randomly selected three time series: W96, W99, and W100, as they all have the same history length. The original timestamp is masked as integer numbers (i.e., 1–2296), I manually converted it back to normal date format with the first date to be January 4th, 1970. The following figure is a preview of W99:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/868/1*lZkEvqYihN8I_RWW-vb4jw.png" /><figcaption>Image by Author</figcaption></figure><h3>Build a Toy Version of Quantile Recurrent Forecaster</h3><p>First, let’s build a quantile forecaster from scratch to understand how the target data flows through the pipeline and how the forecasts are generated. I picked the idea from the paper <a href="https://arxiv.org/pdf/1711.11053">A Multi-Horizon Quantile Recurrent Forecaster</a> by Wen et al. The authors proposed a Multi-Horizon Quantile Recurrent Neural Network (MQ-RNN) framework that combines Sequence-to-Sequence Neural Networks, Quantile Regression, and Direct Multi-Horizon Forecasting for accurate and robust multi-step time series forecasting. By leveraging the expressiveness of neural networks, the nonparametric nature of quantile regression, and a novel training scheme called forking-sequences, the model can effectively handle shifting seasonality, known future events, and cold-start situations in large-scale forecasting applications.</p><p>We cannot reproduce everything in this short blog, but we can try to replicate part of it using the TensorFlow package as a demo. If you are interested in the implementation of the paper, there is an ongoing project that you can leverage: <a href="https://github.com/tianchen101/MQRNN?tab=readme-ov-file">MQRNN</a>.</p><p>Let’s first load the necessary package and define some global parameters. We will use the LSTM model as the core, and we need to do some preprocessing on the data to obtain the rolling windows before fitting. The input_shape is set to (104, 1) meaning we are using two years of data for each training window. In this walkthrough, we will only look into an 80% confidence interval with the median as the point forecast, which means the quantiles = [0.1, 0.5, 0.9]. We will use the last 12 weeks as a test dataset, so the output_steps or horizon is equal to 12 and the cut_off_date will be ‘2013–10–13’.</p><pre># Install the package<br>pip install tensorflow<br><br># Load the package<br>from sklearn.preprocessing import StandardScaler<br>from datetime import datetime<br>from tensorflow.keras.models import Model<br>from tensorflow.keras.layers import Input, LSTM, Dense, concatenate, Layer<br><br># Define Global Parameters<br>input_shape = (104, 1)<br>quantiles = [0.1, 0.9]<br>output_steps = 12<br>cut_off_date = &#39;2013-10-13&#39;<br>tf.random.set_seed(20240710)</pre><p>Next, let’s convert the data to rolling windows which is the desired input shape for RNN-based models:</p><pre># Preprocess The Data<br>def preprocess_data(df, window_size = 104, forecast_horizon = 12):<br>    # Ensure the dataframe is sorted by item and date<br>    <br>    df = df.sort_values(by=[&#39;unique_id&#39;, &#39;ds&#39;])<br>    # List to hold processed data for each item<br>    X, y, unique_id, ds = [], [], [], []<br>    # Normalizer<br>    scaler = StandardScaler()<br>    # Iterate through each item<br>    for key, group in df.groupby(&#39;unique_id&#39;):<br>        demand = group[&#39;y&#39;].values.reshape(-1, 1)<br>        scaled_demand = scaler.fit_transform(demand)<br>        dates = group[&#39;ds&#39;].values<br>        # Create sequences (sliding window approach)    <br>        for i in range(len(scaled_demand) - window_size - forecast_horizon + 1):<br>            X.append(scaled_demand[i:i+window_size])<br>            y.append(scaled_demand[i+window_size:i+window_size+forecast_horizon].flatten())<br>            unique_id.append(key)<br>            ds.append(dates[i+window_size:i+window_size+forecast_horizon])<br>    X = np.array(X)<br>    y = np.array(y)<br>    return X, y, unique_id, ds, scaler</pre><p>Then we split the data into train, val, and test:</p><pre># Split Data<br>def split_data(X, y, unique_id, ds, cut_off_date):<br>    cut_off_date = pd.to_datetime(cut_off_date)<br>    val_start_date = cut_off_date - pd.Timedelta(weeks=12)<br>    train_idx = [i for i, date in enumerate(ds) if date[0] &lt; val_start_date]<br>    val_idx = [i for i, date in enumerate(ds) if val_start_date &lt;= date[0] &lt; cut_off_date]<br>    test_idx = [i for i, date in enumerate(ds) if date[0] &gt;= cut_off_date]<br><br>    X_train, y_train = X[train_idx], y[train_idx]<br>    X_val, y_val = X[val_idx], y[val_idx]<br>    X_test, y_test = X[test_idx], y[test_idx]<br><br>    train_unique_id = [unique_id[i] for i in train_idx]<br>    train_ds = [ds[i] for i in train_idx]<br>    val_unique_id = [unique_id[i] for i in val_idx]<br>    val_ds = [ds[i] for i in val_idx]<br>    test_unique_id = [unique_id[i] for i in test_idx]<br>    test_ds = [ds[i] for i in test_idx]<br>    <br>    return X_train, y_train, X_val, y_val, X_test, y_test, train_unique_id, train_ds, val_unique_id, val_ds, test_unique_id, test_ds</pre><p>The authors of the MQRNN utilized both horizon-specific local context, essential for temporal awareness and seasonality mapping, and horizon-agnostic global context to capture non-time-sensitive information, enhancing the stability of learning and the smoothness of generated forecasts. To build a model that sort of reproduces what the MQRNN is doing, we need to write a quantile loss function and add layers that capture local context and global context. I added an attention layer to it to show you how the attention mechanism can be included in such a process:</p><pre># Attention Layer<br>class Attention(Layer):<br>    def __init__(self, units):<br>        super(Attention, self).__init__()<br>        self.W1 = Dense(units)<br>        self.W2 = Dense(units)<br>        self.V = Dense(1)<br>    def call(self, query, values):<br>        hidden_with_time_axis = tf.expand_dims(query, 1)<br>        score = self.V(tf.nn.tanh(self.W1(values) + self.W2(hidden_with_time_axis)))<br>        attention_weights = tf.nn.softmax(score, axis=1)<br>        context_vector = attention_weights * values<br>        context_vector = tf.reduce_sum(context_vector, axis=1)<br>        return context_vector, attention_weights<br><br># Quantile Loss Function<br>def quantile_loss(q, y_true, y_pred):<br>    e = y_true - y_pred<br>    return tf.reduce_mean(tf.maximum(q*e, (q-1)*e))<br><br>def combined_quantile_loss(quantiles, y_true, y_pred, output_steps):<br>    losses = [quantile_loss(q, y_true, y_pred[:, i*output_steps:(i+1)*output_steps]) for i, q in enumerate(quantiles)]<br>    return tf.reduce_mean(losses)<br><br># Model architecture<br>def create_model(input_shape, quantiles, output_steps):<br>    inputs = Input(shape=input_shape)<br>    lstm1 = LSTM(256, return_sequences=True)(inputs)<br>    lstm_out, state_h, state_c = LSTM(256, return_sequences=True, return_state=True)(lstm1)<br>    context_vector, attention_weights = Attention(256)(state_h, lstm_out)<br>    global_context = Dense(100, activation = &#39;relu&#39;)(context_vector)<br>    forecasts = []<br>    for q in quantiles:<br>        local_context = concatenate([global_context, context_vector])<br>        forecast = Dense(output_steps, activation = &#39;linear&#39;)(local_context)<br>        forecasts.append(forecast)<br>    outputs = concatenate(forecasts, axis=1)<br>    model = Model(inputs, outputs)<br>    model.compile(optimizer=&#39;adam&#39;, loss=lambda y, f: combined_quantile_loss(quantiles, y, f, output_steps))<br>    return model</pre><p>Here are the plotted forecasting results:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*besmyR8Z4sy2v_nXJKzSKg.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*AsbVhsBtpnncH47gdTd7pw.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*3VXe8uw_0kXtZdUXDe4hmA.png" /></figure><p>We also evaluated the SMAPE for each item, as well as the percentage coverage of the interval (how much actual was covered by the interval). The results are as follows:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Nf_QnEzVJI7dDaTuq4cMNg.png" /></figure><p>This toy version can serve as a good baseline to start with quantile forecasting. The distributed training is not configured for this setup nor the model architecture is optimized for large-scale forecasting, thus it might suffer from speed issues. In the next section, we will look into a package that allows you to do quantile forecasts with the most advanced deep-learning models.</p><h3>Quantile Forecasting with the SOTA Models</h3><p>The <a href="https://nixtlaverse.nixtla.io/neuralforecast/index.html">neuralforecast</a> package is an outstanding Python library that allows you to use most of the SOTA deep neural network models for time series forecasting, such as PatchTST, NBEATs, NHITS, TimeMixer, etc. with easy implementation. In this section, I will use <a href="https://arxiv.org/pdf/2211.14730">PatchTST</a> as an example to show you how to perform quantile forecasting.</p><p>First, load the necessary modules and define the parameters for PatchTST. Tuning the model will require some empirical experience and will be project-dependent. If you are interested in getting the potential-optimal parameters for your data, you may look into the auto modules from the neuralforecast. They will allow you to use Ray to perform hyperparameter tuning. And it is quite efficient! The neuralforecast package carries a great set of models that are based on different sampling approaches. The ones with the base_window approach will allow you to use MQLoss or HuberMQLoss, where you can specify the quantile levels you are looking for. In this work, I picked HuberMQLoss as it is more robust to outliers.</p><pre># Install the package<br>pip install neuralforecast<br><br># Load the package<br>from neuralforecast.core import NeuralForecast<br>from neuralforecast.models import PatchTST<br>from neuralforecast.losses.pytorch import HuberMQLoss, MQLoss<br><br># Define Parameters for PatchTST<br>PARAMS = {&#39;input_size&#39;: 104, <br>          &#39;h&#39;: output_steps, <br>          &#39;max_steps&#39;: 6000, <br>          &#39;encoder_layers&#39;: 4,<br>          &#39;start_padding_enabled&#39;: False,<br>          &#39;learning_rate&#39;: 1e-4,<br>          &#39;patch_len&#39;: 52,  # Length of each patch<br>          &#39;hidden_size&#39;: 256,  # Size of the hidden layers<br>          &#39;n_heads&#39;: 4,  # Number of attention heads<br>          &#39;res_attention&#39;: True,<br>          &#39;dropout&#39;: 0.1,  # Dropout rate<br>          &#39;activation&#39;: &#39;gelu&#39;,  # Activation function<br>          &#39;dropout&#39;: 0.1,<br>          &#39;attn_dropout&#39;: 0.1,<br>          &#39;fc_dropout&#39;: 0.1,<br>          &#39;random_seed&#39;: 20240710,<br>          &#39;loss&#39;: HuberMQLoss(quantiles=[0.1, 0.5, 0.9]),<br>          &#39;scaler_type&#39;: &#39;standard&#39;,<br>          &#39;early_stop_patience_steps&#39;: 10}<br><br># Get Training Data<br>train_df = df[df.ds&lt;cut_off_date] <br><br># Fit and predict with PatchTST<br>models = [PatchTST(**PARAMS)]<br>nf = NeuralForecast(models=models, freq=&#39;W&#39;) <br>nf.fit(df=train_df, val_size=12)<br>Y_hat_df = nf.predict().reset_index()</pre><p>Here are plotted forecasts:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*L1mItxMuOZbT-4_OtdUjYw.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*QytBjolXCuWepA9sfkTwdA.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*XzKnqo0RdDhNrGanbwSJ0A.png" /></figure><p>Here are the metrics:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*9kmNri4RKwP0ivM04B6n8A.png" /></figure><p>Through the demo, you can see how easy to implement the model and how the performance of the model has been lifted. However, if you wonder if there are any easier approaches to do this task, the answer is YES. In the next section, we will look into a T5-based model that allows you to conduct zero-shot quantile forecasting.</p><h3>Zero-shot Quantile Forecast with LLMs</h3><p>We have been witnessing a trend where the advancement in NLP will also further push the boundaries for time series forecasting as predicting the next word is a synthetic process for predicting the next period’s value. Given the fast development of large language models (LLMs) for generative tasks, researchers have also started to look into pre-training a large model on millions of time series, allowing users to do zero-shot forecasts.</p><p>However, before we draw an equal sign between the LLMs and Zero-shot Time Series tasks, we have to answer one question: what is the difference between training a language model and training a time series model? It would be “tokens from a finite dictionary versus values from an unbounded.” Amazon recently released a project called <a href="https://github.com/amazon-science/chronos-forecasting?tab=readme-ov-file">Chronos</a> which well handled the challenge and made the large time series model happen. As the authors stated: “Chronos tokenizes time series into discrete bins through simple scaling and quantization of real values. In this way, we can train off-the-shelf language models on this ‘language of time series,’ with no changes to the model architecture”. The original paper can be found <a href="https://arxiv.org/pdf/2403.07815">here</a>.</p><p>Currently, Chronos is available in multiple versions. It can be loaded and used through the <a href="https://auto.gluon.ai/stable/tutorials/timeseries/forecasting-chronos.html">autogluon API</a> with only a few lines of code.</p><pre># Get Training Data and Transform<br>train_df = df[df.ds&lt;cut_off_date] <br>train_df_chronos = TimeSeriesDataFrame(train_df.rename(columns={&#39;ds&#39;: &#39;timestamp&#39;, &#39;unique_id&#39;: &#39;item_id&#39;, &#39;y&#39;: &#39;target&#39;}))<br><br># Zero-shot forecast with Chronos<br>predictor = TimeSeriesPredictor(prediction_length=output_steps, freq=&#39;W&#39;, quantile_levels = [0.1, 0.9]).fit(<br>            train_df_chronos, presets=&quot;chronos_base&quot;, <br>            random_seed = 20240710<br>        )<br>Y_hat_df_chronos = predictor.predict(train_df_chronos).reset_index().rename(columns={&#39;mean&#39;: &#39;Chronos&#39;, <br>                                                                                     &#39;0.1&#39;: &#39;P10&#39;, <br>                                                                                     &#39;0.9&#39;: &#39;P90&#39;,<br>                                                                                     &#39;timestamp&#39;: &#39;ds&#39;,<br>                                                                                     &#39;item_id&#39;: &#39;unique_id&#39;})</pre><p>Here are the plotted forecasts:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*C6pWTqytSRpbj164A3rxaw.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*qi9FNJMtVs34YtjLNuIP3g.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*3KBMNJBfta0MpBpkKB3JtQ.png" /></figure><p>Here are the metrics:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Pb2qbGpefntBS-u89SBVuA.png" /></figure><p>As you can see, Chronos showed a very decent performance compared to PatchTST. However, it does not mean it has surpassed PatchTST, since it is very likely that Chronos has been trained on M4 data. In their original paper, the authors also evaluated their model on the datasets that the model has not been trained on, and Chronos still yielded very comparable results to the SOTA models.</p><p>There are many more large time series models being developed right now. One of them is called <a href="https://docs.nixtla.io/docs/getting-started-about_timegpt">TimeGPT</a> which was developed by NIXTLA. The invention of this kind of model not only made the forecasting task easier, more reliable, and consistent, but it is also a good starting point to make reasonable guesses for time series with limited historical data.</p><h3>Conclusion</h3><p>From building a toy version of a quantile recurrent forecaster to leveraging state-of-the-art models and zero-shot large language models, this blog has demonstrated the power and versatility of quantile forecasting. By incorporating models like TensorFlow’s LSTM, NeuralForecast’s PatchTST, and Amazon’s Chronos, we can achieve accurate, robust, and computationally efficient multi-step time series forecasts. Quantile forecasting not only enhances decision-making by providing a nuanced understanding of future uncertainties but also allows organizations to optimize strategies and resource allocation. The advancements in neural networks and zero-shot learning models further push the boundaries, making quantile forecasting a pivotal tool in modern data-driven industries.</p><p>Note: All the images, numbers and tables are generated by the author. The complete code can be found here: <a href="https://github.com/jinhangjiang/Medium_Demo/blob/4ed201b4130c940106dd70ec7fe8a0d7b5b847a8/From%20Scratch%20to%20Deep%20Quantile%20Forecasting/Quantile_Forecast.ipynb">Quantile Forecasting</a>.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=366d84b7dd22" width="1" height="1" alt=""><hr><p><a href="https://medium.com/data-science/from-scratch-to-deep-quantile-forecasting-366d84b7dd22">From Scratch to Deep Quantile Forecasting</a> was originally published in <a href="https://medium.com/data-science">TDS Archive</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Graph Analytics in Identifying Enduring Effects of Pandemics]]></title>
            <link>https://medium.com/data-science/graph-analytics-in-identifying-enduring-effects-of-pandemics-5225faacfe0e?source=rss-1c41ed254f6d------2</link>
            <guid isPermaLink="false">https://medium.com/p/5225faacfe0e</guid>
            <category><![CDATA[artificial-intelligence]]></category>
            <category><![CDATA[statistics]]></category>
            <category><![CDATA[social-network]]></category>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[healthcare]]></category>
            <dc:creator><![CDATA[Jinhang Jiang]]></dc:creator>
            <pubDate>Mon, 12 Dec 2022 14:20:28 GMT</pubDate>
            <atom:updated>2022-12-13T15:05:48.454Z</atom:updated>
            <content:encoded><![CDATA[<h4>Graph analytics framework designed for identifying enduring effects of COVID-19 on Healthcare Systems using Exponential Random Graph Models (ERGMs)</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*MHKCgs9BW6hJx6c5Gvo3Pg.png" /><figcaption>Image by author</figcaption></figure><h3>Content</h3><ol><li>Introduction</li><li>The Graph Analytics Framework</li><li>Explanatory Modeling with Exponential Random Graph Models</li><li>Implementation of ERGMs</li><li>Conclusion</li></ol><h3>Introduction</h3><p>One of our recent studies published in the ACM Transactions on Management Information Systems (ACM TMIS) has shown the potential of graph analytics in identifying the enduring effects of the COVID-19 pandemic. You may read the original article here:</p><p><a href="https://doi.org/10.1145/3564274">Examining Disease Multimorbidity in U.S. Hospital Visits Before and During COVID-19 Pandemic: A Graph Analytics Approach | ACM Transactions on Management Information Systems</a></p><p>We developed a graph analytics framework to investigate the simultaneous occurrence of multiple diseases in patients, known as disease multimorbidity, before and during the COVID-19 pandemic. By analyzing Electronic Health Records (EHRs) data, we were able to identify patterns and structures that could be used to detect the long-term effects of the virus.</p><p>Our study found that graph analytics can provide more accurate and comprehensive insights into the effects of COVID-19 on patients compared to traditional methods. By capturing complex relationships and dependencies in Electronic Health Records (EHRs) data, graph analytics can help healthcare providers better understand and address the long-term effects of the virus. These findings also have important implications for the ongoing efforts to combat COVID-19 and can support researchers and policymakers in developing effective interventions and treatments.</p><h3>The Graph Analytics Framework</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*b6AHg15iv4R2_8UMm2X-CA.png" /><figcaption>Image by author</figcaption></figure><p>The proposed framework builds upon existing information systems that analyze hospital discharge records. It uses graph analytics for multimorbidity analysis in addition to traditional analytics practices such as ranking disease prevalence or using data mining to predict outcomes. The framework is composed of three interlinked components: exploratory analysis, explanatory modeling, and predictive modeling. It is created empirically from hospital discharge datasets with diagnosis codes (ICD-10 code) and can be converted into an unweighted undirected network by setting a suitable threshold on the edge weight distribution. Within the exploratory analysis, structural information of the network and its nodes can be examined, and newer methods such as DeltaCon or NetSimile can be used for comparison. Explanatory modeling uses mathematical or statistical graph models (such as ERGMs), while predictive modeling uses embedding vectors of nodes and edges of the network to represent latent features or fit a high-dimensional non-linear model for prediction.</p><p>If you are interested in how to get embedding vectors of nodes and edges of the disease network, please refer to the following blog:</p><p><a href="https://medium.com/analytics-vidhya/analyzing-disease-co-occurrence-using-networkx-gephi-and-node2vec-53941da35a0f">Analyzing Disease Co-Occurrence Using NetworkX, Gephi, and Node2Vec</a></p><h3>Explanatory Modeling with Exponential Random Graph Models</h3><p>ERGMs are statistical models that analyze networks and explore the patterns and structures that underlie their formation. ERGMs are based on the concept that the probability of a specific network structure can be expressed as an exponential function of graph statistics, such as the number of edges or triangles in the network. <strong>In this study, we utilized exponential random graph models (ERGMs) to estimate the probability of two diseases forming a link, known as multimorbidity. We then compared the coefficients of different disease categories before and during the COVID-19 pandemic to interpret the enduring effects of the pandemic on healthcare systems.</strong></p><p>One of the key algorithms used in ERGMs is the Markov chain Monte Carlo (MCMC) algorithm, which is a method for sampling from a probability distribution in order to estimate the parameters of the model. MCMC uses a random walk process to explore the space of possible model configurations and iteratively updates the estimates of the model parameters based on the observed data. Another important algorithm used in ERGMs is the Metropolis-Hastings algorithm, which is a variant of the MCMC algorithm that allows for more efficient exploration of the model space.</p><p>One advantage of ERGMs is that they can incorporate exogenous variables, such as individual-level characteristics, to explain the formation of the network. This allows researchers to test hypotheses about how different factors influence network structure and to make predictions about how the network will evolve over time.</p><p>Another benefit of ERGMs is their ability to generate synthetic networks with structural properties similar to the observed data. This can be useful for augmenting limited data sets or simulating the effects of interventions on network structure.</p><p>One disadvantage of ERGMs is their computational complexity, particularly for large networks. They also require a large number of observations to produce reliable estimates, making them unsuitable for small or sparse data sets. The exponential function used in ERGMs can sometimes generate unrealistic predictions, especially for networks with complex structures or when the model is misspecified. ERGMs have other drawbacks as well, such as the potential for inaccurate or misleading results from the exponential function and sensitivity to assumptions and modeling choices. This can make it difficult to compare results across studies or replicate findings using different data sets or methods. Therefore, researchers should carefully consider the limitations and potential biases of ERGMs when using them to analyze network data.</p><h3>Implementation of ERGMs</h3><p>In this section, I will demonstrate some simple codes for the implementation of ERGMs in R. If you are interested in details, please read the original documents here: <a href="https://cran.r-project.org/web/packages/ergm/ergm.pdf">Package ‘ergm’</a>.</p><p>The ergm package in R provides a comprehensive set of functions for fitting and analyzing ERGMs. To fit an ERGM, the ergm() function can be used, which takes as input the data frame containing the network data and a formula specifying the model to be fit. For example, the following code fits an ERGM to a network stored in the network data frame with a model that includes an edge-covariate term and a term for the transitivity of the network.</p><pre>library(ergm)<br><br># Fit the model<br>model &lt;- ergm(formula = network ~ edges + transitivity, data = network)</pre><p>Here are some additional core parameters of the ergm() function:</p><ol><li><strong>nodecov</strong>: This term adds a single network statistic for each quantitative attribute or matrix column to the model equaling the sum of attr(i) and attr(j) for all edges (i, j) in the network. (Numeric)</li><li><strong>nodefactor</strong>: This term adds multiple network statistics to the model, one for each of (a subset of) the unique values of the attr attribute (or each combination of the attributes given). (Categorical)</li><li><strong>control</strong>: This is the user interface for fine-tuning ’ergm’ fitting. For example, inside control, you can set the max iterations, seed, and the number of proposals between sampled statistics.</li></ol><p>With the additional parameters, the codes can be rewritten as the following:</p><pre>library(ergm)<br><br># Fit the model<br>model &lt;- ergm(g1 ~ edges+nodecov(~Numeric_Variable1)<br>                 + nodecov(~Numeric_Variable2)<br>                 + nodefactor(&quot;Categorical_Variable1&quot;, <br>                               levels=c(&quot;a&quot;, &quot;b&quot;, &quot;c&quot;))<br>                 + nodefactor(&quot;Categorical_Variable2&quot;, <br>                               levels=c(&quot;a&quot;,&quot;b&quot;)),  <br>                 control=control.ergm(MCMC.interval = 10000,<br>                                      MCMLE.maxit = 100,<br>                                      seed = 42))</pre><p>The summary() function can be used to obtain summary statistics and diagnostic plots for the fitted model. The function takes as input the fitted model and produces a summary report with information on the model fit, the model coefficients, and the goodness-of-fit measures. For example, the following code generates a summary report for the fitted model:</p><pre># Generate a summary report<br>summary(model)</pre><p>If you are interested in learning how to manipulate the networks in R, please read here:</p><p><a href="https://towardsdatascience.com/network-analysis-in-r-manipulating-network-data-ee388fba7215">Network Analysis with R | Manipulating Network Data</a></p><h3>Conclusion</h3><p>In this study, a graph analytics framework and an ERGM-based explanatory graph model are developed to examine disease multimorbidity recorded in hospital discharge records in Arizona before and during the COVID-19 pandemic. According to our analysis, we observed that while multimorbidity increased by 34.26% and 41.04% for mental disorders and respiratory disorders, respectively, during the peak of the pandemic, the gradients for endocrine diseases and circulatory disorders were not significant. We also found that multimorbidity for acute conditions decreased during the pandemic, while multimorbidity for chronic conditions remained unchanged.</p><p>The framework and model can be used on any standardized Electronic Health Records database containing ordered or unordered lists of diagnosis codes. The study informs future research incorporating multimorbidity into problem scenarios such as disease risk prediction and feature engineering for modeling health outcomes. The explanatory graph model can also be used to complement deep learning and data mining modeling approaches. The study has implications for health analytics researchers and policymakers, providing tools for analyzing disease multimorbidity patterns and guidelines for pre-emptive actions for averting public health crises.</p><h4>To cite the work:</h4><p><strong><em>Srinivasan, Karthik, and Jinhang Jiang. “Examining Disease Multimorbidity in US Hospital Visits Before and During COVID-19 Pandemic: A Graph Analytics Approach.” ACM Transactions on Management Information Systems (2022). </em></strong><a href="https://doi.org/10.1145/3564274"><strong>https://doi.org/10.1145/3564274</strong></a></p><h3><strong>Related Readings</strong></h3><ul><li><a href="https://towardsdatascience.com/network-analysis-in-r-manipulating-network-data-ee388fba7215">Network Analysis with R | Manipulating Network Data</a></li><li><a href="https://medium.com/mlearning-ai/implementation-of-latent-network-models-visualize-network-data-in-r-b25c42f6d95a">Implement Latent Network Models &amp; Visualize Network Data in R</a></li><li><a href="https://towardsdatascience.com/visualize-high-dimensional-network-data-with-3d-360-degree-animated-scatter-plot-d583932d3693">Visualize High-Dimensional Network Data with 3D 360-Degree-Animated Scatter Plot</a></li><li><a href="https://towardsdatascience.com/networkx-code-demo-for-manipulating-subgraphs-e45320581d13">NetworkX: Code Demo for Manipulating Subgraphs</a></li></ul><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=5225faacfe0e" width="1" height="1" alt=""><hr><p><a href="https://medium.com/data-science/graph-analytics-in-identifying-enduring-effects-of-pandemics-5225faacfe0e">Graph Analytics in Identifying Enduring Effects of Pandemics</a> was originally published in <a href="https://medium.com/data-science">TDS Archive</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[All Kinds of Randoms in Python]]></title>
            <link>https://towardsdev.com/all-kinds-of-randoms-in-python-9e382b23fb7b?source=rss-1c41ed254f6d------2</link>
            <guid isPermaLink="false">https://medium.com/p/9e382b23fb7b</guid>
            <category><![CDATA[data-preprocessing]]></category>
            <category><![CDATA[python]]></category>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[statistics]]></category>
            <category><![CDATA[python-programming]]></category>
            <dc:creator><![CDATA[Jinhang Jiang]]></dc:creator>
            <pubDate>Sun, 04 Dec 2022 16:46:56 GMT</pubDate>
            <atom:updated>2022-12-07T12:27:13.712Z</atom:updated>
            <content:encoded><![CDATA[<h4>An all-you-need list of examples for generating random samples in python</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*GjkY_mIzCiz6oB3mAQkliA.png" /><figcaption>image by author</figcaption></figure><p>In this blog, I will demonstrate how to generate sample random numbers in python according to different needs. Random and Numpy are used.</p><pre>import random<br>import numpy as np</pre><h3>Content</h3><p>Scenario 1: Get random integers</p><p>Scenario 2: Get random numbers with decimals</p><p>Scenario 3: Get randoms with known statistics/distributions</p><p>Scenario 4: Play with an existing list of numbers</p><h3>Scenario 1: Get random integers</h3><p>If you just need a random integer between 0 to 10:</p><pre>random.randint(0, 10)<br><br># 7</pre><p>If you need a list (e.g., 15) of random integers between 0 to 10:</p><pre>np.random.randint(0, 10, 15)<br><br># array([4, 4, 3, 0, 0, 1, 6, 9, 5, 3, 1, 6, 8, 5, 9])</pre><p>If you want to get a very random and super-large integer, here’s what you can do:</p><pre>import time<br>int(time.time())<br><br># 1669855560</pre><h3>Scenario 2: Get random numbers with decimals</h3><p>Get a random number between 0 to 1:</p><pre>random.random()<br><br># 0.4353251333114442</pre><p>Get a random number between 0 to 10:</p><pre>random.uniform(0, 10)<br><br># 9.210714292877288</pre><p>If you need a list (e.g., 5) of random numbers between 0 to 10:</p><pre>np.random.uniform(0, 10, 5)<br><br># array([8.70816807, 5.13392407, 0.4998664 , 5.97627955, 2.15261284])</pre><h3>Scenario 3: Get randoms with known statistics/distributions</h3><p>If you already know the mean, mode, and max of your desired sample, you probably want to try the <strong><em>triangular </em></strong>function. It has a cool parameter called <strong><em>mode: t</em></strong>he value where the peak of the distribution occurs. The value must fulfill the condition whereleft &lt;= mode &lt;= right. If we set it to 5, the samples will more likely appear around 5.</p><pre>np.random.triangular(0, 5, 10, 5) # left, mode, right, sample<br><br># [3.7525956  5.16902974 3.79206193 8.46765312 6.23403873]</pre><p>If you want to draw samples from a binomial distribution, you can try a function called <strong><em>binomial</em></strong>. If we set n = 20, p = 0.25, and the size to 2000:</p><pre>sample = np.random.binomial(20, 0.25, 2000)<br>import matplotlib.pyplot as plt<br>plt.hist(sample)</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/375/1*GBEbHhJExEfhYbpomjzb-Q.png" /><figcaption>image by author</figcaption></figure><p>When you want to generate a random number from a given mean (50) and standard deviation (3):</p><pre>random.normalvariate(50,3)<br><br># 49.25030663269399</pre><p>When you want to create a list (e.g., 5) of random numbers, given a known mean (50) and a standard deviation (3):</p><pre>[random.normalvariate(50,3) for _ in range(5)]<br><br># [49.873333367806644, 45.72729827453879, 47.69029669578644, <br># 48.25661520735196, 47.26944802688148]<br><br>np.random.normal(50, 3, 5)<br><br># array([47.18605431, 48.9894249 , 49.50258396, 49.85496526, 48.1503549 ])</pre><p>When you want to generate a list (e.g., 10) of random numbers that sum up to a certain value (1), then you will need the function called <strong><em>dirichlet</em>.</strong></p><pre>sample = np.random.dirichlet(np.ones(10), size=1)<br>print(sample)<br>print(np.sum(sample))<br><br># [[0.23988202 0.08378129 0.19439318 0.04282444 0.01568082 0.04033996<br>#   0.07864523 0.10856321 0.18118435 0.0147055 ]]<br><br># 1.0000000000000002</pre><h3>Scenario 4: Play with an existing list of numbers</h3><p>Suppose we have a list of numbers that looks like the following:</p><pre>sample = np.random.randint(0, 10, 20)<br>sample<br><br># array([0, 3, 1, 1, 0, 2, 5, 1, 0, 7, 3, 6, 6, 9, 3, 7, 0, 5, 6, 6])</pre><p>If you want to randomly shuffle the sample:</p><pre>np.random.shuffle(sample)<br>sample<br><br># array([1, 3, 7, 5, 3, 0, 6, 0, 0, 1, 3, 2, 9, 6, 6, 7, 6, 1, 5, 0])</pre><p>If you want to randomly get 5 elements out of the sample:</p><pre>for _ in range(5):<br>  print(np.random.choice(sample))<br><br># 1<br># 7<br># 6<br># 9<br># 0</pre><p>If you want to randomly permute a sequence or return a permuted range, here are a few examples of what you can do:</p><pre>np.random.permutation(10)<br><br># array([2, 5, 0, 3, 1, 7, 8, 4, 6, 9])<br><br>np.random.permutation(sample)<br><br># array([2, 0, 3, 9, 6, 0, 3, 6, 5, 5, 6, 0, 7, 6, 0, 1, 1, 1, 7, 3])<br><br>arr = np.arange(9).reshape((3, 3))<br>np.random.permutation(arr)<br><br># array([[3, 4, 5],<br>#        [6, 7, 8],<br>#        [0, 1, 2]])</pre><h3>Conclusion</h3><p>This short blog summarizes many popular techniques for generating random numbers in python. Knowing those tricks should help you generate dummy data according to your needs. I will also keep updating this blog as I encounter more scenarios.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=9e382b23fb7b" width="1" height="1" alt=""><hr><p><a href="https://towardsdev.com/all-kinds-of-randoms-in-python-9e382b23fb7b">All Kinds of Randoms in Python</a> was originally published in <a href="https://towardsdev.com/">Towards Dev</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[An End-to-End Guide to Publish Your Python Package]]></title>
            <link>https://medium.com/data-science/an-end-to-end-guide-to-publish-your-python-package-bdb56639662c?source=rss-1c41ed254f6d------2</link>
            <guid isPermaLink="false">https://medium.com/p/bdb56639662c</guid>
            <category><![CDATA[codesharing]]></category>
            <category><![CDATA[mlops]]></category>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[python]]></category>
            <category><![CDATA[engineering]]></category>
            <dc:creator><![CDATA[Jinhang Jiang]]></dc:creator>
            <pubDate>Tue, 14 Jun 2022 18:34:22 GMT</pubDate>
            <atom:updated>2022-07-21T02:02:27.566Z</atom:updated>
            <content:encoded><![CDATA[<h3>A GitHub Repository for Publishing Your Python Package</h3><h4>This is an end-to-end demo aiming to facilitate the process of sharing open-source software or internal work</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*pRW8vDVeVurwIk-bT2sTOw.jpeg" /><figcaption>Photo by <a href="https://unsplash.com/@sigmund?utm_source=unsplash&amp;utm_medium=referral&amp;utm_content=creditCopyText">Sigmund</a> on <a href="https://unsplash.com/s/photos/upload?utm_source=unsplash&amp;utm_medium=referral&amp;utm_content=creditCopyText">Unsplash</a></figcaption></figure><p>This is an end-to-end demo of sharing your code. The project aims to help you understand how to properly publish your python modules/packages on PyPI externally and share your work with your colleagues internally. I have also prepared a GitHub Repo for you to make your life easier.</p><p>You may go directly to this repo for more details: <a href="https://github.com/jinhangjiang/Your-First-Python-Package-on-PyPI">GitHub Repo</a></p><h3>Step 1</h3><p>Go to the following two websites to register your own account, respectively.</p><ul><li>PyPI test: <a href="https://test.pypi.org/">https://test.pypi.org/</a></li><li>PyPI: <a href="https://pypi.org/">https://pypi.org/</a></li></ul><p>Note: You may skip this part if you do not want to publish your package publicly. Also, I highly recommend trying your package on the test site first to avoid mistakes in the uploading process. Since any change, you make to your package on PyPI is not revertable, uploading errors may lead to a malfunctioning patch for your package. You want to avoid that!!</p><h3>Step 2</h3><p>Option 1: Fork <a href="https://github.com/jinhangjiang/Your-First-Python-Package-on-PyPI">this repository</a> to your own GitHub account and make it available in your local. You can make the most changes on GitHub, but you will need to publish your package via cmd with those files available in local.</p><p>Option 2: Create the necessary files manually. And here is the list of the core files you will need:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*U7Yv-bbjveHeYp0QgPfPdw.png" /><figcaption>Image by author</figcaption></figure><p>I know that’s a lot. But bear with me. You only need to make necessary changes to some of them, and the rest will stay as default.</p><h3>OPTIONAL — Multiple Modules</h3><p>Suppose you have multiple classes with functions created in separate files. You want to make the folders (or subfolders) following this convention:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/645/1*gps5AnG1v-a8OTwUt7fNBg.png" /><figcaption>Image by author</figcaption></figure><p>Inside each folder, update the “__init__.py” by doing this:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/042bbab546b003ba57449d85d4d32d23/href">https://medium.com/media/042bbab546b003ba57449d85d4d32d23/href</a></iframe><p>In the main folder, update the “__init__.py” by doing this:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/ebeb9fa5c70c5053578596777da6999c/href">https://medium.com/media/ebeb9fa5c70c5053578596777da6999c/href</a></iframe><p>Then the users will be able to import your library properly like this:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/bc8db2a81386226fb42c899817da35ff/href">https://medium.com/media/bc8db2a81386226fb42c899817da35ff/href</a></iframe><p>If You want to import something from the main (previous) folder, here is what you should do:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/1d9043a1a0758bd7d5e02e91391989df/href">https://medium.com/media/1d9043a1a0758bd7d5e02e91391989df/href</a></iframe><p>Consider adding a list in your “__init__.py” for each sub-lib, so that the users can check what functions are available:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/b11c2bdc81e5d1dba0665b94a2f3a95f/href">https://medium.com/media/b11c2bdc81e5d1dba0665b94a2f3a95f/href</a></iframe><h3>Step 3</h3><p>Install the following python package in your cmd:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/f5f4cb901552b621aa3ded548ed0648d/href">https://medium.com/media/f5f4cb901552b621aa3ded548ed0648d/href</a></iframe><h3>Step 4</h3><p>Do the following changes in ANY order you want:</p><ol><li>Replace your_main_code.py in the src folder with your own python package and leave “__init__.py” as it is if you are only publishing a module.</li><li>Make changes to setup.py, and instructions are included in that file.</li><li>Pick your own license. Open the LICENSE file, click on Edit, click “Choose a license template”, and select the license that fulfills your needs. If you have no idea which license works for you, you can use the MIT license, which is one of the most common choices. Or, you can use this link to pick one: <a href="https://choosealicense.com/">https://choosealicense.com/</a></li><li>Update CHANGELOG.md to reflect version information</li><li>Optional: create a “test.py” and put the file in the tests folder. Or you can remove the whole folder if you are confident that everything works great in your module.</li><li>Delete everything in this “README.md” file, and update the file with the long description of your package.</li></ol><h3>Step 5</h3><p>You have multiple choices for step 5 to perform the rest of the steps. Here are two examples:</p><ol><li>Do it in cmd - Command Prompt</li></ol><ul><li>In your local, open the cmd, navigate to the directory where your package is and type the following:</li></ul><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/4f2c24d1a97cc9c79a089795106e1370/href">https://medium.com/media/4f2c24d1a97cc9c79a089795106e1370/href</a></iframe><p>2. Do it in the Jupyter Notebook terminal:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/7c844c60d5aee3170cb68395f523e12f/href">https://medium.com/media/7c844c60d5aee3170cb68395f523e12f/href</a></iframe><h3>Step 6</h3><p>In this step, we will use the following code in cmd/terminal to build your package:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/f8586ed80fb0929681be87cba878e696/href">https://medium.com/media/f8586ed80fb0929681be87cba878e696/href</a></iframe><p>Once you run the code, you will see the following two folders in the current directory:</p><ul><li>build</li><li>dist</li></ul><p>Under the dist folder, you will see a ‘tar’ file called “TheNameofYourPackage-TheVersionofYourPackage.tar.gz”. Up to this point, if you do not need to publish your code in public; instead, if you just want to share your code with your friends or colleagues, you may just share this file with them. All they need to do is to do “pip install” to use your code:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/c190d07dd2eedfb876420ac3a326ec8e/href">https://medium.com/media/c190d07dd2eedfb876420ac3a326ec8e/href</a></iframe><h3>OPTIONAL — Test Your Package on PyPI</h3><p>Now, you are about to publish your package to PyPI. Before you make it public, one more thing you may want to do is to test if your package will work as expected once people download them. What you can do is to create a folder called “test”, and create a test.py, which includes some sample implementations of your package. Then, <strong>type “pytest” in your cmd/terminal. If everything works fine, it will run your test.py automatically and pass.</strong> Otherwise, it will raise errors, and you should fix the bugs accordingly before moving to the next step.</p><p>And here is one more thing you might want to try to test if the architecture of your package is good to go. In the cmd/terminal, type the following code:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/93a61629ace4b67ff274bdf7f80b1db7/href">https://medium.com/media/93a61629ace4b67ff274bdf7f80b1db7/href</a></iframe><p>You should see something like this:</p><pre>Checking distribution dist/TheNameofYourPackage-TheVersionofYourPackage-1.0.0-py3-none-any.whl: Passed</pre><pre>Checking distribution dist/TheNameofYourPackage-TheVersionofYourPackage.tar.gz: Passed</pre><h3>Step 7</h3><p>Upload your package to TestPyPI:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/4b0c5f1f9aa11f1828c1d39be3a3cdeb/href">https://medium.com/media/4b0c5f1f9aa11f1828c1d39be3a3cdeb/href</a></iframe><p>Then you will see a link leading to the testing version of your package on TestPyPI. Check it out! If there are any typos or incompatible bugs, fix them before uploading your package to the real PyPI.</p><p>And, now, it is the most exciting moment, upload your package to PyPI to help hundreds of thousands of people in our community:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/f3a4bfc2b95dc95a486487601d0c2b5f/href">https://medium.com/media/f3a4bfc2b95dc95a486487601d0c2b5f/href</a></iframe><p>By this point, your package should be officially online and can be “pip install” by anyone at any time from anywhere. Big moment! I still remember how I felt at the moment when I saw my first package is out there. I told myself, that’s why I code!</p><p><strong><em>CONGRATULATIONS!!!</em></strong></p><h3>A few tips</h3><ul><li>Whenever you want to update your package, you should remove the ‘build’ and ‘dist’ folders, make changes to your code, edit the “CHANGLOG.txt” file, and revise the version number in the “setup.py”. And repeat steps 5–7.</li><li>You may upgrade your package after the updates by doing this: <em>pip install YOURPACKAGENAME — upgrade</em></li><li>You can always find your package on PyPi here: <a href="http://pypi.org/project/YOURPACKAGENAME/"><em>http://pypi.org/project/YOURPACKAGENAME/</em></a></li><li>Do not publish packages arbitrarily. Even though there are no hard restrictions on what you can or cannot publish, make sure you are uploading something that is actually meaningful and someone will benefit from your work.</li></ul><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=bdb56639662c" width="1" height="1" alt=""><hr><p><a href="https://medium.com/data-science/an-end-to-end-guide-to-publish-your-python-package-bdb56639662c">An End-to-End Guide to Publish Your Python Package</a> was originally published in <a href="https://medium.com/data-science">TDS Archive</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Linear Regression with Hugging Face]]></title>
            <link>https://medium.com/data-science/linear-regression-with-hugging-face-3883fe729324?source=rss-1c41ed254f6d------2</link>
            <guid isPermaLink="false">https://medium.com/p/3883fe729324</guid>
            <category><![CDATA[nlp]]></category>
            <category><![CDATA[python]]></category>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[transformers]]></category>
            <category><![CDATA[linear-regression]]></category>
            <dc:creator><![CDATA[Jinhang Jiang]]></dc:creator>
            <pubDate>Tue, 07 Jun 2022 19:21:33 GMT</pubDate>
            <atom:updated>2022-06-07T19:21:33.507Z</atom:updated>
            <content:encoded><![CDATA[<h4>This is an exploration of the feasibility of implementing the linear regression analysis with Transformers</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*LO3fhCZB0KHy7EUJYWRitQ.jpeg" /><figcaption>Photo by Scott Webb: <a href="https://www.pexels.com/photo/shapes-and-pattern-2346594/">https://www.pexels.com/photo/shapes-and-pattern-2346594/</a></figcaption></figure><h3>Introduction</h3><p>As we all know, a large amount of effort has been spent on text classification and text clustering problems using the Hugging Face transformers by researchers. Have you wondered if you can leverage the pre-trained transformers in linear regression projects? The Hugging Face API actually supports it! I will walk you through two examples and share some insights at the end.</p><h3>Data</h3><p>First, I found a dataset containing hundreds of corrected essays with scores. The essays were written according to a monthly selection and to train Brazilian high-school students for university entrance exams. The dataset has 695 essays, and I omitted the rows with missing values or the records where total_score = 0. After the preprocessing, 100 essays were removed.</p><ul><li>Here is the link to the data: <a href="https://www.kaggle.com/datasets/danlessa/corrected-essays-with-scores-from-uol">https://www.kaggle.com/datasets/danlessa/corrected-essays-with-scores-from-uol</a>.</li><li>Here is the license of the data: <a href="https://opendatacommons.org/licenses/odbl/1-0/">https://opendatacommons.org/licenses/odbl/1-0/</a>.</li><li>The task is to use their essays (long documents) to predict the total scores.</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*ad17NAEaVGN2RL6iBFuafg.png" /><figcaption>Brazilian Essay Data</figcaption></figure><p>The second dataset is a Women’s Clothing E-Commerce dataset revolving around the reviews written by customers. Its nine supportive features offer a great environment to parse the text through its multiple dimensions. The original dataset has 23,486 customer reviews and no missing values.</p><ul><li>Here is the link to the data: <a href="https://www.kaggle.com/datasets/nicapotato/womens-ecommerce-clothing-reviews">https://www.kaggle.com/datasets/nicapotato/womens-ecommerce-clothing-reviews</a>.</li><li>Here is the license of the data: <a href="https://creativecommons.org/publicdomain/zero/1.0/">https://creativecommons.org/publicdomain/zero/1.0/</a>.</li><li>The task is to use their review comments (short documents) to predict the rating. It sounds more like a multi-class classification problem to me. But it is still acceptable to fit a linear regression model with this data since our goal is only to explore the usage of the Hugging Face API.</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*ATJgh8K5aurYuHyC-SMPTA.png" /><figcaption>Customer Review Data</figcaption></figure><h3>Code</h3><p>FYI, the whole project is conducted in Google Colab. Before we dive into the experiment, let’s check if GPU is available since the entire process will be extremely slow without GPU.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/6d37aa4328ddd6fb5416f667aebf96a5/href">https://medium.com/media/6d37aa4328ddd6fb5416f667aebf96a5/href</a></iframe><figure><img alt="" src="https://cdn-images-1.medium.com/max/835/1*it83j2WvCndC9MLuDrnSrQ.png" /><figcaption>Image by author</figcaption></figure><p>Then, let’s import all the libraries we need:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/7ada5a4aa93a1ea764c44075317f4e97/href">https://medium.com/media/7ada5a4aa93a1ea764c44075317f4e97/href</a></iframe><h4>Step 1: Prepare the Data</h4><p>In this step, we need to split data, encode the text, and convert the data into torch data for the downstream modeling. This is the same process that I would use to prepare the data for the classification problems.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/4598f97423225f2c7bd2911438b44167/href">https://medium.com/media/4598f97423225f2c7bd2911438b44167/href</a></iframe><p><strong><em>Note: Since we are training a linear regression model, in the __getitem__ function, we need to convert the </em></strong><strong><em>label to float. Please refer to line 26.</em></strong></p><h4>Step 2: Load the model</h4><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/465a0d3d94917d0753de958a878bd15f/href">https://medium.com/media/465a0d3d94917d0753de958a878bd15f/href</a></iframe><p>Here is the most important step. According to Hugging Face’s library, when we load the pre-trained models from the Hugging Face API, setting the num_labels to 1 for the AutoModelForSequenceClassification will trigger the linear regression and use MSELoss() as the loss function automatically. You can find the original script <a href="https://github.com/huggingface/transformers/blob/7ae6f070044b0171a71f3269613bf02fd9fca6f2/src/transformers/models/bert/modeling_bert.py#L1564-L1575">HERE</a>.</p><h4>Step 3: Create Compute Metrics for Regression</h4><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/a84bb014c2a840fa437812863f9d592b/href">https://medium.com/media/a84bb014c2a840fa437812863f9d592b/href</a></iframe><p>The logits is actually our prediction. However, it looks very weird in our context. To read more about logits, here is a good study resource from Columbia University: <a href="http://www.columbia.edu/~so33/SusDev/Lecture_9.pdf">Logit/Probit</a>. Even though the default loss function is MSE, I add the loss function here again to verify the loss function is properly implemented.</p><h4>Step 4: Build the Trainer</h4><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/c8286517b0c16ceaaa3911617ce63c8b/href">https://medium.com/media/c8286517b0c16ceaaa3911617ce63c8b/href</a></iframe><h4>Brazilian Essay Score Prediction</h4><p>We need to pick a proper pre-trained model from Hugging Face for each project. Since the Brazilian essays were all written in Portuguese, I picked the “neuralmind/bert-large-portuguese-cased” as the base model.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*NszdxcHQONSA4rTnZ_JQLQ.png" /><figcaption>Image by author</figcaption></figure><p>Since the essays are long documents, the average length is longer than the default max length of the most pre-trained models; I just set the max_length to 512 to capture the most information. The test_size is set to 20%.</p><p>After several experiments, I found that the model has a hard time capturing any linear relationship between the score and the essays, given that the R2 score equals negative numbers. Thus, I decided to take a log10 transformation of the target variable. Since the model had a hard time converging, I ran the Trainer for 100 epochs, and the best model was found at epoch 44.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/362bd81d81a14415afbb90f9e3a77ee3/href">https://medium.com/media/362bd81d81a14415afbb90f9e3a77ee3/href</a></iframe><p>Even though the best R2 is only 0.4643 (which means only roughly 46% of the log10 of total scores can be explained by the essays embeddings.), we can see a clear trend of learning (R2 = -2.4783 means the semantic embeddings of the essays cannot explain the log10 of the total score at all.).</p><h4>Customer Review Project</h4><p>Since everything was written in English for this project, I simply chose ‘bert-base-uncased’ as the base model. I subset the first 3000 reviews as the test set. Empirically speaking, subsetting a larger percent of the data as validation data can increase the performance of the fine-tuned model on the test data. Thus, I set the test_size to 66% for this task. Because the customer reviews are short text data, I set the max_length to 126 for this project.</p><p>As I mentioned at the beginning, this problem sounds more like a multi-class classification problem than a linear regression problem, I added an accuracy metric to replace the SMAPE. And here is how to calculate.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/669af81c568d8a3a0a3bc55430ec8e92/href">https://medium.com/media/669af81c568d8a3a0a3bc55430ec8e92/href</a></iframe><p>I trained the model for five epochs. The best accuracy (0.6694) was attained at epoch three, while the best R2 (0.6422) and RMSE (0.6589) were attained at epoch 2.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/becf02e4f3beda1f7c9986c0c9f2a544/href">https://medium.com/media/becf02e4f3beda1f7c9986c0c9f2a544/href</a></iframe><p>And here we go, the last step: let’s test the fine-tuned model on the test dataset.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/db0cc45b4c11f439e38e83cd00dccb7f/href">https://medium.com/media/db0cc45b4c11f439e38e83cd00dccb7f/href</a></iframe><p>The output is shown below. I would say the results are decent compared to the model’s performance on the validation dataset.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/86b9eab90fc10356af595b1085e2882f/href">https://medium.com/media/86b9eab90fc10356af595b1085e2882f/href</a></iframe><p>For the full code script: <a href="https://github.com/jinhangjiang/Medium_Demo/blob/main/Transformers_Linear_Regression/Transformers_Linear_Regression.ipynb">Transformers_Linear_Regression.ipynb</a></p><h3>Take-aways</h3><ol><li>The Hugging Face API supports linear regression via the ForSequenceClassification interface by setting the num_labels = 1. The problem_type will automatically be set to ‘regression’.</li><li>Since the linear regression is achieved through the classification function, the prediction is kind of confusing. According to this <a href="https://github.com/huggingface/transformers/blob/7ae6f070044b0171a71f3269613bf02fd9fca6f2/src/transformers/pipelines/text_classification.py#L140-L162">document</a> from Hugging Face (line 140 to line 162), they will apply the sigmoid function over the output when the problem_type == “multi_label_classification” or num_labels == 1. My best explanation at this moment is when the problem_type == ‘regression’ , the postprocessing function is triggered to set function_to_apply = ClassificationFunction.NONE, so no additional function is applied over the output.</li><li>The model has a difficult time learning the target when the continuous values are big (referring to what I found in the first example, while I fed the target values like 700, 1000, etc. the model gave me predictions like 15.6, 16.8, …). A log transformation may help in this case.</li></ol><h3>Related Reading</h3><ul><li><a href="https://towardsdatascience.com/sbert-vs-data2vec-on-text-classification-e3c35b19c949">SBERT vs. Data2vec on Text Classification</a></li><li><a href="https://towardsdatascience.com/morethansentiments-a-python-library-for-text-quantification-e57ff9d51cd5">MoreThanSentiments: A Python Library for Text Quantification</a></li><li><a href="https://towardsdatascience.com/customer-churn-accuracy-a-4-6-increase-with-feature-engineering-29bcb1b1ee8f">Customer Churn Accuracy: Increased 4.6% With TextBlob &amp; SBERT</a></li></ul><h3>Reference</h3><p>Hugging Face. (2021). modeling_bert.py. <a href="https://github.com/huggingface/transformers/blob/7ae6f070044b0171a71f3269613bf02fd9fca6f2/src/transformers/models/bert/modeling_bert.py#L1564-L1575">https://github.com/huggingface/transformers/blob/7ae6f070044b0171a71f3269613bf02fd9fca6f2/src/transformers/models/bert/modeling_bert.py#L1564-L1575</a></p><p>Hugging Face. (2021). text_classification.py. <a href="https://github.com/huggingface/transformers/blob/7ae6f070044b0171a71f3269613bf02fd9fca6f2/src/transformers/pipelines/text_classification.py#L140-L162">https://github.com/huggingface/transformers/blob/7ae6f070044b0171a71f3269613bf02fd9fca6f2/src/transformers/pipelines/text_classification.py#L140-L162</a></p><p>La Javaness R&amp;D. (2022). Regression with Text Input Using BERT and Transformers. <a href="https://lajavaness.medium.com/regression-with-text-input-using-bert-and-transformers-71c155034b13">https://lajavaness.medium.com/regression-with-text-input-using-bert-and-transformers-71c155034b13</a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=3883fe729324" width="1" height="1" alt=""><hr><p><a href="https://medium.com/data-science/linear-regression-with-hugging-face-3883fe729324">Linear Regression with Hugging Face</a> was originally published in <a href="https://medium.com/data-science">TDS Archive</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[MoreThanSentiments: Text Analysis Package]]></title>
            <link>https://medium.com/data-science/morethansentiments-a-python-library-for-text-quantification-e57ff9d51cd5?source=rss-1c41ed254f6d------2</link>
            <guid isPermaLink="false">https://medium.com/p/e57ff9d51cd5</guid>
            <category><![CDATA[software-development]]></category>
            <category><![CDATA[nlp]]></category>
            <category><![CDATA[text-mining]]></category>
            <category><![CDATA[open-source]]></category>
            <category><![CDATA[python]]></category>
            <dc:creator><![CDATA[Jinhang Jiang]]></dc:creator>
            <pubDate>Tue, 31 May 2022 14:47:29 GMT</pubDate>
            <atom:updated>2022-12-24T03:24:35.774Z</atom:updated>
            <content:encoded><![CDATA[<h4>A collection of functions that help researchers calculate Boilerplate, Redundancy, Specificity, Relative Prevalence, etc., in python</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*dRfXf70--bxEdfFVbwG5Xw.jpeg" /><figcaption>Photo by <a href="https://unsplash.com/@impatrickt?utm_source=unsplash&amp;utm_medium=referral&amp;utm_content=creditCopyText">Patrick Tomasso</a> on <a href="https://unsplash.com/s/photos/text?utm_source=unsplash&amp;utm_medium=referral&amp;utm_content=creditCopyText">Unsplash</a></figcaption></figure><h3>Introduction</h3><p><a href="https://www.sciencedirect.com/science/article/pii/S2665963822001403">MoreThanSentiments (Jiang and Srinivasan, 2022)</a> is a python library written to help researchers calculate Boilerplate (Lang and Stice-Lawrence, 2015), Redundancy (Cazier and Pfeiffer, 2017), Specificity (Hope et al., 2016), Relative Prevalence (Blankespoor, 2019), etc. Nowadays, people frequently talk about text embedding, semantic similarity, intention detection, and sentiments analysis… However, MoreThanSentiments is inspired by the idea that properly quantifying the text structure will also help researchers extract tons of meaningful information. And this domain-independent package is easy to be implemented in various projects for text quantification tasks.</p><h3>Supported Measurements</h3><h4>Boilerplate</h4><p>In textual analysis, a <em>Boilerplate</em> is a combination of words that can be removed from a sentence without significantly changing the original meaning. In other words, it is a measure of informativeness. It is calculated based on the portion of sentences containing boilerplates against the total number of words.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/531/0*jgD9qDGmEuyLUmN7.png" /><figcaption>Image by author</figcaption></figure><h4>Redundancy</h4><p>Redundancy is a measure of the usefulness of the text. It is defined as the percentage of super-long sentences/phrases (e.g., 10-grams) that occur more than once in each document. Intuitively, if a super-long sentence/phrase is used repetitively, it means the author tries to impose the duplicated information that has been mentioned previously.</p><h4>Specificity</h4><p>Specificity is a measure of the quality of relating uniquely to a particular subject. It is defined as the number of specific entity names, quantitative values, and times/dates all scaled by the total number of words in a document. Currently, the function of Specificity is built on the Named Entity Recognizer from spaCy.</p><h4>Relative Prevalence</h4><p>Relative Prevalence is a measure of hard information. It is the number of numerical values against the length of the whole text. It helps evaluate the portion of quantitative information in a given text.</p><h3>Installation</h3><p>The easiest way to install the toolbox is via pip (pip3 in some distributions):</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/eab6cfd7743eaf133219506fe4fe05a7/href">https://medium.com/media/eab6cfd7743eaf133219506fe4fe05a7/href</a></iframe><h3>Usage</h3><h4>Import the Package</h4><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/38fc08a623f0f20bdeab062ee6389031/href">https://medium.com/media/38fc08a623f0f20bdeab062ee6389031/href</a></iframe><h4>Read data from txt files</h4><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/e7086bf1f3ab2ae830c118d8a340d7ca/href">https://medium.com/media/e7086bf1f3ab2ae830c118d8a340d7ca/href</a></iframe><p>This is a built-in function that helps you read a folder of separated .txt files in python. If you already have all the data stored in a .csv file, you may just read the file with pandas like normal.</p><h4>Sentence Token</h4><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/e0ad8d8a2b22b8ce608b1dd95dcde163/href">https://medium.com/media/e0ad8d8a2b22b8ce608b1dd95dcde163/href</a></iframe><p>If you want to calculate the Boilerplate and Redundancy, it is necessary to tokenize the sentences as the n-grams are generated on the sentence level.</p><h4>Clean Data</h4><p>If you want to clean on the sentence level:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/3486d463926e21c3611af660f5ee2f43/href">https://medium.com/media/3486d463926e21c3611af660f5ee2f43/href</a></iframe><p>If you want to clean on the document level:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/680f8000578122aa91f1d2457428225b/href">https://medium.com/media/680f8000578122aa91f1d2457428225b/href</a></iframe><p>For the data cleaning function, we offer the following options:</p><ul><li>lower: make all the words lowercase</li><li>punctuations: remove all the punctuations in the corpus</li><li>number: remove all the digits in the corpus</li><li>unicode: remove all the Unicode in the corpus</li><li>stop_words: remove the stopwords in the corpus</li></ul><h4>Boilerplate</h4><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/b7ec563857528968a296a09894f99d42/href">https://medium.com/media/b7ec563857528968a296a09894f99d42/href</a></iframe><p>Parameters:</p><ul><li>input_data: this function requires tokenized documents.</li><li>n: number of the ngrams to use. The default is 4.</li><li>min_doc: when building the ngram list, ignore the ngrams that have a document frequency strictly lower than the given threshold. The default is 5 documents. 30% of the number of documents is recommended. The min_doc can also read numbers that are between 0 and 1 as percentages. (e.g., 0.3 will be read as 30%)</li><li>get_ngram: if this parameter is set to “True,” it will return a dataframe with all the ngrams and the corresponding frequency, and the “min_doc” parameter will become ineffective.</li></ul><h4>Redundancy</h4><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/3c31020dcccfa145f58716528a39c1b1/href">https://medium.com/media/3c31020dcccfa145f58716528a39c1b1/href</a></iframe><p>Parameters:</p><ul><li>input_data: this function requires tokenized documents.</li><li>n: number of the n-grams to use. The default is 10.</li></ul><h4>Specificity</h4><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/58d7949a1ee6126c4ad8da2798690247/href">https://medium.com/media/58d7949a1ee6126c4ad8da2798690247/href</a></iframe><p>Parameters:</p><ul><li>input_data: this function requires the documents without tokenization</li></ul><h4>Relative Prevalence</h4><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/c25649f593404cb88bcbd59e3cddcd49/href">https://medium.com/media/c25649f593404cb88bcbd59e3cddcd49/href</a></iframe><p>Parameters:</p><ul><li>input_data: this function requires the documents without tokenization</li></ul><h3>Conclusion</h3><p>MoreThanSentiments is still a developing project. Yet it already shows the potential to help researchers in different domains. This package simplifies the process of quantifying the text structures and provides various text scores for their NLP projects.</p><p>Here are the links to the full examples:</p><ul><li><a href="https://github.com/jinhangjiang/morethansentiments/blob/main/tests/test_code.py">Python Script</a></li><li><a href="https://github.com/jinhangjiang/morethansentiments/blob/main/Boilerplate.ipynb">Python Jupyter Notebook</a></li></ul><h3>Citation</h3><blockquote><strong>If this package was helpful in your work, feel free to cite it as</strong></blockquote><blockquote>Jiang, J., Srinivasan, K. MoreThanSentiments: A text analysis package. Software Impacts, 100456 (2022). <a href="https://doi.org/10.1016/J.SIMPA.2022.100456">https://doi.org/10.1016/J.SIMPA.2022.100456</a></blockquote><h3>Related Reading</h3><p><a href="https://towardsdatascience.com/use-r-to-calculate-boilerplate-for-accounting-analysis-f4a5b64e9b0d">Use R to Calculate Boilerplate for Accounting Analysis</a></p><p><a href="https://towardsdatascience.com/use-r-to-calculate-boilerplate-for-accounting-analysis-f4a5b64e9b0d">Use R to Calculate Boilerplate for Accounting Analysis</a></p><h3>Reference</h3><p>BLANKESPOOR, E. (2019), The Impact of Information Processing Costs on Firm Disclosure Choice: Evidence from the XBRL Mandate. Journal of Accounting Research, 57: 919–967. <a href="https://doi.org/10.1111/1475-679X.12268">https://doi.org/10.1111/1475-679X.12268</a></p><p>Hope, OK., Hu, D. &amp; Lu, H. (2016), The benefits of specific risk-factor disclosures. <em>Rev Account Stud</em> 21<strong>, </strong>1005–1045. <a href="https://doi.org/10.1007/s11142-016-9371-1">https://doi.org/10.1007/s11142-016-9371-1</a></p><p>Richard A. Cazier, Ray J. Pfeiffer. (2017), 10-K Disclosure Repetition and Managerial Reporting Incentives. <em>Journal of Financial Reporting</em>; 2 (1): 107–131. <a href="https://doi.org/10.2308/jfir-51912">https://doi.org/10.2308/jfir-51912</a></p><p>Mark Lang, Lorien Stice-Lawrence. (2015), Textual analysis and international financial reporting: Large sample evidence, Journal of Accounting and Economics, Volume 60, Issues 2–3, Pages 110–135, ISSN 0165–4101. <a href="https://doi.org/10.1016/j.jacceco.2015.09.002">https://doi.org/10.1016/j.jacceco.2015.09.002</a>.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=e57ff9d51cd5" width="1" height="1" alt=""><hr><p><a href="https://medium.com/data-science/morethansentiments-a-python-library-for-text-quantification-e57ff9d51cd5">MoreThanSentiments: Text Analysis Package</a> was originally published in <a href="https://medium.com/data-science">TDS Archive</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[SBERT vs. Data2vec on Text Classification]]></title>
            <link>https://medium.com/data-science/sbert-vs-data2vec-on-text-classification-e3c35b19c949?source=rss-1c41ed254f6d------2</link>
            <guid isPermaLink="false">https://medium.com/p/e3c35b19c949</guid>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[nlp]]></category>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[code]]></category>
            <category><![CDATA[hugging-face]]></category>
            <dc:creator><![CDATA[Jinhang Jiang]]></dc:creator>
            <pubDate>Wed, 18 May 2022 19:38:01 GMT</pubDate>
            <atom:updated>2022-05-26T18:07:47.770Z</atom:updated>
            <content:encoded><![CDATA[<h4>Code demo for text classification with two of the popular pre-trained hugging-face models</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*NHW99vlTagmKDBfH5Ho8MA.png" /><figcaption>Image by author</figcaption></figure><h3>Introduction</h3><p>I personally do believe all the fancy ML research and advanced AI algorithm works have very minimal value if not zero until the date when they can be applied to real-life projects without asking the users for an insane amount of resources and excessive domain knowledge. And Hugging Face builds the bridge. Hugging Face is the home for thousands of pre-trained models which have made great contributions to democratizing artificial intelligence through open source and open science.</p><p>Today, I want to give you an end-to-end code demo to compare two of the most popular pre-trained models by conducting a multi-label text classification analysis.</p><p>The first model is <a href="https://www.sbert.net/">SentenceTransformers (SBERT)</a>. This actually is a collection of a bunch of pre-trained models for various tasks created by the team from the <a href="https://www.informatik.tu-darmstadt.de/ukp/ukp_home/index.en.jsp">Ubiquitous Knowledge Processing Lab at Technical University Darmstadt</a>. I have used the SBERT several times in my previous projects. They even have a python library that offers you the flexibility not to use the Hugging Face API and Pytorch framework. Check out <a href="https://www.sbert.net/docs/installation.html">here</a>.</p><p>The second model is <a href="https://ai.facebook.com/blog/the-first-high-performance-self-supervised-algorithm-that-works-for-speech-vision-and-text/">Data2vec</a>, a powerful pre-trained model offered by the AI team from Meta (Facebook). It is a self-supervised framework (Teacher model → student model) designed for text, audio, and image encoding. If you are interested in how it is developed, you may find the original article <a href="https://arxiv.org/abs/2202.03555">here</a>.</p><h3>Data</h3><p>For the data, I am using a well-known open-source text dataset: BBC News Group (Its license is here: <a href="https://opendatacommons.org/licenses/dbcl/1-0/">https://opendatacommons.org/licenses/dbcl/1-0/</a>). You can either load the data by doing the following:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/9ae5066fba08f60810f070a78d5b8926/href">https://medium.com/media/9ae5066fba08f60810f070a78d5b8926/href</a></iframe><p>Or, you can find a pre-processed CSV version from my GitHub repo: <a href="https://github.com/jinhangjiang/Medium_Demo/blob/main/Data2vec_vs_SBERT/bbc-text.csv">https://github.com/jinhangjiang/Medium_Demo/blob/main/Data2vec_vs_SBERT/bbc-text.csv</a></p><h3>Code Demo</h3><h4>Step 1: Install and import the package we need</h4><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/c9812543b7ff613e3cf28c1b16f77277/href">https://medium.com/media/c9812543b7ff613e3cf28c1b16f77277/href</a></iframe><h4>Step 2: Split the data for validation</h4><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/4a22b54034ea4bd89694648fdd01bdb9/href">https://medium.com/media/4a22b54034ea4bd89694648fdd01bdb9/href</a></iframe><p>Pay attention to one detail here: I am using a CSV file instead of importing the data from sklearn. So I gave the input data as <strong>a list (X.tolist())</strong>. Without doing it, the model will later throw errors.</p><h4>Step 3. Tokenize the text</h4><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/b0f2af4d99a0a0de12751e6f92ec93f5/href">https://medium.com/media/b0f2af4d99a0a0de12751e6f92ec93f5/href</a></iframe><p>Here are some clarifications:</p><p><strong><em>model_name</em></strong>: this parameter should be a string of the name of the pre-trained model that you want to use. You may find the available models:<a href="https://huggingface.co/models">https://huggingface.co/models</a></p><p><strong><em>max_length</em></strong>: this parameter will directly affect the training time and training speed. You would want to specify the length of the text you want the model to process for each document if each document is very long.</p><p><strong><em>padding</em></strong>: set this parameter to True if you have given a max_length. Pad to the longest sequence in the batch (or no padding if only a single sequence is provided)</p><h4>Step 4: Convert embeddings into a torch dataset</h4><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/a4f3a4d8ed4fe892333c669b90a26aa0/href">https://medium.com/media/a4f3a4d8ed4fe892333c669b90a26aa0/href</a></iframe><h4>Step 5: Call the model</h4><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/f90982eb90eef32dfc7abbfc34ec9fbb/href">https://medium.com/media/f90982eb90eef32dfc7abbfc34ec9fbb/href</a></iframe><p><strong><em>AutoModelForSequenceClassification</em></strong>: “AutoModel…” will help you automatically identify the correct model to use. So far, it works fine for me. “…ForSequenceClassification” is specifically used for classification problems.</p><p>to(“cuda”): if GPU is available on your machine, you may append this function at the end to leverage the GPU power. Without this function, the training time typically will significantly increase.</p><p>Step 6: Define evaluation metrics</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/fc31ebca44a302f1608e8746c30a3eab/href">https://medium.com/media/fc31ebca44a302f1608e8746c30a3eab/href</a></iframe><p><strong><em>metrics_name</em></strong>: This should be a string. For our demo, I picked “f1” as the evaluation metric. You can find the available options here: <a href="https://github.com/huggingface/datasets/tree/master/metrics">https://github.com/huggingface/datasets/tree/master/metrics</a></p><p><strong><em>average</em></strong>: I passed this parameter because I am using the f1 score to evaluate a multi-label classification problem. It is NOT a universal parameter.</p><h4>Step 7: Fine-tune the pre-trained model</h4><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/dc6e89043ddb34fa9ffb4bd6808ca4dc/href">https://medium.com/media/dc6e89043ddb34fa9ffb4bd6808ca4dc/href</a></iframe><p>Step 8: Save your fine-tuned model to make it reusable</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/a77807d7ba5b3fd189544aeb009534de/href">https://medium.com/media/a77807d7ba5b3fd189544aeb009534de/href</a></iframe><p>When you want to use your fine-tuned model, all you need to do is to replace the model name string in Step 3 and Step 5 with the path to your own model.</p><p>You have the access to the full code script here: <a href="https://github.com/jinhangjiang/Medium_Demo/blob/main/Data2vec_vs_SBERT/Data2vec_vs_SBERT_512.ipynb">https://github.com/jinhangjiang/Medium_Demo/blob/main/Data2vec_vs_SBERT/Data2vec_vs_SBERT_512.ipynb</a></p><h3>Results</h3><h4>For SBERT (max_length = 512, epoch = 5):</h4><p>Best f1 score: 0.985034</p><p>Average f1 score: 0.959524</p><p>Wall time: 15 min 36 s</p><p>Memory increased: 5.0294 GB</p><h4>For Data2vec (max_length = 512, epoch = 5):</h4><p>Best f1 score: 0.976871</p><p>Average f1 score: 0.957822</p><p>Wall time: 15 min 8 s</p><p>Memory increased: 0.3213 GB</p><p>Each time you run the model, the results may vary a little bit. Overall, after 5 tries, I can conclude that SBERT has a bit better performance in terms of best f1 score while Data2vec used way less memory. The average f1 scores for both models are very close.</p><p>After reading today’s demo, you should have the following takeaways:</p><ol><li>How to tokenize your text data with pre-trained models</li><li>How to properly convert your data to a torch dataset</li><li>How to leverage and fine-tune the pre-trained models</li><li>How to save your model for the future use</li><li>The performance comparison between Data2vec and SBERT</li></ol><p><strong><em>Please feel free to connect with me on </em></strong><a href="https://www.linkedin.com/in/jinhangjiang/"><strong><em>LinkedIn</em></strong></a><strong><em>.</em></strong></p><h3>Reference</h3><p>Baevski et al. (2022). data2vec: A General Framework for Self-supervised Learning in Speech, Vision and Language. <a href="https://arxiv.org/pdf/2202.03555.pdf">https://arxiv.org/pdf/2202.03555.pdf</a></p><p>Hugging Face. (2022). Text classification. <a href="https://huggingface.co/docs/transformers/tasks/sequence_classification">https://huggingface.co/docs/transformers/tasks/sequence_classification</a></p><p>Reimers, N. and Gurevych, I. (2019). Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks. <a href="https://arxiv.org/pdf/1908.10084.pdf">https://arxiv.org/pdf/1908.10084.pdf</a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=e3c35b19c949" width="1" height="1" alt=""><hr><p><a href="https://medium.com/data-science/sbert-vs-data2vec-on-text-classification-e3c35b19c949">SBERT vs. Data2vec on Text Classification</a> was originally published in <a href="https://medium.com/data-science">TDS Archive</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[One Line of Code to Accelerate Your Sklearn Algorithms on Big Data]]></title>
            <link>https://medium.com/data-science/one-line-of-code-to-accelerate-your-sklearn-algorithms-on-big-data-9c26190a0dc5?source=rss-1c41ed254f6d------2</link>
            <guid isPermaLink="false">https://medium.com/p/9c26190a0dc5</guid>
            <category><![CDATA[mlops]]></category>
            <category><![CDATA[python]]></category>
            <category><![CDATA[software-development]]></category>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[machine-learning]]></category>
            <dc:creator><![CDATA[Jinhang Jiang]]></dc:creator>
            <pubDate>Fri, 06 May 2022 13:12:55 GMT</pubDate>
            <atom:updated>2022-05-26T18:07:24.165Z</atom:updated>
            <content:encoded><![CDATA[<h4>The introduction of the intel sklearn extension. Make your Random Forest even faster than XGBoost.</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/972/1*hKuJSM0GmY2pW7jm3BvEyQ.png" /><figcaption>Image by author</figcaption></figure><h3>Introduction</h3><p>Scikit-learn (aka Sklearn) is definitely one of the most used Python libraries for data science nowadays. However, training a big dataset with sklearn algorithms sometimes can be costly. When I was still in school, I remember I trained an SVC model for literally a day. I am sure I’m not the only one who gets frustrated with this kind of situation. And today, I want to introduce the sklearnex to you, an extension developed by the Intel AI team about a year ago, and it was part of the daal4py library originally. This extension has the potential to accelerate your sklearn code 10X-100X.</p><p>According to their documents, “Intel® Extension for Scikit-learn* dynamically patches scikit-learn estimators to use Intel(R) oneAPI Data Analytics Library as the underlying solver while getting the same solution faster through the use of vector instructions, IA hardware-specific memory optimizations, etc. at launch time.” And their team also claims that it can speed various sklearn algorithms up from 10 to 100 times. Doesn’t it sound insane to you? Let’s find out together!</p><h3>Implementation</h3><p>To patch it on top of your sklearn applications without changing the code, you can use the following command line:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/908/1*alaDJJdxYnxZWQoe7eH9Mg.png" /><figcaption>Image by author</figcaption></figure><p>And if you want to use it directly in your notebook, you may do the following:</p><iframe src="https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Fcarbon.now.sh%2Fembed%3Fbg%3Drgba%2528182%252C162%252C145%252C1%2529%26t%3Dzenburn%26wt%3Dbw%26l%3Dpython%26width%3D680%26ds%3Dfalse%26dsyoff%3D20px%26dsblur%3D68px%26wc%3Dtrue%26wa%3Dtrue%26pv%3D56px%26ph%3D56px%26ln%3Dfalse%26fl%3D1%26fm%3DHack%26fs%3D14px%26lh%3D152%2525%26si%3Dfalse%26es%3D2x%26wm%3Dfalse%26code%3Dfrom%252520sklearnex%252520import%252520patch_sklearn%25250Apatch_sklearn%2528%2529%25250A%25250A%252523%252520Don%2527t%252520forget%252520to%252520reimport%252520the%252520package%252520that%252520you%252520want%252520to%252520use%25250Afrom%252520sklearn.tree%252520import%252520RandomForestRegressor%2528%2529&amp;display_name=Carbon&amp;url=https%3A%2F%2Fcarbon.now.sh%2F%3Fbg%3Drgba%252528182%25252C162%25252C145%25252C1%252529%26t%3Dzenburn%26wt%3Dbw%26l%3Dpython%26width%3D680%26ds%3Dfalse%26dsyoff%3D20px%26dsblur%3D68px%26wc%3Dtrue%26wa%3Dtrue%26pv%3D56px%26ph%3D56px%26ln%3Dfalse%26fl%3D1%26fm%3DHack%26fs%3D14px%26lh%3D152%252525%26si%3Dfalse%26es%3D2x%26wm%3Dfalse%26code%3Dfrom%25252520sklearnex%25252520import%25252520patch_sklearn%2525250Apatch_sklearn%252528%252529%2525250A%2525250A%25252523%25252520Don%252527t%25252520forget%25252520to%25252520reimport%25252520the%25252520package%25252520that%25252520you%25252520want%25252520to%25252520use%2525250Afrom%25252520sklearn.tree%25252520import%25252520RandomForestRegressor%252528%252529&amp;image=https%3A%2F%2Fcarbon.now.sh%2Fstatic%2Fbrand%2Fbanner.png&amp;key=a19fcc184b9711e1b4764040d3dc5c07&amp;type=text%2Fhtml&amp;scroll=auto&amp;schema=carbon" width="1024" height="480" frameborder="0" scrolling="no"><a href="https://medium.com/media/ea26e73b8fc52cf897fe77d99b5e249b/href">https://medium.com/media/ea26e73b8fc52cf897fe77d99b5e249b/href</a></iframe><p>If you want to patch it on top of a specific algorithm or a list of specific algorithms, you may pass the names directly in the function like this:</p><iframe src="https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Fcarbon.now.sh%2Fembed%3Fbg%3Drgba%2528182%252C162%252C145%252C1%2529%26t%3Dzenburn%26wt%3Dbw%26l%3Dpython%26width%3D680%26ds%3Dfalse%26dsyoff%3D20px%26dsblur%3D68px%26wc%3Dtrue%26wa%3Dtrue%26pv%3D56px%26ph%3D56px%26ln%3Dfalse%26fl%3D1%26fm%3DHack%26fs%3D14px%26lh%3D152%2525%26si%3Dfalse%26es%3D2x%26wm%3Dfalse%26code%3Dfrom%252520sklearnex%252520import%252520patch_sklearn%25250Apatch_sklearn%2528%25255B%2527RandomForestRegressor%2527%25252C%25250A%252520%252520%252520%252520%252520%252520%252520%252520%252520%252520%252520%252520%252520%252520%252520%2527SVC%2527%25252C%25250A%252520%252520%252520%252520%252520%252520%252520%252520%252520%252520%252520%252520%252520%252520%252520%2527DBSCAN%2527%25255D%2529%25250A%25250A%252523%252520Don%2527t%252520forget%252520to%252520reimport%252520the%252520package%252520that%252520you%252520want%252520to%252520use%25250Afrom%252520sklearn.tree%252520import%252520RandomForestRegressor%2528%2529%25250Afrom%252520sklearn.svm%252520import%252520SVC%25250Afrom%252520sklearn.cluster%252520import%252520DBSCAN&amp;display_name=Carbon&amp;url=https%3A%2F%2Fcarbon.now.sh%2F%3Fbg%3Drgba%252528182%25252C162%25252C145%25252C1%252529%26t%3Dzenburn%26wt%3Dbw%26l%3Dpython%26width%3D680%26ds%3Dfalse%26dsyoff%3D20px%26dsblur%3D68px%26wc%3Dtrue%26wa%3Dtrue%26pv%3D56px%26ph%3D56px%26ln%3Dfalse%26fl%3D1%26fm%3DHack%26fs%3D14px%26lh%3D152%252525%26si%3Dfalse%26es%3D2x%26wm%3Dfalse%26code%3Dfrom%25252520sklearnex%25252520import%25252520patch_sklearn%2525250Apatch_sklearn%252528%2525255B%252527RandomForestRegressor%252527%2525252C%2525250A%25252520%25252520%25252520%25252520%25252520%25252520%25252520%25252520%25252520%25252520%25252520%25252520%25252520%25252520%25252520%252527SVC%252527%2525252C%2525250A%25252520%25252520%25252520%25252520%25252520%25252520%25252520%25252520%25252520%25252520%25252520%25252520%25252520%25252520%25252520%252527DBSCAN%252527%2525255D%252529%2525250A%2525250A%25252523%25252520Don%252527t%25252520forget%25252520to%25252520reimport%25252520the%25252520package%25252520that%25252520you%25252520want%25252520to%25252520use%2525250Afrom%25252520sklearn.tree%25252520import%25252520RandomForestRegressor%252528%252529%2525250Afrom%25252520sklearn.svm%25252520import%25252520SVC%2525250Afrom%25252520sklearn.cluster%25252520import%25252520DBSCAN&amp;image=https%3A%2F%2Fcarbon.now.sh%2Fstatic%2Fbrand%2Fbanner.png&amp;key=a19fcc184b9711e1b4764040d3dc5c07&amp;type=text%2Fhtml&amp;scroll=auto&amp;schema=carbon" width="1024" height="480" frameborder="0" scrolling="no"><a href="https://medium.com/media/fd929cbd2569065106fff5e1efb97fe1/href">https://medium.com/media/fd929cbd2569065106fff5e1efb97fe1/href</a></iframe><p>And if you want to remove it from your code, you can simply unpatch it and reimport your sklearn model again as follow:</p><iframe src="https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Fcarbon.now.sh%2Fembed%3Fbg%3Drgba%2528182%252C162%252C145%252C1%2529%26t%3Dzenburn%26wt%3Dbw%26l%3Dpython%26width%3D680%26ds%3Dfalse%26dsyoff%3D20px%26dsblur%3D68px%26wc%3Dtrue%26wa%3Dtrue%26pv%3D56px%26ph%3D56px%26ln%3Dfalse%26fl%3D1%26fm%3DHack%26fs%3D14px%26lh%3D152%2525%26si%3Dfalse%26es%3D2x%26wm%3Dfalse%26code%3Dfrom%252520sklearnex%252520import%252520unpatch_sklearn%25250Aunpatch_sklearn%2528%2529%25250A%25250A%252523%252520Don%2527t%252520forget%252520to%252520reimport%252520the%252520package%252520that%252520you%252520want%252520to%252520use%25250Afrom%252520sklearn.tree%252520import%252520RandomForestRegressor%2528%2529%25250Afrom%252520sklearn.svm%252520import%252520SVC%25250Afrom%252520sklearn.cluster%252520import%252520DBSCAN&amp;display_name=Carbon&amp;url=https%3A%2F%2Fcarbon.now.sh%2F%3Fbg%3Drgba%252528182%25252C162%25252C145%25252C1%252529%26t%3Dzenburn%26wt%3Dbw%26l%3Dpython%26width%3D680%26ds%3Dfalse%26dsyoff%3D20px%26dsblur%3D68px%26wc%3Dtrue%26wa%3Dtrue%26pv%3D56px%26ph%3D56px%26ln%3Dfalse%26fl%3D1%26fm%3DHack%26fs%3D14px%26lh%3D152%252525%26si%3Dfalse%26es%3D2x%26wm%3Dfalse%26code%3Dfrom%25252520sklearnex%25252520import%25252520unpatch_sklearn%2525250Aunpatch_sklearn%252528%252529%2525250A%2525250A%25252523%25252520Don%252527t%25252520forget%25252520to%25252520reimport%25252520the%25252520package%25252520that%25252520you%25252520want%25252520to%25252520use%2525250Afrom%25252520sklearn.tree%25252520import%25252520RandomForestRegressor%252528%252529%2525250Afrom%25252520sklearn.svm%25252520import%25252520SVC%2525250Afrom%25252520sklearn.cluster%25252520import%25252520DBSCAN&amp;image=https%3A%2F%2Fcarbon.now.sh%2Fstatic%2Fbrand%2Fbanner.png&amp;key=a19fcc184b9711e1b4764040d3dc5c07&amp;type=text%2Fhtml&amp;scroll=auto&amp;schema=carbon" width="1024" height="480" frameborder="0" scrolling="no"><a href="https://medium.com/media/12e518dc4476480a22c1ce6d568ad9b1/href">https://medium.com/media/12e518dc4476480a22c1ce6d568ad9b1/href</a></iframe><h3>Sklearn vs. Sklearnex</h3><ul><li><em>RandomForestRegressor</em></li><li><em>SVC</em></li><li><em>DBSCAN</em></li><li><em>RandomForestClassifier vs. XGBClassifier</em></li></ul><p>Since it is hard to replicate all of their work due to the restrictions of hardware, I plan to compare the boosted version with the original version from three major machine learning algorithms: Regression, Classification, and Clustering.</p><p>Before we jump into the testing, let’s import all the packages we need:</p><iframe src="https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Fcarbon.now.sh%2Fembed%3Fbg%3Drgba%2528182%252C162%252C145%252C1%2529%26t%3Dzenburn%26wt%3Dbw%26l%3Dpython%26width%3D680%26ds%3Dfalse%26dsyoff%3D20px%26dsblur%3D68px%26wc%3Dtrue%26wa%3Dtrue%26pv%3D56px%26ph%3D56px%26ln%3Dfalse%26fl%3D1%26fm%3DHack%26fs%3D14px%26lh%3D152%2525%26si%3Dfalse%26es%3D2x%26wm%3Dfalse%26code%3Dfrom%252520sklearnex%252520import%252520patch_sklearn%25252C%252520unpatch_sklearn%25250Afrom%252520sklearn.datasets%252520import%252520make_classification%25252C%252520make_regression%25252C%252520make_moons%25250Afrom%252520sklearn.metrics%252520import%252520accuracy_score%25250Aimport%252520pandas%252520as%252520pd%25250Aimport%252520seaborn%252520as%252520sns%25250Aimport%252520matplotlib.pyplot%252520as%252520plt%25250Aimport%252520time%25250Afrom%252520xgboost%252520import%252520XGBClassifier&amp;display_name=Carbon&amp;url=https%3A%2F%2Fcarbon.now.sh%2F%3Fbg%3Drgba%252528182%25252C162%25252C145%25252C1%252529%26t%3Dzenburn%26wt%3Dbw%26l%3Dpython%26width%3D680%26ds%3Dfalse%26dsyoff%3D20px%26dsblur%3D68px%26wc%3Dtrue%26wa%3Dtrue%26pv%3D56px%26ph%3D56px%26ln%3Dfalse%26fl%3D1%26fm%3DHack%26fs%3D14px%26lh%3D152%252525%26si%3Dfalse%26es%3D2x%26wm%3Dfalse%26code%3Dfrom%25252520sklearnex%25252520import%25252520patch_sklearn%2525252C%25252520unpatch_sklearn%2525250Afrom%25252520sklearn.datasets%25252520import%25252520make_classification%2525252C%25252520make_regression%2525252C%25252520make_moons%2525250Afrom%25252520sklearn.metrics%25252520import%25252520accuracy_score%2525250Aimport%25252520pandas%25252520as%25252520pd%2525250Aimport%25252520seaborn%25252520as%25252520sns%2525250Aimport%25252520matplotlib.pyplot%25252520as%25252520plt%2525250Aimport%25252520time%2525250Afrom%25252520xgboost%25252520import%25252520XGBClassifier&amp;image=https%3A%2F%2Fcarbon.now.sh%2Fstatic%2Fbrand%2Fbanner.png&amp;key=a19fcc184b9711e1b4764040d3dc5c07&amp;type=text%2Fhtml&amp;scroll=auto&amp;schema=carbon" width="1024" height="480" frameborder="0" scrolling="no"><a href="https://medium.com/media/8615bd39c08f622f22d81776960c424f/href">https://medium.com/media/8615bd39c08f622f22d81776960c424f/href</a></iframe><h4>RandomForestRegressor</h4><p>Step 1: Training</p><p>Since I am using the google colab pro to conduct this demo, only 25G RAM is allocated for use. Therefore, I will test the model with sample sizes ranging from 10 to 100K. It will be enough to see the patterns for bigger data like multi-million+ records.</p><iframe src="https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Fcarbon.now.sh%2Fembed%3Fbg%3Drgba%2528182%252C162%252C145%252C1%2529%26t%3Dzenburn%26wt%3Dbw%26l%3Dpython%26width%3D680%26ds%3Dfalse%26dsyoff%3D20px%26dsblur%3D68px%26wc%3Dtrue%26wa%3Dtrue%26pv%3D56px%26ph%3D56px%26ln%3Dfalse%26fl%3D1%26fm%3DHack%26fs%3D14px%26lh%3D152%2525%26si%3Dfalse%26es%3D2x%26wm%3Dfalse%26code%3Dpatch_sklearn%2528%2529%25250Afrom%252520sklearn.ensemble%252520import%252520RandomForestRegressor%25250A%25250Ascore_1%252520%25253D%252520%25255B%25255D%25250Atime_lapse_1%252520%25253D%252520%25255B%25255D%25250A%25250Afor%252520i%252520in%252520range%25281%25252C6%2529%25253A%25250A%252520%252520print%2528i%2529%25250A%252520%252520X%25252C%252520y%252520%25253D%252520make_regression%2528n_samples%252520%25253D%25252010%252A%252Ai%25252C%252520n_features%25253D%25252030%25252C%252520n_informative%25253D25%25252C%252520random_state%25253D42%2529%25250A%252520%252520tic%252520%25253D%252520time.time%2528%2529%25250A%252520%252520model%252520%25253D%252520RandomForestRegressor%2528%2529.fit%2528X%25252C%252520y%2529%25250A%252520%252520toc%252520%25253D%252520time.time%2528%2529%25250A%252520%252520score_1.append%2528model.score%2528X%25252Cy%2529%2529%25250A%252520%252520time_lapse_1.append%2528round%25281000%252A%2528toc%252520-%252520tic%2529%25252C4%2529%2529%25250A%25250Aunpatch_sklearn%2528%2529%25250Afrom%252520sklearn.ensemble%252520import%252520RandomForestRegressor%25250A%25250Ascore_2%252520%25253D%252520%25255B%25255D%25250Atime_lapse_2%252520%25253D%252520%25255B%25255D%25250A%25250Afor%252520i%252520in%252520range%25281%25252C6%2529%25253A%25250A%252520%252520print%2528i%2529%25250A%252520%252520X%25252C%252520y%252520%25253D%252520make_regression%2528n_samples%252520%25253D%25252010%252A%252Ai%25252C%252520n_features%25253D%25252030%25252C%252520n_informative%25253D25%25252C%252520random_state%25253D42%2529%25250A%252520%252520tic%252520%25253D%252520time.time%2528%2529%25250A%252520%252520model%252520%25253D%252520RandomForestRegressor%2528%2529.fit%2528X%25252C%252520y%2529%25250A%252520%252520toc%252520%25253D%252520time.time%2528%2529%25250A%252520%252520score_2.append%2528model.score%2528X%25252Cy%2529%2529%25250A%252520%252520time_lapse_2.append%2528round%25281000%252A%2528toc%252520-%252520tic%2529%25252C4%2529%2529&amp;display_name=Carbon&amp;url=https%3A%2F%2Fcarbon.now.sh%2F%3Fbg%3Drgba%252528182%25252C162%25252C145%25252C1%252529%26t%3Dzenburn%26wt%3Dbw%26l%3Dpython%26width%3D680%26ds%3Dfalse%26dsyoff%3D20px%26dsblur%3D68px%26wc%3Dtrue%26wa%3Dtrue%26pv%3D56px%26ph%3D56px%26ln%3Dfalse%26fl%3D1%26fm%3DHack%26fs%3D14px%26lh%3D152%252525%26si%3Dfalse%26es%3D2x%26wm%3Dfalse%26code%3Dpatch_sklearn%252528%252529%2525250Afrom%25252520sklearn.ensemble%25252520import%25252520RandomForestRegressor%2525250A%2525250Ascore_1%25252520%2525253D%25252520%2525255B%2525255D%2525250Atime_lapse_1%25252520%2525253D%25252520%2525255B%2525255D%2525250A%2525250Afor%25252520i%25252520in%25252520range%2525281%2525252C6%252529%2525253A%2525250A%25252520%25252520print%252528i%252529%2525250A%25252520%25252520X%2525252C%25252520y%25252520%2525253D%25252520make_regression%252528n_samples%25252520%2525253D%2525252010%25252A%25252Ai%2525252C%25252520n_features%2525253D%2525252030%2525252C%25252520n_informative%2525253D25%2525252C%25252520random_state%2525253D42%252529%2525250A%25252520%25252520tic%25252520%2525253D%25252520time.time%252528%252529%2525250A%25252520%25252520model%25252520%2525253D%25252520RandomForestRegressor%252528%252529.fit%252528X%2525252C%25252520y%252529%2525250A%25252520%25252520toc%25252520%2525253D%25252520time.time%252528%252529%2525250A%25252520%25252520score_1.append%252528model.score%252528X%2525252Cy%252529%252529%2525250A%25252520%25252520time_lapse_1.append%252528round%2525281000%25252A%252528toc%25252520-%25252520tic%252529%2525252C4%252529%252529%2525250A%2525250Aunpatch_sklearn%252528%252529%2525250Afrom%25252520sklearn.ensemble%25252520import%25252520RandomForestRegressor%2525250A%2525250Ascore_2%25252520%2525253D%25252520%2525255B%2525255D%2525250Atime_lapse_2%25252520%2525253D%25252520%2525255B%2525255D%2525250A%2525250Afor%25252520i%25252520in%25252520range%2525281%2525252C6%252529%2525253A%2525250A%25252520%25252520print%252528i%252529%2525250A%25252520%25252520X%2525252C%25252520y%25252520%2525253D%25252520make_regression%252528n_samples%25252520%2525253D%2525252010%25252A%25252Ai%2525252C%25252520n_features%2525253D%2525252030%2525252C%25252520n_informative%2525253D25%2525252C%25252520random_state%2525253D42%252529%2525250A%25252520%25252520tic%25252520%2525253D%25252520time.time%252528%252529%2525250A%25252520%25252520model%25252520%2525253D%25252520RandomForestRegressor%252528%252529.fit%252528X%2525252C%25252520y%252529%2525250A%25252520%25252520toc%25252520%2525253D%25252520time.time%252528%252529%2525250A%25252520%25252520score_2.append%252528model.score%252528X%2525252Cy%252529%252529%2525250A%25252520%25252520time_lapse_2.append%252528round%2525281000%25252A%252528toc%25252520-%25252520tic%252529%2525252C4%252529%252529&amp;image=https%3A%2F%2Fcarbon.now.sh%2Fstatic%2Fbrand%2Fbanner.png&amp;key=a19fcc184b9711e1b4764040d3dc5c07&amp;type=text%2Fhtml&amp;scroll=auto&amp;schema=carbon" width="1024" height="480" frameborder="0" scrolling="no"><a href="https://medium.com/media/9d159874ab246f99991b6b8f22f45d75/href">https://medium.com/media/9d159874ab246f99991b6b8f22f45d75/href</a></iframe><p>Step 2: Collect Results:</p><iframe src="https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Fcarbon.now.sh%2Fembed%3Fbg%3Drgba%2528182%252C162%252C145%252C1%2529%26t%3Dzenburn%26wt%3Dbw%26l%3Dpython%26width%3D680%26ds%3Dfalse%26dsyoff%3D20px%26dsblur%3D68px%26wc%3Dtrue%26wa%3Dtrue%26pv%3D56px%26ph%3D56px%26ln%3Dfalse%26fl%3D1%26fm%3DHack%26fs%3D14px%26lh%3D152%2525%26si%3Dfalse%26es%3D2x%26wm%3Dfalse%26code%3Dresults%252520%25253D%252520pd.DataFrame%2528%25257B%2527Size%2527%25253A%25255B10%252A%252Ai%252520for%252520i%252520in%252520range%25281%25252C6%2529%25255D%25252C%25250A%252520%252520%252520%252520%252520%252520%252520%252520%252520%252520%252520%252520%252520%252520%2527Boost_Score%2527%25253Ascore_1%25252C%252520%25250A%252520%252520%252520%252520%252520%252520%252520%252520%252520%252520%252520%252520%252520%252520%2527Boost_Time%252520%2528ms%2529%2527%25253Atime_lapse_1%25252C%25250A%252520%252520%252520%252520%252520%252520%252520%252520%252520%252520%252520%252520%252520%252520%2527Normal_Score%2527%25253Ascore_2%25252C%252520%25250A%252520%252520%252520%252520%252520%252520%252520%252520%252520%252520%252520%252520%252520%252520%2527Normal_Time%252520%2528ms%2529%2527%25253Atime_lapse_2%25252C%25250A%252520%252520%252520%252520%252520%252520%252520%252520%252520%252520%252520%252520%252520%252520%2527Speed_X%2527%25253A%25255Bround%2528a%25252Fb%25252C4%2529%252520for%252520a%25252Cb%252520in%252520zip%2528time_lapse_2%25252Ctime_lapse_1%2529%25255D%25257D%2529%25250A%25250Adisplay%2528results%2529&amp;display_name=Carbon&amp;url=https%3A%2F%2Fcarbon.now.sh%2F%3Fbg%3Drgba%252528182%25252C162%25252C145%25252C1%252529%26t%3Dzenburn%26wt%3Dbw%26l%3Dpython%26width%3D680%26ds%3Dfalse%26dsyoff%3D20px%26dsblur%3D68px%26wc%3Dtrue%26wa%3Dtrue%26pv%3D56px%26ph%3D56px%26ln%3Dfalse%26fl%3D1%26fm%3DHack%26fs%3D14px%26lh%3D152%252525%26si%3Dfalse%26es%3D2x%26wm%3Dfalse%26code%3Dresults%25252520%2525253D%25252520pd.DataFrame%252528%2525257B%252527Size%252527%2525253A%2525255B10%25252A%25252Ai%25252520for%25252520i%25252520in%25252520range%2525281%2525252C6%252529%2525255D%2525252C%2525250A%25252520%25252520%25252520%25252520%25252520%25252520%25252520%25252520%25252520%25252520%25252520%25252520%25252520%25252520%252527Boost_Score%252527%2525253Ascore_1%2525252C%25252520%2525250A%25252520%25252520%25252520%25252520%25252520%25252520%25252520%25252520%25252520%25252520%25252520%25252520%25252520%25252520%252527Boost_Time%25252520%252528ms%252529%252527%2525253Atime_lapse_1%2525252C%2525250A%25252520%25252520%25252520%25252520%25252520%25252520%25252520%25252520%25252520%25252520%25252520%25252520%25252520%25252520%252527Normal_Score%252527%2525253Ascore_2%2525252C%25252520%2525250A%25252520%25252520%25252520%25252520%25252520%25252520%25252520%25252520%25252520%25252520%25252520%25252520%25252520%25252520%252527Normal_Time%25252520%252528ms%252529%252527%2525253Atime_lapse_2%2525252C%2525250A%25252520%25252520%25252520%25252520%25252520%25252520%25252520%25252520%25252520%25252520%25252520%25252520%25252520%25252520%252527Speed_X%252527%2525253A%2525255Bround%252528a%2525252Fb%2525252C4%252529%25252520for%25252520a%2525252Cb%25252520in%25252520zip%252528time_lapse_2%2525252Ctime_lapse_1%252529%2525255D%2525257D%252529%2525250A%2525250Adisplay%252528results%252529&amp;image=https%3A%2F%2Fcarbon.now.sh%2Fstatic%2Fbrand%2Fbanner.png&amp;key=a19fcc184b9711e1b4764040d3dc5c07&amp;type=text%2Fhtml&amp;scroll=auto&amp;schema=carbon" width="1024" height="480" frameborder="0" scrolling="no"><a href="https://medium.com/media/0280fe23eaab0a88d49c01301d5f5739/href">https://medium.com/media/0280fe23eaab0a88d49c01301d5f5739/href</a></iframe><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*1vbHTyeJ5ai798wwVSBiOA.png" /><figcaption>Image by author</figcaption></figure><p>Based on the outputs, we can see that the performance is very consistent while the training speed increased roughly 6.5X for 100K samples. I consider this is a significant increase in machine learning practice.</p><p>Step 3: Plot</p><iframe src="https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Fcarbon.now.sh%2Fembed%3Fbg%3Drgba%2528182%252C162%252C145%252C1%2529%26t%3Dzenburn%26wt%3Dbw%26l%3Dpython%26width%3D680%26ds%3Dfalse%26dsyoff%3D20px%26dsblur%3D68px%26wc%3Dtrue%26wa%3Dtrue%26pv%3D56px%26ph%3D56px%26ln%3Dfalse%26fl%3D1%26fm%3DHack%26fs%3D14px%26lh%3D152%2525%26si%3Dfalse%26es%3D2x%26wm%3Dfalse%26code%3Dsns.set_style%2528%252522darkgrid%252522%2529%25250Afig%25252C%252520axes%252520%25253D%252520plt.subplots%25281%25252C%2525202%25252C%252520figsize%25253D%252820%25252C%2525206%2529%2529%25250A%252520%25250Afig.suptitle%2528%2527SklearnEX%252520vs.%252520Normal%252520Sklearn%252520for%252520RandomForestRegressor%2527%2529%25250A%25250A%25250Asns.lineplot%2528ax%25253Daxes%25255B0%25255D%25252C%252520data%252520%25253D%252520results%25252C%252520x%25253D%2527Size%2527%25252C%252520y%25253D%2527Boost_Time%252520%2528ms%2529%2527%2529%25250Asns.lineplot%2528ax%25253Daxes%25255B0%25255D%25252C%252520data%252520%25253D%252520results%25252C%252520x%25253D%2527Size%2527%25252C%252520y%25253D%2527Normal_Time%252520%2528ms%2529%2527%2529%25250Asns.lineplot%2528ax%25253Daxes%25255B1%25255D%25252C%252520data%252520%25253D%252520results%25252C%252520x%25253D%2527Size%2527%25252C%252520y%25253D%2527Speed_X%2527%25252C%252520color%25253D%2527green%2527%2529%25250A%25250A%252520%25250Aaxes%25255B0%25255D.set_title%2528%2527Time%252520Used%252520in%252520Training%2527%2529%25250Aaxes%25255B1%25255D.set_title%2528%2527Speed%252520Ratio%2527%2529%25250Aaxes%25255B0%25255D.legend%2528%25255B%2527Boost_Time%252520%2528ms%2529%2527%25252C%252520%2527Normal_Time%252520%2528ms%2529%2527%25255D%2529%25250Aaxes%25255B0%25255D.legend%2528%25255B%2527Speed_X%2527%25255D%2529&amp;display_name=Carbon&amp;url=https%3A%2F%2Fcarbon.now.sh%2F%3Fbg%3Drgba%252528182%25252C162%25252C145%25252C1%252529%26t%3Dzenburn%26wt%3Dbw%26l%3Dpython%26width%3D680%26ds%3Dfalse%26dsyoff%3D20px%26dsblur%3D68px%26wc%3Dtrue%26wa%3Dtrue%26pv%3D56px%26ph%3D56px%26ln%3Dfalse%26fl%3D1%26fm%3DHack%26fs%3D14px%26lh%3D152%252525%26si%3Dfalse%26es%3D2x%26wm%3Dfalse%26code%3Dsns.set_style%252528%25252522darkgrid%25252522%252529%2525250Afig%2525252C%25252520axes%25252520%2525253D%25252520plt.subplots%2525281%2525252C%252525202%2525252C%25252520figsize%2525253D%25252820%2525252C%252525206%252529%252529%2525250A%25252520%2525250Afig.suptitle%252528%252527SklearnEX%25252520vs.%25252520Normal%25252520Sklearn%25252520for%25252520RandomForestRegressor%252527%252529%2525250A%2525250A%2525250Asns.lineplot%252528ax%2525253Daxes%2525255B0%2525255D%2525252C%25252520data%25252520%2525253D%25252520results%2525252C%25252520x%2525253D%252527Size%252527%2525252C%25252520y%2525253D%252527Boost_Time%25252520%252528ms%252529%252527%252529%2525250Asns.lineplot%252528ax%2525253Daxes%2525255B0%2525255D%2525252C%25252520data%25252520%2525253D%25252520results%2525252C%25252520x%2525253D%252527Size%252527%2525252C%25252520y%2525253D%252527Normal_Time%25252520%252528ms%252529%252527%252529%2525250Asns.lineplot%252528ax%2525253Daxes%2525255B1%2525255D%2525252C%25252520data%25252520%2525253D%25252520results%2525252C%25252520x%2525253D%252527Size%252527%2525252C%25252520y%2525253D%252527Speed_X%252527%2525252C%25252520color%2525253D%252527green%252527%252529%2525250A%2525250A%25252520%2525250Aaxes%2525255B0%2525255D.set_title%252528%252527Time%25252520Used%25252520in%25252520Training%252527%252529%2525250Aaxes%2525255B1%2525255D.set_title%252528%252527Speed%25252520Ratio%252527%252529%2525250Aaxes%2525255B0%2525255D.legend%252528%2525255B%252527Boost_Time%25252520%252528ms%252529%252527%2525252C%25252520%252527Normal_Time%25252520%252528ms%252529%252527%2525255D%252529%2525250Aaxes%2525255B0%2525255D.legend%252528%2525255B%252527Speed_X%252527%2525255D%252529&amp;image=https%3A%2F%2Fcarbon.now.sh%2Fstatic%2Fbrand%2Fbanner.png&amp;key=a19fcc184b9711e1b4764040d3dc5c07&amp;type=text%2Fhtml&amp;scroll=auto&amp;schema=carbon" width="1024" height="480" frameborder="0" scrolling="no"><a href="https://medium.com/media/9fd0c05c1e259dd10495a745fc37b08f/href">https://medium.com/media/9fd0c05c1e259dd10495a745fc37b08f/href</a></iframe><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*QYJun0WdbFSecaL3p6R8yg.png" /><figcaption>Image by author</figcaption></figure><h4>SVC</h4><p>Step 1: Training:</p><p><em>Note: pay attention to this part, the training time for the final iteration will approximately take 1 hour and 20 mins to run.</em></p><iframe src="https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Fcarbon.now.sh%2Fembed%3Fbg%3Drgba%2528182%252C162%252C145%252C1%2529%26t%3Dzenburn%26wt%3Dbw%26l%3Dpython%26width%3D680%26ds%3Dfalse%26dsyoff%3D20px%26dsblur%3D68px%26wc%3Dtrue%26wa%3Dtrue%26pv%3D56px%26ph%3D56px%26ln%3Dfalse%26fl%3D1%26fm%3DHack%26fs%3D14px%26lh%3D152%2525%26si%3Dfalse%26es%3D2x%26wm%3Dfalse%26code%3Dpatch_sklearn%2528%2529%25250Afrom%252520sklearn.svm%252520import%252520SVC%25250A%25250Ascore_1%252520%25253D%252520%25255B%25255D%25250Atime_lapse_1%252520%25253D%252520%25255B%25255D%25250A%25250Afor%252520i%252520in%252520range%25281%25252C6%2529%25253A%25250A%252520%252520print%2528i%2529%25250A%252520%252520X%25252C%252520y%252520%25253D%252520make_classification%2528n_samples%252520%25253D%25252010%252A%252Ai%25252C%252520n_features%25253D%25252030%25252C%252520n_informative%25253D25%25252C%252520random_state%25253D42%2529%25250A%252520%252520tic%252520%25253D%252520time.time%2528%2529%25250A%252520%252520model%252520%25253D%252520SVC%2528gamma%25253D%2527auto%2527%2529.fit%2528X%25252C%252520y%2529%25250A%252520%252520toc%252520%25253D%252520time.time%2528%2529%25250A%252520%252520score_1.append%2528model.score%2528X%25252Cy%2529%2529%25250A%252520%252520time_lapse_1.append%2528round%25281000%252A%2528toc%252520-%252520tic%2529%25252C4%2529%2529%25250A%252520%252520%25250Aunpatch_sklearn%2528%2529%25250Afrom%252520sklearn.svm%252520import%252520SVC%25250A%25250Ascore_2%252520%25253D%252520%25255B%25255D%25250Atime_lapse_2%252520%25253D%252520%25255B%25255D%25250A%25250Afor%252520i%252520in%252520range%25281%25252C6%2529%25253A%25250A%252520%252520print%2528i%2529%25250A%252520%252520X%25252C%252520y%252520%25253D%252520make_classification%2528n_samples%252520%25253D%25252010%252A%252Ai%25252C%252520n_features%25253D%25252030%25252C%252520n_informative%25253D25%25252C%252520random_state%25253D42%2529%25250A%252520%252520tic%252520%25253D%252520time.time%2528%2529%25250A%252520%252520model%252520%25253D%252520SVC%2528gamma%25253D%2527auto%2527%2529.fit%2528X%25252C%252520y%2529%25250A%252520%252520toc%252520%25253D%252520time.time%2528%2529%25250A%252520%252520score_2.append%2528model.score%2528X%25252Cy%2529%2529%25250A%252520%252520time_lapse_2.append%2528round%25281000%252A%2528toc%252520-%252520tic%2529%25252C4%2529%2529&amp;display_name=Carbon&amp;url=https%3A%2F%2Fcarbon.now.sh%2F%3Fbg%3Drgba%252528182%25252C162%25252C145%25252C1%252529%26t%3Dzenburn%26wt%3Dbw%26l%3Dpython%26width%3D680%26ds%3Dfalse%26dsyoff%3D20px%26dsblur%3D68px%26wc%3Dtrue%26wa%3Dtrue%26pv%3D56px%26ph%3D56px%26ln%3Dfalse%26fl%3D1%26fm%3DHack%26fs%3D14px%26lh%3D152%252525%26si%3Dfalse%26es%3D2x%26wm%3Dfalse%26code%3Dpatch_sklearn%252528%252529%2525250Afrom%25252520sklearn.svm%25252520import%25252520SVC%2525250A%2525250Ascore_1%25252520%2525253D%25252520%2525255B%2525255D%2525250Atime_lapse_1%25252520%2525253D%25252520%2525255B%2525255D%2525250A%2525250Afor%25252520i%25252520in%25252520range%2525281%2525252C6%252529%2525253A%2525250A%25252520%25252520print%252528i%252529%2525250A%25252520%25252520X%2525252C%25252520y%25252520%2525253D%25252520make_classification%252528n_samples%25252520%2525253D%2525252010%25252A%25252Ai%2525252C%25252520n_features%2525253D%2525252030%2525252C%25252520n_informative%2525253D25%2525252C%25252520random_state%2525253D42%252529%2525250A%25252520%25252520tic%25252520%2525253D%25252520time.time%252528%252529%2525250A%25252520%25252520model%25252520%2525253D%25252520SVC%252528gamma%2525253D%252527auto%252527%252529.fit%252528X%2525252C%25252520y%252529%2525250A%25252520%25252520toc%25252520%2525253D%25252520time.time%252528%252529%2525250A%25252520%25252520score_1.append%252528model.score%252528X%2525252Cy%252529%252529%2525250A%25252520%25252520time_lapse_1.append%252528round%2525281000%25252A%252528toc%25252520-%25252520tic%252529%2525252C4%252529%252529%2525250A%25252520%25252520%2525250Aunpatch_sklearn%252528%252529%2525250Afrom%25252520sklearn.svm%25252520import%25252520SVC%2525250A%2525250Ascore_2%25252520%2525253D%25252520%2525255B%2525255D%2525250Atime_lapse_2%25252520%2525253D%25252520%2525255B%2525255D%2525250A%2525250Afor%25252520i%25252520in%25252520range%2525281%2525252C6%252529%2525253A%2525250A%25252520%25252520print%252528i%252529%2525250A%25252520%25252520X%2525252C%25252520y%25252520%2525253D%25252520make_classification%252528n_samples%25252520%2525253D%2525252010%25252A%25252Ai%2525252C%25252520n_features%2525253D%2525252030%2525252C%25252520n_informative%2525253D25%2525252C%25252520random_state%2525253D42%252529%2525250A%25252520%25252520tic%25252520%2525253D%25252520time.time%252528%252529%2525250A%25252520%25252520model%25252520%2525253D%25252520SVC%252528gamma%2525253D%252527auto%252527%252529.fit%252528X%2525252C%25252520y%252529%2525250A%25252520%25252520toc%25252520%2525253D%25252520time.time%252528%252529%2525250A%25252520%25252520score_2.append%252528model.score%252528X%2525252Cy%252529%252529%2525250A%25252520%25252520time_lapse_2.append%252528round%2525281000%25252A%252528toc%25252520-%25252520tic%252529%2525252C4%252529%252529&amp;image=https%3A%2F%2Fcarbon.now.sh%2Fstatic%2Fbrand%2Fbanner.png&amp;key=a19fcc184b9711e1b4764040d3dc5c07&amp;type=text%2Fhtml&amp;scroll=auto&amp;schema=carbon" width="1024" height="480" frameborder="0" scrolling="no"><a href="https://medium.com/media/edbc3e85a932a4c35de69d9aaae75a71/href">https://medium.com/media/edbc3e85a932a4c35de69d9aaae75a71/href</a></iframe><p>Since the code will be extremely similar to what I have done with RandomForestRegressor, I will omit steps 2 and 3. Here are the results:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*89F8ctC9xk_oJpOgroOs-g.png" /><figcaption>Image by author</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*W_86NRrDbz7oiDk5wC5-3w.png" /><figcaption>Image by author</figcaption></figure><p>So far, SVC has shown the most striking contrast in this demo. With the sklearnex patched on top, we can observe 32X faster on a dataset with 100K samples. Due to the SVC’s time-consuming issue, this algorithm has been abandoned by me for most cases regardless of its unique algorithmic uniqueness. However, with the enhancement given by sklearnex, I think SVC will start to gain more attention back.</p><h4>DBSCAN</h4><p>Step 1: Training:</p><iframe src="https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Fcarbon.now.sh%2Fembed%3Fbg%3Drgba%2528182%252C162%252C145%252C1%2529%26t%3Dzenburn%26wt%3Dbw%26l%3Dpython%26width%3D680%26ds%3Dfalse%26dsyoff%3D20px%26dsblur%3D68px%26wc%3Dtrue%26wa%3Dtrue%26pv%3D56px%26ph%3D56px%26ln%3Dfalse%26fl%3D1%26fm%3DHack%26fs%3D14px%26lh%3D152%2525%26si%3Dfalse%26es%3D2x%26wm%3Dfalse%26code%3Dscore_1%252520%25253D%252520%25255B%25255D%25250Atime_lapse_1%252520%25253D%252520%25255B%25255D%25250Ascore_2%252520%25253D%252520%25255B%25255D%25250Atime_lapse_2%252520%25253D%252520%25255B%25255D%25250A%25250Afor%252520i%252520in%252520%25255B5000%25252C25000%25252C50000%25252C70000%25255D%25253A%25250A%252520%252520print%2528i%2529%25250A%252520%252520patch_sklearn%2528%2529%25250A%252520%252520from%252520sklearn.cluster%252520import%252520DBSCAN%25250A%252520%252520X%25252C%252520y%252520%25253D%252520make_moons%2528n_samples%252520%25253D%252520i%25252C%252520noise%25253D0.1%25252C%252520random_state%25253D42%2529%25250A%252520%252520tic%252520%25253D%252520time.time%2528%2529%25250A%252520%252520model%252520%25253D%252520DBSCAN%2528eps%25253D1%2529.fit_predict%2528X%2529%25250A%252520%252520toc%252520%25253D%252520time.time%2528%2529%25250A%252520%252520score_1.append%2528accuracy_score%2528y%25252Cmodel%2529%2529%25250A%252520%252520time_lapse_1.append%2528round%25281000%252A%2528toc%252520-%252520tic%2529%25252C4%2529%2529%25250A%25250A%252520%252520unpatch_sklearn%2528%2529%25250A%252520%252520from%252520sklearn.cluster%252520import%252520DBSCAN%25250A%252520%252520tic%252520%25253D%252520time.time%2528%2529%25250A%252520%252520model%252520%25253D%252520DBSCAN%2528eps%25253D1%2529.fit_predict%2528X%2529%25250A%252520%252520toc%252520%25253D%252520time.time%2528%2529%25250A%252520%252520score_2.append%2528accuracy_score%2528y%25252Cmodel%2529%2529%25250A%252520%252520time_lapse_2.append%2528round%25281000%252A%2528toc%252520-%252520tic%2529%25252C4%2529%2529&amp;display_name=Carbon&amp;url=https%3A%2F%2Fcarbon.now.sh%2F%3Fbg%3Drgba%252528182%25252C162%25252C145%25252C1%252529%26t%3Dzenburn%26wt%3Dbw%26l%3Dpython%26width%3D680%26ds%3Dfalse%26dsyoff%3D20px%26dsblur%3D68px%26wc%3Dtrue%26wa%3Dtrue%26pv%3D56px%26ph%3D56px%26ln%3Dfalse%26fl%3D1%26fm%3DHack%26fs%3D14px%26lh%3D152%252525%26si%3Dfalse%26es%3D2x%26wm%3Dfalse%26code%3Dscore_1%25252520%2525253D%25252520%2525255B%2525255D%2525250Atime_lapse_1%25252520%2525253D%25252520%2525255B%2525255D%2525250Ascore_2%25252520%2525253D%25252520%2525255B%2525255D%2525250Atime_lapse_2%25252520%2525253D%25252520%2525255B%2525255D%2525250A%2525250Afor%25252520i%25252520in%25252520%2525255B5000%2525252C25000%2525252C50000%2525252C70000%2525255D%2525253A%2525250A%25252520%25252520print%252528i%252529%2525250A%25252520%25252520patch_sklearn%252528%252529%2525250A%25252520%25252520from%25252520sklearn.cluster%25252520import%25252520DBSCAN%2525250A%25252520%25252520X%2525252C%25252520y%25252520%2525253D%25252520make_moons%252528n_samples%25252520%2525253D%25252520i%2525252C%25252520noise%2525253D0.1%2525252C%25252520random_state%2525253D42%252529%2525250A%25252520%25252520tic%25252520%2525253D%25252520time.time%252528%252529%2525250A%25252520%25252520model%25252520%2525253D%25252520DBSCAN%252528eps%2525253D1%252529.fit_predict%252528X%252529%2525250A%25252520%25252520toc%25252520%2525253D%25252520time.time%252528%252529%2525250A%25252520%25252520score_1.append%252528accuracy_score%252528y%2525252Cmodel%252529%252529%2525250A%25252520%25252520time_lapse_1.append%252528round%2525281000%25252A%252528toc%25252520-%25252520tic%252529%2525252C4%252529%252529%2525250A%2525250A%25252520%25252520unpatch_sklearn%252528%252529%2525250A%25252520%25252520from%25252520sklearn.cluster%25252520import%25252520DBSCAN%2525250A%25252520%25252520tic%25252520%2525253D%25252520time.time%252528%252529%2525250A%25252520%25252520model%25252520%2525253D%25252520DBSCAN%252528eps%2525253D1%252529.fit_predict%252528X%252529%2525250A%25252520%25252520toc%25252520%2525253D%25252520time.time%252528%252529%2525250A%25252520%25252520score_2.append%252528accuracy_score%252528y%2525252Cmodel%252529%252529%2525250A%25252520%25252520time_lapse_2.append%252528round%2525281000%25252A%252528toc%25252520-%25252520tic%252529%2525252C4%252529%252529&amp;image=https%3A%2F%2Fcarbon.now.sh%2Fstatic%2Fbrand%2Fbanner.png&amp;key=a19fcc184b9711e1b4764040d3dc5c07&amp;type=text%2Fhtml&amp;scroll=auto&amp;schema=carbon" width="1024" height="480" frameborder="0" scrolling="no"><a href="https://medium.com/media/35b53993704bbf213c36eb1cbb15aeb4/href">https://medium.com/media/35b53993704bbf213c36eb1cbb15aeb4/href</a></iframe><p>Result:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1014/1*TZsxGvqtDY7mTuN98PIAoQ.png" /><figcaption>Image by author</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*unLOd2NStNkoAYFzmJodmg.png" /><figcaption>Image by author</figcaption></figure><p>According to Intel’s document, their extension can help DBSCAN become 322X faster on a dataset with 500K samples and 50 features. However, my google colab will crash with a sample size larger than 80K for DBSCAN, so I was not able to replicate this number. Based on what I can get, on average, DBSCAN got 1.25X faster with the help of sklearnex.</p><h4>RandomForest vs. XGBoost</h4><p>For many data science projects, we have seen people pick XGBoost over Random Forest. This kind of decision involves a lot of tradeoffs, and certainly, the training time on a large dataset is one of them. So I am about to show you something really exciting:</p><p>Step 1: Training</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*l48EyABd1Ueguu8z_y0Vgg.png" /><figcaption>Code by author</figcaption></figure><p>Result:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*IkBmh4T9KbMVbZj1Jc7t4Q.png" /><figcaption>Image by author</figcaption></figure><p><em>Note:</em></p><p><em>Spped_X_RR is the comparison of training time for the original Random Forest over the Random Forest with extension</em></p><p><em>Spped_X_RX is the comparison of training time for the XGBoost over the Random Forest with extension</em></p><p><em>Spped_X_RR is the comparison of training time for the original Random Forest over the XGBoost</em></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*ce7VaVbMBggC8R3ciYjUmw.png" /><figcaption>Image by author</figcaption></figure><p>Based on the results, we can see that the boosted Random Forest Classifier is approximately 3.5X faster than XGBClassifier on a given dataset with 10K and 100K, respectively.</p><p>You may find the complete code here: <a href="https://github.com/jinhangjiang/Medium_Demo/blob/main/Intel_Sklearn/Intel_Extension_for_Sklearn.ipynb">https://github.com/jinhangjiang/Medium_Demo/blob/main/Intel_Sklearn/Intel_Extension_for_Sklearn.ipynb</a></p><h3>Conclusion</h3><p>Given the simplicity of the implementation of this Intel extension, I believe it can be applied in a very large scope of projects in data science. Especially, given the limited testing results, we can clearly observe the scalability of this extension. To conclude, as simple as this: the bigger the data, the better the performance. You may find the details of the complete comparison conducted by Intel here: <a href="https://intel.github.io/scikit-learn-intelex/acceleration.html">https://intel.github.io/scikit-learn-intelex/acceleration.html</a>.</p><p>Of course, there are some limitations to this extension currently. For example, the extension does not support multi-output and sparse data for the Random Forest and K-Nearest Neighbors. For more details, check here: <a href="https://intel.github.io/scikit-learn-intelex/algorithms.html">https://intel.github.io/scikit-learn-intelex/algorithms.html</a>.</p><p><strong><em>Please feel free to connect with me on </em></strong><a href="https://www.linkedin.com/in/jinhangjiang/"><strong><em>LinkedIn</em></strong></a><strong><em>.</em></strong></p><h3>Reference</h3><p>Intel Corporation. (2021). Intel(R) Extension for Scikit-learn* 2021.5 documentation. <a href="https://intel.github.io/scikit-learn-intelex/blogs.html">https://intel.github.io/scikit-learn-intelex/blogs.html</a>.</p><p>Smirnov, E. (2021) Save Time and Money with Intel Extension for Scikit-learn. <a href="https://medium.com/intel-analytics-software/save-time-and-money-with-intel-extension-for-scikit-learn-33627425ae4">https://medium.com/intel-analytics-software/save-time-and-money-with-intel-extension-for-scikit-learn-33627425ae4</a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=9c26190a0dc5" width="1" height="1" alt=""><hr><p><a href="https://medium.com/data-science/one-line-of-code-to-accelerate-your-sklearn-algorithms-on-big-data-9c26190a0dc5">One Line of Code to Accelerate Your Sklearn Algorithms on Big Data</a> was originally published in <a href="https://medium.com/data-science">TDS Archive</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
    </channel>
</rss>