<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Maziar Izadi on Medium]]></title>
        <description><![CDATA[Stories by Maziar Izadi on Medium]]></description>
        <link>https://medium.com/@maziarizadi?source=rss-19c9d8bdde6------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/2*kejaqlrN38WKtVYiypYHyQ.png</url>
            <title>Stories by Maziar Izadi on Medium</title>
            <link>https://medium.com/@maziarizadi?source=rss-19c9d8bdde6------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Wed, 18 Nov 2026 16:39:11 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@maziarizadi/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[Covid-19: Genetic Mutation or Engineering]]></title>
            <link>https://medium.com/analytics-vidhya/covid-19-genetic-mutation-or-engineering-a55afe359659?source=rss-19c9d8bdde6------2</link>
            <guid isPermaLink="false">https://medium.com/p/a55afe359659</guid>
            <category><![CDATA[python]]></category>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[data-visualization]]></category>
            <category><![CDATA[covid19]]></category>
            <category><![CDATA[motion-charts]]></category>
            <dc:creator><![CDATA[Maziar Izadi]]></dc:creator>
            <pubDate>Sat, 18 Apr 2020 17:17:25 GMT</pubDate>
            <atom:updated>2020-04-18T17:17:25.939Z</atom:updated>
            <content:encoded><![CDATA[<h4>Motion Chart Visualisation in Python</h4><blockquote><a href="https://www.smh.com.au/politics/federal/virus-will-leave-an-economic-impact-for-decades-20200406-p54hk9.html">Virus will leave an economic impact for decades</a>.</blockquote><p><em>Using the </em><strong><em>long-term</em></strong><em> </em><strong><em>interest rate</em></strong><em> </em><strong><em>data</em></strong><em> and other sources to look at the economic impact of 12 pandemics in history, researchers found “significant macroeconomic after-effects” from pandemics that lasted for 40 years.</em></p><h3>In this article, I let data talk for itself.</h3><p>A <strong>visual</strong> <strong>motion chart</strong> which <strong>illustrates the trend of outbreak</strong> in different countries from 3 perspectives has a lot to say…</p><ol><li>Number of Confirmed cases</li><li>Number of Recovered cases</li><li>Number of deceased</li></ol><h4>Data Source</h4><p>I forked data from <a href="https://github.com/maziarizadi/COVID-19">data source</a> operated by the<strong> Johns Hopkins University Center for Systems Science and Engineering (JHU CSSE)</strong>. Also, Supported by <strong>ESRI Living Atlas Team</strong> and the <strong>Johns Hopkins University Applied Physics Lab (JHU APL)</strong>.</p><iframe src="https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Fwww.youtube.com%2Fembed%2FhRUr9-o-roo%3Ffeature%3Doembed&amp;display_name=YouTube&amp;url=https%3A%2F%2Fwww.youtube.com%2Fwatch%3Fv%3DhRUr9-o-roo&amp;image=https%3A%2F%2Fi.ytimg.com%2Fvi%2FhRUr9-o-roo%2Fhqdefault.jpg&amp;key=a19fcc184b9711e1b4764040d3dc5c07&amp;type=text%2Fhtml&amp;schema=youtube" width="854" height="480" frameborder="0" scrolling="no"><a href="https://medium.com/media/97956cac38dd2a5ecedc8d25864438aa/href">https://medium.com/media/97956cac38dd2a5ecedc8d25864438aa/href</a></iframe><p>Python code for this work is completely available on my <a href="https://github.com/maziarizadi/COVID-19">GitHub</a>, so I try to keep it here more generic.</p><p>Motion chart above is created with 5 dimensions:</p><ol><li>x-axis: number of <strong>confirmed</strong> cases</li><li>y-axis: number of <strong>recovered</strong> cases</li><li>Circles’ colour: <strong>Countries</strong></li><li>Circles size: number of <strong>deaths</strong></li><li><strong>Time</strong></li></ol><h3>Let’s get data in motion</h3><iframe src="https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Fgiphy.com%2Fembed%2FMcVcE1YJrsze1qtgnP%2Ftwitter%2Fiframe&amp;display_name=Giphy&amp;url=https%3A%2F%2Fmedia.giphy.com%2Fmedia%2FMcVcE1YJrsze1qtgnP%2Fgiphy.gif&amp;image=https%3A%2F%2Fi.giphy.com%2Fmedia%2FMcVcE1YJrsze1qtgnP%2Fgiphy.gif&amp;key=a19fcc184b9711e1b4764040d3dc5c07&amp;type=text%2Fhtml&amp;schema=giphy" width="435" height="244" frameborder="0" scrolling="no"><a href="https://medium.com/media/ed6f303bd7c43b2900abd3c3c2efe445/href">https://medium.com/media/ed6f303bd7c43b2900abd3c3c2efe445/href</a></iframe><p>Given data is being forked from <a href="https://github.com/CSSEGISandData/COVID-19"><strong>JHU CSSE COVID-19</strong></a><strong> </strong>into my Github, using os library, I look into the github folder and print the contents.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*44dMs50u9vYqwZATqErUlQ.png" /><figcaption>Print Github folder’s contents</figcaption></figure><p>Among all the contents, I read_csv the three marked files into three separate pandas dataframe from which I’ve sample printed one below.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*2wMSX7jxeZEnMb0FpQGBjw.png" /><figcaption>time_series_covid19_deaths_global.csv to dataframe first 5 rows (<strong>before preprocessing</strong>)</figcaption></figure><h3>Preprocessing data</h3><p>As usual, we need to prepare data for consumption. Our desired dataframe must look like the one below which has 5 columns.</p><p>I will explain the process for one out of the three data frames considering that they are all done exactly the same way.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/954/1*FFLI6S6sgzEgpeKGBLFG3Q.png" /><figcaption>Desirable data format for motion chart creation (<strong>after preprocessing</strong>)</figcaption></figure><p>Looking at the two data structures, it’s clear that we need to do some preparation. For that, let’s drop extra columns from the original data frame.</p><pre>new_data_conf = data_conf.drop(columns=[&#39;Province/State&#39;,&#39;Lat&#39;, &#39;Long&#39;])</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/904/1*7v5cOrof8_lAhxY6ZsoEtA.png" /></figure><p>Next, we need to convert everything into columns. There are numerous ways to do that. My approach is using groupby based on country names to unify each of them into unique row. (remember that in the original data set, due to “province/state” field, same country was repeated in multiple rows).</p><blockquote>A quick note on groupby(). It’s required to use a mathematical function for numerical columns for which I have used mean().</blockquote><pre>new_data_conf = new_data_conf.groupby([&#39;Country/Region&#39;])[new_data_conf.columns].sum()</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/800/1*Yrl-SR8Y4fSqCyhLqouYmg.png" /></figure><p>Now, to convert date into a column, I used transpose(), reset_index(), and finally rename().</p><pre>new_data_conf=new_data_conf.transpose()</pre><pre>new_data_conf.reset_index(level=0, inplace=True)</pre><pre>new_data_conf.rename(columns={&#39;index&#39; : &#39;Date&#39;}, inplace=True)</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*6UZ8TZW-VqY1vqkhitia9g.png" /><figcaption>time_series_covid19_deaths_global.csv to dataframe first 5 rows</figcaption></figure><p>Next, we need to melt the data frame to create another column to include country names so that we can add the values of ‘confirmed’, ‘recovered’, and ‘death’ cases as three separate columns to come up with our desirable structure.</p><ol><li>melt each data frame separately:</li></ol><pre>cols = new_data_conf.columns.tolist()</pre><pre>new_data_conf = new_data_conf.melt(id_vars=&#39;Date&#39;, value_vars=cols[2:])</pre><p>2. Join them by using merge one by one:</p><pre><strong>new_df</strong> = pd.merge(<strong>new_data_conf</strong>, <strong>new_data_reco</strong>,  how=&#39;left&#39;, left_on=[&#39;Date&#39;,&#39;Country/Region&#39;], right_on = [&#39;Date&#39;,&#39;Country/Region&#39;])</pre><pre><strong>final_df</strong> = pd.merge(<strong>new_df</strong>, <strong>new_data_death</strong>,  how=&#39;left&#39;, left_on=[&#39;Date&#39;,&#39;Country/Region&#39;], right_on = [&#39;Date&#39;,&#39;Country/Region&#39;])</pre><p>We get our desirable data frame 💪</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*OkDGLF-v5Dlg6j6hVvPjSg.png" /><figcaption>five-columns desirable data frame</figcaption></figure><p>There’s one last step remained and that is drawing the actual <strong><em>Motion Chart</em></strong>. But before that, to get a better outcome, I would like to filter data based on countries with highest number of ‘Confirmed’ cases.</p><pre>top_countries = final_df.loc[final_df[&#39;Confirmed_Cases&#39;] &gt; 15000]<br>top_countries_set = set(top_countries[&#39;Country/Region&#39;])</pre><blockquote>The output list reflects the list of countries as of this date (Apr 15th).</blockquote><p>3. select rows:</p><p>Having the list of selected countries into a list as a set(), I used loc() function to select required fields only and save them in top_countries_df data frame.</p><pre>top_countries_df = final_df.loc[final_df[&#39;Country/Region&#39;].isin(top_countries_set)]<br>top_countries_df</pre><h3>Motion Chart</h3><p>to draw the motion chart, we need to fill in the hyperparameters listed below and the rest is with the MotionChart library that we import</p><pre>from motionchart.motionchart import MotionChart</pre><pre>mChart = MotionChart(df = top_countries_df, key=&#39;Date&#39;, x=&#39;Confirmed_Cases&#39;, y=&#39;Recovered_Cases&#39;, xscale=&#39;linear&#39;, yscale=&#39;linear&#39;, size=&#39;Deaths&#39;, color=&#39;Country/Region&#39;, category=&#39;Country/Region&#39;)</pre><p>key: ‘Date’ would be the main drive of my chart</p><p>xaxis:chose ‘confirmed’ cases</p><p>yaxis: chose ‘recovered’ cases</p><p>Bubble size: chose the number of ‘death’s in each scenario</p><p>category: The country list</p><p>These were my preference and anyone can, definitely, make any change based on their taste.</p><h4>Publish</h4><p>To publish the chart, it depends on your environment. As I was using a Jupyter Notebook, I used .to_notebook(). To publish the same result on web, you need to use .to_browser().</p><pre>mChart.to_notebook()</pre><p>And here is final result:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*htmw9vB1BBvAVLOVkSb-jA.png" /><figcaption>Final graph as of Apr 17 2020</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*xIdWDU5e-WwvHCl4MzbX0Q.png" /><figcaption>Final graph including the sliding bar opened</figcaption></figure><p>Please comment below should you have any questions/feedback/comments.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/480/1*XK0PLI-onm8LAL4Kv-jeng.gif" /><figcaption>source: giphy.com</figcaption></figure><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=a55afe359659" width="1" height="1" alt=""><hr><p><a href="https://medium.com/analytics-vidhya/covid-19-genetic-mutation-or-engineering-a55afe359659">Covid-19: Genetic Mutation or Engineering</a> was originally published in <a href="https://medium.com/analytics-vidhya">Analytics Vidhya</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Financial Transaction Fraud Detection]]></title>
            <link>https://medium.com/analytics-vidhya/financial-transaction-fraud-detection-6e1384169398?source=rss-19c9d8bdde6------2</link>
            <guid isPermaLink="false">https://medium.com/p/6e1384169398</guid>
            <category><![CDATA[xgboost]]></category>
            <category><![CDATA[fraud-detection]]></category>
            <category><![CDATA[tensorflow-keras]]></category>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[machine-learning]]></category>
            <dc:creator><![CDATA[Maziar Izadi]]></dc:creator>
            <pubDate>Mon, 09 Mar 2020 12:33:23 GMT</pubDate>
            <atom:updated>2020-03-09T12:33:23.838Z</atom:updated>
            <content:encoded><![CDATA[<h4>Logistic Regression, TensorFlow Keras, or XGBoost</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/540/1*7wHNa_9rn11reuxetp03Dg.gif" /><figcaption>Source: giphy.com</figcaption></figure><ul><li><a href="https://www.auspaynet.com.au/sites/default/files/2019-08/AustralianPaymentCardFraud2019_0.pdf">Australia lost <strong>$574 Million</strong></a> on fraudulent transactions in 2018, out of which $487.5 Million occurred by card-not-present channels.</li><li>In the same year, <a href="https://shiftprocessing.com/credit-card-fraud-statistics/"><strong>$24.26 Billion</strong> was lost</a> due to payment card fraud worldwide.</li><li>In <a href="https://www.ukfinance.org.uk/system/files/Fraud%20The%20Facts%202019%20-%20FINAL%20ONLINE.pdf">FRAUD THE FACTS 2019</a>, the UK government reported that “unauthorised financial fraud losses across payment cards, remote banking and cheques totalled <strong>£844.8 million</strong> in 2018, an <strong>increase of 16 per cent 😱</strong> compared to 2017”.</li></ul><p>Above mentioned matters explicitly express the substantial importance of fraud detection capabilities in banking and financial sector. Nonetheless, it is unfortunate that financial institutions are reluctant to switch to more advanced technologies such as machine-learning and deep-learning engines due to restrictions posed by regulatory parties and banks have sticked to original (and fair-to-say well-proven) rule-based systems.</p><p>Hopefully🤞, thanks to recent computing enablement and data availability, we’ll soon witness changes in this trend.</p><p>Here, I’m going to compare the performance of multiple tools, which are also mathematically different, in detecting fraudulent transactions. For that, I have used dataset provided by <a href="https://www.kaggle.com/mlg-ulb">Machine Learning Group — ULB</a> as part of <a href="https://www.kaggle.com/mlg-ulb/creditcardfraud">Credit Card Fraud Detection Data on Kaggle</a>.</p><p>The datasets contains transactions made by credit cards in September 2013 by European cardholders which presents transactions that occurred in two days, where we have 492 frauds out of 284,807 transactions.</p><blockquote><strong>The dataset is highly unbalanced</strong>, the positive class (frauds) account for <strong>0.172% of all transactions</strong>. This matter is the root of extreme complexity in this topic — extremely imbalanced data that I have explained how to overcome further down.</blockquote><figure><img alt="" src="https://cdn-images-1.medium.com/max/500/1*cZsud-l2f--Pm4KZujdpZQ.gif" /><figcaption>Source: Giphy.com</figcaption></figure><p>Notes on the current article:</p><ul><li>For the first time, I used Google Colab to write Python code. However, I can’t highlight any wow in the experience as oppose to Jupyter Notebook. The reason I used Colab was TensorFlow mainly.</li><li><strong>Logistic Regresion</strong>, <strong>tf.Keras</strong>, and <strong>Xgboost</strong> algorithms are used to predict fraudulent transaction and <strong>results are compared in terms of precision and recall</strong>.</li></ul><blockquote>Complete python code is available on <a href="https://github.com/maziarizadi/KaggleFraudDetection/blob/master/FraudDetect-MultipleSolutions-Compare.ipynb">my Github</a>.</blockquote><p>Let’s jump into it 😏</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/480/1*CYjIXFQFE7gWFH_XFf_2-A.gif" /><figcaption>Source: Giphy.com</figcaption></figure><h3>1. Import data from Google Drive</h3><p>Load in the csv file from my Google Drive and save into a Pandas data frame.</p><pre># Code to read csv file into Colaboratory:</pre><pre>!pip install -U -q PyDrive</pre><pre>from pydrive.auth import GoogleAuth</pre><pre>from pydrive.drive import GoogleDrive</pre><pre>from google.colab import auth</pre><pre>from oauth2client.client import GoogleCredentials</pre><pre>------</pre><pre># Authenticate and create the PyDrive client.</pre><pre>auth.authenticate_user()</pre><pre>gauth = GoogleAuth()</pre><pre>gauth.credentials = GoogleCredentials.get_application_default()</pre><pre>drive = GoogleDrive(gauth)</pre><p>In the process of importing the csv file from Google Drive, you need to enter the file ID . Look at the 3rd method introduced <a href="https://towardsdatascience.com/3-ways-to-load-csv-files-into-colab-7c14fcbdcb92">here</a> if you’d like to learn the details.</p><pre>#you need to enter your file id</pre><pre>id = &#39;1grwIZR_LdcdyirULSoJ_VFhtuPpv00AB&#39;</pre><pre>----<br>downloaded = drive.CreateFile({&#39;id&#39;:id})</pre><pre>downloaded.GetContentFile(&#39;Filename.csv&#39;)</pre><pre>----<br>data = pd.read_csv(&#39;Filename.csv&#39;)</pre><pre># Dataset is now stored in a Pandas Dataframe</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/376/1*LhcfOWEydECa9DZMyuRUag.png" /></figure><p>Our data has <strong>284,807 records in 31 columns</strong> out of which 30 columns encompass independent variables which theoretically explain the changes in our dependant variable. In our scenario, dependant variable is a binary column to show whether a transaction was fraudulent or genuine.</p><p>Based on the data description (image below), there’s a variety of ranges in our data set. As a result, Data Normalisation is required to change the values of numeric columns in the dataset to a common scale.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*hyYAqhZqdgPmDuD6kKTDSQ.png" /><figcaption>Dataframe description</figcaption></figure><h3>2. Scaling the data frame</h3><p>I separated dependant variable from independent variables. Remember that normalisation is only done on independent variables.</p><pre>X_data = data.iloc[:,0:30]</pre><pre>y_data = data.iloc[:,-1]</pre><p>… and then, used <strong>Standard Scaler</strong> to normalise<strong> training</strong> dataset.</p><pre>standard_scaler = preprocessing.StandardScaler()</pre><pre>X_standard_scaled_df = standard_scaler.fit_transform(X_data)</pre><h3>3. Feature Extraction</h3><p>Obviously in ML algorithms you are dealing with the variance-bias trade-off and it’s always a challenge to overcome overfitting problem. I used a couple of methods and found out <strong>Principal Component Analysis</strong> contributed to the best output in terms of dealing with the trade-off.</p><p>Towards Data Science has provided a brief and effective explanation on <a href="https://towardsdatascience.com/a-one-stop-shop-for-principal-component-analysis-5582fb7e0a9c"><strong>PCA</strong></a><strong>. </strong>If you are on your outset to become a data scientist, I suggest you have a look…good stuff!</p><pre># Make an instance of the Model</pre><pre>pca = PCA(10)</pre><pre><br># fit and transform data frame in one jump</pre><pre>pca_selected = pca.fit_transform(X_standard_scaled_df)</pre><p>Result of PCA is interesting! There are 10 features extracted out of the function. I convert the result into a Pandas data frame and have a look at the first 5 rows.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*kaRVNzMxyMzCzIak7NT88A.png" /><figcaption>data set first 5 rows</figcaption></figure><h3><strong>4. Train and Test Data Split</strong></h3><p>As mentioned before, the trickiest point about fraud data sets is the extreme imbalanced distribution of positive and negative instances. For instance, let’s have a look at our current data set by using .value_counts() function and then illustrating it.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*HjsdLXAvFuKruxTNM9a0CQ.png" /><figcaption>positive and negative value counts</figcaption></figure><blockquote>Results show that there are only 492 instances of fraudulent transactions out of the total 284,807 records which holds only %0.1727 of all the samples.</blockquote><p><em>There it goes splitting an extremely imbalanced data set…</em></p><p>Since I need to make sure that the handful number of positive class is distributed proportionally in both training and test data, I take the following steps:</p><ol><li>separate all the positive records from negative</li></ol><pre>data_class_0 = ready_data[ready_data[&#39;Class&#39;]==0]</pre><pre>data_class_1 = ready_data[ready_data[&#39;Class&#39;]==1]</pre><p>2. spilt each class into train and test set, %67 to %33 respectively</p><pre># Since the number of fraud transactions are too little in compare to non-fraud,</pre><pre># I make sure that they are distributed proportionally in both train and test set</pre><pre>X_0 = data_class_0.iloc[:,0:-1]  #independent columns</pre><pre>y_0 = data_class_0.iloc[:,-1]    #target column i.e Class</pre><pre>X_1 = data_class_1.iloc[:,0:-1]  #independent columns</pre><pre>y_1 = data_class_1.iloc[:,-1]    #target column i.e Class</pre><pre>X_train_0, X_test_0, y_train_0, y_test_0 = train_test_split(X_0, y_0, test_size=0.33, random_state=42)</pre><pre>X_train_1, X_test_1, y_train_1, y_test_1 = train_test_split(X_1, y_1, test_size=0.33, random_state=42)</pre><p>3. Join them back to have one train and one test set which proportionally encompass both positive and negative class</p><pre>X_train = pd.concat([X_train_0, X_train_1])</pre><pre>y_train = pd.concat([y_train_0, y_train_1])</pre><pre>X_test = pd.concat([X_test_0 , X_test_1])</pre><pre>y_test = pd.concat([y_test_0 , y_test_1])</pre><p>At this point, if we look at data set, we have have the following:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/946/1*LWkUQEy3OLVnIHgOTilfxA.png" /><figcaption>Train and Test data sets</figcaption></figure><p>Considering the imbalanced data set, next step is to balance our independent training set X_train .</p><h3>5. Balance training data set</h3><p>Before getting into what I did, you might like to have a look on the concept of <strong>Over-sampling</strong> and <strong>Under-sampling</strong> in general. <a href="https://machinelearningmastery.com/random-oversampling-and-undersampling-for-imbalanced-classification/">Here</a> is a good explanation by “Machine Learning Mastery”.</p><p>I used <strong>SMOTE()</strong> function for balancing current model. In a nutshell, SMOTE corresponds to the desired ratio of the number of samples in the minority class over the number of samples in the majority class after resampling. Check <a href="https://imbalanced-learn.readthedocs.io/en/stable/generated/imblearn.over_sampling.SMOTE.html">here</a> for full documentation. To do so, we simply enter:</p><pre>sm = SMOTE(random_state=42)</pre><pre>X_res, y_res = sm.fit_resample(X_train, y_train)</pre><p>Now if we compare the dataset before and after <strong>SMOTE</strong>, here we see the magic.</p><pre>Output:</pre><pre>Original dataset shape Counter({0: 190491, 1: 329})</pre><pre>Resampled dataset shape Counter({0: 190491, 1: 190491})</pre><p>At this stage, we have (…finally 😫) got data set ready for modelling.</p><h3>6. Fraud Detection Model</h3><p>For this purpose, I used 3 algorithms and compared their results:</p><h4>6.1. Logistic Regression</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/571/0*iMrH_O2eSitdg33y.png" /><figcaption><a href="https://www.saedsayad.com/logistic_regression.htm">Source</a></figcaption></figure><p>As discussed above, logistic regression is the most highly accepted and used algorithm among the others in the real-life banking industry.</p><pre>logisticRegr = LogisticRegression()</pre><pre>logit_model = logisticRegr.fit(X_train, y_train)</pre><pre>logit_predict = logisticRegr.predict(X_test)</pre><p>Here is the accuracy output of Logistic Regression:</p><pre>In:<br>print(classification_report(y_test, logit_predict))</pre><pre>Out:</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*AjcDStc4jWsHmZSEvAHdAA.png" /><figcaption>Logistic Regression classification report</figcaption></figure><h4>6.2. Deep Learning Neural Network — TensorFLow.Keras</h4><figure><img alt="Image result for deep learning keras png" src="https://cdn-images-1.medium.com/proxy/1*ZMzAVt_1ZL4AaN-hGW1-0w.png" /></figure><p>The second algorithm is the Artificial Neural Network for which I used TensorFLow Keras.</p><blockquote><strong>Input and hidden layers</strong></blockquote><ul><li>Looking at the code on my Github, you’ll see that I tried building the neural network twice. Once, I set the units = 10 for hidden layers and second time, set it =32 (which is more common practice). The network with 32 units of hidden layer resulted in higher accuracy.</li><li>I did also play with the number of epochs which is another influential hyper-parameter for managing overfitting problem. I found 10 epochs end up with an overfitted model which 5 was reasonably acceptable.</li><li>The other hyper-parameter is Activation Function. At the most basic level, an activation function decides whether a <strong>neuron</strong> should be fired or not. It accepts the weighted sum of the inputs and bias as input to any activation function. <strong><em>Step function</em></strong>, <strong><em>Sigmoid</em></strong>, <strong><em>ReLU</em></strong>, <strong><em>Tanh</em></strong>, and <strong><em>Softmax</em></strong> are examples of activation functions. MissingLink has provided a good summary of their story in <a href="https://missinglink.ai/guides/neural-network-concepts/7-types-neural-network-activation-functions-right/#:~:text=Role%20of%20the%20Activation%20Function%20in%20a%20Neural%20Network%20Model&amp;text=The%20activation%20function%20is%20a,on%20a%20rule%20or%20threshold.">7 Types of Neural Network Activation Functions</a>.</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/844/1*Kt-itKi8N15mTxskzt96aA.png" /><figcaption>ANN Activation functions</figcaption></figure><blockquote><strong>Output layer</strong></blockquote><p>The output will be a one binary layer for which I set the Sigmoid(👆) activation function.</p><pre># Initialising the ANN<br>classifier = keras.Sequential()</pre><pre><br># Adding the input layer and the first hidden layer<br>classifier.add(keras.layers.Dense(units =32 , kernel_initializer = &#39;uniform&#39;, activation = &#39;relu&#39;, input_dim =10))</pre><pre># Adding the output layer<br>classifier.add(keras.layers.Dense(units = 1, kernel_initializer = &#39;uniform&#39;, activation = &#39;sigmoid&#39;))</pre><pre># Compiling the ANN</pre><pre>classifier.compile(optimizer = &#39;adam&#39;, loss = &#39;binary_crossentropy&#39;, metrics = [&#39;accuracy&#39;])</pre><pre># And finally<br># Fitting the ANN to the Training set<br>model = classifier.fit(X_train.values, y_train.values, batch_size = 128, epochs = 5)</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*du_ga3JwoyRiDvCk1taxxg.png" /><figcaption>Model is being trained through 5 Epochs</figcaption></figure><pre># Predicting the Test set results<br>y_pred = classifier.predict(X_test)<br>y_pred = (y_pred &gt; 0.5)</pre><pre>Output:<br>93987/93987 [==============================] - 3s 31us/sample - loss: 0.0041 - accuracy: 0.9992</pre><p>Let’s see how our model performed</p><pre>print(classification_report(y_test, y_pred))</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1014/1*dDE9tcToc7sqEyiGqyZ2kg.png" /><figcaption>tf.Keras classification report</figcaption></figure><blockquote>A couple of extras:</blockquote><ul><li><strong>Confusion Matrix</strong></li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*z0qUxnr7mMGOtQIcxMop9w.png" /><figcaption>Keras confusion matrix</figcaption></figure><ul><li><strong>ROC AUC Diagram</strong></li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/880/1*xV8v_whjRrSSkDRNqJBFzg.png" /><figcaption>Keras ROC AUC</figcaption></figure><h4>6.3. XGBoost</h4><p>XGBoost stands for e<strong>X</strong>treme <strong>G</strong>radient <strong>B</strong>oosting and is the next level of features of the scikit-learn and R implementations, with new additions like regularisation. Again, have a look at MachineLearningMastery for <a href="https://machinelearningmastery.com/gentle-introduction-xgboost-applied-machine-learning/">further explanations</a>.</p><blockquote>XGBoost is,</blockquote><blockquote>an algorithm that has recently been dominating applied machine learning and Kaggle competitions for structured or tabular data.</blockquote><blockquote>an implementation of gradient boosted decision trees designed for speed and performance.</blockquote><p>There are two important hyper-parameters that you need to have eyes on which are learning_rate and n-estimators. They help you challenge overfitting problem and improve accuracy. My learning was %1 learning rate produced and better “recall” as oppose to %10. And to be honest, I believe 10,000 n_estimators was a overkill but I did it anyway 😅.</p><pre># Learning rate = 0.01</pre><pre>XGB_classifier = XGBClassifier(n_estimators=10000, learning_rate=.01, maximize=True)</pre><pre>XGB_classifier.fit(X_train,y_train, eval_metric = &#39;aucpr&#39;)</pre><p>Output:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Wx80AA546FCgxTsUTmdGTQ.png" /><figcaption>Xgboost trained model</figcaption></figure><p>Next is using the model to predict dependant variables based on independent test data…</p><pre>XGB_classifier_predict_smote = XGB_classifier.predict(X_test)</pre><p>… and compare the prediction with actual dependent test set.</p><pre>classification_report(y_test,XGB_classifier_predict_smote)</pre><p>Finally goes the result 👇</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/946/1*36ycMY8qd7YLyhxht4fsGw.png" /><figcaption>Xgboost classification report</figcaption></figure><h3>Comparing results and <em>conclusion</em></h3><blockquote>“what is the best accuracy measure in our scenario?”</blockquote><p><strong>hmmm… 🤔 any idea?!</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/480/1*h4t_Eb3-Ku-ab7DuwQtW8A.gif" /><figcaption>Source: Giphy.com</figcaption></figure><p>Why I even posed this question? Because remember…banking is a sensitive industry and there’s huge loss every year.</p><p>To answer this question, I would like to refer you back to our friends who build the Confusion_Matrix… <strong>the four (in)famous TP, FP, TN, and FN</strong>.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/723/0*8-X_-oZHMr9AkAa9.png" /></figure><p>If you think about which one of the four is most important to us, you’d know what the best accuracy measure is. Basically, we are dealing with fraudulent transaction, it’s critically important that we do not flag a fraud transaction as genuine. The opposite (flagging a non-fraud as fraud) is also costly but not as critical.</p><p>Therefore, <strong>priority would be minimising the number of False Negatives.</strong> Meaning to say that <em>minimising the number of “fraudulent transactions which are marked as genuine”</em>. Accordingly, we know that we have to improve <strong>Recall</strong>’s amount as high as possible.</p><p>This <a href="https://hackernoon.com/idiots-guide-to-precision-recall-and-confusion-matrix-b32d36463556">Idiot’s Guide to Precision, Recall and Confusion Matrix</a> helped me big time. You can also have a look if you feel like you’re not following what I’m saying here.</p><blockquote><strong>Comparing</strong> results of the three algorithms, we see that <strong><em>Precision</em></strong> hasn’t changed drastically while there’s been a <strong>substantial improvement </strong>in <strong><em>Recall</em></strong> from Logistic Regression to tf.Keras and from tf.Keras to XGBoost.</blockquote><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Ym5Uu7sSpbWct63vVIKjxw.png" /><figcaption>Final comparison of 3 algorithms</figcaption></figure><blockquote>… and the prize🥇🏆goes to Xgboost for its humble, mathematically powerful gradient boosted engine.</blockquote><figure><img alt="" src="https://cdn-images-1.medium.com/max/400/1*8RZtRF3z0mVVoO8RxiOMbw.gif" /><figcaption>Source: Giphy.com</figcaption></figure><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=6e1384169398" width="1" height="1" alt=""><hr><p><a href="https://medium.com/analytics-vidhya/financial-transaction-fraud-detection-6e1384169398">Financial Transaction Fraud Detection</a> was originally published in <a href="https://medium.com/analytics-vidhya">Analytics Vidhya</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Easily Share your ML Model with Stakeholders via the Web]]></title>
            <link>https://medium.com/analytics-vidhya/easily-share-your-ml-model-with-stakeholders-via-the-web-f9328eccbaa6?source=rss-19c9d8bdde6------2</link>
            <guid isPermaLink="false">https://medium.com/p/f9328eccbaa6</guid>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[web-development]]></category>
            <category><![CDATA[python]]></category>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[flask]]></category>
            <dc:creator><![CDATA[Maziar Izadi]]></dc:creator>
            <pubDate>Mon, 27 Jan 2020 05:05:47 GMT</pubDate>
            <atom:updated>2020-01-27T12:49:39.482Z</atom:updated>
            <content:encoded><![CDATA[<h4>Add a few lines of Flask and expose your code on the web</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/600/1*fvkuhDujOA9tReonKgmJwg.gif" /></figure><p>As discussed in my previous story, as a Data Scientist, I’ve been wondering:</p><ol><li>Does the model need to be trained every single time someone wants to see the result?</li></ol><p>2. what if, I need to share my code with a stakeholder who doesn’t use Jupyter?</p><p>3. What if they don’t even have Python installed? How can a new dataset feed into the model and get output?</p><p>In <a href="https://medium.com/@maziarizadi/pickle-your-model-in-python-2bbe7dba2bbb">Pickle your model in Python</a>, I discussed the means of packaging your model using pickle, however, questions 2 and 3 from the list above were not answered. In this article, we cover how to easily expose our application (either a simple or the most sophisticated one) as a web app.</p><blockquote>If you haven’t yet, make sure you read <a href="https://medium.com/@maziarizadi/pickle-your-model-in-python-2bbe7dba2bbb">previous article</a> first.</blockquote><blockquote>The intention is by introducing these simple components, in the future we will be able to come up with an end-to-end solution to deploy a Machine Learning model so that users can interact with the model via their browsers in a responsive way. 😎…cool, right?</blockquote><p><strong>How?</strong></p><p>To do so, we need to use the following components:</p><ol><li>Pickle (<a href="https://medium.com/@maziarizadi/pickle-your-model-in-python-2bbe7dba2bbb"><em>previous article</em></a>)</li><li>Flask (current article)</li><li>Docker (coming next 😉)</li></ol><p>🛑 Please note that I use all my work on a Mac OS.</p><h3>Flask</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*rDX-VDncF81e0iVJ75SDMw.png" /></figure><blockquote><a href="https://flask.palletsprojects.com/en/master/tutorial/">Flask</a> is a lightweight <a href="https://wsgi.readthedocs.io/en/latest/what.html">WSGI</a> web application framework. It is designed to make getting started quick and easy, with the ability to scale up to complex applications. It has become one of the most popular Python web application frameworks.</blockquote><h4>What problem does it solve?</h4><p>Let’s say that I have a Python application (either a simple “Hello World” or a sophisticated “Spam Detection AI”), and I want to expose it on the web. Flask is my to-go 👍</p><h4>Why Flask?</h4><p>When it comes to <a href="https://jeffknupp.com/blog/2014/03/03/what-is-a-web-framework/">web frameworks</a>, there are <a href="https://hackr.io/blog/top-10-web-development-frameworks-in-2020">various discussions</a> on the web. But why Flask? 🤔</p><p>In a nutshell, Flask is extremely flexible and easy to use which makes it a <em>perfect candidate for Data Scientists like me who do not have that much of a programming background</em>. What’ s more, it is <a href="https://towardsdatascience.com/how-to-be-pythonic-and-why-you-should-care-188d63a5037e">Pythonic</a> which uses the Python constructs and data structures with clean, readable idioms.</p><h4>Let’s get started with Flask</h4><blockquote>Full documentation is available <a href="https://flask.palletsprojects.com/en/master/installation/">here</a>.</blockquote><h4>Create an environment</h4><p>Use a <strong>virtual environment</strong> to manage the dependencies for your project, both in development and in production.</p><ul><li>What problem does a virtual environment solve?</li></ul><p>The more Python projects you have, the more likely it is that you need to work with different versions of Python libraries, or even Python itself. Newer versions of libraries for one project can break compatibility in another project.</p><p>Virtual environments are independent groups of Python libraries, one for each project. Packages installed for one project will not affect other projects or the operating system’s packages.</p><p>Open a new terminal and navigate to your project folder.</p><blockquote>A cool way to navigate directly to a folder on Mac Terminal 👇👇👇</blockquote><figure><img alt="" src="https://cdn-images-1.medium.com/max/670/1*EVTQ2YgU_9oea8kv8zeZ2A.gif" /><figcaption>GIF <a href="https://techwiser.com/wp-content/uploads/2019/04/GIF-FINAL.gif">Source</a></figcaption></figure><p>Using the Terminal, create a project folder and a venv folder within:</p><pre>mkdir SpamDetect<br>cd SpamDetect<br>python3 -m venv venv</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*S3qqw7x0TkNh6Q0kblinSA.png" /></figure><p>If you browse to that folder now, you see the new venv folder is created.</p><h4>Activate the environment</h4><pre>. venv/bin/activate</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*0pKenNDdnexIjZrFaywm1w.png" /></figure><p>We can see that the venv is activated.</p><h4>Install Flask</h4><pre>pip install Flask</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*MhdLeMco91TUg6iDBo6AoQ.png" /></figure><p>Since I do already have Flask installed, it has suggested that I <strong>upgrade</strong> to the higher version.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*M9PbN6Fk9tRlwQbM3bTXqQ.png" /></figure><h4>Your Flask application</h4><p>In the current article for simplicity purpose, before jumping into the Machine Learning model and seeing complex codes all at the same time, I would like to show you a simple Flask app.</p><p>Step 1. Create a new file called hello.py with Python extension.</p><pre>$ touch hello.py</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*HNepplfOmYJWmk0nUY4lKA.png" /></figure><p>Step 2. A new file appears in your folder:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*qZxfqnH3g_m7dghxAZU1VQ.png" /></figure><p>Open the file and save the Flask app in the code.</p><pre># A minimal Flask app</pre><pre><strong>from</strong> flask <strong>import</strong> Flask, escape, request</pre><pre>app = Flask(__name__)</pre><pre>@app.route(&#39;/&#39;)<br><strong>def</strong> hello():<br>   name = request.args.get(&quot;name&quot;, &quot;World&quot;)<br>   <strong>return</strong> f&#39;Hello, {escape(name)}!&#39;</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*ZUVzyFGqdwlEQCYcJ3alYw.png" /></figure><h4>What did that code do?</h4><ol><li>First we imported the <a href="https://flask.palletsprojects.com/en/1.1.x/api/#flask.Flask"><strong>Flask</strong></a> , flask’s <a href="https://flask.palletsprojects.com/en/1.0.x/api/#flask.escape"><strong>escape</strong></a> and <a href="https://flask.palletsprojects.com/en/1.0.x/api/#incoming-request-data"><strong>request</strong></a> class.</li><li>Next we create an instance of this class. The first argument is the name of the application’s module or package. If you are using a single module (as in this example), you should use __name__ because depending on if it’s started as application or imported as module the name will be different (&#39;__main__&#39; versus the actual import name). This is needed so that Flask knows where to look for templates, static files, and so on. For more information have a look at the <a href="https://flask.palletsprojects.com/en/1.1.x/api/#flask.Flask"><strong>Flask</strong></a> documentation.</li><li>We then use the <a href="https://flask.palletsprojects.com/en/1.1.x/api/#flask.Flask.route"><strong>route()</strong></a> decorator to tell Flask what URL should trigger our function. In the documentations, you see that flask has its own <a href="https://flask.palletsprojects.com/en/1.0.x/api/#flask.escape"><strong>escape</strong></a> and <a href="https://flask.palletsprojects.com/en/1.0.x/api/#incoming-request-data"><strong>request</strong></a> format.</li><li>The function is given a name which is also used to generate URLs for that particular function, and returns the message we want to display in the user’s browser.</li></ol><p>Step 3. Export FLASK_APP env variable</p><p>To run the application, use the <strong>flask run</strong>command or <strong>python -m flask</strong>. Before you can do that you need to tell your terminal the application to work with by exporting the FLASK_APP environment variable:</p><pre>$ export FLASK_APP=hello.py<br>$ flask run</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*TJiw00eeT3LQsBcdZKLWow.png" /></figure><p>Step 4. Go to browser and enter the URL provided in your Terminal.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/646/1*0qLVjACQQecRjAmQciE7OQ.png" /></figure><h3>Flask Summary</h3><p>We learned:</p><blockquote>how to create an environment and activate it</blockquote><blockquote>how to install Flask</blockquote><blockquote>how to call a simple Flask application and see the result in browser</blockquote><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*2CYlpU8_9Snl3GdpUJlLCw.png" /></figure><h3>Upcoming Next…</h3><p>Now we know how flask can become handy when we wanna provide a UI to our ML model users. So we park flask here and in the next article move to Docker and see how Flask is included in your dockerised container.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=f9328eccbaa6" width="1" height="1" alt=""><hr><p><a href="https://medium.com/analytics-vidhya/easily-share-your-ml-model-with-stakeholders-via-the-web-f9328eccbaa6">Easily Share your ML Model with Stakeholders via the Web</a> was originally published in <a href="https://medium.com/analytics-vidhya">Analytics Vidhya</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Pre-processing of Topically Coherent Text Segments in Python]]></title>
            <link>https://medium.com/analytics-vidhya/pre-processing-of-topically-coherent-text-segments-in-python-58f9b258596c?source=rss-19c9d8bdde6------2</link>
            <guid isPermaLink="false">https://medium.com/p/58f9b258596c</guid>
            <category><![CDATA[reqular-expression]]></category>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[python]]></category>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[naturallanguageprocessing]]></category>
            <dc:creator><![CDATA[Maziar Izadi]]></dc:creator>
            <pubDate>Wed, 15 Jan 2020 12:53:37 GMT</pubDate>
            <atom:updated>2020-01-24T14:00:47.804Z</atom:updated>
            <content:encoded><![CDATA[<h3>Pre-processing of Topically Coherent Text Segments in Python 💬</h3><h4>How to use Natural Language Toolkit to pre-process a set of transcripts and convert them into numerical representations</h4><p>Complete <strong>Jupyter Notebook</strong> and files are available on my <a href="https://github.com/maziarizadi/TextPreProcessingPy"><strong>GitHub page</strong></a>.</p><h4>Introduction</h4><p>Text documents, such as long recordings and meeting transcripts, are usually comprised of topically coherent text segments, each of which contains some number of text passages. Within each topically coherent segment, one would expect that the word usage demonstrates more consistent lexical distributions than that across segments. <strong>Natural Language Processing(NLP)</strong>, more specifically, a linear partition of texts into topic segments can be used for text analysis tasks, such as passage retrieval in IR, document summarization, and discourse analysis. In the current exercise, we will review how to write a Python code to <strong>preprocess </strong>a set of transcripts and <strong>convert them into numerical representations</strong> suitable for input into <strong>topic segmentation algorithms</strong>.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*PGB0w1JZslqA-hM0xGrmJw.gif" /><figcaption><a href="https://s3.amazonaws.com/codecademy-content/courses/NLP/Natural_Language_Processing_Overview.gif">Source</a> of the image</figcaption></figure><p>The source of this article is derived from one of the assignments I fulfilled as part of my <strong>Data Science Graduate Diploma at Monash University</strong>. I have also made some changes to make the original tasks more interesting.</p><h4>What can be the Use Case and how can NLP help?</h4><p>Nowadays there are many job hunting websites including seek.com.au and au.indeed.com. These job hunting sites all manage a job search system, where job hunters could search for relevant jobs based on keywords, salary, and categories. Very often, the category of an advertised job is manually entered by the advertiser (e.g., the employer). There might be mistakes made for category assignment. <strong>As a result, the jobs in the wrong class will not get enough exposure to relevant candidate groups</strong>.</p><p>Advances in text analysis, automated job classification could become feasible and sensible suggestions for job categories can be made to potential advertisers. This can help reduce human data entry error, increase the job exposure to relevant candidates, and also improve the user experience of the job hunting site. In order to do so, we need an<strong> automated job ads classification </strong>system that is trained on existing job advertisement data set with normalized job categories and <strong>predicts the class labels of newly entered job advertisements</strong>.</p><blockquote>Current example touches on the first step in handling job advertisement text data, i.e., parsing the job advertisement text into a more appropriate format.</blockquote><p>The job advertisement data that we provide contains a significant amount of redundant information represented in the simple txt format. We should properly preprocess the job advertisement text data to improve the<br>performance of classification algorithms.</p><h3>Problem statement 💡</h3><p>We are required to write Python code to extract a set of words (e.g., unigrams) that are indicative of the content of each job advertisement, and <strong>convert each advertisement description into a numeric representation</strong>: count vector that can be directly used as the input to many of the classification algorithms.</p><h4>What are the steps we are going to take?</h4><ul><li>Extract the IDs and descriptions of all the job advertisements in<br>the data file data.txt (about 500 job advertisements).</li><li>Process and store these job advertisement text as sparse count vectors.</li></ul><p>In order to achieve the above-mentioned, we will:</p><ul><li>Exclude words with length less than 4</li><li>Remove stopwords using the provided stop words list (i.e, stopwords_en.txt )</li><li>Remove the words that appear only once in one job advertisement description, save them ( No duplication) as a txt file (refer to the required output)</li><li>Exclude those words in the generated vocabulary</li><li>Find the frequent words that appear in more than 100 advertisement<br>description, save them as a txt file (refer to the required output)</li><li>Exclude them in the generated vocabulary</li></ul><p>We will not:</p><ul><li>Generate multi-word phrases (i.e., collocations, Ngrams)</li></ul><p>By the end of the exercise, we will have several outputs listed below including their requirements:</p><p>1. vocab.txt: it contains the unigram vocabulary in the following format:word_string:integer_index</p><ul><li>Words in the vocabulary must be sorted in alphabetical order . This file is the key to interpret the sparse encoding. For instance, in the following example, word abbie is the 12th word (the corresponding integer_index = 11) in the vocabulary (note that numbers and words in the following are not indicative).</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/344/1*IamDO4qHbniGJWVAXSRuyQ.jpeg" /><figcaption>vocab.txt file output format</figcaption></figure><p>2. highFreq.txt This file contains frequent words that appear in more than 100 advertisement descriptions . In the output txt file, each line should contain only one word. The order of the unigrams is based on their frequency, i.e., the number of advertisements containing that word, from high to low.</p><p>3. lowFreq.txt This file contains the words that appear only once in one job advertisement description in the alphabetical order . In the output txt file, each line should contain one word.</p><p>4.sparse.txt Each line of this file corresponds to one advertisement. So, they start with advertisement ID . The rest of each line is the sparse representation of the corresponding description in the form of word_index:word_freq separated by a comma. The order of the lines must match the order of the advertisements in the input file.</p><p><strong>Note:</strong> word_freq here refers to the frequency of the unigram in the corresponding description rather than the whole document. For example, word number 11 (which is ‘abbie’ according to the above example) appears exactly once in the description of advertisement 12612628 (numbers are not indicative) :</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/842/1*cUkXXD2uaCQhY06U2et3gQ.png" /><figcaption>sparse.txt file output format</figcaption></figure><h3>Solution ⛳️</h3><p>So we always begin with importing required libraries. Owing to the nature of this exercise, followings are required:</p><h4>Import libraries</h4><ul><li><strong>Regular Expressions</strong></li></ul><p>First one is the Regular Expressions which they call ReGex in a short format. If you haven’t used them, I strongly suggest you pick it up and get some cool stuff done. Further down, I’ve provided some details to start with.</p><pre># Regular Expressions (ReGeX)</pre><pre>import re</pre><ul><li><strong>Natural Language Toolkit</strong></li></ul><p>NLTK is a leading platform for building Python programs to work with human language data. It provides easy-to-use interfaces such as WordNet, along with a suite of text processing libraries for <strong>classification, tokenization, stemming, tagging, parsing, and semantic reasoning, wrappers for industrial-strength NLP libraries</strong>.</p><p>nltk.probability provides classes for representing and processing probabilistic information such as FreqDist which we’ll use later.</p><pre># Natural Language Toolkit</pre><pre>import nltk</pre><pre>from nltk.probability import *</pre><pre>from nltk.corpus import stopwords</pre><ul><li><strong>Itertools</strong></li></ul><p>The Python itertools module is a collection of tools for handling iterators. Simply put, iterators are data types that can be used in a for loop. The most common iterator in Python is the list.</p><pre># Functions creating iterators for efficient looping</pre><pre>import itertools</pre><pre>from itertools import chain</pre><pre>from itertools import groupby</pre><h4>Let’s write some code 🔥</h4><p>We import data first. File named data.txt is on GitHub for your reference. I have it saved on my local computer in the same folder as my Jupyter Notebook file.</p><p>Before reading the file, we define an empty list and call it data, for convenience.</p><pre>data = []</pre><p>Then we simply read data.txt and save it in the list data. Make sure you define the<strong> encoding format</strong> utf8 , otherwise you might get an error.</p><ul><li>Sample error:</li></ul><pre>UnicodeDecodeError: &#39;charmap&#39; codec can&#39;t decode byte 0x9d in position 260893: character maps to &lt;undefined&gt;</pre><p>Another <strong>consideration </strong>is that we directly convert the text into lower for consistency purpose using .lower() function.</p><pre>with open(&#39;data.txt&#39;, encoding=&quot;utf8&quot;) as f:<br>    data = f.read().lower()</pre><h4>Formatting and cleansing ✂️ 🔨 📌</h4><p>Now, we need to begin the process of tokenizing the text. The task of breaking a character sequence into pieces is known as tokenization.</p><p>Firstly, we have to remove all the noise such as /-*#@ or any other non-word character or extra spaces from the text and we do that with the powerful ReGex tool.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*zrjjtCTZQSbnZXiMSrHJng.jpeg" /></figure><p>In order to run the formatting using ReGex, there are two steps you need to take;</p><p>(1) Create the pattern,</p><p>(2) Run the pattern using Python code and find the matches.</p><pre># (1) create a pattern for REGEX to find and keep matching words only</pre><pre>pattern = re.compile(r&quot;[a-zA-Z]+(?:[-&#39;][a-zA-Z]+)?&quot;)</pre><pre># (2)tokenise the words: match the pattern to file&#39;s content <br># and tokenize the content</pre><pre>tokenised = pattern.findall(data)</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/600/1*PIKVuAWpPhpXWfbDXiKsfQ.gif" /><figcaption>Image <a href="https://www.google.com/url?sa=i&amp;source=images&amp;cd=&amp;cad=rja&amp;uact=8&amp;ved=2ahUKEwjf4MHO4YTnAhWLyDgGHW9YDI0QjRx6BAgBEAQ&amp;url=http%3A%2F%2Fnlp.cs.tamu.edu%2F&amp;psig=AOvVaw0kkE5JmXeahMHoc6Uvi0S9&amp;ust=1579148397647336">source</a></figcaption></figure><p>There are heaps of online resource on Regex but the one I found the most interesting was <a href="https://regex101.com/">https://regex101.com/</a>. Not only does it help you match your text with the pattern, but also it provides short and sweet informative content. In Picture 1, I’ve provided a brief list of functionalities they provide on their page.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*pwcYEMluczE-AXfHdrjbcw.jpeg" /><figcaption>Picture 1, Functionalities provided by <a href="https://regex101.com/">regex101</a>.com</figcaption></figure><p><strong>Useful Regex Resources for Python:</strong></p><ul><li><a href="https://www.dataquest.io/blog/regular-expressions-data-scientists/">Python Regex Tutorial for Data Science</a></li><li><a href="https://docs.python.org/3/library/re.html">Python 3 re module documentation</a></li><li><a href="https://regex101.com/">Online regex tester and debugger</a></li></ul><h4><strong>Indexing the tokenized list 📇</strong></h4><p>Now, I’ve indexed the tokens based on id and title in each job ad:</p><pre># pass the length of the &#39;tokenised&#39; series into a variable</pre><pre>tokenised_len = len(tokenised)</pre><pre><br># indexing the tokens based on the position of &quot;id&quot; and &quot;title&quot;</pre><pre>indexes = [i for i, v in enumerate(tokenised) if v==&#39;id&#39; and i+1 &lt; tokenised_len and tokenised[i+1]==&#39;title&#39;]</pre><p>Next, we create a function from <strong>itertools </strong><a href="https://docs.python.org/3/library/itertools.html">recipes </a>which iterates through the list of tokens and <strong>creates sub list to include tokens related to one job ad only</strong>. The output will be a data dictionary.</p><pre># from itertools recipes<br>def pairwise(iterable, fillvalue=None):<br>    &quot;&quot;&quot;<br>       This function iterates through the list of tokens and <br>       creates sub list to include tokens related to one job ad only<br>    &quot;&quot;&quot;<br>    a, b = iter(iterable), iter(iterable)<br>    next(b, None)<br>    return itertools.zip_longest(a, b, fillvalue=fillvalue)</pre><pre><br># pairwise based on indexes in the last block and store in the &#39;tokenised&#39; as a list</pre><pre>tokenised = [tokenised[i:j] for i,j in pairwise(indexes)]</pre><p>To create the data dict, I used Python <strong>itertools</strong>. <strong>Jason Rigdel</strong> has introduced a good explanation and a set of examples on the topic of <strong>itertools </strong>in Python.</p><ul><li><a href="https://medium.com/@jasonrigden/a-guide-to-python-itertools-82e5a306cdf8">A Guide to Python Itertools</a></li></ul><p>However, the list contains a lot of functional words, such as “to”, “in”, “the”, “is” and so on.</p><blockquote>These functional words usually <em>do not contribute much to the semantics of the text</em>, except for increase the dimensionality of the data in text analysis.</blockquote><blockquote>Also, note that our goal, usually, is to build a prediction classification model. Thus, we are more interested in the meaning of the report than the syntax. Therefore, we can choose to remove those words, which is your next task.</blockquote><p>I will exclude all tokens less than 4 characters by keeping those which include more than 3 characters and append the rest to a list I named to_remove. This list will be added to a generic English stopwords list later.</p><pre>tokenised = [[word if len(word) &gt; 3 else &quot;to_remove&quot; for word in job] for job in tokenised]</pre><h4>Removing StopWords ✂️</h4><p>Stop words carry<em> little lexical content</em>.</p><blockquote>They are often functional words in English, for example, articles, pronouns, particles, and so on. In NLP and IR, we usually exclude stop words from the vocabulary. Otherwise, we will face the <a href="https://towardsdatascience.com/the-curse-of-dimensionality-f07c66128fe1">curse of dimensionality</a>.</blockquote><p>There are some exceptions, such as syntactic analysis like parsing, we choose to keep those functional words. However, you are going to remove all the stop words in the above list by using the stop word list in <strong>NLTK</strong>, which is:</p><pre>nltk.download(&#39;stopwords&#39;)</pre><pre>stopwords_list = stopwords.words(&#39;english&#39;)</pre><p>For this example, I have already provided the stopwords_en.txt file on my GitHub where you can download. We add to_remove list that we created above first to stopwords_en.txt file, read the file, and then save them as a set().</p><pre># adding&#39;to_removed&#39; string to the list of stopwords</pre><pre>stopwords = []</pre><pre>with open(&#39;stopwords_en.txt&#39;,&quot;a&quot;) as f:<br>    f.write(&quot;\nto_remove&quot;) #\n to shift to next line</pre><pre><br>with open(&#39;stopwords_en.txt&#39;) as f:<br>    stopwords = f.read().splitlines() #reading stopwords line and create stopwords as a list</pre><pre># convert stopwords into set</pre><pre>stopwordsset = set(stopwords)</pre><p>You might be wondering why we saved stopwords as a set . That’s a good question…Python set is better choice than a list because set runs much faster than lists in terms of searching a large number of <strong><em>hashable</em></strong> items.</p><p>Next, I’ve created a function called purifier() which essentially purifies tokens by removing stopwords , and then run the tokenised list through.</p><pre>def purifier(tokenList,remove_token):<br>    &quot;&quot;&quot;<br>        This function takes two input (list of current tokens <br>        and list of tokens to be removed)<br>        The function converts the list into set to improve the <br>        performance<br>        and return a list of sets each of which include purified <br>        tokens and remove_token lists are removed<br>    &quot;&quot;&quot;<br>    return [set(word for word in job if word not in remove_token) for job in tokenList]</pre><pre># running the &#39;purifier&#39; function</pre><pre>tokenised = purifier(tokenised,stopwordsset)</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/227/1*cWfW76Tdy3iWAoKeW5rRtg.png" /><figcaption>Pic from shutterstock library</figcaption></figure><p>Next is to remove the words that appear only once in one job advertisement description, save them ( No duplication) as a txt file (refer to the required output). In order to do that you will need to exclude those words in the generated vocabulary.</p><p>To do that, we begin by using <strong>chain()</strong> function to join all the words in all the job ads together by making a list. In “<a href="https://medium.com/@jasonrigden/a-guide-to-python-itertools-82e5a306cdf8">A Guide to Python Itertools</a>”, there’s a good explanation on how chain() function works.</p><pre>stop_wrds_removed_words = list(chain.from_iterable([word for word in job] for job in tokenised))</pre><p>Convert the list of words into set to remove duplicates and create the set of vocabulary</p><pre>stop_wrds_removed_vocab = set(stop_wrds_removed_words)</pre><p>Next is passing the words in <strong>FreqDisrt()</strong> function to count the number of token.</p><blockquote>The FreqDist class is used to encode “frequency distributions”, which count the number of times that each outcome of an experiment occurs. It is one of the classes under nltk.probability Module.</blockquote><blockquote>According to <a href="https://devopedia.org/text-corpus-for-nlp">developedia</a>, typically, each text corpus is a collection of text sources. There are dozens of such corpora for a variety of NLP tasks. This article ignores speech corpora and considers only those in text form. In our example, text corpus refers to the combination of all job ads together (… and not each job separately).</blockquote><p>The following line function counts the number of times a word occurs in the whole corpus regardless which ad it is in.</p><pre>fd = FreqDist(stop_wrds_removed_words)</pre><h4>Low Frequent Tokens</h4><p>To find the Less Frequent Tokens, I create a list of tokens which have occurred once only and convert the list into set for performance improvement.</p><pre>once_only = set([k for k, v in fd.items() if v == 1])</pre><pre># sort the set into alphabetical order</pre><pre>once_only = sorted(once_only)</pre><pre><br>set(once_only)</pre><p>To create the lowFreq.txt file, I’ve saved the sorted set of the words that appear “once only” in one job advertisement description into a file with the same name.</p><pre>out_file = open(&quot;lowFreq.txt&quot;, &#39;w&#39;)<br>for d in once_only:<br>    out_file.write(&#39;&#39;.join(d) + &#39;\n&#39;)<br>out_file.close()</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Rm8HmCWd017_LbUSKgh3Tg.png" /></figure><h4>High Frequent Tokens</h4><p>At this stage, I repeat the same steps above, however, this time the intention is to find the high frequent words and save them in file named highFreq.txt.<br>I start by removing lowFreq tokens from the list of tokens by running the purifier() function which we defined earlier.</p><pre>tokenised = purifier(tokenised,once_only)</pre><p>Next is to create a new list of words after once_only words are removed.</p><pre>LowFreqRemoved_Words = list(chain.from_iterable([word for word in job] for job in tokenised))</pre><pre><br>LowFreqRemoved_vocab = set(LowFreqRemoved_Words)</pre><pre><br>LowFreqRemoved_fd = FreqDist(LowFreqRemoved_Words)</pre><p>For high frequent words, I have selected 100 threshold. You can definitely choose any threshold accordingly to the context of your work.</p><pre>highFreq = set([k for k, v in LowFreqRemoved_fd.items() if v &gt; 100])</pre><p>Now, going to save the sorted list of high frequent words that appear in more than 100 job advertisement description to a file.</p><pre>out_file = open(&quot;highFreq.txt&quot;, &#39;w&#39;)<br>for d in highFreq:<br>    out_file.write(&#39;&#39;.join(d) + &#39;\n&#39;)<br>out_file.close()</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*6GLm05BN8s3TMBUMjQO1zw.png" /></figure><p>Once again, we run the purifier() function to remove highFreq data set and create a new list.</p><pre>tokenised = purifier(tokenised,highFreq)</pre><pre><br>HighFreqRemoved_words = list(chain.from_iterable([word for word in job] for job in tokenised))</pre><pre>HighFreqRemoved_vocab = set(HighFreqRemoved_words)</pre><h4>Note</h4><p>You might be wondering what’s the difference between words and vocab lists in my code and why every time I created a words list, then a vocab is also created. The reason goes back to the difference between list and set in Python. Yet bottom line is that in vocab each word is listed only once while words might have duplicates.</p><p>Next is just a simple checkpoint to see the progress of purification of tokens:</p><pre>print(f&quot;Length of words: {len(stop_wrds_removed_words)}&quot;)</pre><pre>print(f&quot;Length of vocab: {len(stop_wrds_removed_vocab)}&quot;)</pre><pre>print(f&quot;Length of LowFreqRemoved_Words: {len(LowFreqRemoved_Words)}&quot;)</pre><pre>print(f&quot;Length of LowFreqRemoved_vocab: {len(LowFreqRemoved_vocab)}&quot;)</pre><pre>print(f&quot;Length of HighFreqRemoved_words: {len(HighFreqRemoved_words)}&quot;)</pre><pre>print(f&quot;Length of HighFreqRemoved_vocab: {len(HighFreqRemoved_vocab)}&quot;)</pre><p>…which gives us the following output:</p><pre>Length of words: 474345<br>Length of vocab: 18619<br>Length of LowFreqRemoved_Words: 465779<br>Length of LowFreqRemoved_vocab: 10053<br>Length of HighFreqRemoved_words: 126491<br>Length of HighFreqRemoved_vocab: 9103</pre><p>Next is creating a file of all vocabs named vocab.txt.</p><pre>HighFreqRemoved_vocab = list(HighFreqRemoved_vocab)</pre><pre><br># list of final vocabs</pre><pre>vocab = {HighFreqRemoved_vocab[i]:i for i in range(0,len(HighFreqRemoved_vocab))}</pre><p>building a function to create the vocab.txt file and finally build and sort the file by calling the functions:</p><pre>def vaocab_output(file):<br>    with open (file, &quot;a&quot;) as f:<br>        for key in sorted(vocab.keys()):<br>            f.write(&quot;%s:%s\n&quot; % (key, vocab[key]))</pre><pre># calling the function to build the file</pre><pre>vaocab_output(&quot;vocab.txt&quot;)</pre><p>vocab.txt output:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*z0WliTu2MnmQ4TX-acSSkA.png" /></figure><p>Until this point, I tried to keep the code as simple as possible for practice purpose. However, for this step, I created a bit more complex, yet more efficient, piece of code. Please leave any questions in comments and I make sure they are answered.</p><p>The final activity is to sparse representation of the corresponding description in the form of word_index:word_freq separated by a comma and create the file sparse.txt.</p><pre>data = {}<br>id = None<br>with open(&#39;data.txt&#39;, &#39;r&#39;,encoding=&quot;utf8&quot;) as f:<br>    for i, line in enumerate(f): # create the iteration in the range of imported file&#39;s length<br>        line = line.lower() <br>        line = line.strip()<br>        if not line:<br>            continue<br>        section = line.split(&#39;:&#39;)[0] # define &#39;section&#39; as a method to manupilate each line based on how the line begins<br>        content = &#39;:&#39;.join(line.split(&#39;:&#39;)[1:]).strip() # define &#39;content&#39; a method to capture tokens<br>        if section == &#39;id&#39;: # id section:<br>            if id: # Error handle if theres some bad formatting: multiple ids<br>                raise ValueError(&#39;unable to parse file at line %d, multiple ids&#39; % i)<br>            id = content[1:] # capture the job id<br>            if id in data.keys():# Error handle if theres some bad formatting: duplicates<br>                raise ValueError(&#39;unable to parse file at line %d, duplicate id&#39; % i)<br>        elif section == &#39;description&#39;: #capture job description per each job ad<br>            if not id:# Error handle if theres some bad formatting: missing id<br>                raise ValueError(&#39;unable to parse file at line %d, missing id&#39; % i)<br>            content = pattern.findall(line)<br>            content = [value for value in content if len(value) &gt; 3] # remove short character token<br>            content = [value for value in content if value not in stopwordsset] # remove stopwords<br>            content = [value for value in content if value not in once_only] # remove lowFreq token<br>            content = [value for value in content if value not in highFreq] # remove highFreq tokens<br>            data[id] = content # creates data dictionary<br>            id = None<br>        elif section == &#39;title&#39;: # if the line start with &#39;title&#39; do nothing<br>            continue<br>        else:<br>            raise ValueError(&#39;unable to parse file at line %d, unexpected section name&#39; % i)</pre><p>And finally build the sparse.txt file.</p><pre>with open(&#39;sparse.txt&#39;,&quot;w&quot;) as f:<br>    for jobID,content in data.items(): # go through data dictionary created in the last block<br>        fd_parse = FreqDist(content) # count number of times each token occured in the same job ad<br>        tmp = &quot;&quot; # create a placeholder for word_index:word_freq<br>        for (x,y) in fd_parse.items(): # iterate through each frequencies<br>            tmp += f&quot;{vocab[x]}:{y},&quot; # build the dictionary of word_index:word_freq in the placeholder<br>        f.write(f&quot;#{jobID},{tmp[:-1]}\n&quot;) # write in the file line by line</pre><p>Output of sparse.txt :</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*edqWIkdSj9hkE5uELDynEQ.png" /></figure><p>Further knowledge</p><ul><li><a href="https://www.kaggle.com/adamschroeder/countvectorizer-tfidfvectorizer-predict-comments">CountVectorizer, TfidfVectorizer, Predict Comments Tutorial on Kaggle</a></li><li><a href="https://scikit-learn.org/stable/modules/generated/sklearn.feature_extraction.text.CountVectorizer.html#sklearn.feature_extraction.text.CountVectorizer">Convert a collection of text documents to a matrix of token counts</a></li></ul><p>— -end — -</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=58f9b258596c" width="1" height="1" alt=""><hr><p><a href="https://medium.com/analytics-vidhya/pre-processing-of-topically-coherent-text-segments-in-python-58f9b258596c">Pre-processing of Topically Coherent Text Segments in Python</a> was originally published in <a href="https://medium.com/analytics-vidhya">Analytics Vidhya</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Pickle your model in Python]]></title>
            <link>https://medium.com/@maziarizadi/pickle-your-model-in-python-2bbe7dba2bbb?source=rss-19c9d8bdde6------2</link>
            <guid isPermaLink="false">https://medium.com/p/2bbe7dba2bbb</guid>
            <category><![CDATA[joblib]]></category>
            <category><![CDATA[pickles]]></category>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[serialization]]></category>
            <category><![CDATA[python]]></category>
            <dc:creator><![CDATA[Maziar Izadi]]></dc:creator>
            <pubDate>Mon, 06 Jan 2020 03:55:32 GMT</pubDate>
            <atom:updated>2020-01-06T03:55:32.574Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/601/1*PPIp7twJJUknfohZqtL8pQ.png" /></figure><p>As a data scientist, I am a big fan of Jupyter Notebook as it provides a user friendly and easy-to-use UI to write your code and get feedback straight after. However, I’ve been wondering about the means of packaging my work and make it reusable.</p><ul><li>what if, I need to share my code with a stakeholder who doesn’t use Jupyter?</li><li>Does the model need to be trained every single time someone wants to see the result?</li><li>What if they don’t even have Python installed? How can a new dataset feed into the model and get output?</li></ul><p>On the other hand, different format of data such as Dictionaries, DataFrames, Lists and so on as well as modules and libraries are used in your code which might need to be saved to a file so that they can be used later on.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/482/1*RJnLo03uVPfr9ktElOmazw.jpeg" /></figure><p>I will answer to all these questions in my upcoming articles. But a solution as part of these problems came in where I came across what was called Python’s <strong>Pickle</strong>. As it is explained clearly by <a href="https://www.geeksforgeeks.org/understanding-python-pickling-example/">GeeksforGeeks</a>:</p><blockquote>Python pickle module is used for serialising and de-serialising a Python object structure. Any object in Python can be pickled so that it can be saved on disk. What pickle does is that it “serialises” the object first before writing it to file. Pickling is a way to convert a python object (list, dict, etc.) into a character stream. The idea is that this character stream contains all the information necessary to reconstruct the object in another python script.</blockquote><blockquote>The whole idea is to use the Pickle module to implement binary protocols for serialising and de-serialising a Python object structure so that the Python object is converted into a byte stream.</blockquote><p>The main benefit of Pickle for data scientists is when they are working with machine learning algorithms. Imagine that you have split your data set, used Logistic Regression to build a model and trained your model, tested it, re-trained it, tested and went through all the cycle to enhance to your desirable level of accuracy and precision.</p><ul><li>What happens when you need to make a new prediction in the future using the same model?</li></ul><p><em>Pickle helps you to be able to do that without having to rewrite everything or train the model all over again.</em></p><h3>What can be Pickled?</h3><p>In order to store data with Pickle, you should know what can be pickled directly and what are the workarounds for those which cannot directly. As it is clearly mentioned by <a href="https://www.datacamp.com/community/tutorials/pickle-python-tutorial">DataCamp</a>, the following can be pickled:</p><ul><li>Booleans,</li><li>Integers,</li><li>Floats,</li><li>Complex numbers,</li><li>(normal and Unicode) Strings,</li><li>Tuples,</li><li>Lists,</li><li>Sets, and</li><li>Dictionaries that contain picklable objects.</li><li>You can also pickle classes and functions, if they are defined at the top level of a module.</li></ul><p><strong>Not everything can be pickled (easily), though</strong>: examples of this are generators, inner classes, lambda functions and defaultdicts. In the case of lambda functions, you need to use an additional package named dill. With defaultdicts, you need to create them with a module-level function. For more info on dill, please check <a href="https://pypi.org/project/dill/">PYPI documentation</a>.</p><h3>Pickling use-cases</h3><p><a href="https://pythontips.com/2013/08/02/what-is-pickle-in-python/">PythonTips</a> has gathered a few use cases from stackoverflow which I have cited below. It is very useful when you want to dump some object while coding in the python shell. So after dumping whenever you restart the python shell you can import the pickled object and deserialize it.</p><ol><li>Saving a program’s state data to disk so that it can carry on where it left off when restarted (persistence)</li><li>Sending python data over a TCP connection in a multi-core or distributed system (marshalling)</li><li>Storing python objects in a database</li><li>Converting an arbitrary python object to a string so that it can be used as a dictionary key (e.g. for caching &amp; memoization).</li></ol><h4>Other advantages of using Pickle Module, cited <a href="https://www.geeksforgeeks.org/understanding-python-pickling-example/">from</a>:</h4><ol><li><strong>Recursive objects (objects containing references to themselves):</strong> Pickle keeps track of the objects it has already serialised, so later references to the same object won’t be serialised again. (The marshal module breaks for this.)</li><li><strong>Object sharing (references to the same object in different places):</strong> This is similar to self- referencing objects; pickle stores the object once, and ensures that all other references point to the master copy. Shared objects remain shared, which can be very important for mutable objects.</li><li><strong>User-defined classes and their instances:</strong> Marshal does not support these at all, but pickle can save and restore class instances transparently. The class definition must be importable and live in the same module as when the object was stored.</li></ol><h3>Pickle vs cPickle vs Marshal vs Joblib vs JSON</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*ZKC28qWwE3Fz0oK39ojJ4w.jpeg" /><figcaption>A comparison between Pickle and other similar tool</figcaption></figure><p>The cPickle module implements the same algorithm as pickle, in C instead of Python. It is many times faster than the Python implementation (some reference argue 1000 times), but does not allow the user to subclass from Pickle. If subclassing is not important for your use, you probably want to use cPickle.</p><p>For more info on Joblib, I suggest reading their documentations <a href="https://joblib.readthedocs.io/en/latest/">here</a>.</p><h3>How to Pickle files</h3><p>As usual, you have to start pickling by importing it</p><pre>import pickle</pre><p>For this article, I am just going to keep it simple with an easy example. However, I will write another article soon after this to introduce flask and how it makes showcasing super convenient for data scientist (stay tuned, very fun stuff). In the future article, you will see a more complex use of pickle.</p><p>For now, let’s assume that we have a fitted Linear Regression model and we’ve called it regressor using the following sample:</p><pre>regressor = LinearRegression()</pre><pre>#Fitting model with training data<br>regressor.fit(X, y)</pre><p>now we want to save the model to disk. We simple use the dump() function in pickle and save the model, as follow:</p><pre>pickle.dump(regressor, open(&#39;model.pkl&#39;,&#39;wb&#39;))</pre><p>There are a couple of notes that you must pay attention to.</p><ol><li>A file name has to be passed in the function. This file name will be used by the code to generate the pickled file. File name does not necessarily require a .pkl extension. It can also be written as open(&#39;model&#39;,&#39;wb&#39;)</li><li>When using the open() function, &#39;wb&#39; is required. As explained on top, pickle module uses binary protocol and as a result, the file mode should also be in write and binary format.</li></ol><p>In addition to that, dump function is used to write the pickled representation of the object (the model in our example) to the open file. Dump format is as below and for further details you can refer to Python docos <a href="https://docs.python.org/3/library/pickle.html">here</a>.</p><pre>pickle.<strong>dump</strong>(<em>obj</em>, <em>file</em>, <em>protocol=None</em>, <em>*</em>, <em>fix_imports=True</em>, <em>buffer_callback=None</em>)</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/283/1*FE2xzFzOb66O4tjlA_Z0Nw.jpeg" /></figure><h3>De-pickling files</h3><p>Let’s say that I have my model pickled (or received a pickled file from another data scientist) and need to load it back into my Python program.</p><p>It is very simple and similar to what we saw earlier in pickling process. We need to use the open() function again, but this time with &#39;rb&#39; as second argument. Obviously, r stands for read mode and b is binary, like before.</p><p>The function which is used to de-pickle the file is load() from the pickle library. As below:</p><pre># Loading model to compare the results<br>model = pickle.load(open(&#39;model.pkl&#39;,&#39;rb&#39;))</pre><h3>In a nutshell</h3><p>As a data scientist, you need to serialise your code (including multiple components) for several reasons such as saving your fitted model on the disk. Python has provided the pickle library which makes the life much easier for data scientists who work with ML algorithms all the time. Using pickle, simply save your model on disc with dump() function and de-pickle it into your python code with load() function. Use open() function to create and/or read from a .pkl file and make sure you open the file in the binary format by wb for write and rb for read mode.</p><h3>Upcoming next</h3><ul><li>Joblib and how to use it</li><li>Python Flask — Model output presentation has never been so easy</li></ul><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=2bbe7dba2bbb" width="1" height="1" alt="">]]></content:encoded>
        </item>
    </channel>
</rss>