<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Maksym Balatsko on Medium]]></title>
        <description><![CDATA[Stories by Maksym Balatsko on Medium]]></description>
        <link>https://medium.com/@mbalatsko?source=rss-fa6012a510db------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/1*I-WD3U0HRYyGTxBY4cobRQ@2x.jpeg</url>
            <title>Stories by Maksym Balatsko on Medium</title>
            <link>https://medium.com/@mbalatsko?source=rss-fa6012a510db------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Wed, 18 Nov 2026 05:05:19 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@mbalatsko/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[Your ML code is well structured… Or is it?]]></title>
            <link>https://medium.com/data-science/your-ml-code-is-well-structured-or-is-it-c1ef49aa5ba1?source=rss-fa6012a510db------2</link>
            <guid isPermaLink="false">https://medium.com/p/c1ef49aa5ba1</guid>
            <category><![CDATA[pipeline]]></category>
            <category><![CDATA[project-structure]]></category>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[software-development]]></category>
            <category><![CDATA[machine-learning]]></category>
            <dc:creator><![CDATA[Maksym Balatsko]]></dc:creator>
            <pubDate>Fri, 20 Mar 2020 19:06:06 GMT</pubDate>
            <atom:updated>2020-03-20T19:06:06.124Z</atom:updated>
            <content:encoded><![CDATA[<h4>How to structure code for your Machine Learning projects</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*_oMHIBYHvizpE0aMnBI-lw.jpeg" /><figcaption>Photo by <a href="https://unsplash.com/@seiseisei?utm_source=medium&amp;utm_medium=referral">Seika I</a> on <a href="https://unsplash.com/?utm_source=medium&amp;utm_medium=referral">Unsplash</a></figcaption></figure><p>I came to Machine Learning sphere from Web development. And I was really upset when I realized that there aren’t any clearly defined principles of how the project should be structured. Maybe it is better to say that it is <strong>nearly</strong> <strong>impossible</strong> to find best practices of project structure and entire application modules interrogation for this domain-specific sphere (Machine learning/Data Science). The information about MVC is much easier to find. Now I’ve been working on ML projects for a while, and in this post I’ll try to sum up my experience and knowledge concerning this topic.</p><p>Let’s begin with the basics. In most cases, the ML project consists of <strong>the core model </strong>and some <strong>wrapper</strong> (<em>GUI, HTTP API, CLI interface</em>) through which the end-user will use it. Let’s concentrate on the core <strong>model</strong> part and leave the wrapper for the guys that already know what the have to do : ) We all know that software development process in ML world is in most cases dictated by <strong>CRISP-DM</strong> (Cross-industry standard process for data mining) or its elder brother from IBM <strong>ASUM-DM</strong> (Analytics Solutions Unified Method for Data Mining/Predictive Analytics).</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*JYbymHifAk7aQ1pHm_IdMQ.png" /><figcaption>CRISP-DM (<a href="https://www.datasciencecentral.com/profiles/blogs/crisp-dm-a-standard-methodology-to-ensure-a-good-outcome">source</a>)</figcaption></figure><p>From these two approaches, we can highlight 3 main SW development phases: <strong>Data preparation, Modeling, and Evaluation</strong>, which are cycled during the whole of the application existence. No matter if you are only starting the project or you are adding a new feature, you have to pass all these three stages. Ok, now let’s define what is core model or what I mean when I say the core model?</p><p><strong>The core model</strong> is an algorithm, which performs finite and well defined subsequent data transformations of the input, with clearly defined data structure, and as the result returns output, with the clearly defined data structure. I think this definition suits practically all ML projects. So now let’s define how this algorithm is affected on all 3 stages.</p><p><strong>Data preparation</strong>: here we have to validate and clean data we got from the client(or from elsewhere). Actually what we are doing is deciding how the input of the algorithm would look like.</p><p><strong>Modeling</strong>: here we have to define data transformation steps, which would be performed on the input data and on the output will return data in the required structure. Important note: here we create not only what we are used to call the ML model (kNN, Logistic regression, etc.), but also data preprocessing, feature engineering and other possible data transformation steps. Defacto anything that we are used to calling the “ML model” is nothing more than data transformer.</p><p><strong>Evaluation</strong>: here we have to evaluate using some metric the relevance of the algorithm outputs.</p><p>Uff, now we have it defined. So what about code? How these three stages affect the codebase?</p><p>When you are doing <strong>data preparation</strong>, you should always expect that the data should be changing in time and the problem you are currently solving might be extended. <strong>Example</strong>: you have to write a model that will detect particles of 2 types on the microscope images. You have somehow prepared data and said to the customer to send the input to the model in some format. The thing you have to think of before making the final decision is: <em>will it work if the customer would like to add a new type of particles for detection?</em></p><p>When you are doing <strong>modeling</strong>, you should always expect that the results of the evaluation or the results in the production might be bad and you’ll have to make drastic changes in the model and you’ll have to do them as quickly as possible. <em>How long would take to change the model structure?</em></p><p>When you are creating the <strong>evaluation</strong> code you should always expect that evaluation metric, data on which evaluation is performed or even model structure might change. <em>How long would it take to change these things and run evaluation again?</em></p><p>So now we see that a lot of sh*t might happen during the development and our main aim is to tackle it as quickly as we can. So how we should write our code to be able to do that easily? We all know the main code quality metrics in SW development are <em>Reliability</em>, <em>Maintainability</em>, <em>Testability</em>, <em>Portability</em>, <em>Reusability</em>. But these buzz words could mean anything without being domain-specific.</p><p>We have already defined the core model and I think that the best technical realization of such a model is the <strong>pipeline</strong>. The <strong>pipeline</strong> is a set of data processing steps connected in series, where the output of one step is the input of the next one. The pipeline can be constructed from several pipelines.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/571/1*c5KASzqI0A5UQJQqsm3JQA.png" /><figcaption>Text preprocessing pipeline example</figcaption></figure><p>Now I want to name and describe the main principles which I have defined for myself and which I think have to be fulfilled by every Machine Learning/Data Science project:</p><ol><li><strong>Pipeline consists of clearly defined and ordered data processing steps. </strong>Data has a clear processing path with a strictly defined order. This partially guarantees application output stability (<em>Reliability</em>)</li><li><strong>High-level pipeline structure transparency</strong>. Any developer unfamiliar with the project has to be able to understand (or maybe better write down an ordered list of) all steps, which are performed in the pipeline of the application. This increases the maintainability of the entire application and vastly simplifies the onboarding of new people to the project. (<em>Maintainability</em>)</li><li><strong>Data processing steps atomicity</strong>. Every step has a clearly defined data structure of its input and output, and its internal state is isolated from other steps. This simplifies pipeline steps unit testing. Also, it guarantees the possibility of the pipeline step reusability in the different places of the pipeline or in another pipeline. (<em>Maintainability, Testability, Reusability</em>)</li><li><strong>Accessibility of pipeline step parameters</strong>. Any parameter of every data processing step has to be easily accessible and possible to be modified. This is very important both for manual and automated model parameters tuning and evaluation. It simplifies the data processing step unit testing. And also provides reusability with other settings, where it might be useful. (<em>Maintainability</em>)</li><li><strong>Possibility of addition and deletion of pipeline steps</strong>. Every step of the pipeline can be deleted if the output of the previous step and the input of the next step have the same structure. A new data processing step can be added between two steps if the input and the output of a new step are compatible with their inputs and outputs. This speeds up the process of modeling the entire data processing flow and in the future speeds up corrections of an existing model according to the feedback. (<em>Maintainability</em>)</li></ol><p>I think following these 5 rules will help you to write<strong> clean, testable and maintainable code</strong> for your ML projects.</p><p>In case ofsklearn , you can use its Pipeline implementation, it is really awesome. If you want to have some transformer, which is not in sklearn , just google how to write custom transformer and integrate it into your pipeline, it is really easy.</p><p>If you use something else than sklearn you have two options: transform your solution to sklearn or to write your own pipeline interface, all rules it has to fulfill you got on the table. No matter you are working with textual, numeric, image or mixed data, these rules are universal.</p><p>To match words with action, I’ll show how these <strong>principles</strong> work in practice. Let’s imagine we have a <strong>dataset of posts</strong>. Every post has an image, text, category, tags, rating from critics, authors rating, price. Let’s imagine that image and text fields contains filename from which we have to retrieve data. Our aim is to estimate the posts’ price. And I’d like you to understand that this example <em>doesn’t have any connection with reality</em>. I use this example to show you, that you can combine any amount of tools with different interfaces and purposes, and connect them into the <strong>pipeline, which fulfills all aforementioned rules.</strong> Let’s imagine that for simplicity we have chosen stack: <em>pandas</em>, <em>sklearn</em>, <em>spacy</em> for NLP of text attribute, <em>OpenCV</em> for image processing and <em>YOLO</em> for object detection on the image.</p><p>Now I’ll show you a piece of code of the core model if you don’t understand its high-level structure then everything I wrote before doesn’t work and you can immediately leave this page ;)</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/44ac20d9fe6a3c1b36e3f4c2328b43f8/href">https://medium.com/media/44ac20d9fe6a3c1b36e3f4c2328b43f8/href</a></iframe><p>So, if you are still reading this text, I suppose that you have understood the code. Doesn’t it look great and really maintainable? The first thing I have to highlight is that, of course,<strong> not all transformers exist in sklearn.</strong> But as I said before, you can make one on your own. You can take a look at my previous post, at the end you’ll find custom sklearn transformer which uses spacy API:</p><p><a href="https://towardsdatascience.com/text-preprocessing-steps-and-universal-pipeline-94233cb6725a">Text preprocessing steps and universal pipeline</a></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/781/1*iGrHgiLJn4Az_9ZoDOSwpw.png" /><figcaption>Pipeline scheme</figcaption></figure><p>Now let’s name all the steps. <strong>High-level representation:</strong></p><ol><li>Preprocessing</li><li>Feature selection</li><li>Regression model</li></ol><p><strong>Preprocessing</strong>: Independently applying different preprocessing steps on different groups of columns by data representation type and then merge all transformed features into one dataframe.</p><p><strong>Image data:</strong></p><ol><li>Read images from files using <em>OpenCV</em></li><li>Convert colormap from <em>OpenCV</em> <em>BGR</em> to <em>RGB</em></li><li>Detect cats, dogs and people using <em>YOLO</em> and return an array of detected objects for each image</li><li>Transform output arrays to an indicator matrix, where 1 means object is on the image, 0 object is not on the image</li></ol><p><strong>Text data:</strong></p><ol><li>Read texts from files</li><li>Perform lemmatization of all words</li><li>Vectorize using TF-IDF algorithm, perform tokenization by words and use bag of words with a maximum size of 10000</li></ol><p><strong>Numeric data:</strong></p><ol><li>Impute missing values using median strategy</li><li>Perform standardization</li></ol><p><strong>Categorical data:</strong></p><ol><li>Impute missing values with word ‘missing’</li><li>Perform one-hot encoding</li></ol><p><strong>Categorical multilabel data:</strong> transform arrays to an indicator matrix, where 1 means object is in the list, 0 object is not in the list.</p><p>Now you can see how fancy your model could look like. This code fulfills all the aforementioned rules.</p><p>I hope you liked this post and I’m looking forward to your feedback! I really want to begin a discussion on this topic, so do not hesitate to write comments or email me directly.</p><p>If you have problems with converting your model into pipeline write a comment or email me directly, I’ll try to help you.</p><p>My email: maksym.balatsko@blindspot.ai</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=c1ef49aa5ba1" width="1" height="1" alt=""><hr><p><a href="https://medium.com/data-science/your-ml-code-is-well-structured-or-is-it-c1ef49aa5ba1">Your ML code is well structured… Or is it?</a> was originally published in <a href="https://medium.com/data-science">TDS Archive</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Setup version increment and automated release process]]></title>
            <link>https://medium.com/data-science/setup-version-increment-and-automated-release-process-591d87ea1221?source=rss-fa6012a510db------2</link>
            <guid isPermaLink="false">https://medium.com/p/591d87ea1221</guid>
            <category><![CDATA[automation]]></category>
            <category><![CDATA[versioning]]></category>
            <category><![CDATA[release-management]]></category>
            <category><![CDATA[process-automation]]></category>
            <category><![CDATA[machine-learning]]></category>
            <dc:creator><![CDATA[Maksym Balatsko]]></dc:creator>
            <pubDate>Wed, 18 Mar 2020 02:53:43 GMT</pubDate>
            <atom:updated>2020-03-18T02:53:43.689Z</atom:updated>
            <content:encoded><![CDATA[<h4>I’ll show you how easy could be the automation of the release process and version increment</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*iQ47Rj061zRZOV5Y" /><figcaption>Photo by <a href="https://unsplash.com/@yancymin?utm_source=medium&amp;utm_medium=referral">Yancy Min</a> on <a href="https://unsplash.com/?utm_source=medium&amp;utm_medium=referral">Unsplash</a></figcaption></figure><p>I’m keen on process automation and I strongly believe that time and money investments into process automation in the development phase is a primary need, no matter how long the project is going to last. When you invest in processes automation you vastly decrease further maintenance expenses.</p><p>I think one of the topics, which is not properly covered in ML articles is release process automation and project version increment. I don’t want to discuss possible release options in this post, it is not about it. Here we will discuss how to set up the release process to your CI/CD tool (Bitbucket pipelines, Gitlab pipelines, etc.) I think it is safe to say that in most cases release is a file or set of files, which will be somehow integrated into client existing solution (no matter if it is a jar file with the spring boot app inside, set of python scripts which are used for model training and predictions or a docker image in which core application runs). Problem statement: We want to manage product versions and we want to be able to return to the concrete version to be able to test and diagnose problems that happened on the production.</p><h4>Versioning</h4><p>To begin with, let’s define what does number in version string means.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/991/1*Nk3OlP6d79iZAN3uWhvW7A.png" /><figcaption>Version string</figcaption></figure><p>Definition from Semantic versioning (<a href="https://semver.org/">https://semver.org/</a>)</p><p>Given a version number <strong>MAJOR.MINOR.PATCH-pre_release-label</strong>, increment the:</p><ol><li><strong>MAJOR</strong> version when you make incompatible API changes</li><li><strong>MINOR</strong> version when you add functionality in a backward-compatible manner</li><li><strong>PATCH</strong> version when you make backward-compatible bug fixes.</li></ol><p><strong>Pre-release-label</strong> is the placeholder of the version, which is kept before release.</p><p>This is a best-practice definition, but of course, you can choose your own versioning scheme. So according to this definition, we need to define three tasks:</p><ol><li>Release and increment the major version</li><li>Release and increment the minor version</li><li>Release and increment the patch version</li></ol><p>The next step is to define how the release would be represented in GIT history except for the git tag. There are several techniques to achieve this, the most popular are separate release branch or detached commit. In this post, we will be discussing detached commit because it could much easier be automated (no merge needed).</p><h4>Detached commit</h4><p>Let’ begin with the theory. Detached commit is a commit that doesn&#39;t belong to any branch and can be pointed only directly. How to make it searchable in the git history without knowing the commit hash? You have just to assign a git tag to it. Then you can filter git history by tags in any VCS UI.</p><h4>VERSION file</h4><p>Somehow you have to keep actual version string. The easiest way is to keep it in a text file in the root of the repository. The policies of VERSION file:</p><ol><li>File contents can’t be manually changed</li><li>It can’t be changed in a feature-branch (the best practice is to create a pipeline step, which compares it against master)</li><li>File changes only after release and automatically committed directly to master branch.</li></ol><h4>Set up CI/CD pipelines</h4><p>To your CI/CD pipelines file, you have to add three custom/manual steps/jobs, which would perform release and an appropriate increment. And optionally add step/job which runs only on feature branches and which checks if VERSION file was changed against master.</p><p>Bash script to check if VERSION file changed against master:</p><pre>git diff --exit-code VERSION</pre><p>And now the most interesting part:</p><ol><li>We have to create a script which manages VERSION file with functions: get version, get version with placeholder, increment version major/minor/patch</li><li>We have to define release branch restriction policy (release could be made only on master branch) and create detached commit push flow.</li><li>Define CI/CD release step (Here we will define it for Bitbucket pipelines and Gitlab pipelines)</li></ol><p>I hate writing bash scripts. I use bash only to define high-level steps and run python scripts from it, because I strongly believe that bash scripts have ugly notation, are not readable for less experienced Linux users and are not maintainable in the long term perspective. fire library can help you easily write good CLI-runnable python scripts. Check out my previous post:</p><p><a href="https://towardsdatascience.com/a-simple-way-to-create-python-cli-app-1a4492c164b6">A simple way to create python CLI app</a></p><p>VERSION file management script (nothing interesting to discuss, just file writing operations):</p><pre>#!/usr/bin/env python<br>import os<br>import re<br><br>import fire<br><br>pre_release_placeholder = &#39;SNAPSHOT&#39;<br>version_filepath = os.path.join(&#39;.&#39;, &#39;VERSION&#39;)<br>version_pattern = re.compile(fr&#39;^\d+.\d+.\d+(-{pre_release_placeholder})?$&#39;)<br><br><br>def get(with_pre_release_placeholder: bool = False):<br>    with open(version_filepath, &#39;r&#39;) as version_file:<br>        version_lines = version_file.readlines()<br>        assert len(version_lines) == 1, &#39;Version file is malformed&#39;<br>        version = version_lines[0]<br>        assert version_pattern.match(version), &#39;Version string is malformed&#39;<br>        if with_pre_release_placeholder:<br>            return version<br>        else:<br>            return version.replace(f&#39;-{pre_release_placeholder}&#39;, &#39;&#39;)<br><br><br>def write_version_file(major: int, minor: int, patch: int):<br>    version = f&#39;{major}.{minor}.{patch}-{pre_release_placeholder}&#39;<br>    with open(version_filepath, &#39;w&#39;) as version_file:<br>        version_file.write(version)<br><br><br>def inc_patch():<br>    version = get()<br>    major, minor, patch = version.split(&#39;.&#39;)<br>    write_version_file(major, minor, int(patch) + 1)<br><br><br>def inc_minor():<br>    version = get()<br>    major, minor, patch = version.split(&#39;.&#39;)<br>    write_version_file(major, int(minor) + 1, patch)<br><br><br>def inc_major():<br>    version = get()<br>    major, minor, patch = version.split(&#39;.&#39;)<br>    write_version_file(int(major) + 1, minor, patch)<br><br><br>if __name__ == &quot;__main__&quot;:<br>    fire.Fire({<br>        &#39;get&#39;: get,<br>        &#39;inc-patch&#39;: inc_patch,<br>        &#39;inc-minor&#39;: inc_minor,<br>        &#39;inc-major&#39;: inc_major<br>    })</pre><p>Now let’s develop the release script. Firstly we have to check whether the release script is running on the right branch:</p><pre># Bitbucket<br>commit=${BITBUCKET_COMMIT:-$(git rev-parse HEAD)}</pre><pre># Gitlab<br>commit=${CI_COMMIT_SHA:-$(git rev-parse HEAD)}</pre><pre># Define ALLOWED_RELEASE_BRANCH in VCS pipelines secret variables.<br>branch=${ALLOWED_RELEASE_BRANCH:-master}<br><br>if ! git branch -a --contains &quot;${commit}&quot; | grep -e &quot;^[* ]*remotes/origin/${branch}\$&quot;<br>then<br>  echo -e &quot;###\n### Not on ${branch}. Only ${branch} commits can be released.\n###&quot;<br>  exit 1<br>else<br>  echo -e &quot;###\n### Releasing of ${commit} on ${branch}\n###&quot;<br>fi</pre><p>Then you have to write some release stuff such as: publish docker image or copy to S3 bucket or whatever you need. Don’t forget to do it with the right VERSION file:</p><pre>version=$(./scripts/version.py get)<br>version_file=VERSION</pre><pre>echo ${version} &gt; ${version_file}</pre><p>Then we have to push a detached commit with modified in a previous step VERSION file:</p><pre># Define some cool release pusher to ditinguish from amnual commits :)<br>git config user.name &quot;Elon Musk&quot;<br>git config user.email &quot;elon.musk@spacex.com&quot;<br><br>echo &quot;Pushing detached tag of new version&quot;<br>git add ${version_file}<br>git commit -m &quot;Release version ${version}&quot;<br>git tag  -a ${version} -m &quot;Release version ${version} tag&quot;<br>git push origin ${version}</pre><p>Fetch back to master branch and increment version with a pre-release placeholder:</p><pre>echo &quot;Pushing new version to ${branch}&quot;<br>git fetch origin &quot;${branch}:${branch}&quot; || git pull<br>git checkout &quot;${branch}&quot;</pre><pre># Parameter of the script<br>release_type=${1}<br>./scripts/version.py inc-${release_type}<br><br>next_working_version=$(./scripts/version.py get --with-pre-release-placeholder)<br>git add ${version_file}<br>git commit -m &quot;Incrementing working version to ${next_working_version} after ${version} release.&quot;<br>git push origin ${branch}</pre><p>And that&#39;s it. We have created a clean release script. Now let’s develop Bitbucket and Gitlab pipeline steps snippets.</p><p><strong>Bitbucket</strong>:</p><pre>image:<br>  name: python:3.7.6-slim-buster<br><br>pipelines:<br>  default:<br>    - step:<br>        - name: Test version file not changed<br>        - script:<br>            - git diff --exit-code VERSION<br><br>  custom:<br>    release-inc-patch:<br>      - step:<br>          caches:<br>            - pip<br>          name: Release current version and increment patch version<br>          script:<br>            - apt-get update &amp;&amp; apt-get install -y git<br>            - pip install fire==0.2.1<br>            - ./scripts/release.sh patch<br><br>    release-inc-minor:<br>      - step:<br>          caches:<br>            - pip<br>          name: Release current version and increment minor version<br>          script:<br>            - apt-get update &amp;&amp; apt-get install -y git<br>            - pip install fire==0.2.1<br>            - ./scripts/release.sh minor<br>    release-inc-major:<br>      - step:<br>          caches:<br>            - pip<br>          name: Release current version and increment major version<br>          script:<br>            - apt-get update &amp;&amp; apt-get install -y git<br>            - pip install fire==0.2.1<br>            - ./scripts/release.sh major</pre><p>Running custom pipeline steps: <a href="https://confluence.atlassian.com/bitbucket/run-pipelines-manually-861242583.html">https://confluence.atlassian.com/bitbucket/run-pipelines-manually-861242583.html</a></p><p><strong>Gitlab</strong>:</p><pre>test-version-changed:<br>  stage: test<br>  name: Test version file not changed<br>  script:<br>    - git diff --exit-code VERSION<br><br>release-inc-patch:<br>  stage: deploy<br>  image: python:3.7.6-slim-buster<br>  caches:<br>    - pip<br>  name: Release current version and increment patch version<br>  script:<br>    - apt-get update &amp;&amp; apt-get install -y git<br>    - pip install fire==0.2.1<br>    - ./scripts/release.sh patch<br>  when: manual<br>  only:<br>    - master<br><br>release-inc-minor:<br>  stage: deploy<br>  image: python:3.7.6-slim-buster<br>  caches:<br>    - pip<br>  name: Release current version and increment minor version<br>  script:<br>    - apt-get update &amp;&amp; apt-get install -y git<br>    - pip install fire==0.2.1<br>    - ./scripts/release.sh minor<br>  when: manual<br>  only:<br>    - master<br><br>release-inc-major:<br>  stage: deploy<br>  image: python:3.7.6-slim-buster<br>  caches:<br>    - pip<br>  name: Release current version and increment major version<br>  script:<br>    - apt-get update &amp;&amp; apt-get install -y git<br>    - pip install fire==0.2.1<br>    - ./scripts/release.sh major<br>  when: manual<br>  only:<br>    - master</pre><p>Run manual jobs: <a href="https://forum.gitlab.com/t/gitlab-ci-run-pipeline-manually/13797">https://forum.gitlab.com/t/gitlab-ci-run-pipeline-manually/13797</a></p><p>That’s it! All sources you can find on my GitHub: <a href="https://github.com/mbalatsko/release-version-increment"><strong>https://github.com/mbalatsko/release-version-increment</strong></a></p><p>I hope this article will be useful and informative. I’m looking forward to your feedback!</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=591d87ea1221" width="1" height="1" alt=""><hr><p><a href="https://medium.com/data-science/setup-version-increment-and-automated-release-process-591d87ea1221">Setup version increment and automated release process</a> was originally published in <a href="https://medium.com/data-science">TDS Archive</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[A simple way to create python CLI app]]></title>
            <link>https://medium.com/data-science/a-simple-way-to-create-python-cli-app-1a4492c164b6?source=rss-fa6012a510db------2</link>
            <guid isPermaLink="false">https://medium.com/p/1a4492c164b6</guid>
            <category><![CDATA[snippet]]></category>
            <category><![CDATA[python]]></category>
            <category><![CDATA[cli-app]]></category>
            <category><![CDATA[cli]]></category>
            <category><![CDATA[machine-learning]]></category>
            <dc:creator><![CDATA[Maksym Balatsko]]></dc:creator>
            <pubDate>Mon, 16 Mar 2020 20:17:24 GMT</pubDate>
            <atom:updated>2020-03-16T20:17:24.633Z</atom:updated>
            <content:encoded><![CDATA[<h4>An easy example of wrapping Machine Learning application to the command line interface</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*hTjBjk0Jq0EWoHOY" /><figcaption>Photo by <a href="https://unsplash.com/@cdr6934?utm_source=medium&amp;utm_medium=referral">Chris Ried</a> on <a href="https://unsplash.com/?utm_source=medium&amp;utm_medium=referral">Unsplash</a></figcaption></figure><p>When we create any type of application, we have to provide an interface using which other people will have the opportunity to use it. I want to show you an easy example of wrapping ML application to the CLI interface.</p><p>I think it is safe to say that we want the best results from the least effort. In this case, fire library will be the best choice. fire is a library, which is developed and widely used by Google. Let’s explore its functionality on concrete examples:</p><pre>main.py</pre><pre>import fire<br><br><br>def add(a: int, b: int):<br>    <em>&quot;&quot;&quot;<br>    Returns sum of a and b<br>    </em><strong><em>:param</em></strong><em> a: first argument<br>    </em><strong><em>:param</em></strong><em> b: second argument<br>    </em><strong><em>:return</em></strong><em>: sum of a and b<br>    &quot;&quot;&quot;<br>    </em>return a+b<br><br><br>if __name__ == &quot;__main__&quot;:<br>    fire.Fire({<br>        &quot;sum&quot;: add<br>    })</pre><p>Here fire creates CLI command sum from function add. From python doc, it creates a command description.</p><p>Output of python main.py sum --help :</p><pre>NAME<br>    main.py sum - Returns sum of a and b</pre><pre>SYNOPSIS<br>    main.py sum A B</pre><pre>DESCRIPTION<br>    Returns sum of a and b</pre><pre>POSITIONAL ARGUMENTS<br>    A<br>        first argument<br>    B<br>        second argument</pre><pre>NOTES<br>    You can also use flags syntax for POSITIONAL ARGUMENTS</pre><p>So you now can use this command in several ways:</p><p>python main.py 1 2</p><p>python main.py --a 1 --b 2</p><p>Also, you can define optional command arguments, by defining default value in a function definition:</p><pre>def add(a: int, b: int = 2):<br>    <em>&quot;&quot;&quot;<br>    Returns sum of a and b<br>    </em><strong><em>:param</em></strong><em> a: first argument<br>    </em><strong><em>:param</em></strong><em> b: second argument (default: 2)<br>    </em><strong><em>:return</em></strong><em>: sum of a and b<br>    &quot;&quot;&quot;<br>    </em>return a+b</pre><p>Now the result of python main.py sum --help is:</p><pre>NAME<br>    main.py sum - Returns sum of a and b</pre><pre>SYNOPSIS<br>    main.py sum A &lt;flags&gt;</pre><pre>DESCRIPTION<br>    Returns sum of a and b</pre><pre>POSITIONAL ARGUMENTS<br>    A<br>        first argument</pre><pre>FLAGS<br>    --b=B<br>        second argument (default: 2)</pre><pre>NOTES<br>    You can also use flags syntax for POSITIONAL ARGUMENTS</pre><p>So you now can use this command in several ways:</p><p>python main.py 1</p><p>python main.py 1 2</p><p>python main.py 1 --b 2</p><p>python main.py --a 1 --b 2</p><p>No more words needed, you can see how easy and quick CLI application configuration could be.</p><p>Now let’s create a program skeleton for a common use case for ML application. Let’s assume that you need to create a CLI application with two functions train and predict . The train function inputs are train data and some model parameters, the outputs is a trained model file. Predict takes on input data for predictions, the trained model, and saves predictions to the output file.</p><p>Except for functions mapping, fire can create a CLI interface from python class instance, which in this case could be a better option. In our case the code could be:</p><pre>import fire<br>import pickle<br>import logging<br><br>import pandas as pd<br>from sklearn.neighbors import KNeighborsClassifier<br><br>logging.basicConfig(level=logging.INFO)<br>logger = logging.getLogger(__name__)<br><br>def load_model(model_path: str):<br>    <em>&quot;&quot;&quot;Loads model from `model_path`&quot;&quot;&quot;<br>    </em>with open(model_path, &#39;rb&#39;) as file:<br>        saved_model = pickle.load(file)<br>        return saved_model<br><br>def save_model(model, model_path: str):<br>    <em>&quot;&quot;&quot;Saves `model` to `model_path`&quot;&quot;&quot;<br>    </em>with open(model_path, &#39;wb&#39;) as file:<br>        pickle.dump(model, file)<br><br>class Classifier:<br>    <em>&quot;&quot;&quot;<br>    Some classifier, that makes some random classifications.<br>    &quot;&quot;&quot;<br><br>    </em>def train(self, train_data_path: str, model_path: str, k: int = 5):<br>        <em>&quot;&quot;&quot;<br>        Trains model on `train_data_path` data and saves trained model to `model_path`.<br>        Additionaly you can set KNN classifier `k` parameter.<br>        </em><strong><em>:param</em></strong><em> train_data_path: path to train data in csv format<br>        </em><strong><em>:param</em></strong><em> model_path: path to save model to.<br>        </em><strong><em>:param</em></strong><em> k: k-neighbors parameter of model.<br>        &quot;&quot;&quot;<br>        </em>logger.info(f&quot;Loading train data from {train_data_path} ...&quot;)<br>        df = pd.read_csv(train_data_path)<br>        X = df.drop(columns=[&#39;y&#39;])<br>        y = df[&#39;y&#39;]<br><br>        logger.info(&quot;Running model training...&quot;)<br>        model = KNeighborsClassifier(n_neighbors=k)<br>        model.fit(X, y)<br><br>        logger.info(f&quot;Saving model to {model_path} ...&quot;)<br>        save_model(model, model_path)<br><br>        logger.info(&quot;Successfully trained model.&quot;)<br><br>    def predict(self, predict_data_path: str, model_path: str, output_path: str):<br>        <em>&quot;&quot;&quot;<br>        Predicts `predict_data_path` data using `model_path` model and saves predictions to `output_path`<br>        </em><strong><em>:param</em></strong><em> predict_data_path: path to data for predictions<br>        </em><strong><em>:param</em></strong><em> model_path: path to trained model<br>        </em><strong><em>:param</em></strong><em> output_path: path to save predictions<br>        &quot;&quot;&quot;<br>        </em>logger.info(f&quot;Loading data for predictions from {predict_data_path} ...&quot;)<br>        X = pd.read_csv(predict_data_path)<br><br>        logger.info(f&quot;Loading model from {model_path} ...&quot;)<br>        model = load_model(model_path)<br><br>        logger.info(&quot;Running model predictions...&quot;)<br>        y_pred = model.predict(X)<br><br>        logger.info(f&quot;Saving predictions to {output_path} ...&quot;)<br>        pd.DataFrame(y_pred).to_csv(output_path)<br><br>        logger.info(&quot;Successfully predicted.&quot;)<br><br>if __name__ == &quot;__main__&quot;:<br>    fire.Fire(Classifier)</pre><p>The output of python main.py is:</p><pre>NAME<br>    main.py - Some classifier, that makes some random classifications.</pre><pre>SYNOPSIS<br>    main.py COMMAND</pre><pre>DESCRIPTION<br>    Some classifier, that makes some random classifications.</pre><pre>COMMANDS<br>    COMMAND is one of the following:</pre><pre>predict<br>       Predicts `predict_data_path` data using `model_path` model and saves predictions to `output_path`</pre><pre>train<br>       Trains model on `train_data_path` data and saves trained model to `model_path`. Additionaly you can set KNN classifier `k` parameter.</pre><p>And the corresponding command documentation:</p><p>python main.py train --help :</p><pre>NAME<br>    main.py train - Trains model on `train_data_path` data and saves trained model to `model_path`. Additionaly you can set KNN classifier `k` parameter.</pre><pre>SYNOPSIS<br>    main.py train TRAIN_DATA_PATH MODEL_PATH &lt;flags&gt;</pre><pre>DESCRIPTION<br>    Trains model on `train_data_path` data and saves trained model to `model_path`. Additionaly you can set KNN classifier `k` parameter.</pre><pre>POSITIONAL ARGUMENTS<br>    TRAIN_DATA_PATH<br>        path to train data in csv format<br>    MODEL_PATH<br>        path to save model to.</pre><pre>FLAGS<br>    --k=K<br>        k-neighbors parameter of model.</pre><pre>NOTES<br>    You can also use flags syntax for POSITIONAL ARGUMENTS</pre><p>python main.py predict --help :</p><pre>NAME<br>    main.py predict - Predicts `predict_data_path` data using `model_path` model and saves predictions to `output_path`</pre><pre>SYNOPSIS<br>    main.py predict PREDICT_DATA_PATH MODEL_PATH OUTPUT_PATH</pre><pre>DESCRIPTION<br>    Predicts `predict_data_path` data using `model_path` model and saves predictions to `output_path`</pre><pre>POSITIONAL ARGUMENTS<br>    PREDICT_DATA_PATH<br>        path to data for predictions<br>    MODEL_PATH<br>        path to trained model<br>    OUTPUT_PATH<br>        path to save predictions</pre><pre>NOTES<br>    You can also use flags syntax for POSITIONAL ARGUMENTS</pre><p>Usage example:</p><pre>python main.py train train.csv knn.sav --k 7<br>python main.py predict test.csv knn.sav out.csv</pre><p>Now you can use this snippet for the creation of your own CLI application. Some tips for further improvement:</p><ol><li>Use logging instead of printing</li><li>Create arguments validation</li><li>Create README template with a brief application usage description for the end-user.</li><li>Explore more cool features on <a href="https://github.com/google/python-fire">https://github.com/google/python-fire</a></li></ol><p>I hope this article will be useful and informative. I’m looking forward to your feedback!</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=1a4492c164b6" width="1" height="1" alt=""><hr><p><a href="https://medium.com/data-science/a-simple-way-to-create-python-cli-app-1a4492c164b6">A simple way to create python CLI app</a> was originally published in <a href="https://medium.com/data-science">TDS Archive</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[All you want to know about preprocessing: Data preparation]]></title>
            <link>https://medium.com/data-science/all-you-want-to-know-about-preprocessing-data-preparation-b6c2866071d4?source=rss-fa6012a510db------2</link>
            <guid isPermaLink="false">https://medium.com/p/b6c2866071d4</guid>
            <category><![CDATA[categorical-data]]></category>
            <category><![CDATA[data-preparation]]></category>
            <category><![CDATA[data-preprocessing]]></category>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[data-cleaning]]></category>
            <dc:creator><![CDATA[Maksym Balatsko]]></dc:creator>
            <pubDate>Wed, 29 May 2019 23:47:02 GMT</pubDate>
            <atom:updated>2019-05-29T23:47:02.118Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*eWYm3Dm7s0t8lLWYRW4pow.jpeg" /></figure><h4>This is an introduction part, where we are going to discuss how to check and prepare your data for further preprocessing.</h4><p>Nowadays, almost all ML/data mining projects workflow run on a standard CRISP-DM (Cross-industry standard process for data mining) or its IBM enhance ASUM-DM (Analytics Solutions Unified Method for Data Mining/Predictive Analytics). The longest and the most important step in this workflow is <strong>Data preparation/preprocessing</strong>, which approximately takes 70% of the time. This step is important because in most situations data provided by the customer has a bad quality or just cannot be directly fed to some kind of ML model. My favorite byword, that I&#39;ll mention in all my posts, concerning data preprocessing, says: Garbage in, garbage out (GIGO). In other words, if you feed your model with miserable data, don&#39;t expect it to perform well. In this post we are going to discuss:</p><ol><li>Data types</li><li>Data validation</li><li>Handling dates</li><li>Handling nominal and ordinal categorical values</li></ol><p>In the next posts we are going to talk about more advanced preprocessing techniques:</p><ol><li>Data cleaning and standardization: Normalization and standartization, Handling missing data, Handling outliers</li><li>Feature selection and dataset balancing: Dataset balancing, Feature extraction, Feature selection.</li></ol><p>This post series represents the usual preprocessing flow order, but, in fact, all these parts are divided in such a way to be independent and not to require the knowledge of the previous parts.</p><p>This is an introduction part, where we are going to discuss how to check and prepare your data for further preprocessing.</p><h3>Data types</h3><p>To start, let’s define what data types exist and what measurement scales they have:</p><h4>Numeric</h4><ol><li>Discrete - integer values. Example: number of products bought in the shop</li><li>Continuos - any value in some admissable range (float, double). Example: average length of words in text</li></ol><h4>Categorical</h4><p>The variable value selected from a predefined number of categories</p><ol><li>Ordinal - categories could be meaningfully ordered. Example: grade (A, B, C, D, E, F)</li><li>Nominal - categories don&#39;t have any order. Example: religion (Christian, Muslim, Hindu, etc.)</li><li>Dichotomous/Binary - the special case of nominal, with only 2 possible categories. Example: gender (male, female)</li></ol><h4>Date</h4><p>String, python datetime, timestamp. Example: 12.12.2012</p><h4>Text</h4><p>Multidimensional data, more about text preprocessing see in my previous <a href="https://towardsdatascience.com/text-preprocessing-steps-and-universal-pipeline-94233cb6725a">post</a></p><h4>Images</h4><p>Multidimensional data, more about image preprocessing see in my next posts</p><h4>Time series</h4><p>Data points indexed in the time order, more about time series preprocessing see in my next posts.</p><h3>Data validation</h3><p>The first step is the simplest and the most obvious: you have to investigate and validate your data. To be able to validate the data you have to have a deep understanding of your data. Easy rule: Don&#39;t dismiss the description of the dataset. Validation step consists of:</p><h4>Data type and data representation consistency check</h4><p>Same things have to be represented in the same way and in the same format. Examples:</p><ol><li>Dates have the same format. Several times in my practice I’ve got data where a part of dates was in American format, the other part in European.</li><li>Integers are really integers, not strings or floats</li><li>Categorical data doesn’t have duplicates because of whitespaces, lower/upper cases</li><li>Other data representations don’t contain an error</li></ol><h4>Data domain check</h4><p>Data is in range of permissible values. Example: numerical variables are in admissable (min, max) range.</p><h4>Data integrity check</h4><p>Check permitted relationships and fulfillment of the constraints. Examples:</p><ol><li>Check name titles with sex, age of birth with age</li><li>Historical data have the right chronology. Delivery after purchase, Bank account opening before the first payment, etc.</li><li>The actions are made by allowed entities. The mortgage could be approved only for people older than 18 years old, etc.</li></ol><h4>Ok, we have found some errors, what could we do?</h4><ol><li>Correct them, if you are sure, what the problem is, or consult with the specialist or data provider if possible.</li><li>Discard samples with errors, in many cases it is a good choice because you aren&#39;t able to fulfill 1.</li><li>Do nothing, this, of course, could cause undesired effects in future steps.</li></ol><h3>Handling dates</h3><p>Different systems stores dates in different formats: 11.12.2019, 2016-02-12, Sep 24, 2003 etc. But for building models on dates data, we need to somehow convert it to a numeric format.</p><p>To start, I&#39;ll show you an example of how to convert a date string into python datetime type, which is much more convenient for further steps. The example is demonstrated on pandas dataframe. Let&#39;s assume that date_string column contains dates in strings:</p><pre># Converts date string column to python datetime type<br># `infer_datetime_format=True` says method to guess date format from string</pre><pre>df[&#39;datetime&#39;] = pd.to_datetime(df[&#39;date_string&#39;], infer_datetime_format=True)</pre><pre># Converts date string column to python datetime type<br># `format` argument specifies the format of date to parse, fails on errors</pre><pre>df[&#39;datetime&#39;] = pd.to_datetime(df[&#39;date_string&#39;], format=&#39;%Y.%m.%d&#39;)</pre><p>Frequently, just the year (YYYY) is sufficient. But if we want to store months, days or even more detailed data, our numeric format has to fulfill 1 sufficient constraint, it has to save intervals, it means that for example, Monday — Friday in one week has to have the same difference as 1. — 5. of any month. So YYYYMMDD format will be not an option, because the last day of the month and the first day of the next month have a bigger distance than the first and the second day of the month. Actually, there are 4 most common methods to transform date to numeric format:</p><h4>Unix timestamp</h4><p>Number of seconds since 1970</p><p><strong>Pros:</strong></p><ol><li>perfectly preserve intervals</li><li>good if hours, minutes and seconds matters</li></ol><p><strong>Cons:</strong></p><ol><li>values are non-obvious</li><li>don’t help intuition and knowledge discovery</li><li>harder to verify, easier to make an error</li></ol><p>Converting datetime column to timestamp in pandas:</p><pre># Coverts column in python datetime type to timestamp<br>df[&#39;timestamp&#39;] = df[&#39;datetime&#39;].values.astype(np.int64) // 10 ** 9</pre><h4>KSP date format</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/362/0*ZfgmYdZfvbI4FbKG" /></figure><p><strong>Pros:</strong></p><ol><li>the year and quarter are obvious</li><li>easy intuition and knowledge discovery</li><li>can be extended to include time</li></ol><p><strong>Cons:</strong></p><ol><li>preserves intervals (almost)</li></ol><p>Converting python datetime column to KSP format in pandas:</p><pre>import datetime as dt<br>import calendar</pre><pre>def to_ksp_format(datetime):<br>    year = datetime.year<br>    day_from_jan_1 = (datetime - dt.datetime(year, 1, 1)).days<br>    is_leap_year = int(calendar.isleap(year))<br>    return year + (day_from_jan_1 - 0.5) / (365 + is_leap_year)</pre><pre>df[&#39;ksp_date&#39;] = df[&#39;datetime&#39;].apply(to_ksp_format)</pre><h4>Divide into several features</h4><p>Year, month, days, etc.</p><p><strong>Cons:</strong></p><ol><li>perfectly preserve intervals</li><li>easy intuition and knowledge discovery</li></ol><p><strong>Pros:</strong></p><ol><li>more dimensions you add, more complex your model could get, but it is not always bad.</li></ol><h4>Construct new feature</h4><p>Construct a new feature based on date features. For example:</p><p>date of birth -&gt; age</p><p>date order created and date order delivered -&gt; time to delivery.</p><p><strong>Cons:</strong></p><ol><li>easy intuition and knowledge discovery</li></ol><p><strong>Pros:</strong></p><ol><li>manual feature construction might lead to important information loss</li></ol><h3>Handling categorical values</h3><p>Flashback: Categorical - the variable value selected from a predefined number of categories. Categorical values, like any other non-numeric types, has also to be converted into numeric values. How to do it right?</p><h4>Ordinal</h4><p>Categories could be meaningfully ordered. Can be converted into numeric values saving its natural order. Grades: A+ - 4.0, A- - 3.7, B+ - 3.3, B - 3.0, etc.</p><p>Demonstration in pandas:</p><pre>grades = {<br>    &#39;A+&#39;: 4.0,<br>    &#39;A-&#39;: 3.7,<br>    &#39;B+&#39;: 3.3,<br>    &#39;B&#39; : 3.0<br>}<br>df[&#39;grade_numeric&#39;] = df[&#39;grade&#39;].apply(lambda x: grades[x])</pre><h4>Dichotomous/Binary</h4><p>Only one of two possible categories. In this case, you can convert values into indicator values 1/0. For example: Male - 1 or Female - 0, or you can do it oppositely.</p><p>Demonstration in pandas:</p><pre>df[&#39;gender_indicator&#39;] = df[&#39;gender&#39;].apply(lambda x: int(x.lower() == &#39;Male&#39;))</pre><h4>Nominal</h4><p>One or more of all possible categories. In this case, One hot encoding have to be used. This method assumes creating an indicator value for every category(1- the sample is in the category, 0 - if not). This method is applicable also for Dichotomous/Binary categorical values. <strong>NEVER USE ORDINAL REPRESENTATION FOR NOMINAL VALUES</strong>, it would cause terrible side effects and your model will not be able to handle the categorical feature in the right way.</p><p>Demonstration in pandas:</p><pre># Pandas `.get_dummies()` method<br>df = pd.concat([df, pd.get_dummies(df[&#39;category&#39;], prefix=&#39;category&#39;)],axis=1)</pre><pre># now drop the original &#39;category&#39; column (you don&#39;t need it anymore)<br>df.drop([&#39;category&#39;],axis=1, inplace=True)</pre><p>Demonstration in sklearn and pandas:</p><pre>from sklearn.preprocessing import LabelEncoder, OneHotEncoder</pre><pre>prefix = &#39;category&#39;</pre><pre>ohe = OneHotEncoder(sparse=False)<br>ohe = ohe.fit(df[[&#39;category&#39;]])<br>onehot_encoded = ohe.transform(df[[&#39;category&#39;]])</pre><pre>features_names_prefixed = [ f&quot;{prefix}_{category}&quot; for category in onehot_encoder.categories_[0]]</pre><pre>df = pd.concat([df, pd.DataFrame(onehot_encoded, columns=features_names_prefixed)], axis=1)</pre><pre># now drop the original &#39;category&#39; column (you don&#39;t need it anymore)<br>df.drop([&#39;category&#39;],axis=1, inplace=True)</pre><p>I hope you’ll like my post. Feel free to ask questions in comments.</p><p>P.S. These are very and very basic and simple things, but they are very important in practice. Much more interesting stuff is coming in the next posts!</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=b6c2866071d4" width="1" height="1" alt=""><hr><p><a href="https://medium.com/data-science/all-you-want-to-know-about-preprocessing-data-preparation-b6c2866071d4">All you want to know about preprocessing: Data preparation</a> was originally published in <a href="https://medium.com/data-science">TDS Archive</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[4 easy steps to improve your machine learning code performance]]></title>
            <link>https://medium.com/data-science/4-easy-steps-to-improve-your-machine-learning-code-performance-88a0b0eeffa8?source=rss-fa6012a510db------2</link>
            <guid isPermaLink="false">https://medium.com/p/88a0b0eeffa8</guid>
            <category><![CDATA[mapreduce]]></category>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[parallel-computing]]></category>
            <category><![CDATA[distributed-systems]]></category>
            <category><![CDATA[optimization]]></category>
            <dc:creator><![CDATA[Maksym Balatsko]]></dc:creator>
            <pubDate>Tue, 21 May 2019 15:09:26 GMT</pubDate>
            <atom:updated>2019-05-24T15:14:19.969Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*HXDn5JnRI8LnV3B5NkyHRQ.png" /></figure><h4>I want to show you 4 easy steps, that will help you, without any knowledge about optimization, speed-up your code at least twice and make it more efficient and readable.</h4><p>It is really astonishing, how many new machine learning libraries and models are created every day. But it is such a disappointment that most of them are just not properly designed. Actually sometimes great thing with great ideas behind it is just not being used, because it is not well documented, has difficult or not intuitive API or just because the thing doesn’t perform well and is too slow for the production environment. A great example of this would the library that I mentioned in my previous post — <a href="https://github.com/EFord36/normalise">normalize</a>. Such a beautiful idea behind it and it produces the results that really can make the difference in some nlp tasks, but there are some unexpected bugs in it and API is not flexible enough. OK, you say, these two things are not so crucial and actually, you can survive it. But the thing you cannot survive is that its performance is awful, it is just <strong>SLOW</strong> and so can’t be used in a production environment.</p><p>I want to encourage everyone, who creates machine learning libraries or just shows his own pieces of reusable code to try to optimize it as much as it is possible. Because everybody, who does it more or less wants to make the difference or to make this world a little bit better, helping other people shorten the way they passed and have better results immediately. I want to show you 4 easy steps, that will help you, without any knowledge about optimization, speed-up your code at least twice and make it more efficient and readable.</p><h3>Calculate your math efficiently</h3><p>Captain Obvious says: “Use math functions of the library you work with or just use numpy and you’ll be always alright”. He is right because in most cases the author of the library optimized all the functions in a way, to make his code as fast as it is possible. But almost every python ML library is based on numpy. Numpy <a href="https://www.numpy.org/">official web page</a> says:</p><blockquote>NumPy is the fundamental package for scientific computing with Python. It contains among other things:</blockquote><blockquote>a powerful N-dimensional array object</blockquote><blockquote>sophisticated (broadcasting) functions</blockquote><blockquote>tools for integrating C/C++ and Fortran code</blockquote><blockquote>useful linear algebra, Fourier transform, and random number capabilities</blockquote><p>Actually, in ML we just making some matrix operations, again and again. Numpy uses all possible state-of-the-art optimization techniques you can imagine to make all these operations efficient. So rule number one:</p><blockquote>Use numpy instead of python math, use numpy arrays instead of python arrays.</blockquote><p>If you follow just this first simple rule, you’ll soon feel the difference.</p><h3>Vectorize your functions</h3><p>How to get rid of annoying for loops that just applies the same function on a set of inputs? Just vectorize this function! Numpy allows you to vectorize a function, which means that newly created function will be applied on list inputs and produce an array of function results. According to this <a href="https://towardsdatascience.com/data-science-with-python-turn-your-conditional-loops-to-numpy-vectors-9484ff9c622e">post</a>, it might speed up your computation for at least twice. I highly recommend to read it, because there author shows the beautiful comparison of forloops and numpy vectorize and shows measurements as the proofs that numpy vectorize just works faster! Let’s just see some simple code snippets, to show how it works:</p><pre>import numpy as np<br>from math import sin, cos, gamma</pre><pre>def func(x):<br>    if x &gt; 10 * sin(x) * cos(x):<br>        return sin(x) * cos(x)<br>    elif x**100 &lt; (x-1)**101:<br>        return x**89<br>    elif x**100 - gamma(x) &lt; (x-1)**101 * gamma(x):<br>        return gamma(x)<br>    else:<br>        return x**56</pre><pre>arr = np.random.randn(10000)</pre><pre># FoorLoop method<br>res = []</pre><pre>for i in arr:<br>    res.append(func(i))</pre><pre># Numpy Vectorization</pre><pre>func_vect = np.vectorize(func)<br>res_vet = func_vect(arr)</pre><p>Numpy vectorization code, in this case, runs 2x faster! Vectorization not only can help to speed-up your code but also make it cleaner. For example, make stemming in nltk to run on all tokens, instead of only one:</p><pre>import numpy as np<br>from nltk.stem import PorterStemmer<br>from nltk.tokenize import word_tokenize</pre><pre>tokens = word_tokenize(text)<br>porter = PorterStemmer()</pre><pre># vectorizing function to able to call on list of tokens<br>stem_words = np.vectorize(porter.stem)</pre><pre>tokens_stems = stem_words(tokens)</pre><p>It looks fancier than ForLoop alternative:</p><pre>import numpy as np<br>from nltk.stem import PorterStemmer<br>from nltk.tokenize import word_tokenize</pre><pre>tokens = word_tokenize(text)<br>porter = PorterStemmer()</pre><pre>def stem_words(tokens):<br>    stems = []<br>    for t in tokens:<br>        stems.append(porter.stem(t))<br>    return stems</pre><pre>tokens_stems = stem_words(tokens)</pre><p>So to conclude, vectorization is a very powerful mechanism, that will help you not only speed-up your code but also to make it look cleaner.</p><h3>n_jobs in sklearn</h3><p>Almost every computer nowadays has more than one core. Not using all of them, if you have paid for it, as for me is a crime. Actually, you have to take from your computer everything it can give you. For ML developer his computer does not have to be a pretty and fancy thing, first of all, it is a hard-working miner, which have to work in three shifts like Stakhanov in USSR. You just have to take from your computer everything it can give you, if you still don’t understand why, just remember one simple thing: “You have paid for it”. The easiest way for ML developer to use more than core in his pipelines is just not to forget to set <strong>n_jobs</strong> parameter in every sklearn model he uses. <strong>n_jobs</strong> tells sklearn how many jobs to run in parallel both to fit and predict on the model. Most sklearn models support parallelized training and transformations. This will help you to save much time while GridSearching the best parameters or training models on large or high dimensional datasets. Beware of setting <strong>n_jobs</strong> to <strong>-1</strong>, because parallelizations save your time, not RAM. Yeah, actually parallelization might be memory extensive but using at least 2 jobs instead of 1 will already help you to optimize your model training and prediction performance.</p><h3>MapReduce</h3><p>Wikipedia says:</p><blockquote><strong>MapReduce</strong> is a programming model and an associated implementation for processing and generating big data sets with a parallel, distributed algorithm on a cluster.</blockquote><p>Looks like something complex, like something you could use only on a big cluster or on an enormous dataset, right? But it is very easy, I’ll explain it in a couple of sentences and 1 example.</p><p>MapReduce is divided into two parts:</p><ol><li>Map — function, which somehow modifies an input into an output. This function is called on every sample of data, which have to be processed. This step is parallelized, because for each sample, in fact, you do the same operation.</li><li>Reduce — function, that aggregates all output of the map function. This step is usually sequential.</li></ol><p>For example, you have to calculate the sum of squares of an array of data. Sequential implementation would look like:</p><pre>arr = [1, 2, 3, 4, 5]</pre><pre>result = 0<br>for i in arr:<br>    result += i**2</pre><p>Let’s convert this task into MapReduce:</p><ol><li>Map — calculate the square of the number</li><li>Reduce — calculate the sum of map results</li></ol><p>Sequential implementation in python:</p><pre>from functools import reduce</pre><pre>arr = [1, 2, 3, 4, 5]</pre><pre>map_arr = map(lambda x: x**2, arr)<br>result = reduce(lambda x, y: x + y, map_arr)</pre><p>As was aforementioned the map part will be parallelized, reduce would stay sequential. Let’s take a glance on parallelized implementation:</p><pre>import multiprocessing as mp<br>from functools import reduce</pre><pre>def pow2(x):<br>    return x**2</pre><pre>arr = [1, 2, 3, 4, 5]</pre><pre># count of avaliable cores, of course you can use less for computations<br>cores = mp.cpu_count()<br>pool = mp.Pool(cores)</pre><pre>map_arr = pool.map(pow2, arr)<br>result = reduce(lambda x, y: x + y, map_arr)</pre><pre>pool.close()<br>pool.join()</pre><p>So now for this task, you use all the power of your computer! <strong>.map </strong>method of pool object assumes that the map function has only one argument, there is an equivalent <strong>.starmap</strong> ,<strong> </strong>which assumes more than one argument in map function, the full description of these functions you can find <a href="https://docs.python.org/3.7/library/multiprocessing.html">here</a>. Good to mention is that both <strong>.map</strong> and <strong>.starmap</strong> returns mapped values in the same order as input parameters.</p><p>Sometimes it is more efficient to compute map function not only on one sample but on a batch of samples. If the computation task is simple and quick enough, the main overhead of the multiprocessing computation will be transferring data between processes, which could slow down the process a lot. As in the previous part of my post we were talking about <strong>n_jobs</strong> parameter of sklearn models, let&#39;s create universal code snippet of base transformer, which could parallelize the process of its transformations, in this case, batch processing will be the best choice.</p><p>Here is a code snippet of pandas apply parallelization inside a base sklearn transformer, which you could use in you further pipelines:</p><pre>import multiprocessing as mp<br>from sklearn.base import TransformerMixin, BaseEstimator</pre><pre>class ParallelTransformer(BaseEstimator, TransformerMixin):<br>    def __init__(self,<br>                 n_jobs=1):<br>        &quot;&quot;&quot;<br>        n_jobs - parallel jobs to run<br>        &quot;&quot;&quot;<br>        self.variety = variety<br>        self.user_abbrevs = user_abbrevs<br>        self.n_jobs = n_jobs</pre><pre>    def fit(self, X, y=None):<br>        return self</pre><pre>    def transform(self, X, *_):<br>        X_copy = X.copy()</pre><pre>        cores = mp.cpu_count()<br>        partitions = 1<br>        <br>        if self.n_jobs &lt;= -1:<br>            partitions = cores<br>        elif self.n_jobs &lt;= 0:<br>            partitions = 1<br>        else:<br>            partitions = min(self.n_jobs, cores)<br>        <br>        if partitions == 1:<br>            # transform sequentially<br>            return X_copy.apply(self._transform_one)<br>        <br>        # splitting data into batches<br>        data_split = np.array_split(X_copy, partitions)<br>        <br>        pool = mp.Pool(cores)<br>        <br>        # Here reduce function - concationation of transformed batches<br>        data = pd.concat(<br>            pool.map(self._preprocess_part, data_split)<br>        )<br>        <br>        pool.close()<br>        pool.join()</pre><pre>        return data</pre><pre>    def _transform_part(self, df_part):<br>        return df_part.apply(self._transform_one)</pre><pre>    def _transform_one(self, line):</pre><pre>        # some kind of transformations here<br>        return line</pre><p>I’ll summarize this code. Determine whether to process dataset in parallel or sequential. If in parallel, divide into <strong>n_jobs</strong> batches all the data and transform in parallel, then concatenate back transformed batches of the dataset. And that’s all! I hope this code snippet, will help you to boost your transformation pipelines.</p><p>I hope you liked my post, if you have further questions, ask in comments I’ll be glad to answer it. I’m looking forward to your feedback!</p><p>P.S. If you are an experienced developer, who wants to contribute to open source, one of your possible tasks could be helping good but not optimized libraries become fast and production ready. If you don’t know, where to start, you can help <a href="https://github.com/EFord36/normalise">normalize</a> library to optimize its performance. I don’t have any connections with the authors of this library, they don’t pay me and it is not an advertisement. Just another library, which needs your help ;)</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=88a0b0eeffa8" width="1" height="1" alt=""><hr><p><a href="https://medium.com/data-science/4-easy-steps-to-improve-your-machine-learning-code-performance-88a0b0eeffa8">4 easy steps to improve your machine learning code performance</a> was originally published in <a href="https://medium.com/data-science">TDS Archive</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Text preprocessing steps and universal pipeline]]></title>
            <link>https://medium.com/data-science/text-preprocessing-steps-and-universal-pipeline-94233cb6725a?source=rss-fa6012a510db------2</link>
            <guid isPermaLink="false">https://medium.com/p/94233cb6725a</guid>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[nlp]]></category>
            <category><![CDATA[data-preprocessing]]></category>
            <category><![CDATA[naturallanguageprocessing]]></category>
            <category><![CDATA[pipeline]]></category>
            <dc:creator><![CDATA[Maksym Balatsko]]></dc:creator>
            <pubDate>Mon, 20 May 2019 16:25:51 GMT</pubDate>
            <atom:updated>2019-05-21T15:06:47.539Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*QGWyxDaFhavZa495eJBO9Q.jpeg" /></figure><h3>Text preprocessing steps and universal reusable pipeline</h3><h4>The description of all text preprocessing steps and creation of a reusable text preprocessing pipeline</h4><p>Before feeding any ML model some kind data, it has to be properly preprocessed. You must have heard the byword: Garbage in, garbage out (GIGO). Text is a specific kind of data and can&#39;t be directly fed to most ML models, so before feeding it to a model you have to somehow extract numerical features from it, in another word vectorize. Vectorization is not the topic of this tutorial, but the main thing you have to understand is that GIGO is also applicable on vectorization too, you can extract qualitative features only from the qualitatively preprocessed text.</p><p>Things we are going to discuss:</p><ol><li>Tokenization</li><li>Cleaning</li><li>Normalization</li><li>Lemmatization</li><li>Steaming</li></ol><p>Finally, we’ll create a reusable pipeline, which you’ll be able to use in your applications.</p><p>Kaggle kernel: <a href="https://www.kaggle.com/balatmak/text-preprocessing-steps-and-universal-pipeline">https://www.kaggle.com/balatmak/text-preprocessing-steps-and-universal-pipeline</a></p><p>Let’s assume this example text:</p><pre>An explosion targeting a tourist bus has injured at least 16 people near the Grand Egyptian Museum, <br>next to the pyramids in Giza, security sources say E.U.<br><br>South African tourists are among the injured. Most of those hurt suffered minor injuries, <br>while three were treated in hospital, N.A.T.O. say.<br><br>http://localhost:8888/notebooks/Text%20preprocessing.ipynb<br><br>@nickname of twitter user and his email is email@gmail.com . <br><br>A device went off close to the museum fence as the bus was passing on 16/02/2012.</pre><h3>Tokenization</h3><p>Tokenization - text preprocessing step, which assumes splitting text into tokens(words, sentences, etc.)</p><p>Seems like you can use somkeind of simple seperator to achieve it, but you don’t have to forget that there are a lot of different situations, where separators just don’t work. For example, . separator for tokenization into sentences will fail if you have abbreviations with dots. So you have to have a more complex model to achieve good enough result. Commonly this problem is solved using nltk or spacy nlp libraries.</p><p>NLTK:</p><pre>from nltk.tokenize import sent_tokenize, word_tokenize<br><br>nltk_words = word_tokenize(example_text)<br>display(f&quot;Tokenized words: <strong>{nltk_words}</strong>&quot;)</pre><p>Output:</p><pre>Tokenized words: [&#39;An&#39;, &#39;explosion&#39;, &#39;targeting&#39;, &#39;a&#39;, &#39;tourist&#39;, &#39;bus&#39;, &#39;has&#39;, &#39;injured&#39;, &#39;at&#39;, &#39;least&#39;, &#39;16&#39;, &#39;people&#39;, &#39;near&#39;, &#39;the&#39;, &#39;Grand&#39;, &#39;Egyptian&#39;, &#39;Museum&#39;, &#39;,&#39;, &#39;next&#39;, &#39;to&#39;, &#39;the&#39;, &#39;pyramids&#39;, &#39;in&#39;, &#39;Giza&#39;, &#39;,&#39;, &#39;security&#39;, &#39;sources&#39;, &#39;say&#39;, &#39;E.U&#39;, &#39;.&#39;, &#39;South&#39;, &#39;African&#39;, &#39;tourists&#39;, &#39;are&#39;, &#39;among&#39;, &#39;the&#39;, &#39;injured&#39;, &#39;.&#39;, &#39;Most&#39;, &#39;of&#39;, &#39;those&#39;, &#39;hurt&#39;, &#39;suffered&#39;, &#39;minor&#39;, &#39;injuries&#39;, &#39;,&#39;, &#39;while&#39;, &#39;three&#39;, &#39;were&#39;, &#39;treated&#39;, &#39;in&#39;, &#39;hospital&#39;, &#39;,&#39;, &#39;N.A.T.O&#39;, &#39;.&#39;, &#39;say&#39;, &#39;.&#39;, &#39;http&#39;, &#39;:&#39;, &#39;//localhost:8888/notebooks/Text&#39;, &#39;%&#39;, &#39;20preprocessing.ipynb&#39;, &#39;@&#39;, &#39;nickname&#39;, &#39;of&#39;, &#39;twitter&#39;, &#39;user&#39;, &#39;and&#39;, &#39;his&#39;, &#39;email&#39;, &#39;is&#39;, &#39;email&#39;, &#39;@&#39;, &#39;gmail.com&#39;, &#39;.&#39;, &#39;A&#39;, &#39;device&#39;, &#39;went&#39;, &#39;off&#39;, &#39;close&#39;, &#39;to&#39;, &#39;the&#39;, &#39;museum&#39;, &#39;fence&#39;, &#39;as&#39;, &#39;the&#39;, &#39;bus&#39;, &#39;was&#39;, &#39;passing&#39;, &#39;on&#39;, &#39;16/02/2012&#39;, &#39;.&#39;]</pre><p>Spacy:</p><pre>import spacy<br>import en_core_web_sm<br><br>nlp = en_core_web_sm.load()<br><br>doc = nlp(example_text)<br>spacy_words = [token.text for token <strong>in</strong> doc]<br>display(f&quot;Tokenized words: <strong>{spacy_words}</strong>&quot;)</pre><p>Output:</p><pre>Tokenized words: [&#39;\\n&#39;, &#39;An&#39;, &#39;explosion&#39;, &#39;targeting&#39;, &#39;a&#39;, &#39;tourist&#39;, &#39;bus&#39;, &#39;has&#39;, &#39;injured&#39;, &#39;at&#39;, &#39;least&#39;, &#39;16&#39;, &#39;people&#39;, &#39;near&#39;, &#39;the&#39;, &#39;Grand&#39;, &#39;Egyptian&#39;, &#39;Museum&#39;, &#39;,&#39;, &#39;\\n&#39;, &#39;next&#39;, &#39;to&#39;, &#39;the&#39;, &#39;pyramids&#39;, &#39;in&#39;, &#39;Giza&#39;, &#39;,&#39;, &#39;security&#39;, &#39;sources&#39;, &#39;say&#39;, &#39;E.U.&#39;, &#39;\\n\\n&#39;, &#39;South&#39;, &#39;African&#39;, &#39;tourists&#39;, &#39;are&#39;, &#39;among&#39;, &#39;the&#39;, &#39;injured&#39;, &#39;.&#39;, &#39;Most&#39;, &#39;of&#39;, &#39;those&#39;, &#39;hurt&#39;, &#39;suffered&#39;, &#39;minor&#39;, &#39;injuries&#39;, &#39;,&#39;, &#39;\\n&#39;, &#39;while&#39;, &#39;three&#39;, &#39;were&#39;, &#39;treated&#39;, &#39;in&#39;, &#39;hospital&#39;, &#39;,&#39;, &#39;N.A.T.O.&#39;, &#39;say&#39;, &#39;.&#39;, &#39;\\n\\n&#39;, &#39;http://localhost:8888/notebooks&#39;, &#39;/&#39;, &#39;Text%20preprocessing.ipynb&#39;, &#39;\\n\\n&#39;, &#39;@nickname&#39;, &#39;of&#39;, &#39;twitter&#39;, &#39;user&#39;, &#39;and&#39;, &#39;his&#39;, &#39;email&#39;, &#39;is&#39;, &#39;email@gmail.com&#39;, &#39;.&#39;, &#39;\\n\\n&#39;, &#39;A&#39;, &#39;device&#39;, &#39;went&#39;, &#39;off&#39;, &#39;close&#39;, &#39;to&#39;, &#39;the&#39;, &#39;museum&#39;, &#39;fence&#39;, &#39;as&#39;, &#39;the&#39;, &#39;bus&#39;, &#39;was&#39;, &#39;passing&#39;, &#39;on&#39;, &#39;16/02/2012&#39;, &#39;.&#39;, &#39;\\n&#39;]</pre><p>In spacy output tokenization, but not in nltk:</p><pre>{&#39;E.U.&#39;, &#39;\\n&#39;, &#39;Text%20preprocessing.ipynb&#39;, &#39;email@gmail.com&#39;, &#39;\\n\\n&#39;, &#39;N.A.T.O.&#39;, &#39;http://localhost:8888/notebooks&#39;, &#39;@nickname&#39;, &#39;/&#39;}</pre><p>In nltk but not in spacy:</p><pre>{&#39;nickname&#39;, &#39;//localhost:8888/notebooks/Text&#39;, &#39;N.A.T.O&#39;, &#39;:&#39;, &#39;@&#39;, &#39;gmail.com&#39;, &#39;E.U&#39;, &#39;http&#39;, &#39;20preprocessing.ipynb&#39;, &#39;%&#39;}</pre><p>We see that spacy tokenized some weird stuff like \n, \n\n, but was able to handle URLs, emails and Twitter-like mentions. Also, we see that nltk tokenized abbreviations without the last .</p><h3>Cleaning</h3><p>Cleaning is step assumes removing all undesirable content.</p><h4>Punctuation removal</h4><p>Punctuation removal might be a good step, when punctuation does not brings additional value for text vectorization. Punctuation removal is better to be done after the tokenization step, doing it before might cause undesirable effects. Good choice for TF-IDF, Count, Binary vectorization.</p><p>Let’s assume this text for this step:</p><pre>@nickname of twitter user, and his email is email@gmail.com .</pre><p>Before tokenization:</p><pre>text_without_punct = text_with_punct.translate(str.maketrans(&#39;&#39;, &#39;&#39;, string.punctuation))<br>display(f&quot;Text without punctuation: <strong>{text_without_punct}</strong>&quot;)</pre><p>Output:</p><pre>Text without punctuation: nickname of twitter user and his email is emailgmailcom</pre><p>Here you can see that important symbols for correct tokenizations were removed. Now email can’t be properly detected. As you could mention from the Tokenization step, punctuation symbols were parsed as single tokens, so better way would be to tokenize first and then remove punctuation symbols.</p><pre>import spacy<br>import en_core_web_sm<br><br>nlp = en_core_web_sm.load()</pre><pre>doc = nlp(text_with_punct)<br>tokens = [t.text for t <strong>in</strong> doc]</pre><pre><em># python based removal</em><br>tokens_without_punct_python = [t for t <strong>in</strong> tokens if t <strong>not</strong> <strong>in</strong> string.punctuation]<br>display(f&quot;Python based removal: <strong>{tokens_without_punct_python}</strong>&quot;)</pre><pre># spacy based removal<br>tokens_without_punct_spacy = [t.text for t <strong>in</strong> doc if t.pos_ != &#39;PUNCT&#39;]<br>display(f&quot;Spacy based removal: <strong>{tokens_without_punct_spacy}</strong>&quot;)</pre><p>Python based removal result:</p><pre>[&#39;@nickname&#39;, &#39;of&#39;, &#39;twitter&#39;, &#39;user&#39;, &#39;and&#39;, &#39;his&#39;, &#39;email&#39;, &#39;is&#39;, &#39;email@gmail.com&#39;]</pre><p>Spacy based removal:</p><pre>[&#39;of&#39;, &#39;twitter&#39;, &#39;user&#39;, &#39;and&#39;, &#39;his&#39;, &#39;email&#39;, &#39;is&#39;, &#39;email@gmail.com&#39;]</pre><p>Here you see that python-based removal worked even better than spacy because spacy tagged @nicname as PUNCT part-of-speech.</p><h4>Stop words removal</h4><p>Stop words usually refers to the most common words in a language, which usually does not bring additional meaning. There is no single universal list of stop words used by all nlp tools because this term has a very fuzzy definition. Although the practice has shown, that this step is must have when preparing a text for indexing, but might be tricky for text classification purposes.</p><p>Spacy stop words count: 312</p><p>NLTK stop words count: 179</p><p>Let’s assume this text for this step:</p><pre>This movie is just not good enough</pre><p>Spacy:</p><pre>import spacy<br>import en_core_web_sm<br><br>nlp = en_core_web_sm.load()</pre><pre>text_without_stop_words = [t.text for t <strong>in</strong> nlp(text) if <strong>not</strong> t.is_stop]<br>display(f&quot;Spacy text without stop words: <strong>{text_without_stop_words}</strong>&quot;)</pre><p>Spacy text without stop words:</p><pre>[&#39;movie&#39;, &#39;good&#39;]</pre><p>NLTK:</p><pre>import nltk<br><br>nltk_stop_words = nltk.corpus.stopwords.words(&#39;english&#39;)<br>text_without_stop_words = [t for t <strong>in</strong> word_tokenize(text) if t <strong>not</strong> <strong>in</strong> nltk_stop_words]<br>display(f&quot;nltk text without stop words: <strong>{text_without_stop_words}</strong>&quot;)</pre><p>NLTK text without stop words:</p><pre>[&#39;This&#39;, &#39;movie&#39;, &#39;good&#39;, &#39;enough&#39;]</pre><p>Here you see that nltk and spacy have different vocabulary size, so the results of filtering are different. But the main thing I want to underline is that the word not was filtered, which in most cases will be alright, but in the case when you want to determine the polarity of this sentence not will bring additional meaning.</p><p>For such cases, you are able to set stop words you can ignore in spacy library. In the case of nltk you cat just remove or add custom words to nltk_stop_words, it is just a list.</p><pre>import en_core_web_sm<br><br>nlp = en_core_web_sm.load()<br><br>customize_stop_words = [<br>    &#39;not&#39;<br>]<br><br>for w <strong>in</strong> customize_stop_words:<br>    nlp.vocab[w].is_stop = False<br><br>text_without_stop_words = [t.text for t <strong>in</strong> nlp(text) if <strong>not</strong> t.is_stop]<br>display(f&quot;Spacy text without updated stop words: <strong>{text_without_stop_words}</strong>&quot;)</pre><p>Spacy text without updated stop words:</p><pre>[&#39;movie&#39;, &#39;not&#39;, &#39;good&#39;]</pre><h3>Normalization</h3><p>Like any data text requires normalization. In case of text it is:</p><ol><li>Converting dates to text</li><li>Numbers to text</li><li>Currency/Percent signs to text</li><li>Expanding of abbreviations (content dependent) NLP — Natural Language Processing, Neuro-linguistic programming, Non-Linear programming</li><li>Spelling mistakes correction</li></ol><p>To summarize, normalization is a conversion of any non-text information into textual equivalent.</p><p>For this purposes exists a great library — <a href="https://github.com/EFord36/normalise">normalize</a>. I’ll show you the usage of this library from its README. This library is based on nltk package, so it expects nltk word tokens.</p><p>Let’s assume this text for this step:</p><pre>On the 13 Feb. 2007, Theresa May announced on MTV news that the rate of childhod obesity had risen from 7.3-9.6<strong>% i</strong>n just 3 years , costing the N.A.T.O £20m</pre><p>Code:</p><pre>from normalise import normalise<br><br>user_abbr = {<br>    &quot;N.A.T.O&quot;: &quot;North Atlantic Treaty Organization&quot;<br>}<br><br>normalized_tokens = normalise(word_tokenize(text), user_abbrevs=user_abbr, verbose=False)<br>display(f&quot;Normalized text: {&#39; &#39;.join(normalized_tokens)}&quot;)</pre><p>Output:</p><pre>On the thirteenth of February two thousand and seven , Theresa May announced on M T V news that the rate of childhood obesity had risen from seven point three to nine point six % in just three years , costing the North Atlantic Treaty Organization twenty million pounds</pre><p>The worst thing in this library is that for now you can’t disable some modules like abbreviation expanding, and it causes things like MTV -&gt; M T V. But I have already added an appropriate issue on this repository, maybe it would be fixed in a while.</p><h3>Lemmatization and Steaming</h3><p>Stemming is the process of reducing inflection in words to their root forms such as mapping a group of words to the same stem even if the stem itself is not a valid word in the Language.</p><p>Lemmatization, unlike Stemming, reduces the inflected words properly ensuring that the root word belongs to the language. In Lemmatization root word is called Lemma. A lemma (plural lemmas or lemmata) is the canonical form, dictionary form, or citation form of a set of words.</p><p>Let’s assume this text for this step:</p><pre>On the thirteenth of February two thousand and seven , Theresa May announced on M T V news that the rate of childhood obesity had risen from seven point three to nine point six % in just three years , costing the North Atlantic Treaty Organization twenty million pounds</pre><p>NLTK stemmer:</p><pre>import numpy as np<br>from nltk.stem import PorterStemmer<br>from nltk.tokenize import word_tokenize</pre><pre>tokens = word_tokenize(text)<br>porter=PorterStemmer()</pre><pre># vectorizing function to able to call on list of tokens<br>stem_words = np.vectorize(porter.stem)</pre><pre>stemed_text = &#39; &#39;.join(stem_words(tokens))<br>display(f&quot;Stemed text: <strong>{stemed_text}</strong>&quot;)</pre><p>Stemmed text:</p><pre>On the thirteenth of februari two thousand and seven , theresa may announc on M T V news that the rate of childhood obes had risen from seven point three to nine point six % in just three year , cost the north atlant treati organ twenti million pound</pre><p>NLTK lemmatization:</p><pre>import numpy as np<br>from nltk.stem import WordNetLemmatizer<br>from nltk.tokenize import word_tokenize</pre><pre>tokens = word_tokenize(text)</pre><pre>wordnet_lemmatizer = WordNetLemmatizer()</pre><pre># vectorizing function to able to call on list of tokens<br>lemmatize_words = np.vectorize(wordnet_lemmatizer.lemmatize)</pre><pre>lemmatized_text = &#39; &#39;.join(lemmatize_words(tokens))<br>display(f&quot;nltk lemmatized text: <strong>{lemmatized_text}</strong>&quot;)</pre><p>NLTK lemmatized text:</p><pre>On the thirteenth of February two thousand and seven , Theresa May announced on M T V news that the rate of childhood obesity had risen from seven point three to nine point six % in just three year , costing the North Atlantic Treaty Organization twenty million pound</pre><p>Spacy lemmatization:</p><pre>import en_core_web_sm<br><br>nlp = en_core_web_sm.load()</pre><pre>lemmas = [t.lemma_ for t <strong>in</strong> nlp(text)]<br>display(f&quot;Spacy lemmatized text: {&#39; &#39;.join(lemmas)}&quot;)</pre><p>Spacy lemmatized text:</p><pre>On the thirteenth of February two thousand and seven , Theresa May announce on M T v news that the rate of childhood obesity have rise from seven point three to nine point six % in just three year , cost the North Atlantic Treaty Organization twenty million pound</pre><p>We see that spacy lemmatized much better than nltk, one of the examples risen -&gt; rise, only spacy handled that.</p><h3>Reusable pipeline</h3><p>And now my favorite part! We are going to create a reusable pipeline, which you could use on any of your projects.</p><pre>import numpy as np<br>import multiprocessing as mp<br><br>import string<br>import spacy <br>import en_core_web_sm<br>from nltk.tokenize import word_tokenize<br>from sklearn.base import TransformerMixin, BaseEstimator<br>from normalise import normalise<br><br>nlp = en_core_web_sm.load()<br><br><br>class <strong>TextPreprocessor</strong>(BaseEstimator, TransformerMixin):<br>    def __init__(self,<br>                 variety=&quot;BrE&quot;,<br>                 user_abbrevs={},<br>                 n_jobs=1):<br>        <em>&quot;&quot;&quot;</em><br><em>        Text preprocessing transformer includes steps:</em><br><em>            1. Text normalization</em><br><em>            2. Punctuation removal</em><br><em>            3. Stop words removal</em><br><em>            4. Lemmatization</em><br><em>        </em><br><em>        variety - format of date (AmE - american type, BrE - british format) </em><br><em>        user_abbrevs - dict of user abbreviations mappings (from normalise package)</em><br><em>        n_jobs - parallel jobs to run</em><br><em>        &quot;&quot;&quot;</em><br>        self.variety = variety<br>        self.user_abbrevs = user_abbrevs<br>        self.n_jobs = n_jobs<br><br>    def fit(self, X, y=None):<br>        return self<br><br>    def transform(self, X, *_):<br>        X_copy = X.copy()<br><br>        partitions = 1<br>        cores = mp.cpu_count()<br>        if self.n_jobs &lt;= -1:<br>            partitions = cores<br>        elif self.n_jobs &lt;= 0:<br>            return X_copy.apply(self._preprocess_text)<br>        else:<br>            partitions = min(self.n_jobs, cores)<br><br>        data_split = np.array_split(X_copy, partitions)<br>        pool = mp.Pool(cores)<br>        data = pd.concat(pool.map(self._preprocess_part, data_split))<br>        pool.close()<br>        pool.join()<br><br>        return data<br><br>    def _preprocess_part(self, part):<br>        return part.apply(self._preprocess_text)<br><br>    def _preprocess_text(self, text):<br>        normalized_text = self._normalize(text)<br>        doc = nlp(normalized_text)<br>        removed_punct = self._remove_punct(doc)<br>        removed_stop_words = self._remove_stop_words(removed_punct)<br>        return self._lemmatize(removed_stop_words)<br><br>    def _normalize(self, text):<br>        <em># some issues in normalise package</em><br>        try:<br>            return &#39; &#39;.join(normalise(text, variety=self.variety, user_abbrevs=self.user_abbrevs, verbose=False))<br>        except:<br>            return text<br><br>    def _remove_punct(self, doc):<br>        return [t for t <strong>in</strong> doc if t.text <strong>not</strong> <strong>in</strong> string.punctuation]<br><br>    def _remove_stop_words(self, doc):<br>        return [t for t <strong>in</strong> doc if <strong>not</strong> t.is_stop]<br><br>    def _lemmatize(self, doc):<br>        return &#39; &#39;.join([t.lemma_ for t <strong>in</strong> doc])</pre><p>This code could be used in sklearn pipeline.</p><p>Measured performance: 2225 texts were processed on 4 processes for 22 minutes. Not even close to being fast! This causes the normalization part, the library is not sufficiently optimized, but produces rather interesting results and can bring additional value for further vectorization, so it is up to you, whether to use it or not.</p><p>I hope you liked this post and I’m looking forward to your feedback!</p><p>Kaggle kernel: <a href="https://www.kaggle.com/balatmak/text-preprocessing-steps-and-universal-pipeline">https://www.kaggle.com/balatmak/text-preprocessing-steps-and-universal-pipeline</a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=94233cb6725a" width="1" height="1" alt=""><hr><p><a href="https://medium.com/data-science/text-preprocessing-steps-and-universal-pipeline-94233cb6725a">Text preprocessing steps and universal pipeline</a> was originally published in <a href="https://medium.com/data-science">TDS Archive</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
    </channel>
</rss>