<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Shivanee Jaiswal on Medium]]></title>
        <description><![CDATA[Stories by Shivanee Jaiswal on Medium]]></description>
        <link>https://medium.com/@shivaneej02?source=rss-f3986f7d8ce8------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/1*P3kxkXB64Hqa1QWhkZYw4w.jpeg</url>
            <title>Stories by Shivanee Jaiswal on Medium</title>
            <link>https://medium.com/@shivaneej02?source=rss-f3986f7d8ce8------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Fri, 20 Nov 2026 18:11:02 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@shivaneej02/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[Contrast Enhancement of Grayscale Images using Morphological Operators]]></title>
            <link>https://medium.com/data-science/contrast-enhancement-of-grayscale-images-using-morphological-operators-de6d483545a1?source=rss-f3986f7d8ce8------2</link>
            <guid isPermaLink="false">https://medium.com/p/de6d483545a1</guid>
            <category><![CDATA[morphological-operations]]></category>
            <category><![CDATA[opencv]]></category>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[python]]></category>
            <category><![CDATA[image-processing]]></category>
            <dc:creator><![CDATA[Shivanee Jaiswal]]></dc:creator>
            <pubDate>Thu, 13 Jan 2022 06:24:02 GMT</pubDate>
            <atom:updated>2022-01-13T06:24:02.947Z</atom:updated>
            <content:encoded><![CDATA[<h3>Contrast Enhancement of Grayscale Images Using Morphological Operators</h3><h4>An easy way to enhance contrast of grayscale images using Morphological Operators in OpenCV</h4><figure><img alt="A grayscale image" src="https://cdn-images-1.medium.com/max/1024/0*00n2nQhl4-pWH59J" /><figcaption>Photo by <a href="https://unsplash.com/@miladmoafi?utm_source=medium&amp;utm_medium=referral">Milad Moafi</a> on <a href="https://unsplash.com/?utm_source=medium&amp;utm_medium=referral">Unsplash</a></figcaption></figure><h3>Introduction</h3><p>Contrast Enhancement is a very common image processing technique for enhancing features in low contrast images. Several methods like <em>Contrast Stretching</em>, <em>Histogram Equalization</em>, <em>Adaptive Histogram Equalization</em>, <em>Contrast-Limited Adaptive Histogram Equalization </em>or <em>CLAHE</em>, etc. have been used for enhancing the contrast of images. In this article, we will look at another method of Contrast Enhancement which is performed using a combination of Morphological Transformations.</p><h3>What are Morphological Transformations or Operators?</h3><p>Morphological Transformations or Morphological Operators are simple image transformations that are usually applied on binary images, but can be applied to grayscale images as well. There are various types of Morphological Transformations like <em>Erosion</em>, <em>Dilation</em>, <em>Opening</em>, <em>Closing</em>, <em>Gradient</em>, <em>Top Hat </em>and the <em>Black Hat</em>. The two main components of these transformations are the input image and a kernel which is known as <strong>Structuring Element (SE)</strong>. Before we jump on to the different types of Morphological Transformations in detail, let us understand the Structuring Element.</p><h4>What is the Structuring Element (SE)?</h4><p>The <strong>Structuring Element (SE) </strong>is the neighborhood around each pixel that is examined while performing the morphological operations. Structuring Element can be of different shapes and sizes, and changing it can significantly impact the performance of the transformation.</p><p>There are three shapes of the Structuring Element provided by OpenCV — <em>Rectangular</em>, <em>Elliptical</em> and <em>Cross-Shaped</em>. The figure below shows these three shapes.</p><figure><img alt="Three shapes of Structuring Element — Rectangular, Elliptical and Cross Shaped" src="https://cdn-images-1.medium.com/max/1024/1*qwLimOLM3AXjZ3GWeOrlQQ.png" /><figcaption>Different Shapes of Structuring Element for size 5 x 5 (Image by Author)</figcaption></figure><h4>Types of Morphological Operators</h4><p>The different types of Morphological Operators are:</p><ol><li><strong><em>Erosion — </em></strong>According to the OpenCV Documentation, the Erosion operator works similar to soil erosion. It “erodes”, or in simpler words, removes the boundaries of the foreground object in the image. The pixels at the boundary of the foreground object are removed. It reduces the size of the object in the image.</li><li><strong><em>Dilation — </em></strong>Dilation works exactly opposite to erosion. It “dilates” or expands the boundaries of the foreground object which in turn increases the size of the object in the image. This is useful if we want to join broken parts of the foreground object.</li><li><strong><em>Opening — </em></strong>The Opening operation is used in noise removal. It is done by first applying the erosion operation, followed by the dilation operation. The erosion operation first removes all the small noisy areas from the image and then dilation is applied to restore the original size of the object.</li><li><strong><em>Closing — </em></strong>The Closing is the opposite of Opening, which means that dilation is applied first followed by erosion. This is useful for joining or filling small spots that are present in the foreground object, while retaining the size of the object.</li><li><strong><em>Gradient — </em></strong>Morphological<strong><em> </em></strong>Gradient of an image is obtained by subtracting the erosion result from the dilation result. The result of the Gradient operation is the outline of the foreground object in the image.</li><li><strong><em>Top Hat / White Hat— </em></strong>Top Hat Transform, also known as the <em>White Hat Transform</em>, is obtained by removing or subtracting the Opening of the image from the original image. This operator gives us the bright features in the image that are smaller than the Structuring Element.</li><li><strong><em>Black Hat / Bottom Hat— </em></strong>The Black Hat Transform, also known as the <em>Bottom Hat Transform</em>, is obtained by removing or subtracting the Closing of the image from the original image. This operator gives us the dark features in the image that are smaller than the Structuring Element.</li></ol><blockquote><strong>Note:</strong> The Top Hat and the Black Hat transforms are more suited for grayscale images.</blockquote><h4>How do Morphological Transforms help in Contrast Enhancement?</h4><p>Contrast Enhancement, in simple words, requires the following to be done:</p><ol><li>Making the bright regions in the image brighter.</li><li>Making the dark regions in the image darker.</li></ol><p>As we had seen earlier, the result of the Top Hat Transform is an image consisting of all the bright features in the input image and the result of the Black Hat Transform is an image consisting of all the dark features in the input image. Thus, for the purpose of Contrast Enhancement, we will need the Top and the Black Hat Transforms of the input image.</p><p>After obtaining the Top and Black Hat Transforms of the input image, we will <strong>add</strong> the Top Hat Transform to the input image in order to make its bright regions brighter, and <strong>subtract</strong> the Black Hat Transform from the input image to make its dark regions darker.</p><p>The step mentioned above can be represented as an equation shown below:</p><figure><img alt="Equation for Contrast Enhancement using Top and Black Hat Morphological Operators R = I + T — B" src="https://cdn-images-1.medium.com/max/144/1*uqGuOo4CNOAjeOHIQ2jt2A.png" /><figcaption>Equation for Contrast Enhancement</figcaption></figure><p>where <em>R</em> is the result image, <em>I</em> is the input image, <em>T</em> and <em>B</em> are the Top Hat and the Black Hat transforms respectively.</p><p>The flowchart below depicts the steps we will follow to enhance the contrast.</p><figure><img alt="Flowchart showing the steps of Contrast Enhancement" src="https://cdn-images-1.medium.com/max/1024/1*NARZVy-1RL3YN51skjeHqA.png" /><figcaption>Steps for Contrast Enhancement using Morphological Transforms (Image by Author)</figcaption></figure><h3>Code</h3><p>We will implement this Contrast Enhancement technique using Python and OpenCV. We will need to first install opencv-python using pip.</p><pre>pip install opencv-python</pre><p>After installing OpenCV, we will import the library in our code.</p><pre>import cv2 as cv</pre><p>We will use the below image for our code, which is taken from the <a href="https://github.com/ml-workgroup/covid-19-image-repository">COVID-19 Image Repository on GitHub</a>.</p><blockquote>The image data in this repository has been collected from the Institute for Diagnostic and Interventional Radiology, Hannover Medical School, Hannover, Germany and are licensed under the Creative Commons Attribution 3.0 Unported.</blockquote><figure><img alt="Input Image — An X-ray image of the chest" src="https://cdn-images-1.medium.com/max/320/1*F_tzg8Rbp4uTqKDBKMYV8Q.png" /><figcaption>Input Image</figcaption></figure><p>I have scaled down the image by 50% to reduce the size of the image.</p><p>To read this image, we will use the imread function by OpenCV.</p><pre>filename = # path to the image file<br>img = cv.imread(filename,0)</pre><p>Now that we have our image, we will obtain the Top and the Black Hat Transforms of this image. Before this, we need to construct our Structuring Element or the kernel. We can use the getStructuringElement function provided by OpenCV for this purpose.</p><pre>kernel = cv.getStructuringElement(cv.MORPH_ELLIPSE,(5,5))</pre><p>In the above snippet, we have constructed an elliptical Structuring Element of size (5,5). You can experiment by changing these parameters and observe the effect on the output.</p><p>The next step is to get the transforms of the input image using the kernel we constructed in the previous step.</p><pre># Top Hat Transform<br>topHat = cv.morphologyEx(img, cv.MORPH_TOPHAT, kernel)</pre><pre># Black Hat Transform<br>blackHat = cv.morphologyEx(img, cv.MORPH_BLACKHAT, kernel)</pre><p>Once we have our transforms, we will apply the equation that we had seen earlier.</p><pre>res = img + topHat - blackHat</pre><figure><img alt="Output image -An x-ray image of the chest with improved contrast using Structuring Element of size (5,5)" src="https://cdn-images-1.medium.com/max/320/1*-GjRCvU82f2qeRrVmKspWA.png" /><figcaption>Output Image, for elliptical kernel of size (5,5)</figcaption></figure><p>We can see the contrast of the input image has improved a bit. Some of the minute features, that were not prominent in the input image, are now visible. However, this technique also adds some noise to the image if the Structuring Element is not chosen carefully.</p><p>For example, the image below shows the output when an elliptical Structuring Element was chosen of size (15,15). The minute features in the lungs and the edges of the bones are now more prominent and clear than the earlier output, but we can see some noisy areas in the output image, near the boundaries of the body, i.e. the white patches in the background.</p><figure><img alt="Output image -An x-ray image of the chest with improved contrast using Structuring Element of size (15,15)" src="https://cdn-images-1.medium.com/max/320/1*ItV7iI42JbzyhzAY5Zkiyw.png" /><figcaption>Output image, for elliptical kernel of size (15,15)</figcaption></figure><p>As you go on increasing the size of the Structuring Element, the foreground features will become more prominent but the background will start becoming more and more noisy. The output below with Structuring Element of size (35,35) has more noisy area in the background.</p><figure><img alt="Output image -An x-ray image of the chest with improved contrast using Structuring Element of size (35,35)" src="https://cdn-images-1.medium.com/max/320/1*v6a2fyDYERI-trAoI--Lfg.png" /><figcaption>Output image, for elliptical kernel of size (35,35)</figcaption></figure><h3>Conclusion</h3><p>Thus, we saw how to enhance the contrast of grayscale images using a combination of Top Hat and Black Hat Morphological Operations. Some of the minute features were greatly enhanced and became more prominent in our output image as seen above.</p><h4>Points to remember</h4><ol><li>This method might not work as efficiently as the original Contrast Stretching method due to the noise it introduces in the image, as we go on increasing the size of our Structuring Element.</li><li>As the Structuring Element is basically the size of the neighborhood to consider while applying the transformations, the output will also depend on the size of the input image. For example, a (35,35) kernel for an image of size 1000 x 1000 will form a smaller region as compared to a (35,35) kernel for a 250 x 250 image. Thus, resizing the image will also affect the output of this method.</li><li>As Top and Black Hat Transforms give us the brighter and darker features respectively that are <strong>smaller than the Structuring Element, </strong>this method of Contrast Enhancement will work well if the size of the features to enhance are smaller than the size of our chosen Structuring Element. Thus, it becomes difficult to accurately enhance the contrast of larger features, as greater sizes of our kernel will introduce larger noisy areas in our image background.</li></ol><p>Thank you for reading! :) You can connect with me on <a href="https://www.linkedin.com/in/shivaneej/">LinkedIn</a> if you have any questions.</p><h3>References</h3><ol><li>OpenCV: Morphological Transformations <a href="https://docs.opencv.org/3.4/d9/d61/tutorial_py_morphological_ops.html">https://docs.opencv.org/3.4/d9/d61/tutorial_py_morphological_ops.html</a></li><li>Kushol R., Nishat R. M., Rahman A. B. M. A., Salekin M. M., “Contrast Enhancement of Medical X-Ray Image Using Morphological Operators with Optimal Structuring Element,” arXiv:1905.08545v1 [cs.CV] 27 May 2019</li><li>Hinrich B. Winther, Hans Laser, Svetlana Gerbel, Sabine K. Maschke, Jan B. Hinrichs, Jens Vogel-Claussen, Frank K. Wacker, Marius M. Höper, Bernhard C. Meyer, “COVID-19 Image Repository,” DOI: 10.6084/m9.figshare.12275009</li></ol><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=de6d483545a1" width="1" height="1" alt=""><hr><p><a href="https://medium.com/data-science/contrast-enhancement-of-grayscale-images-using-morphological-operators-de6d483545a1">Contrast Enhancement of Grayscale Images using Morphological Operators</a> was originally published in <a href="https://medium.com/data-science">TDS Archive</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Natural Language Processing — Dependency Parsing]]></title>
            <link>https://medium.com/data-science/natural-language-processing-dependency-parsing-cf094bbbe3f7?source=rss-f3986f7d8ce8------2</link>
            <guid isPermaLink="false">https://medium.com/p/cf094bbbe3f7</guid>
            <category><![CDATA[hands-on-tutorials]]></category>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[nlp]]></category>
            <category><![CDATA[python]]></category>
            <category><![CDATA[naturallanguageprocessing]]></category>
            <dc:creator><![CDATA[Shivanee Jaiswal]]></dc:creator>
            <pubDate>Sun, 01 Aug 2021 17:51:55 GMT</pubDate>
            <atom:updated>2021-08-01T18:28:46.279Z</atom:updated>
            <content:encoded><![CDATA[<h4><a href="https://towardsdatascience.com/tagged/hands-on-tutorials">Hands-On Tutorials</a></h4><h3>Natural Language Processing — Dependency Parsing</h3><h4>Different ways for Dependency Parsing using spaCy, NLTK with Stanford CoreNLP and Stanza</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*QolbwbdqxGSmKzuu" /><figcaption>Photo by <a href="https://unsplash.com/@siora18?utm_source=medium&amp;utm_medium=referral">Siora Photography</a> on <a href="https://unsplash.com/?utm_source=medium&amp;utm_medium=referral">Unsplash</a></figcaption></figure><h3>What is Dependency Parsing?</h3><p>Dependency Parsing is the process to analyze the grammatical structure in a sentence and find out related words as well as the type of the relationship between them.</p><p>Each relationship:</p><ol><li>Has one <strong>head</strong><em> </em>and a <strong>dependent </strong>that modifies the <strong>head</strong>.</li><li>Is labeled according to the nature of the dependency between the <strong>head </strong>and the <strong>dependent</strong>. These labels can be found at <a href="https://universaldependencies.org/u/dep/">Universal Dependency Relations</a>.</li></ol><figure><img alt="" src="https://cdn-images-1.medium.com/max/328/1*NrOGJABfU_dp2qOasutr_Q.png" /><figcaption>Simple dependency relation between two words</figcaption></figure><p>In the above diagram, there exists a relationship between <em>car</em><strong><em> </em></strong>and <em>black </em>because <em>black </em>modifies the meaning of <em>car</em>. Here, <em>car </em>acts as the <strong>head </strong>and <em>black</em> is a <strong>dependent </strong>of the head. The nature of the relationship here is <strong>amod </strong>which stands for “Adjectival Modifier”. It is an adjective or an adjective phrase that modifies a noun.</p><h3>Implementation</h3><p>There are different ways to implement dependency parsing in Python. In this article, we will look at three ways.</p><h4>Method 1: Using spaCy</h4><p>spaCy is an open-source Python library for Natural Language Processing.</p><p>To get started, first install spaCy and load the required language model.</p><pre>pip install -U pip setuptools wheel<br>pip install -U spacy<br>python -m spacy download en_core_web_sm</pre><p><strong>en_core_web_sm </strong>is the smallest English model available in spaCy with a size of 12MB. Refer <a href="https://spacy.io/models/en">spaCy English Models</a> to view other available models.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/7eea45eaf3f1f2a9aa3612dcba1b6773/href">https://medium.com/media/7eea45eaf3f1f2a9aa3612dcba1b6773/href</a></iframe><figure><img alt="" src="https://cdn-images-1.medium.com/max/821/1*BIWcMmbtcQz2cnrZOJu2Nw.png" /><figcaption>Dependency Parsing using spaCy</figcaption></figure><p>spaCy also provides a built-in dependency visualizer called <strong>displaCy </strong>that you can use to generate dependency graph for sentences.</p><p>displacy.render() function will generate the visualization for the sentence.</p><blockquote><strong>Note:</strong> If you are running the code in Jupyter Notebook or Google Colab, use jupyter = True in render() function.</blockquote><figure><img alt="" src="https://cdn-images-1.medium.com/max/877/1*mfcStHLTzMZC1evPaSJaag.png" /><figcaption>Output of Dependency Visualization</figcaption></figure><h4>Method 2: Using NLTK with Stanford CoreNLP</h4><p>CoreNLP, created by Stanford NLP Group, provides NLP tools in Java. This Java library can be used with NLTK to parse dependencies in Python.</p><p>The first step is to download the Stanford CoreNLP zip file and Stanford CoreNLP model jar file from the <a href="https://stanfordnlp.github.io/CoreNLP/download.html">CoreNLP website</a>.</p><p>Running these three commands will download the required libraries and unzip the zip file.</p><pre>wget <a href="https://nlp.stanford.edu/software/stanford-corenlp-4.2.2.zip">https://nlp.stanford.edu/software/stanford-corenlp-4.2.2.zip</a><br>wget <a href="https://nlp.stanford.edu/software/stanford-corenlp-4.2.2-models-english.jar">https://nlp.stanford.edu/software/stanford-corenlp-4.2.2-models-english.jar</a><br>unzip /content/stanford-corenlp-4.2.2.zip</pre><p>Once these libraries are downloaded, we can import StanfordDependencyParser from NLTK.</p><pre>from nltk.parse.stanford import StanfordDependencyParser</pre><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/b0a6918cb7c45b3ba2682a42db8d3c9a/href">https://medium.com/media/b0a6918cb7c45b3ba2682a42db8d3c9a/href</a></iframe><figure><img alt="" src="https://cdn-images-1.medium.com/max/903/1*gakpXaH5miMKZLua-a82-A.png" /><figcaption>Dependency Parsing using NLTK and Stanford CoreNLP</figcaption></figure><p>To visualize the dependency generated by CoreNLP, we can either extract a labeled and directed NetworkX Graph object using dependency.nx_graph() function or we can generate a DOT definition in Graph Description Language using dependency.to_dot() function. The DOT definition can be visualized as a graph using <a href="http://graphs.grevian.org/graph">GraphViz</a>.</p><p><strong>Visualize using NetworkX</strong></p><pre># importing networkx<br>import networkx as nx<br>import matplotlib.pyplot as plt</pre><pre># Using reverse() to reverse the direction of edges as nx_graph() returns inverted edges<br>G = dependency.nx_graph().reverse()</pre><pre># nx_graph() returns numeric node labels starting from 1<br># Create a dictionary to map numeric nodes and words in the sentence<br>words = sentence.split(&quot; &quot;)<br>labels = {index + 1: words[index] for index in range(len(words))}</pre><pre>nx.draw(G, with_labels=True, labels=labels, node_size=2500, node_color=&#39;#B5EAD7&#39;, font_size=10)</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/446/1*TeFVGbmlzR45CfRV7MAfyQ.png" /><figcaption>Dependency Graph created from NetworkX Object</figcaption></figure><p><strong>Visualize using DOT Definition</strong></p><pre>!pip install graphviz</pre><pre>from graphviz import Source<br>dot_def = dependency.to_dot()</pre><pre>&#39;&#39;&#39;<br>The below lines require graphviz executables to be installed to your system. If that does not work, print the dot_def object and paste the output on http://graphs.grevian.org/graph/<br>&#39;&#39;&#39;<br>source = Source(dot_def, filename=&quot;dependency_graph&quot;, format=&quot;png&quot;)<br>source.view()</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/266/0*1vOZtc8unoB2h8Cl" /><figcaption>Dependency Graph created from DOT Definition</figcaption></figure><h4>Method 3: Using Stanza</h4><p>Stanford NLP Group have also developed Stanza. It provides a Neural Network NLP Pipeline that can be customized and a Python wrapper over Stanford CoreNLP package, making it easier to use the CoreNLP features without downloading the jar files.</p><p>To get started, install Stanza</p><pre>pip install stanza</pre><p>The next step is to import stanza and download the required language model. You can view all the available models <a href="https://stanfordnlp.github.io/stanza/available_models.html">here</a>.</p><pre>import stanza<br>stanza.download(&#39;en&#39;)</pre><p>Initialize the neural pipeline using stanza.Pipeline() function. The first parameter is the language to use. An optional parameter processors can be passed which can be a dictionary or a comma separated string to configure the processors to use in the pipeline.</p><pre>nlp = stanza.Pipeline(&#39;en&#39;,<br>                     processors = &#39;tokenize,mwt,pos,lemma,depparse&#39;)</pre><p>You can find the list of all processors on <a href="https://stanfordnlp.github.io/stanza/pipeline.html#processors">Pipeline and Processors</a>. Some of the processors might require to be preceded by some other processor in the pipeline, else they won’t work. For example, pos processor requires tokenizeand mwt processors, so we need to use these two processors in the pipeline as well if we want to use pos processor.</p><p>We will now pass our sentence through the pipeline and store all the results in doc variable.</p><pre>doc = nlp(sentence)</pre><p>If we print doc.sentences , we will see a list for each of the sentence that was passed through the pipeline. Each list contains result of all token information and linguistic features.</p><p>For example, each item in the list is an object with properties denoting <em>lemma</em>, <em>universal part-of-speech</em>, <em>treebank-specific part-of-speech</em>, <em>morphological features</em>, <em>index of head</em>, <em>position in the sentence</em>, etc.</p><pre>{<br>    &quot;id&quot;: 1,   -----&gt; index of word in the sentence, starting from 1<br>    &quot;text&quot;: &quot;Deemed&quot;, -----&gt; actual word<br>    &quot;lemma&quot;: &quot;Deem&quot;,  -----&gt; lemma of the word<br>    &quot;upos&quot;: &quot;VERB&quot;, -----&gt; universal POS<br>    &quot;xpos&quot;: &quot;VBN&quot;, -----&gt; treebank specific POS<br>    &quot;feats&quot;: &quot;Tense=Past|VerbForm=Part&quot;,-----&gt; morphological feature<br>    &quot;head&quot;: 2, -----&gt; index of the head<br>    &quot;deprel&quot;: &quot;amod&quot;, -----&gt; dependency relationship nature<br>    &quot;start_char&quot;: 0, -----&gt; start index of word in the sentence<br>    &quot;end_char&quot;: 6  -----&gt; end index of word in the sentence<br>}</pre><p>For all the sentence in the doc object, we can call the print_dependencies() function. This function will print tuples with three values — the token, the index of the head and the relationship nature.</p><blockquote><strong>Note:</strong> In our case as we have only one sentence, we directly access it using doc.sentences[0]</blockquote><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/d007cae84a02ba302e8f089684c8808b/href">https://medium.com/media/d007cae84a02ba302e8f089684c8808b/href</a></iframe><figure><img alt="" src="https://cdn-images-1.medium.com/max/340/1*I8S9-6mbqeovqKzDnI6FPA.png" /><figcaption>Output of print_dependencies() function</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/546/1*YP7Ey_dqnHJ_8iOW--h5uQ.png" /><figcaption>Dependency Parsing using Stanza</figcaption></figure><p>Thus in this article, we saw three ways to implement dependency parsing for text in Python. Thank you for reading! :) You can connect with me on <a href="https://www.linkedin.com/in/shivaneej/">LinkedIn</a> if you have questions about the implementation.</p><h3>References</h3><ol><li>spaCy Dependency Parser — <a href="https://spacy.io/usage/linguistic-features#dependency-parse">https://spacy.io/usage/linguistic-features#dependency-parse</a></li><li>Stanford CoreNLP — <a href="https://stanfordnlp.github.io/CoreNLP/index.html">https://stanfordnlp.github.io/CoreNLP/index.html</a></li><li>Stanza Dependency Parser — <a href="https://stanfordnlp.github.io/stanza/depparse.html">https://stanfordnlp.github.io/stanza/depparse.html</a></li></ol><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=cf094bbbe3f7" width="1" height="1" alt=""><hr><p><a href="https://medium.com/data-science/natural-language-processing-dependency-parsing-cf094bbbe3f7">Natural Language Processing — Dependency Parsing</a> was originally published in <a href="https://medium.com/data-science">TDS Archive</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Build a Player Recommendation System For Cricket Using K-Nearest Neighbor Algorithm]]></title>
            <link>https://medium.com/data-science/build-a-player-recommendation-system-for-cricket-using-k-nearest-neighbor-algorithm-3742d2bfc657?source=rss-f3986f7d8ce8------2</link>
            <guid isPermaLink="false">https://medium.com/p/3742d2bfc657</guid>
            <category><![CDATA[recommendation-system]]></category>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[nearest-neighbors]]></category>
            <category><![CDATA[knn]]></category>
            <dc:creator><![CDATA[Shivanee Jaiswal]]></dc:creator>
            <pubDate>Sat, 24 Jul 2021 16:29:11 GMT</pubDate>
            <atom:updated>2021-07-24T16:29:11.241Z</atom:updated>
            <content:encoded><![CDATA[<h4>A beginner’s guide to using k-Nearest Neighbor Algorithm to build a simple recommender system.</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*hEqqeteRsDOmPIPd" /><figcaption>Photo by <a href="https://unsplash.com/@bushmush?utm_source=medium&amp;utm_medium=referral">michael weir</a> on <a href="https://unsplash.com/?utm_source=medium&amp;utm_medium=referral">Unsplash</a></figcaption></figure><p>In this article, we will build a simple cricket player recommendation system that will suggest a list of batsmen for the team based on the statistics of players that have been playing for the team in the past.</p><blockquote>We will build the recommender system for batsmen only which can be extended for bowlers and other types of players as well by calculating their respective metrics.</blockquote><h3>Overview of k-nearest neighbors</h3><p>In simple terms, k-nearest neighbors (kNN) algorithm finds out <em>k</em> neighbors nearest to a data point based on any distance metric. It is very similar to k-means in the way how similarity of data points is calculated. We will use kNN algorithm to recommend players that are nearest to the current team members.</p><h3><strong>Data Collection</strong></h3><p>The dataset used for this system was downloaded from <a href="https://www.kaggle.com/harsha547/indian-premier-league-csv-dataset">Indian Premier League CSV Dataset</a> on Kaggle. It consists of 6 CSV files that summarize the ball by ball information of 577 IPL matches up to IPL Season 9.</p><h4>Understanding the data</h4><p>We will use Pandas to read the following CSV files.</p><ol><li><strong>Ball_by_Ball.csv —</strong> This file has the data about every ball of the matches. We can extract the id of the players at the striker and non striker end, runs scored, etc. We will use this file to calculate batsmen statistics for our recommender system.</li><li><strong>Match.csv — </strong>This file stores information about the match like the venue, teams, result, umpire details, etc. We will need this file to extract the association between a Match_Id and the Season_Id .</li><li><strong>Player.csv — </strong>This file contains data about all the players, i.e. their name, country, date of birth, etc. These fields will be used to build our recommender system using k-nearest neighbors algorithm.</li><li><strong>Player_Match.csv — </strong>This file associates the Player_Id with the matches they have played. We will use this file to understand the features of the players in the current team.</li></ol><h3>Data Cleaning</h3><p>We will create another data frame called player_datato store batsmen statistics and other relevant features from the existing playerdataframe. As the player dataframe has two columns Is_Umpire and unnamed:7 that are insignificant for our use case, we will drop them and copy the other columns to player_data .</p><pre>player_data = player.drop([&quot;Is_Umpire&quot;, &quot;Unnamed: 7&quot;], axis = 1)</pre><h3>Feature Extraction</h3><h4>Derive Season from Match_Id</h4><p>We will derive the performance statistics of the players in every season. match dataframe has the fields Match_Id and Season_Id that can be used to derive the season number from the Match_Id .</p><pre>NUMBER_OF_SEASONS = 9<br>season_info = pd.DataFrame(columns = [&#39;Season&#39;, &#39;Match_Id_start&#39;, &#39;Match_Id_end&#39;])</pre><pre>for season in range(1, NUMBER_OF_SEASONS + 1):<br>    match_info = match.loc[match[&#39;Season_Id&#39;] == season][&#39;Match_Id&#39;]<br>    season_info = season_info.append({<br>        &#39;Season&#39; : season,<br>        &#39;Match_Id_start&#39; : match_info.min(), <br>        &#39;Match_Id_end&#39; : match_info.max()<br>    }, ignore_index=True)</pre><p>The above code snippet will find the range of Match_Id for every season.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/511/1*Bv7RmqtpPN0PQr9c7DjMPw.png" /><figcaption>Range of Match_Id per season</figcaption></figure><p>Based on the above results, we will create a function that will return the correct season number based on the id of the match.</p><pre>def get_season_from_match_id(match_id):<br>    season = season_info.loc[<br>        (season_info[&#39;Match_Id_start&#39;] &lt;= match_id) &amp; <br>        (season_info[&#39;Match_Id_end&#39;] &gt;= match_id)] [&#39;Season&#39;]<br>    # Return the integer value of the season else return -1 if<br>      season is not found   <br>    return season.item() if not season.empty else -1</pre><h4>Calculation of Batting Performance per season</h4><p>From the ball_by_ball data, we will calculate the following features for all players per season:</p><ol><li>Runs Scored</li><li>Number of innings played</li><li>Number of innings where the player was not out</li><li>Balls Faced</li><li>Number of fours</li><li>Number of sixes</li></ol><p>Calculating runs scored, balls faced, number of fours and number of sixes is pretty straightforward. In the ball_by_balldataframe, we can simply check the values in theStriker_Id and Batsman_Scored columns and accordingly increment these features.</p><p>The difficulty lies in calculating the number of innings played per season and the number of innings where the player was not out. For this, we not only need to look at the current row in the dataframe, but also at the previous row. The innings for a player should be incremented in the following cases:</p><ol><li>Match_Id of the previous ball is different than that of the current ball. This means that the current row belongs to a new match, so a new inning will begin. We will increment the innings count for both the striker and the non striker.</li><li>Match_Id is the same for the previous and the current ball but the Innings_Id changes. This means that it is the second innings of the same match. We will increment the innings count for both the striker and the non-striker.</li><li>Both the Match_Id and Innings_Id are same in the previous and current ball but the current Striker_Id is not equal to previous Striker_Id or Non_Striker_Id . This means that a new player has come to bat, so we will increase the count of innings only for the player with id equal to Striker_Id. Similar logic would be applicable for the current Non_Striker_Id.</li></ol><p>We will also track the Player_dismissedcolumn to find out whether the player was not out in a particular inning.</p><h4>Calculation of Batting Statistics</h4><p>The final step is the calculation of batting statistics like <strong>Batting Strike Rate (BSR)</strong>, <strong>Batting Average (BA)</strong> and <strong>Boundary Runs per Inning (BRPI)</strong>. The BSR, BA and BRPI are first calculated per season and then the mean of these values are calculated with respect to the number of seasons played by the players. While calculating the average metrics, only those seasons are considered in which the batsman had actually played. This removes the bias towards players who have played in all the past seasons.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/442/1*x9qvHrDwISNV47cZYUuM2Q.png" /><figcaption>Calculation of Batting Average per season</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/290/1*wOCFixITTdyAbkFwY2Kljg.png" /><figcaption>Calculation of Batting Strike Rate per season</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/464/1*TMrBbDPhAYzHqi1nYWKujA.png" /><figcaption>Calculation of Boundary Runs Per Inning per season</figcaption></figure><p>Once we have calculated these statistics, we also derive the age of the player and whether they are domestic or international players. These attributes are important because of the assumption that players who are currently over the age of 40 won’t be playing the next season. Also, with the restrictions around choosing international players in an IPL team, the attribute for domestic players will be important.</p><h3>Implementation of K-Nearest Neighbors</h3><p>For the implementation of k-nearest neighbors, we will use scikit learn and build a numpy array with only those features will contribute to the team selection, i.e BA, BSR, BRPI, Age and Nationality.</p><p>X is the numpy array representing those features.</p><pre>from sklearn.neighbors import NearestNeighbors</pre><pre>nbrs = NearestNeighbors(n_neighbors=3, algorithm=&#39;ball_tree&#39;).fit(X)<br>distances, indices = nbrs.kneighbors(X)</pre><p>indices will give the indices of the nearest neighbors for each of the rows in X.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/446/1*NrJuZWmUBbW_wt_3f29VNw.png" /><figcaption>Visualization of first few players and their nearest neighbors. The nodes numbers are id of the players</figcaption></figure><p>For all of the players who have played in a particular team in the current season, kNN will return 3 nearest neighbors that are similar to the existing player.</p><blockquote>Out of the 3 players suggested, the first player will be the same as the existing one because the distance of a player with himself is 0.</blockquote><p>All the recommended players are added to an ordered collection and sorted by the highest BSR to lowest. We can program to return the top <em>n</em> batsmen where <em>n</em> can be changed as required.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*HPZ-uTY8jfQRwWTl-75H1A.png" /><figcaption>Final Output recommending 15 players sorted by Batting Strike Rate.</figcaption></figure><p>This was a very simple implementation of the batsmen recommendation system. You can view the GitHub repository for the code <a href="https://github.com/shivaneej/Batsmen-Recommendation-System">here</a>. Thank you for reading! :)</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=3742d2bfc657" width="1" height="1" alt=""><hr><p><a href="https://medium.com/data-science/build-a-player-recommendation-system-for-cricket-using-k-nearest-neighbor-algorithm-3742d2bfc657">Build a Player Recommendation System For Cricket Using K-Nearest Neighbor Algorithm</a> was originally published in <a href="https://medium.com/data-science">TDS Archive</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
    </channel>
</rss>