<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Mustafa Aytuğ Kaya on Medium]]></title>
        <description><![CDATA[Stories by Mustafa Aytuğ Kaya on Medium]]></description>
        <link>https://medium.com/@m.aytugkaya?source=rss-de9f6a439ca2------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/0*V71ftfTeuvkXkKrf.jpg</url>
            <title>Stories by Mustafa Aytuğ Kaya on Medium</title>
            <link>https://medium.com/@m.aytugkaya?source=rss-de9f6a439ca2------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Sat, 21 Nov 2026 01:00:03 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@m.aytugkaya/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[SpaCy or Spark NLP — A Benchmarking Comparison]]></title>
            <link>https://medium.com/spark-nlp/spacy-or-spark-nlp-a-benchmarking-comparison-23202f12a65c?source=rss-de9f6a439ca2------2</link>
            <guid isPermaLink="false">https://medium.com/p/23202f12a65c</guid>
            <category><![CDATA[spark]]></category>
            <category><![CDATA[spark-nlp]]></category>
            <dc:creator><![CDATA[Mustafa Aytuğ Kaya]]></dc:creator>
            <pubDate>Thu, 27 Aug 2020 06:28:18 GMT</pubDate>
            <atom:updated>2020-08-28T09:31:25.232Z</atom:updated>
            <content:encoded><![CDATA[<h3>SpaCy or Spark NLP — A Benchmarking Comparison</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*qmvnS9Hs_cNK4Eej" /><figcaption><a href="https://www.pexels.com/photo/beautiful-country-creek-environment-259735/">Photo: Pixabay</a></figcaption></figure><p>The aim of this article is to run a realistic Natural Language Processing scenario to compare the leading linguistic programming libraries: enterprise-grade John Snow Labs’ <a href="https://nlp.johnsnowlabs.com/docs/en/quickstart">Spark NLP</a> and Explosion AI’s industrial-strength library <a href="https://spacy.io/">spaCy</a>, both of which are open-source with commercially permissive licenses.</p><p>Comparing two different libraries is not as simple as it sounds. Each library has different implementation methods and thus will have different use cases, data pipelines, and characteristics. In this study, a detailed Spark NLP pipeline will be designed and a parallel code mimicking that will be written using spaCy, focusing primarily on runtime speed. We will then compare the results in terms of memory usage, speed, and accuracy.</p><p>Spacy is one of the best-documented libraries I have seen and now they include <a href="https://course.spacy.io/en/">a free course</a>, which I have taken previously and would highly recommend for a quick brush-up. Spark NLP has a well-designed <a href="https://nlp.johnsnowlabs.com/docs/en/quickstart">website</a> loaded with tons of useful information. For a quick start, I would recommend reading <a href="https://towardsdatascience.com/introduction-to-spark-nlp-foundations-and-basic-components-part-i-c83b7629ed59">this article</a> followed by <a href="https://medium.com/spark-nlp/spark-nlp-101-document-assembler-500018f5f6b5">this article</a>. Many useful notebooks can be found in John Snow Labs’ <a href="https://github.com/JohnSnowLabs/spark-nlp-workshop/tree/master/tutorials/Certification_Trainings/Public">repo</a> for practice and a better understanding of Spark NLP’s dynamics.</p><p><strong>Data:</strong></p><p>We will be using a library consisting of seven different classics downloaded from Gutenberg.org. The corpus consists of approximately 4.9 M characters and 97 thousand sentences.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/478/1*pU88CEHtKCV5kRmbr5AAkg.png" /></figure><p>The whole notebook of the comparison and the corpus data can be found in my <a href="https://github.com/aytugkaya/spark_nlp/tree/master/spaCy_SparkNLP_comparison">GitHub repo</a>. Let’s start by examining the spaCy way.</p><pre>### 2.1 Clean Tabs and Whitespace<br>clean_shrink = lambda text : text.replace(r&#39;\n|\t|\s+&#39;,&#39; &#39;).replace(&#39;\s+&#39;,&#39; &#39;).strip()<br>df.loc[:,&#39;document&#39;]=df.text.map(clean_shrink)</pre><pre>### 2.2 Tokenizer<br>sentence_tokenizer = lambda sent : [token for token in nlp(sent)]<br>df.loc[:,&#39;token&#39;]=df.document.map(sentence_tokenizer)</pre><pre>### 2.3 Normalizer<br>normalizer = lambda tokens : [re.sub(punct,&#39;&#39;,token.text)  for token in tokens if re.sub(punct,&#39;&#39;,token.text) != &#39;&#39;]<br>df.loc[:,&#39;normalized&#39;]=df.token.map(normalizer)</pre><pre>### 2.4 Remove Stop Words<br>normalizer_and_stop = lambda tokens : [re.sub(punct,&#39;&#39;,token.text)  for token in tokens if re.sub(punct,&#39;&#39;,token.text) != &#39;&#39; and not token.is_stop]<br>df.loc[:,&#39;cleanTokens&#39;]=df.token.map(normalizer_and_stop)</pre><pre>### 2.5 Lemmatize<br>normalizer_and_stop_lemma = lambda tokens : [re.sub(punct,&#39;&#39;,token.lemma_)  for token in tokens if re.sub(punct,&#39;&#39;,token.text) != &#39;&#39; and not token.is_stop]<br>df.loc[:,&#39;lemma&#39;]=df.token.map(normalizer_and_stop_lemma)</pre><pre>### 2.6 Stemmer<br>stemmer = PorterStemmer()<br>stems = lambda tokens : [stemmer.stem(token.text) if len(tokens)&gt;0 else [] for token in tokens]<br>df.loc[:,&#39;stem&#39;]=df.token.map(stems)</pre><pre>### 2.7 Part of Speech Tagging<br>normalizer_and_stop_pos = lambda tokens : [re.sub(punct,&#39;&#39;,token.pos_)  for token in tokens if re.sub(punct,&#39;&#39;,token.text) != &#39;&#39; and not token.is_stop]<br>df.loc[:,&#39;pos&#39;]=df.cleanTokens.map(normalizer_and_stop_pos)</pre><pre>### 2.8 Token Assembler<br>token_assembler = lambda tokens : &quot; &quot;.join(tokens)<br>df.loc[:,&#39;clean_text&#39;]=df.cleanTokens.map(token_assembler)</pre><pre>### 2.9 Tagger<br>tagger = lambda text : [(ent.text, ent.label_) for ent in nlp(text).ents]<br>df.loc[:,&#39;ner_chunks&#39;]=df.loc[:,&#39;document&#39;].map(tagger)</pre><pre>### 2.10 Regex Parser<br>noun_chunker = lambda text : [(chnk,(chnk[0].pos_,chnk[1].pos_,chnk[2].tag_ ))for chnk in nlp(text).noun_chunks if len(chnk.text.split())==3\<br>                              and  chnk.text.replace(&#39; &#39;,&#39;&#39;).isalpha()   and chnk[0].pos_ == &#39;DET&#39;and chnk[1].pos_ == &#39;ADJ&#39; and chnk[2].tag_ in [&#39;NN&#39;,&#39;NNP&#39;]<br>                             ]<br>df.loc[:,&#39;RegexpParser&#39;] =df.loc[:,&#39;document&#39;].map(noun_chunker)</pre><p>Let’s take a brief pause here and observe the Regex Parser output. We asked for the noun chunker to return chunks that consist of a <strong>Determiner</strong>, an <strong>Adjective</strong>, and a <strong>Noun</strong> (proper, singular or plural).</p><pre>[chunk for chunk in df.RegexpParser.values if chunk!=[]]</pre><p>The results look pretty good.</p><pre>[[(My dear Robinson, (&#39;DET&#39;, &#39;ADJ&#39;, &#39;NNP&#39;))],<br> [(this accidental souvenir, (&#39;DET&#39;, &#39;ADJ&#39;, &#39;NN&#39;))],<br> [(a great deal, (&#39;DET&#39;, &#39;ADJ&#39;, &#39;NN&#39;))],<br> [(a great amount, (&#39;DET&#39;, &#39;ADJ&#39;, &#39;NN&#39;))],<br> [(the local hunt, (&#39;DET&#39;, &#39;ADJ&#39;, &#39;NN&#39;)),<br>  (some surgical assistance, (&#39;DET&#39;, &#39;ADJ&#39;, &#39;NN&#39;))],<br> [(a remarkable power, (&#39;DET&#39;, &#39;ADJ&#39;, &#39;NN&#39;))],<br> [(a good deal, (&#39;DET&#39;, &#39;ADJ&#39;, &#39;NN&#39;))],<br> [(a fresh basis, (&#39;DET&#39;, &#39;ADJ&#39;, &#39;NN&#39;)),<br>  (this unknown visitor, (&#39;DET&#39;, &#39;ADJ&#39;, &#39;NN&#39;))],<br> [(the obvious conclusion, (&#39;DET&#39;, &#39;ADJ&#39;, &#39;NN&#39;))],<br> [(their good will, (&#39;DET&#39;, &#39;ADJ&#39;, &#39;NN&#39;))],<br> [(my dear Watson, (&#39;DET&#39;, &#39;ADJ&#39;, &#39;NNP&#39;)),<br>  (a young fellow, (&#39;DET&#39;, &#39;ADJ&#39;, &#39;NN&#39;))],<br> [(a favourite dog, (&#39;DET&#39;, &#39;ADJ&#39;, &#39;NN&#39;))],<br> [(the latter part, (&#39;DET&#39;, &#39;ADJ&#39;, &#39;NN&#39;))],<br> [(that local hunt, (&#39;DET&#39;, &#39;ADJ&#39;, &#39;NN&#39;))],<br> [(a heavy stick, (&#39;DET&#39;, &#39;ADJ&#39;, &#39;NN&#39;))],<br> [(a professional brother, (&#39;DET&#39;, &#39;ADJ&#39;, &#39;NN&#39;))],<br> [(the dramatic moment, (&#39;DET&#39;, &#39;ADJ&#39;, &#39;NN&#39;))],<br> [(a long nose, (&#39;DET&#39;, &#39;ADJ&#39;, &#39;NN&#39;))],<br>................</pre><p>Let’s continue building our blocks.</p><pre>### 2.11 N-Gram Generator<br>ngram_generator = lambda input_list: [*zip(*[input_list[i:] for i in range(n)])]<br>n=3<br>df.loc[:,&#39;triGrams&#39;] = df.loc[:,&#39;token&#39;].map(ngram_generator)</pre><pre>### 2.12 Word2Vec Embeddings<br>vector = lambda tokens: [(token.text, token.has_vector, token.vector, token.is_oov) for token in tokens]<br>df.loc[:,&#39;vectors&#39;] = df.loc[:,&#39;token&#39;].map(vector)</pre><pre>### 2.13 Regex Matcher<br>rules = r&#39;&#39;&#39;\b[A-Z]\w+ly\b|Stephen\s(?!Proto|Cardinal)[A-Z]\w+|Simon\s[A-Z]\w+&#39;&#39;&#39;<br>regex_matchers = lambda text : re.findall(rules,text)<br>df.loc[:,&#39;Regex_matches&#39;] =df.loc[:,&#39;document&#39;].map(regex_matchers)</pre><p>Let’s take a look at regex matches</p><pre>df.Regex_matches[df.Regex_matches.map(len)&gt;1]</pre><p>Here are the results:</p><pre>13123                      [Polly, Polly]<br>25669                      [Sally, Sally]<br>27262                      [Sally, Sally]<br>27273                      [Polly, Sally]<br>27340                      [Polly, Sally]<br>28311                      [Pelly, Dolly]<br>42016                      [Feely, Feely]<br>49802                    [Finally, Emily]<br>52129                    [Lively, Lively]<br>58295                      [Silly, Milly]<br>62141                      [Silly, Milly]<br>64811                       [Only, Molly]<br>71650                        [Hely, Daly]<br>74427                      [Healy, Dolly]<br>77404                      [Molly, Milly]<br>77437                      [Milly, Molly]<br>81557                     [Molly, Reilly]<br>84023          [Szombathely, Szombathely]<br>89594                       [Healy, Joly]<br>92206    [Simon Dedalus, Stephen Dedalus]<br>92980      [Firstly, Nelly, Nelly, Nelly]<br>93046               [Szombathely, Karoly]<br>94402       [Reilly, Simon Dedalus, Hely]<br>94489    [Stephen Dedalus, Simon Dedalus]</pre><p>Let’s Venture Into The Characters…</p><p>Now that we have a dataset with many features, we have a plethora of options to dive into. Let’s examine the characters that are in the books…Let’s find <strong>NER Chunks</strong> that have a <strong>‘PERSON’</strong> tag, consisting of 2 words.</p><pre>flatlist = lambda l : [re.sub(&quot;[^a-zA-Z\s\&#39;]&quot;,&quot;&quot;,item[0]).title().strip()  for sublist in l for item in sublist if item[1]==&#39;PERSON&#39; and len(item[0].split())==2]<br>ner_chunks = df.ner_chunks.to_list()<br>names=(flatlist(ner_chunks))<br>len(sorted(names))</pre><p>The code above returned 4832 names, which looks a bit suspicious since this number is high. Let’s inspect the result of the Counter object:</p><pre>(&#39;St Clare&#39;, 306),<br> (&#39;Buck Mulligan&#39;, 96),<br> (&#39;Aunt Chloe&#39;, 76),<br> (&#39;Martin Cunningham&#39;, 74),<br> (&#39;Masr George&#39;, 45),<br> (&#39;Ned Lambert&#39;, 44),<br> (&#39;Solomon Northup&#39;, 43),<br> (&#39;Tom Sawyer&#39;, 42),<br> (&#39;Aunt Sally&#39;, 37),<br> (&#39;Uncle Tom&#39;, 37),<br> (&#39;Ben Dollard&#39;, 36),<br> (&#39;Myles Crawford&#39;, 35),<br> (&#39;Blazes Boylan&#39;, 28),<br> (&#39;Uncle Abram&#39;, 24),<br> (&#39;J J&#39;, 22),<br>...... <br> (&#39;L L&#39;, 5),<br> (&#39;Mark Twain&#39;, 5),<br> (&quot;Tom Sawyer&#39;S&quot;, 5),<br> (&#39;Chapter Xi&#39;, 5),<br> (&#39;Chapter Xix&#39;, 5),<br> (&#39;Dey Wuz&#39;, 5),<br> (&#39;George Jackson&#39;, 5),<br> (&#39;Levi Bell&#39;, 5),<br> (&#39;King Lear&#39;, 5),<br> (&#39;Simon Legree&#39;, 5),<br> (&#39;Garrett Deasy&#39;, 5),<br> (&#39;A E&#39;, 5),<br> (&#39;S D&#39;, 5),<br> (&#39;Josie Powell&#39;, 5),<br> (&#39;Mrs Purefoy&#39;, 5),<br> (&#39;Ben Howth&#39;, 5),<br> (&#39;Bald Pat&#39;, 5),<br> (&#39;Barney Kiernan&#39;, 5),<br> (&#39;Michael Gunn&#39;, 5),<br> (&#39;C C&#39;, 5),</pre><p>Many tags are inaccurate, unfortunately. Please observe some chapter titles as well as capitalized initials that are inaccurately returned as PER tags.</p><p><em>In writing the above code, mappings were used to ensure fast pacing, and the Spark NLP pipeline that will be implemented below was mimicked. Once we run a similar code in Spark NLP, we will compare results in terms of memory usage, speed, and accuracy.</em></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/564/1*Ev6_2rEeJP2JGxflYpWWmw.png" /></figure><p>Time to do things Spark NLP way!</p><pre>documentAssembler = DocumentAssembler()\<br>.setInputCol(“text”)\<br>.setOutputCol(“document”)\<br>.setCleanupMode(“shrink”)</pre><pre>sentenceDetector = SentenceDetector().\<br>    setInputCols([&#39;document&#39;]).\<br>    setOutputCol(&#39;sentences&#39;)</pre><pre>tokenizer = Tokenizer() \<br>    .setInputCols([&quot;sentences&quot;]) \<br>    .setOutputCol(&quot;token&quot;)</pre><pre>ngrams = NGramGenerator() \<br>            .setInputCols([&quot;token&quot;]) \<br>            .setOutputCol(&quot;ngrams&quot;) \<br>            .setN(3) \<br>            .setEnableCumulative(False)\<br>            .setDelimiter(&quot;_&quot;) # Default is space</pre><pre>normalizer = Normalizer() \<br>    .setInputCols([&quot;token&quot;]) \<br>    .setOutputCol(&quot;normalized&quot;)\<br>    .setLowercase(False)\<br>    .setCleanupPatterns([&quot;[^\w\d\s\.\!\?]&quot;])</pre><pre>stopwords_cleaner = StopWordsCleaner()\<br>      .setInputCols(&quot;normalized&quot;)\<br>      .setOutputCol(&quot;cleanTokens&quot;)\<br>      .setCaseSensitive(False)\</pre><pre>lemma = LemmatizerModel.pretrained(&#39;lemma_antbnc&#39;) \<br>    .setInputCols([&quot;cleanTokens&quot;]) \<br>    .setOutputCol(&quot;lemma&quot;)</pre><pre>stemmer = Stemmer() \<br>    .setInputCols([&quot;token&quot;]) \<br>    .setOutputCol(&quot;stem&quot;)</pre><pre>pos = PerceptronModel.pretrained(&quot;pos_anc&quot;, &#39;en&#39;)\<br>      .setInputCols(&quot;clean_text&quot;, &quot;cleanTokens&quot;)\<br>      .setOutputCol(&quot;pos&quot;)</pre><pre>chunker = Chunker()\<br>    .setInputCols([&quot;sentences&quot;, &quot;pos&quot;])\<br>    .setOutputCol(&quot;chunk&quot;)\<br>    .setRegexParsers([&quot;&lt;DT&gt;+&lt;JJ&gt;*&lt;NN&gt;&quot;])  ## Determiner - adjective - singular noun</pre><pre>tokenassembler = TokenAssembler()\<br>    .setInputCols([&quot;sentences&quot;, &quot;cleanTokens&quot;]) \<br>    .setOutputCol(&quot;clean_text&quot;)\</pre><pre>tokenizer2 = Tokenizer() \<br>    .setInputCols([&quot;clean_text&quot;]) \<br>    .setOutputCol(&quot;token2&quot;)</pre><pre>glove_embeddings = WordEmbeddingsModel.pretrained(&#39;glove_100d&#39;)\<br>          .setInputCols([&quot;document&quot;, &quot;lemma&quot;])\<br>          .setOutputCol(&quot;embeddings&quot;)\<br>          .setCaseSensitive(False)</pre><pre>onto_ner = NerDLModel.pretrained(&quot;onto_100&quot;, &#39;en&#39;) \<br>          .setInputCols([&quot;document&quot;, &quot;token&quot;, &quot;embeddings&quot;]) \<br>          .setOutputCol(&quot;ner&quot;)</pre><pre>ner_converter = NerConverter() \<br>  .setInputCols([&quot;sentences&quot;, &quot;token&quot;, &quot;ner&quot;]) \<br>  .setOutputCol(&quot;ner_chunk&quot;)</pre><pre>rules = r&#39;&#39;&#39;<br>\b[A-Z]\w+ly\b, starting with a capital letter ending with &#39;ly&#39;<br>Stephen\s(?!Proto|Cardinal)[A-Z]\w+, followed by &quot;Stephen&quot;<br>Simon\s[A-Z]\w+, followed by &quot;Simon&quot;<br>&#39;&#39;&#39;</pre><pre>with open(&#39;ulyses_regex_rules.txt&#39;, &#39;w&#39;) as f:<br>    <br>    f.write(rules)</pre><pre>regex_matcher = RegexMatcher()\<br>    .setInputCols(&#39;sentences&#39;)\<br>    .setStrategy(&quot;MATCH_ALL&quot;)\<br>    .setOutputCol(&quot;regex_matches&quot;)\<br>    .setExternalRules(path=&#39;./ulyses_regex_rules.txt&#39;, delimiter=&#39;,&#39;)</pre><p>Now that our pieces are ready, let’s define the assembly line.</p><pre>nlpPipeline = Pipeline(stages=[<br> documentAssembler,<br> sentenceDetector,<br> tokenizer,<br> ngrams,<br> normalizer,<br> stopwords_cleaner,<br> lemma,<br> stemmer,<br> tokenassembler,<br> tokenizer2,<br> pos,<br> chunker,<br> glove_embeddings,<br> onto_ner,<br> ner_converter,<br> regex_matcher<br> <br> ])</pre><pre>empty_df = spark.createDataFrame([[‘’]]).toDF(“text”)</pre><pre>pipelineModel = nlpPipeline.fit(empty_df)</pre><pre>lib_result = pipelineModel.transform(library)</pre><p>Let’s check regex matches according to our search criteria:<br>- A whole word that begins with a capital letter and ends with ‘ly’,<br>- ‘Stephen’ that is not followed by ‘Cardinal’ or ‘Proto’, but followed by a word that starts with a capital letter.<br>- ‘Simon’ is followed by a word that starts with a capital letter.<br>We are looking for at least two occurrences in each sentence…</p><pre>match_df.filter(F.size(&#39;finished_regex_matches&#39;)&gt;1).show(truncate = 50)</pre><p>Let’s take a look at the results…</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/672/1*ngIEMgZ0Ol6AODRJIXALjg.png" /></figure><p>The chunker annotator in our pipeline is going to return chunks that consist of a Determiner, an Adjective, and a singular Noun.</p><pre>lib_result.withColumn(<br>    &quot;tmp&quot;, <br>    F.explode(&quot;chunk&quot;)) \<br>    .select(&quot;tmp.*&quot;).select(&quot;begin&quot;,&quot;end&quot;,&quot;result&quot;,&quot;metadata.sentence&quot;).show(20,truncate = 100)</pre><p>Here are the top 20 rows of the chunker results.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/486/1*wJmnuRapGnbwX4PTl4FiLg.png" /></figure><p>Let’s Venture Into The Characters…Spark NLP Way.</p><p>Here we examine the characters that are in the books…This time we will be using Spark NLP mechanics. Please note differences in accuracy as compared to spaCy.</p><pre>l = result_ner.filter(result_ner.ner_label == &quot;PERSON&quot;).select(F.expr(&quot;ner_chunk&quot;)).collect()</pre><pre>names = list([re.sub(&quot;[^a-zA-Z\s\&#39;]&quot;,&quot;&quot;,l_[0]).title() for l_ in l if l_[0].replace(&#39; &#39;,&#39;&#39;).isalpha() and len(l_[0].strip().split())==2 and &quot;’&quot; not in l_[0]])</pre><pre>len(set(names))</pre><p>This time, the number of characters we have are limited to 1284. Let’s take a look at the most common 350 names.</p><pre>(&#39;Buck Mulligan&#39;, 93),<br> (&#39;Aunt Chloe&#39;, 82),<br> (&#39;Martin Cunningham&#39;, 71),<br> (&#39;Bayou Boeuf&#39;, 48),<br> (&#39;Aunt Sally&#39;, 39),<br> (&#39;Ned Lambert&#39;, 39),<br> (&#39;Mary Jane&#39;, 38),<br> (&#39;Solomon Northup&#39;, 36),<br> (&#39;John Thornton&#39;, 34),<br> (&#39;Myles Crawford&#39;, 33),<br> (&#39;Ben Dollard&#39;, 31),<br> (&#39;Sherlock Holmes&#39;, 30),<br> (&#39;Tom Sawyer&#39;, 30),<br> (&#39;John Eglinton&#39;, 29),<br> (&#39;Nosey Flynn&#39;, 28),<br> (&#39;Corny Kelleher&#39;, 27),<br> (&#39;Mrs Breen&#39;, 27),<br> (&#39;Father Conmee&#39;, 26),<br> (&#39;Uncle Tom&#39;, 25),<br> (&#39;John Wyse&#39;, 24),<br> (&#39;Henry Baskerville&#39;, 23),<br> (&#39;Uncle Abram&#39;, 22),<br> (&#39;Blazes Boylan&#39;, 19),<br> (&#39;Bob Doran&#39;, 18),<br> (&#39;Davy Byrne&#39;, 18),<br> (&#39;Coombe Tracey&#39;, 17),<br> (&#39;Aunt Phebe&#39;, 17),<br> (&#39;Simon Dedalus&#39;, 17),<br>........................<br> (&#39;Sandy Hill&#39;, 8),<br> (&#39;Theophilus Freeman&#39;, 8),<br> (&#39;Father Cowley&#39;, 8),<br> (&#39;Gregory B&#39;, 7),<br> (&#39;George Harris&#39;, 7),<br> (&#39;Rachel Halliday&#39;, 7),<br> (&#39;George Shelby&#39;, 7),<br> (&#39;Anne Hampton&#39;, 7),<br> (&#39;Peter Tanner&#39;, 7),<br> (&#39;Almidano Artifoni&#39;, 7),<br> (&#39;Hugo Baskerville&#39;, 6),<br> (&#39;Laura Lyons&#39;, 6),<br> (&#39;Aunt Polly&#39;, 6),<br> (&#39;Peter Wilks&#39;, 6),<br>.........................</pre><p>The names list looks much more accurate. No wonder why Spark NLP is enterprise preferred!</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/620/1*yyXS8XGlokTVH7tstakqCw.png" /><figcaption>SpaCy is good, but Spark NLP is better…</figcaption></figure><p><strong>Let’s Talk about Resource Consumption:</strong><br>The system used for this study is an 8 core Intel(R) Core(TM) i7–9700K CPU @ 3.60GHz with 32820MB Memory. The operating system is Ubuntu 20.04.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/3d213d22d8eb34be9be32368d17be0fa/href">https://medium.com/media/3d213d22d8eb34be9be32368d17be0fa/href</a></iframe><p><strong><em>Spark NLP uses less memory and runs twice as fast when compared to spaCy. This fact, being coupled with higher accuracy of the Spark NLP provides good reasons to master this library!</em></strong></p><p>In this article, we compared the NLP pipeline in both libraries. While implementing the same process flow is very difficult using two completely different libraries, code was mimicked to the maximum extent possible. As expected, Spark NLP proved to be faster and more accurate in terms of Named Entity Recognition. However, for small-sized datasets, spaCy may be more practical and possibly even faster, but when the data size increases, Spark NLP’s speed becomes clearly visible. No wonder Spark NLP is the weapon of first choice for enterprises.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=23202f12a65c" width="1" height="1" alt=""><hr><p><a href="https://medium.com/spark-nlp/spacy-or-spark-nlp-a-benchmarking-comparison-23202f12a65c">SpaCy or Spark NLP — A Benchmarking Comparison</a> was originally published in <a href="https://medium.com/spark-nlp">spark-nlp</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[How to Wrap Your Head Around Spark NLP]]></title>
            <link>https://medium.com/data-science/how-to-wrap-your-head-around-spark-nlp-a6f6a968b7e8?source=rss-de9f6a439ca2------2</link>
            <guid isPermaLink="false">https://medium.com/p/a6f6a968b7e8</guid>
            <category><![CDATA[spark]]></category>
            <dc:creator><![CDATA[Mustafa Aytuğ Kaya]]></dc:creator>
            <pubDate>Tue, 25 Aug 2020 21:19:54 GMT</pubDate>
            <atom:updated>2020-08-25T21:19:54.894Z</atom:updated>
            <content:encoded><![CDATA[<h4>*Follow up on <em>“How to Get Started with SparkNLP in 2 Weeks — Part I”</em></h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*Y0id1KPSHlieo2QN" /><figcaption><a href="https://www.pexels.com/photo/pine-trees-by-lake-in-forest-against-sky-247474/">Photo by: Pixabay</a></figcaption></figure><p>Welcome to the second part of the Spark NLP article. In the first part, the objective was to present an ice breaker for NLP practitioners and warm-up minds towards Spark NLP. The strongest bias against any Spark-based library comes from the school of thought that states <em>“Spark code is a bit different than your regular Python script”</em>. To fight this prejudice, learning strategies were shared, and if you have followed them, you are ready for the next level.</p><p>In this part of the article, we will compare spaCy to Spark NLP and dive deeper into Spark NLP modules and pipelines. I created a notebook that uses both spaCy and Spark NLP for doing the same things for picture-perfect comparison. SpaCy performs well, but in terms of speed, memory consumption, and accuracy Spark NLP outperforms it. Therefore, in terms of ease of use, once initial friction of Spark is overcome, I found it to be at least on par with spaCy, thanks to the convenience that comes with pipelines.</p><p>Usage of smart, <a href="https://github.com/JohnSnowLabs/spark-nlp-workshop/tree/master/tutorials/Certification_Trainings/Public">comprehensive notebooks</a>, prepared by Spark NLP creators, which provide numerous examples for real-life scenarios together with the <a href="https://github.com/aytugkaya/spark_nlp">repo</a> I created for practice is highly recommended to excel in the required skill sets.</p><p><strong>Day 8/9: Understanding Annotators /Transformers in Spark NLP and </strong><a href="https://github.com/JohnSnowLabs/spark-nlp-workshop/blob/master/tutorials/Certification_Trainings/Public/2.Text_Preprocessing_with_SparkNLP_Annotators_Transformers.ipynb"><strong>Text Preprocessing </strong></a><strong>with Spark</strong></p><p>Built natively on Apache Spark and <a href="https://www.analyticsindiamag.com/a-hands-on-primer-to-tensorflow/">TensorFlow</a>, Spark NLP library provides simple, performant as well as accurate NLP notations for machine learning pipelines which can scale easily in a distributed environment. This library reuses the Spark ML pipeline along with integrating NLP functionality.</p><p>The library covers many common NLP tasks, including tokenization, stemming, lemmatization, part of speech tagging, sentiment analysis, spell checking, named entity recognition, all of which are included as open-source and can be used by training models with your data. Spark NLP’s annotators utilize rule-based algorithms, machine learning, and some <a href="https://www.tensorflow.org/">Tensorflow</a> running under the hood to power specific deep learning implementations.</p><p>In Spark NLP, all <strong>Annotators</strong> are either Estimators or Transformers just like Spark ML, consisting of two types: <strong>AnnotatorApproach</strong> and <strong>AnnotatorModel. </strong>Any annotator training on a DataFrame that produces a model is an AnnotatorApproach. Those transforming a DataFrame into another DataFrame through some models are <em>AnnotatorModels</em> (e.g. <em>WordEmbeddingsModel</em>). Normally, an annotator does not take <em>Model </em>suffix if it doesn’t rely on a pre-trained annotator while transforming a DataFrame (e.g. <em>Tokenizer</em>). Here is a list of Annotators and their descriptions:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/51842f448aa6dd53585a73d386925776/href">https://medium.com/media/51842f448aa6dd53585a73d386925776/href</a></iframe><p>To get through the NLP process, we need to get raw data preprocessed. In addition to SQL filters, transformations, and user-defined functions, Spark NLP comes with powerful tools for the task. <strong><em>DocumentAssembler </em></strong>is a special transformer that creates the first annotation of type Document which may be used by follow up annotators in the pipeline.</p><p><strong><em>TokenAssembler </em></strong>reconstructs a Document type annotation from tokens, usually after these have been normalized, lemmatized, normalized, spell checked, etc, to use this document annotation in further annotators.</p><p><strong><em>Doc2Chunk </em></strong>converts Document type annotations into Chunk type with the contents of a chunk Col, while <strong><em>Chunk2Doc </em></strong>converts a CHUNK type column back into DOCUMENT. Useful when trying to re-tokenize or do further analysis on a CHUNK result.</p><p><strong><em>Finisher </em></strong>outputs annotation(s) values into a string for ease of use. Once we have our NLP pipeline ready to go, we might want to use our annotation results somewhere else where it is practically available.</p><p>As stated earlier, a Pipeline is specified as a sequence of stages, and each stage is either a Transformer or an Estimator. These stages are run in order, and the input DataFrame is transformed as it passes through each stage.</p><p>And here is how this pipeline is coded in Spark NLP.</p><pre>from pyspark.ml import Pipeline</pre><pre>document_assembler = DocumentAssembler()\<br> .setInputCol(“text”)\<br> .setOutputCol(“document”)</pre><pre>sentenceDetector = SentenceDetector()\<br> .setInputCols([“document”])\<br> .setOutputCol(“sentences”)tokenizer = Tokenizer() \<br> .setInputCols([“sentences”]) \<br> .setOutputCol(“token”)normalizer = Normalizer()\<br> .setInputCols([“token”])\<br> .setOutputCol(“normal”)</pre><pre>word_embeddings=WordEmbeddingsModel.pretrained()\<br> .setInputCols([“document”,”normal”])\<br> .setOutputCol(“embeddings”)</pre><pre>nlpPipeline = Pipeline(stages=[<br> document_assembler, <br> sentenceDetector,<br> tokenizer,<br> normalizer,<br> word_embeddings,<br> ])</pre><pre>pipelineModel = nlpPipeline.fit(df)</pre><p>When we fit() on the pipeline with Spark data frame (df), its <em>text</em> column is fed into DocumentAssembler() transformer at first and then a new column “<em>document</em>” is created in <em>Document</em> type (<em>AnnotatorType</em>). As mentioned before, this transformer is the initial entry point to Spark NLP for any Spark data frame. Then its document column is fed into SentenceDetector()<em> </em>(<em>AnnotatorApproach</em>) and the text is split into an array of sentences and a new column “<em>sentences” in Document </em>type<em> </em>is created. Then the “sentences” column is fed into Tokenizer() (<em>AnnotatorModel</em>) and each sentence is tokenized and a new column “<em>token” in Token </em>type<em> </em>is created. And so on. Please refer to the <a href="https://github.com/JohnSnowLabs/spark-nlp-workshop/blob/master/tutorials/Certification_Trainings/Public/2.Text_Preprocessing_with_SparkNLP_Annotators_Transformers.ipynb">Spark NLP workshop preprocessing notebook</a> and <a href="https://github.com/aytugkaya/spark_nlp">my repo</a> for seeing annotators and transformers in action.</p><p><strong>Day 10: Regex</strong><br><em>Regular expressions are essentially a tiny, highly specialized programming language embedded inside Python and made available through the </em><a href="https://docs.python.org/3/library/re.html#module-re"><em>re module</em></a><em>.</em> <em>Using this little language, you specify the rules for the set of possible strings that you want to match; this set might contain English sentences, or e-mail addresses, clinical entities, or anything you like. You can also use regexes to modify a string or to split it apart in various ways.</em></p><p>Regex is one of the major building blocks of NLP, which we will be using not only for preprocessing but also as a part of RegexMatcher in the Spark NLP pipeline. As it can sometimes be quite complicated, I recommend this <a href="https://learn.datacamp.com/courses/regular-expressions-in-python">course</a> by Maria Eugenia Inzaugarat to have a quick grasp on advanced topics, and please refer to the <a href="https://docs.python.org/3/library/re.html">official python documentation</a> as well as the brilliant python <a href="https://docs.python.org/3/howto/regex.html#regex-howto">regex how to</a> tutorial. Another key component I used for mastering regex was <a href="http://regex101.com/">Regex101</a>, a very powerful API that enables you to test almost any regular expression on text chunks.</p><p>Most of the time Regex is overlooked. Trust me: you will need it and you will use it. The more you know, the better.</p><p><strong>Day 11: Pretrained Models</strong></p><p>It was previously mentioned that trained annotators are called <em>AnnotatorModel</em> and the goal here is to transform one DataFrame into another through the specified model (trained annotator). Spark NLP offers pre-trained models in many languages and all you need to do is load the pre-trained model and configure its parameters accordingly. Why worry about training a new model from scratch when you can enjoy the direct application of pre-trained SOTA algorithms with a single transform()method? In the <a href="https://nlp.johnsnowlabs.com/docs/en/models">official documentation</a>, you can find detailed information regarding how these models are trained by using which algorithms and datasets. <a href="https://www.johnsnowlabs.com/spark-nlp-in-action/">Sample notebooks can be found here</a>.</p><pre>MODEL_NAME=&#39;sentimentdl_use_twitter&#39;</pre><pre>documentAssembler = DocumentAssembler()\<br>    .setInputCol(&quot;text&quot;)\<br>    .setOutputCol(&quot;document&quot;)</pre><pre>use = UniversalSentenceEncoder.pretrained(name=&quot;tfhub_use&quot;, lang=&quot;en&quot;)\<br> .setInputCols([&quot;document&quot;])\<br> .setOutputCol(&quot;sentence_embeddings&quot;)</pre><pre>sentimentdl = SentimentDLModel.pretrained(name=MODEL_NAME, lang=&quot;en&quot;)\<br>    .setInputCols([&quot;sentence_embeddings&quot;])\<br>    .setOutputCol(&quot;pr_sentiment&quot;)</pre><pre>nlpPipeline = Pipeline(<br>      stages = [<br>          documentAssembler,<br>          use,<br>          sentimentdl<br>      ])</pre><pre>empty_df = spark.createDataFrame([[&#39;&#39;]]).toDF(&quot;text&quot;)</pre><pre>pipelineModel = nlpPipeline.fit(empty_df)</pre><pre>result = pipelineModel.transform(spark_df1)</pre><p><strong>Day 12/13: Text Classification</strong></p><p>There are several text classification options in Spark NLP:</p><ul><li>Text preprocessing in Spark NLP and using ML algorithms from <a href="https://spark.apache.org/docs/latest/ml-guide.html">Spark ML</a></li><li>Text preprocessing and word embeddings (Glove, Bert, Elmo) in Spark NLP and ML algorithms from Spark ML</li><li>Text preprocessing and sentence embeddings (Universal Sentence Encoders) in Spark NLP and ML algorithms from Spark ML</li><li>Text preprocessing and ClassifierDL (TensorFlow based DL architecture) in Spark NLP</li></ul><p>As discussed thoroughly in <a href="https://towardsdatascience.com/introduction-to-spark-nlp-foundations-and-basic-components-part-i-c83b7629ed59">this article about Spark NLP</a>, all these text processing steps before ClassifierDL can be implemented in a pipeline that is specified as a sequence of stages, and each stage is either a Transformer or an Estimator. These stages are run in order, and the input DataFrame is transformed as it passes through each stage. That is, the data are passed through the fitted pipeline in order. Each stage’s transform()<em> </em>method updates the dataset and passes it to the next stage. With the help of Pipelines, we can ensure that training and test data go through identical feature processing steps.</p><p>You can build a text classifier with Bert, Elmo, Glove and Universal Sentence Encoders in Spark NLP with this ClassiferDL. Sample classifier notebooks can be found <a href="https://github.com/JohnSnowLabs/spark-nlp-workshop/blob/master/tutorials/Certification_Trainings/Public/5.Text_Classification_with_ClassifierDL.ipynb">here</a> and <a href="https://github.com/aytugkaya/spark_nlp/tree/master/toxicity">here</a>.</p><pre>document_assembler = DocumentAssembler() \<br>    .setInputCol(&quot;comment_text&quot;) \<br>    .setOutputCol(&quot;document&quot;)\<br>    .setCleanupMode(&#39;shrink&#39;)</pre><pre>tokenizer = Tokenizer() \<br>  .setInputCols([&quot;document&quot;]) \<br>  .setOutputCol(&quot;token&quot;)<br>    <br>normalizer = Normalizer() \<br>    .setInputCols([&quot;token&quot;]) \<br>    .setOutputCol(&quot;normalized&quot;)\<br>    .setLowercase(True)</pre><pre>stopwords_cleaner = StopWordsCleaner()\<br>      .setInputCols(&quot;normalized&quot;)\<br>      .setOutputCol(&quot;cleanTokens&quot;)\<br>      .setCaseSensitive(False)</pre><pre>lemma = LemmatizerModel.pretrained(&#39;lemma_antbnc&#39;) \<br>    .setInputCols([&quot;cleanTokens&quot;]) \<br>    .setOutputCol(&quot;lemma&quot;)</pre><pre>bert = BertEmbeddings.pretrained(&#39;bert_base_uncased&#39;, &#39;en&#39;) \<br>      .setInputCols(&quot;document&quot;, &quot;lemma&quot;) \<br>      .setOutputCol(&quot;embeddings&quot;)\<br>      .setPoolingLayer(0) # default 0</pre><pre>embeddingsSentence = SentenceEmbeddings() \<br>      .setInputCols([&quot;document&quot;, &quot;embeddings&quot;]) \<br>      .setOutputCol(&quot;sentence_embeddings&quot;) \<br>      .setPoolingStrategy(&quot;AVERAGE&quot;)</pre><pre>classsifierdl = ClassifierDLApproach()\<br>  .setInputCols([&quot;sentence_embeddings&quot;])\<br>  .setOutputCol(&quot;prediction&quot;)\<br>  .setLabelColumn(&quot;toxic&quot;)\<br>  .setMaxEpochs(5)\<br>  .setEnableOutputLogs(True)\<br>  .setBatchSize(32)\<br>  .setValidationSplit(0.1)\<br>  .setDropout(0.75)\<br>  .setLr(0.0035)\<br>  #.setOutputLogsPath(&#39;logs&#39;)</pre><pre>clf_pipeline = Pipeline(<br>    stages=[document_assembler, <br>            tokenizer,<br>            normalizer,<br>            stopwords_cleaner, <br>            lemma, <br>            bert,<br>            embeddingsSentence,<br>            classsifierdl<br>           ])</pre><pre>clf_pipelineModel = clf_pipeline.fit(sdf)</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*kx7OVEyRvBQk02zbKgfnow.png" /></figure><p><strong>Day 13–14: SpaCy or Spark NLP — A Benchmarking Comparison</strong></p><p>In this section, we will get a lot of exercise coding! We will be using a library consisting of seven different classics downloaded from Gutenberg.org. The corpus consists of approximately 4.9 M characters and 97 thousand sentences. We will build the same output data frames using similar processes both by spaCy and Spark NLP and then compare the results.</p><p>Spacy is one of the best-documented libraries I have seen and I was pleasantly surprised that now they include <a href="https://course.spacy.io/en/">a free course</a>, which I have taken previously. A good chance to brush up your skills, if required.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/478/1*pU88CEHtKCV5kRmbr5AAkg.png" /></figure><p>The whole notebook of the comparison and the corpus data can be found in my <a href="https://github.com/aytugkaya/spark_nlp/tree/master/spaCy_SparkNLP_comparison">GitHub repo</a>. Let’s start by examining the spaCy way.</p><pre>### 2.1 Clean Tabs and Whitespace<br>clean_shrink = lambda text : text.replace(r&#39;\n|\t|\s+&#39;,&#39; &#39;).replace(&#39;\s+&#39;,&#39; &#39;).strip()<br>df.loc[:,&#39;document&#39;]=df.text.map(clean_shrink)</pre><pre>### 2.2 Tokenizer<br>sentence_tokenizer = lambda sent : [token for token in nlp(sent)]<br>df.loc[:,&#39;token&#39;]=df.document.map(sentence_tokenizer)</pre><pre>### 2.3 Normalizer<br>normalizer = lambda tokens : [re.sub(punct,&#39;&#39;,token.text)  for token in tokens if re.sub(punct,&#39;&#39;,token.text) != &#39;&#39;]<br>df.loc[:,&#39;normalized&#39;]=df.token.map(normalizer)</pre><pre>### 2.4 Remove Stop Words<br>normalizer_and_stop = lambda tokens : [re.sub(punct,&#39;&#39;,token.text)  for token in tokens if re.sub(punct,&#39;&#39;,token.text) != &#39;&#39; and not token.is_stop]<br>df.loc[:,&#39;cleanTokens&#39;]=df.token.map(normalizer_and_stop)</pre><pre>### 2.5 Lemmatize<br>normalizer_and_stop_lemma = lambda tokens : [re.sub(punct,&#39;&#39;,token.lemma_)  for token in tokens if re.sub(punct,&#39;&#39;,token.text) != &#39;&#39; and not token.is_stop]<br>df.loc[:,&#39;lemma&#39;]=df.token.map(normalizer_and_stop_lemma)</pre><pre>### 2.6 Stemmer<br>stemmer = PorterStemmer()<br>stems = lambda tokens : [stemmer.stem(token.text) if len(tokens)&gt;0 else [] for token in tokens]<br>df.loc[:,&#39;stem&#39;]=df.token.map(stems)</pre><pre>### 2.7 Part of Speech Tagging<br>normalizer_and_stop_pos = lambda tokens : [re.sub(punct,&#39;&#39;,token.pos_)  for token in tokens if re.sub(punct,&#39;&#39;,token.text) != &#39;&#39; and not token.is_stop]<br>df.loc[:,&#39;pos&#39;]=df.cleanTokens.map(normalizer_and_stop_pos)</pre><pre>### 2.8 Token Assembler<br>token_assembler = lambda tokens : &quot; &quot;.join(tokens)<br>df.loc[:,&#39;clean_text&#39;]=df.cleanTokens.map(token_assembler)</pre><pre>### 2.9 Tagger<br>tagger = lambda text : [(ent.text, ent.label_) for ent in nlp(text).ents]<br>df.loc[:,&#39;ner_chunks&#39;]=df.loc[:,&#39;document&#39;].map(tagger)</pre><pre>### 2.10 Regex Parser<br>noun_chunker = lambda text : [(chnk,(chnk[0].pos_,chnk[1].pos_,chnk[2].tag_ ))for chnk in nlp(text).noun_chunks if len(chnk.text.split())==3\<br>                              and  chnk.text.replace(&#39; &#39;,&#39;&#39;).isalpha()   and chnk[0].pos_ == &#39;DET&#39;and chnk[1].pos_ == &#39;ADJ&#39; and chnk[2].tag_ in [&#39;NN&#39;,&#39;NNP&#39;]<br>                             ]<br>df.loc[:,&#39;RegexpParser&#39;] =df.loc[:,&#39;document&#39;].map(noun_chunker)</pre><p>Let’s take a brief pause here and observe the Regex Parser output. We asked for the noun chunker to return chunks that consist of a <strong>Determiner</strong>, an <strong>Adjective</strong>, and a <strong>Noun</strong> (proper, singular or plural).</p><pre>[chunk for chunk in df.RegexpParser.values if chunk!=[]]</pre><p>The results look pretty good.</p><pre>[[(My dear Robinson, (&#39;DET&#39;, &#39;ADJ&#39;, &#39;NNP&#39;))],<br> [(this accidental souvenir, (&#39;DET&#39;, &#39;ADJ&#39;, &#39;NN&#39;))],<br> [(a great deal, (&#39;DET&#39;, &#39;ADJ&#39;, &#39;NN&#39;))],<br> [(a great amount, (&#39;DET&#39;, &#39;ADJ&#39;, &#39;NN&#39;))],<br> [(the local hunt, (&#39;DET&#39;, &#39;ADJ&#39;, &#39;NN&#39;)),<br>  (some surgical assistance, (&#39;DET&#39;, &#39;ADJ&#39;, &#39;NN&#39;))],<br> [(a remarkable power, (&#39;DET&#39;, &#39;ADJ&#39;, &#39;NN&#39;))],<br> [(a good deal, (&#39;DET&#39;, &#39;ADJ&#39;, &#39;NN&#39;))],<br> [(a fresh basis, (&#39;DET&#39;, &#39;ADJ&#39;, &#39;NN&#39;)),<br>  (this unknown visitor, (&#39;DET&#39;, &#39;ADJ&#39;, &#39;NN&#39;))],<br> [(the obvious conclusion, (&#39;DET&#39;, &#39;ADJ&#39;, &#39;NN&#39;))],<br> [(their good will, (&#39;DET&#39;, &#39;ADJ&#39;, &#39;NN&#39;))],<br> [(my dear Watson, (&#39;DET&#39;, &#39;ADJ&#39;, &#39;NNP&#39;)),<br>  (a young fellow, (&#39;DET&#39;, &#39;ADJ&#39;, &#39;NN&#39;))],<br> [(a favourite dog, (&#39;DET&#39;, &#39;ADJ&#39;, &#39;NN&#39;))],<br> [(the latter part, (&#39;DET&#39;, &#39;ADJ&#39;, &#39;NN&#39;))],<br> [(that local hunt, (&#39;DET&#39;, &#39;ADJ&#39;, &#39;NN&#39;))],<br> [(a heavy stick, (&#39;DET&#39;, &#39;ADJ&#39;, &#39;NN&#39;))],<br> [(a professional brother, (&#39;DET&#39;, &#39;ADJ&#39;, &#39;NN&#39;))],<br> [(the dramatic moment, (&#39;DET&#39;, &#39;ADJ&#39;, &#39;NN&#39;))],<br> [(a long nose, (&#39;DET&#39;, &#39;ADJ&#39;, &#39;NN&#39;))],<br>................</pre><p>Let’s continue building our blocks.</p><pre>### 2.11 N-Gram Generator<br>ngram_generator = lambda input_list: [*zip(*[input_list[i:] for i in range(n)])]<br>n=3<br>df.loc[:,&#39;triGrams&#39;] = df.loc[:,&#39;token&#39;].map(ngram_generator)</pre><pre>### 2.12 Word2Vec Embeddings<br>vector = lambda tokens: [(token.text, token.has_vector, token.vector, token.is_oov) for token in tokens]<br>df.loc[:,&#39;vectors&#39;] = df.loc[:,&#39;token&#39;].map(vector)</pre><pre>### 2.13 Regex Matcher<br>rules = r&#39;&#39;&#39;\b[A-Z]\w+ly\b|Stephen\s(?!Proto|Cardinal)[A-Z]\w+|Simon\s[A-Z]\w+&#39;&#39;&#39;<br>regex_matchers = lambda text : re.findall(rules,text)<br>df.loc[:,&#39;Regex_matches&#39;] =df.loc[:,&#39;document&#39;].map(regex_matchers)</pre><p>Let’s take a look at regex matches</p><pre>df.Regex_matches[df.Regex_matches.map(len)&gt;1]</pre><p>Here are the results:</p><pre>13123                      [Polly, Polly]<br>25669                      [Sally, Sally]<br>27262                      [Sally, Sally]<br>27273                      [Polly, Sally]<br>27340                      [Polly, Sally]<br>28311                      [Pelly, Dolly]<br>42016                      [Feely, Feely]<br>49802                    [Finally, Emily]<br>52129                    [Lively, Lively]<br>58295                      [Silly, Milly]<br>62141                      [Silly, Milly]<br>64811                       [Only, Molly]<br>71650                        [Hely, Daly]<br>74427                      [Healy, Dolly]<br>77404                      [Molly, Milly]<br>77437                      [Milly, Molly]<br>81557                     [Molly, Reilly]<br>84023          [Szombathely, Szombathely]<br>89594                       [Healy, Joly]<br>92206    [Simon Dedalus, Stephen Dedalus]<br>92980      [Firstly, Nelly, Nelly, Nelly]<br>93046               [Szombathely, Karoly]<br>94402       [Reilly, Simon Dedalus, Hely]<br>94489    [Stephen Dedalus, Simon Dedalus]</pre><p>Let’s Venture Into The Characters…</p><p>Now that we have a dataset with many features, we have a plethora of options to dive into. Let’s examine the characters that are in the books…Let’s find <strong>NER Chunks</strong> that have a <strong>‘PERSON’</strong> tag, consisting of 2 words.</p><pre>flatlist = lambda l : [re.sub(&quot;[^a-zA-Z\s\&#39;]&quot;,&quot;&quot;,item[0]).title().strip()  for sublist in l for item in sublist if item[1]==&#39;PERSON&#39; and len(item[0].split())==2]<br>ner_chunks = df.ner_chunks.to_list()<br>names=(flatlist(ner_chunks))<br>len(sorted(names))</pre><p>The code above returned 4832 names, which looks a bit suspicious since this number is high. Let’s inspect the result of the Counter object:</p><pre>(&#39;St Clare&#39;, 306),<br> (&#39;Buck Mulligan&#39;, 96),<br> (&#39;Aunt Chloe&#39;, 76),<br> (&#39;Martin Cunningham&#39;, 74),<br> (&#39;Masr George&#39;, 45),<br> (&#39;Ned Lambert&#39;, 44),<br> (&#39;Solomon Northup&#39;, 43),<br> (&#39;Tom Sawyer&#39;, 42),<br> (&#39;Aunt Sally&#39;, 37),<br> (&#39;Uncle Tom&#39;, 37),<br> (&#39;Ben Dollard&#39;, 36),<br> (&#39;Myles Crawford&#39;, 35),<br> (&#39;Blazes Boylan&#39;, 28),<br> (&#39;Uncle Abram&#39;, 24),<br> (&#39;J J&#39;, 22),<br>...... <br> (&#39;L L&#39;, 5),<br> (&#39;Mark Twain&#39;, 5),<br> (&quot;Tom Sawyer&#39;S&quot;, 5),<br> (&#39;Chapter Xi&#39;, 5),<br> (&#39;Chapter Xix&#39;, 5),<br> (&#39;Dey Wuz&#39;, 5),<br> (&#39;George Jackson&#39;, 5),<br> (&#39;Levi Bell&#39;, 5),<br> (&#39;King Lear&#39;, 5),<br> (&#39;Simon Legree&#39;, 5),<br> (&#39;Garrett Deasy&#39;, 5),<br> (&#39;A E&#39;, 5),<br> (&#39;S D&#39;, 5),<br> (&#39;Josie Powell&#39;, 5),<br> (&#39;Mrs Purefoy&#39;, 5),<br> (&#39;Ben Howth&#39;, 5),<br> (&#39;Bald Pat&#39;, 5),<br> (&#39;Barney Kiernan&#39;, 5),<br> (&#39;Michael Gunn&#39;, 5),<br> (&#39;C C&#39;, 5),</pre><p>Many tags are inaccurate, unfortunately. Please observe some chapter titles as well as capitalized initials that are inaccurately returned as PER tags.</p><p>In writing the above code, mappings were used to ensure fast pacing, and the Spark NLP pipeline that will be implemented below was mimicked. Once we run a similar code in Spark NLP, we will compare results in terms of memory usage, speed, and accuracy.</p><p>Time to do things Spark NLP way!</p><pre>documentAssembler = DocumentAssembler()\<br>.setInputCol(“text”)\<br>.setOutputCol(“document”)\<br>.setCleanupMode(“shrink”)</pre><pre>sentenceDetector = SentenceDetector().\<br>    setInputCols([&#39;document&#39;]).\<br>    setOutputCol(&#39;sentences&#39;)</pre><pre>tokenizer = Tokenizer() \<br>    .setInputCols([&quot;sentences&quot;]) \<br>    .setOutputCol(&quot;token&quot;)</pre><pre>ngrams = NGramGenerator() \<br>            .setInputCols([&quot;token&quot;]) \<br>            .setOutputCol(&quot;ngrams&quot;) \<br>            .setN(3) \<br>            .setEnableCumulative(False)\<br>            .setDelimiter(&quot;_&quot;) # Default is space</pre><pre>normalizer = Normalizer() \<br>    .setInputCols([&quot;token&quot;]) \<br>    .setOutputCol(&quot;normalized&quot;)\<br>    .setLowercase(False)\<br>    .setCleanupPatterns([&quot;[^\w\d\s\.\!\?]&quot;])</pre><pre>stopwords_cleaner = StopWordsCleaner()\<br>      .setInputCols(&quot;normalized&quot;)\<br>      .setOutputCol(&quot;cleanTokens&quot;)\<br>      .setCaseSensitive(False)\</pre><pre>lemma = LemmatizerModel.pretrained(&#39;lemma_antbnc&#39;) \<br>    .setInputCols([&quot;cleanTokens&quot;]) \<br>    .setOutputCol(&quot;lemma&quot;)</pre><pre>stemmer = Stemmer() \<br>    .setInputCols([&quot;token&quot;]) \<br>    .setOutputCol(&quot;stem&quot;)</pre><pre>pos = PerceptronModel.pretrained(&quot;pos_anc&quot;, &#39;en&#39;)\<br>      .setInputCols(&quot;clean_text&quot;, &quot;cleanTokens&quot;)\<br>      .setOutputCol(&quot;pos&quot;)</pre><pre>chunker = Chunker()\<br>    .setInputCols([&quot;sentences&quot;, &quot;pos&quot;])\<br>    .setOutputCol(&quot;chunk&quot;)\<br>    .setRegexParsers([&quot;&lt;DT&gt;+&lt;JJ&gt;*&lt;NN&gt;&quot;])  ## Determiner - adjective - singular noun</pre><pre>tokenassembler = TokenAssembler()\<br>    .setInputCols([&quot;sentences&quot;, &quot;cleanTokens&quot;]) \<br>    .setOutputCol(&quot;clean_text&quot;)\</pre><pre>tokenizer2 = Tokenizer() \<br>    .setInputCols([&quot;clean_text&quot;]) \<br>    .setOutputCol(&quot;token2&quot;)</pre><pre>glove_embeddings = WordEmbeddingsModel.pretrained(&#39;glove_100d&#39;)\<br>          .setInputCols([&quot;document&quot;, &quot;lemma&quot;])\<br>          .setOutputCol(&quot;embeddings&quot;)\<br>          .setCaseSensitive(False)</pre><pre>onto_ner = NerDLModel.pretrained(&quot;onto_100&quot;, &#39;en&#39;) \<br>          .setInputCols([&quot;document&quot;, &quot;token&quot;, &quot;embeddings&quot;]) \<br>          .setOutputCol(&quot;ner&quot;)</pre><pre>ner_converter = NerConverter() \<br>  .setInputCols([&quot;sentences&quot;, &quot;token&quot;, &quot;ner&quot;]) \<br>  .setOutputCol(&quot;ner_chunk&quot;)</pre><pre>rules = r&#39;&#39;&#39;<br>\b[A-Z]\w+ly\b, starting with a capital letter ending with &#39;ly&#39;<br>Stephen\s(?!Proto|Cardinal)[A-Z]\w+, followed by &quot;Stephen&quot;<br>Simon\s[A-Z]\w+, followed by &quot;Simon&quot;<br>&#39;&#39;&#39;</pre><pre>with open(&#39;ulyses_regex_rules.txt&#39;, &#39;w&#39;) as f:<br>    <br>    f.write(rules)</pre><pre>regex_matcher = RegexMatcher()\<br>    .setInputCols(&#39;sentences&#39;)\<br>    .setStrategy(&quot;MATCH_ALL&quot;)\<br>    .setOutputCol(&quot;regex_matches&quot;)\<br>    .setExternalRules(path=&#39;./ulyses_regex_rules.txt&#39;, delimiter=&#39;,&#39;)</pre><p>Now that our pieces are ready, let’s define the assembly line.</p><pre>nlpPipeline = Pipeline(stages=[<br> documentAssembler,<br> sentenceDetector,<br> tokenizer,<br> ngrams,<br> normalizer,<br> stopwords_cleaner,<br> lemma,<br> stemmer,<br> tokenassembler,<br> tokenizer2,<br> pos,<br> chunker,<br> glove_embeddings,<br> onto_ner,<br> ner_converter,<br> regex_matcher<br> <br> ])</pre><pre>empty_df = spark.createDataFrame([[‘’]]).toDF(“text”)</pre><pre>pipelineModel = nlpPipeline.fit(empty_df)</pre><pre>lib_result = pipelineModel.transform(library)</pre><p>Let’s check regex matches according to our search criteria:<br>- A whole word that begins with a capital letter and ends with ‘ly’,<br>- ‘Stephen’ that is not followed by ‘Cardinal’ or ‘Proto’, but followed by a word that starts with a capital letter.<br>- ‘Simon’ is followed by a word that starts with a capital letter.<br>We are looking for at least two occurrences in each sentence…</p><pre>match_df.filter(F.size(&#39;finished_regex_matches&#39;)&gt;1).show(truncate = 50)</pre><p>Let’s take a look at the results…</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/672/1*ngIEMgZ0Ol6AODRJIXALjg.png" /></figure><p>The chunker annotator in our pipeline is going to return chunks that consist of a Determiner, an Adjective, and a singular Noun.</p><pre>lib_result.withColumn(<br>    &quot;tmp&quot;, <br>    F.explode(&quot;chunk&quot;)) \<br>    .select(&quot;tmp.*&quot;).select(&quot;begin&quot;,&quot;end&quot;,&quot;result&quot;,&quot;metadata.sentence&quot;).show(20,truncate = 100)</pre><p>Here are the top 20 rows of the chunker results.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/486/1*wJmnuRapGnbwX4PTl4FiLg.png" /></figure><p>Let’s Venture Into The Characters…Spark NLP Way.</p><p>Here we examine the characters that are in the books…This time we will be using Spark NLP mechanics. Please note differences in accuracy as compared to spaCy.</p><pre>l = result_ner.filter(result_ner.ner_label == &quot;PERSON&quot;).select(F.expr(&quot;ner_chunk&quot;)).collect()</pre><pre>names = list([re.sub(&quot;[^a-zA-Z\s\&#39;]&quot;,&quot;&quot;,l_[0]).title() for l_ in l if l_[0].replace(&#39; &#39;,&#39;&#39;).isalpha() and len(l_[0].strip().split())==2 and &quot;’&quot; not in l_[0]])</pre><pre>len(set(names))</pre><p>This time, the number of characters we have are limited to 1284. Let’s take a look at the most common 350 names.</p><pre>(&#39;Buck Mulligan&#39;, 93),<br> (&#39;Aunt Chloe&#39;, 82),<br> (&#39;Martin Cunningham&#39;, 71),<br> (&#39;Bayou Boeuf&#39;, 48),<br> (&#39;Aunt Sally&#39;, 39),<br> (&#39;Ned Lambert&#39;, 39),<br> (&#39;Mary Jane&#39;, 38),<br> (&#39;Solomon Northup&#39;, 36),<br> (&#39;John Thornton&#39;, 34),<br> (&#39;Myles Crawford&#39;, 33),<br> (&#39;Ben Dollard&#39;, 31),<br> (&#39;Sherlock Holmes&#39;, 30),<br> (&#39;Tom Sawyer&#39;, 30),<br> (&#39;John Eglinton&#39;, 29),<br> (&#39;Nosey Flynn&#39;, 28),<br> (&#39;Corny Kelleher&#39;, 27),<br> (&#39;Mrs Breen&#39;, 27),<br> (&#39;Father Conmee&#39;, 26),<br> (&#39;Uncle Tom&#39;, 25),<br> (&#39;John Wyse&#39;, 24),<br> (&#39;Henry Baskerville&#39;, 23),<br> (&#39;Uncle Abram&#39;, 22),<br> (&#39;Blazes Boylan&#39;, 19),<br> (&#39;Bob Doran&#39;, 18),<br> (&#39;Davy Byrne&#39;, 18),<br> (&#39;Coombe Tracey&#39;, 17),<br> (&#39;Aunt Phebe&#39;, 17),<br> (&#39;Simon Dedalus&#39;, 17),<br>........................<br> (&#39;Sandy Hill&#39;, 8),<br> (&#39;Theophilus Freeman&#39;, 8),<br> (&#39;Father Cowley&#39;, 8),<br> (&#39;Gregory B&#39;, 7),<br> (&#39;George Harris&#39;, 7),<br> (&#39;Rachel Halliday&#39;, 7),<br> (&#39;George Shelby&#39;, 7),<br> (&#39;Anne Hampton&#39;, 7),<br> (&#39;Peter Tanner&#39;, 7),<br> (&#39;Almidano Artifoni&#39;, 7),<br> (&#39;Hugo Baskerville&#39;, 6),<br> (&#39;Laura Lyons&#39;, 6),<br> (&#39;Aunt Polly&#39;, 6),<br> (&#39;Peter Wilks&#39;, 6),<br>.........................</pre><p>The names list looks much more accurate. No wonder why Spark NLP is enterprise preferred!</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/620/1*yyXS8XGlokTVH7tstakqCw.png" /><figcaption>SpaCy is good, but Spark NLP is better…</figcaption></figure><p><strong>Let’s Talk about Resource Consumption:</strong><br>The system used for this study is an 8 core Intel(R) Core(TM) i7–9700K CPU @ 3.60GHz with 32820MB Memory. The operating system is Ubuntu 20.04.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/e99a6acc6f33213ba3ce68791edba906/href">https://medium.com/media/e99a6acc6f33213ba3ce68791edba906/href</a></iframe><p><em>Spark NLP uses less memory and runs twice as fast when compared to spaCy. This fact, being coupled with higher accuracy of the Spark NLP provides good reasons to master this library!</em></p><p>That’s all for now. The main purpose of this article was to give momentum to intermediate level practitioners a guide on how to achieve their goals in the NLP domain using Spark. Some Spark NLP concepts may be difficult to learn in the beginning, however, the articles, courses, and sample notebooks presented in this paper will provide a good starting point. Initial overhead with Spark may be high, but spaCy — Spark NLP comparison shows the difference. It is not easy, but not too hard either to master Spark NLP…and it is definitely a skill you will have shining on your resume if you aim for the enterprise domain!</p><p>Please follow the <a href="https://towardsdatascience.com/how-to-get-started-with-sparknlp-in-2-weeks-cb47b2ba994d">link to the first part of the article.</a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=a6f6a968b7e8" width="1" height="1" alt=""><hr><p><a href="https://medium.com/data-science/how-to-wrap-your-head-around-spark-nlp-a6f6a968b7e8">How to Wrap Your Head Around Spark NLP</a> was originally published in <a href="https://medium.com/data-science">TDS Archive</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[How to Get Started with SparkNLP in 2 Weeks]]></title>
            <link>https://medium.com/data-science/how-to-get-started-with-sparknlp-in-2-weeks-cb47b2ba994d?source=rss-de9f6a439ca2------2</link>
            <guid isPermaLink="false">https://medium.com/p/cb47b2ba994d</guid>
            <category><![CDATA[spark-nlp]]></category>
            <category><![CDATA[spark]]></category>
            <dc:creator><![CDATA[Mustafa Aytuğ Kaya]]></dc:creator>
            <pubDate>Wed, 19 Aug 2020 00:24:45 GMT</pubDate>
            <atom:updated>2020-08-25T23:27:26.350Z</atom:updated>
            <content:encoded><![CDATA[<h3>How to Get Started with Spark NLP in 2 Weeks — Part I</h3><h4><em>Well begun is half done…</em></h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*F-bAn2f9vtl92AXH" /><figcaption><a href="https://www.pexels.com/photo/green-mountain-surrounded-by-body-of-water-photo-225203/">Photo: Borislav Krutsev</a></figcaption></figure><p>If you want to make a head start in enterprise NLP, but have no clue about Spark, this article is for you. I have seen many colleagues wanting to step to this domain but disheartened due to the initial learning overhead that comes with Spark. It may seem inconspicuous at first glance since Spark code is a bit different than your regular Python script. However, Spark and Spark NLP basics aren’t really hard to learn. If you axiomatically accept this assertion, I will show you how easy the basics are and will provide a road map to pave the way to learn key elements, which will satisfy most use cases of an intermediate level practitioner. Due to impeccable modularity that comes with Spark NLP pipelines, for an average learner, -mark my words- two weeks will be enough to build basic models. Roll up your sleeves, here we start!</p><p><strong>Why Spark NLP?</strong><br>Supply and Demand is the answer: It Is The Most Widely Used Library In Enterprises! Here are a <a href="https://www.kdnuggets.com/2019/06/spark-nlp-getting-started-with-worlds-most-widely-used-nlp-library-enterprise.html">few reasons</a> why. Common NLP packages today have been designed by academics and they favor ease of prototyping over runtime performance, eclipsing scalability, error handling, target frugal memory consumption and code reuse. Although some libraries like ‘the industrial-strength NLP library — spaCy’ might be considered an exception (since they are designed to get things done rather than doing research), they may fall short of enterprise targets when it comes to dealing with data in volume.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/905/1*IlLAFarDtcfS_rlxmCuH8w.png" /><figcaption><em>Usage of tools designed for research and smaller scale tasks cannot be handy</em></figcaption></figure><p>We are going to have <strong><em>a different strategy</em></strong> here. Rather than following the crowds in the routine, we will use basic libraries to brush up ‘basics’ and then jump directly to address the enterprise sector. Our final aim is to target the niche by building continental pipelines, which are impossible to resolve with standard libraries, albeit their capacity in their league.</p><p>If you are not convinced yet, please read this article for <a href="https://analyticsindiamag.com/5-reasons-why-spark-nlp-is-the-most-widely-used-library-in-enterprises/">benchmarking</a> and comparison with spaCy, which will give you five good reasons to start with Spark NLP. First of all Spark NLP has the innate trait of <strong>scalability</strong> it inherits from Spark, which was primarily used for distributed applications, it is designed <em>to be scalable</em>. Spark NLP benefits from this since it can scale on any Spark cluster as well as on-premise and with any cloud provider. Furthermore, Spark NLP optimizations are done in such a way that it could run <strong>orders of magnitude faster</strong> than what the inherent design limitations of legacy libraries allow. It provides the concepts of annotators and it includes more than what other NLP libs include. It includes sentence detection, tokenization, stemming, lemmatization, POS Tagger, NER, dependency parse, text matcher, date matcher, chunking, context-aware spell checking, sentiment detector, pre-trained models, and training models with <strong>very high accuracy</strong> according to academic peer-reviewed results. Spark NLP also includes production-ready implementation of BERT embeddings for named entity recognition. For example, it makes much fewer errors on NER compared to spaCy, which we tested in the second part of this article. Also, worthy of notice, Spark NLP includes features that provide full Python API, supports training on GPU, user-defined deep learning networks, Spark, and Hadoop.</p><p>The library comes with a production-ready implementation of BERT embeddings and uses transfer learning for data extraction. Transfer learning is a highly-effective method of extracting data that can leverage even small amounts of data. As a result, there’s no need to collect large amounts of data to train SOTA models.</p><p>Also, John Snow labs Slack channel provides top tier support that is beneficial because developers and new learners tend to band together and create resources that every one of them can benefit from. You will get answers to your questions right from the developers with dedication. I have been there a few times, can attest that they are quick and accurate in their response. Additionally, anyone finding themselves stuck can quickly get help from people that have had similar problems through Stack Overflow or similar platforms.</p><p><strong>Famous “Facts” About Spark That Are Wrong</strong></p><p>- <em>Spark is cluster computing so it can’t be run on local machines</em></p><p>Wrong! You can run Spark on your local machine, and each CPU core will be used to the core! That’s how it looks like when Spark NLP is in action:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/694/1*eLS0-gVdiNmpFxL-FuPouw.png" /><figcaption>Spark NLP is running…I paid for all those CPU cores, and it pleases me to see all of them run.</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/697/1*qKczFyKm_U4HKaGWF8PEOQ.png" /><figcaption>While it is possible to use <a href="https://spacy.io/usage/examples#multi-processing">Joblib for multiprocessing</a>, this is how typically spaCy processes would look like.</figcaption></figure><p>What’s even more, you can run Spark on GPU.</p><p>- <em>Spark is yet another language to learn!</em></p><p>Well, if you know SQL, PySpark and Spark NLP is not going to feel like another language at all. SQL and Regex are all languages on their own, but it wasn’t hard to learn their basics, right?</p><p>- <em>Spark is built for big data, so Spark NLP is only good for big data.</em></p><p>Yes, there is a significant overhead used for Spark internals, but Spark NLP introduces a ‘Light Pipeline’ for smaller datasets.</p><p><strong>At first, Spark seems like another challenging language to learn. Spark isn’t the easiest library to comprehend, but what we will be doing in the NLP domain has been skillfully crafted in Spark NLP’s infrastructure provided by a simple API that can be easily interacted with.</strong></p><p>While being one of the sharpest pencils in a data scientist’s toolbox, Pandas uses only a single CPU core, and in essence, it is not fast or robust enough to handle bigger datasets. Spark was designed to hurdle those deficiencies using cluster computing, <strong>and you can even run it on your local machine</strong>, assigning as many CPU cores as you want! Unfortunately, even though it runs on a local machine, pip install Pyspark is not enough to set it up, and a series of dependencies must be installed on PC and Mac. For those who want to jump into action as quickly as possible, I recommend the use of Google Colab.</p><p>1) Setting up on Mac or Linux<br>To utilize Spark NLP, Apache Spark version 2.4.3 and higher must be installed. Assuming that you haven’t installed Apache Spark yet, let’s start with Java installation at first. Just go to the <a href="https://www.oracle.com/technetwork/java/javase/downloads/jdk8-downloads-2133151.html">official website</a> and from <em>“Java SE Development Kit 8u191”</em>, and install<strong> JDK 8</strong>. A necessary command-line script can be found in detail <a href="https://medium.com/spark-nlp/introduction-to-spark-nlp-installation-and-getting-started-part-ii-d009f7a177f3">here</a>.</p><p>2) Setting up Spark on a PC can be a little bit tricky. Having tried many methods, I found <a href="https://medium.com/@GalarnykMichael/install-spark-on-windows-pyspark-4498a5d8d66c">this article</a> to be the only one that works. It also includes a clear and concise video. In addition to the provided documentation, this <a href="https://changhsinlee.com/install-pyspark-windows-jupyter/">blog page</a> also helped me.</p><p>3) <em>Unless you have to run Spark on your local machine, the easiest way to get started with PySpark is using Google Colab.</em> Since it is essentially a Jupyter Notebook that runs on Google server, you don’t need to install anything in our system locally.</p><p>To run spark in Colab, first, we need to install all the dependencies in the Colab environment such as Apache Spark 2.3.2 with Hadoop 2.7, Java 8, and Findspark to locate the spark in the system. Please refer to <a href="https://towardsdatascience.com/pyspark-in-google-colab-6821c2faf41c">this article</a> for further details.</p><pre><strong>import</strong> <strong>os</strong></pre><pre><em># Install java</em><br>! apt-get install -y openjdk-8-jdk-headless -qq &gt; /dev/null<br>os.environ[&quot;JAVA_HOME&quot;] = &quot;/usr/lib/jvm/java-8-openjdk-amd64&quot;<br>os.environ[&quot;PATH&quot;] = os.environ[&quot;JAVA_HOME&quot;] + &quot;/bin:&quot; + os.environ[&quot;PATH&quot;]<br>! java -version</pre><pre><em># Install pyspark</em><br>! pip install --ignore-installed -q pyspark==2.4.4</pre><pre><em># Install Spark NLP</em><br>! pip install --ignore-installed -q spark-nlp==2.4.5</pre><p>For PC users who don’t want to run their notebooks on Colab, I recommend installing Spark NLP on a dual boot Linux system or using <a href="https://towardsdatascience.com/dual-boot-is-dead-windows-and-linux-are-now-one-27555902a128">WSL 2</a>, however, <a href="https://www.phoronix.com/scan.php?page=article&amp;item=windows-10-wsl2&amp;num=2">this benchmarking article</a> reflects some performance loss with WSL 2. To be more precise, I had my best Spark NLP experience after a dual boot Ubuntu 20.04 installation.</p><p><strong>Day 2: Spark Basics, RDD Structure, and NLP Baby Steps with Spark</strong></p><p>As previously mentioned, Apache Spark is a distributed cluster computing framework that is highly efficient for large data sets using in-memory computations for lightning-fast data processing. It is also considered to be more efficient than MapReduce for the complex application running on Disk.</p><p><strong>While libraries like pandas are adequate for most everyday operations, big data requires a cluster computing framework due to the </strong><em>volume, variety, and velocity</em>, which are three defining properties or dimensions of <strong>big data</strong>.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/901/1*rfK_R98Y-hrOyylkqmRP8g.png" /></figure><p>At the center of Spark is the Spark Core — the foundation of the overall project, on top of which the rest of Spark libraries are built. Spark SQL lets you query structured data inside Spark programs, using either SQL or a familiar <a href="https://spark.apache.org/docs/latest/sql-programming-guide.html">DataFrame API</a>.</p><p>MLlib fits into <a href="https://spark.apache.org/">Spark</a>’s APIs and interoperates with <a href="http://www.numpy.org/">NumPy</a> in Python and R libraries. You can use any Hadoop data source, making it easy to plug into Hadoop workflows. Spark excels at iterative computation, enabling MLlib to run fast — up to 100x faster than MapReduce.</p><p><strong>Spark Mllib</strong> contains the legacy API built on top of RDDs. Although I find Spark Mllib and RDD structure easier to use as a Python practitioner, as of Spark 2.0, the RDD-based APIs in the Spark.MLlib package has entered maintenance mode. The primary Machine Learning API for Spark is now the DataFrame-based API in the Spark ML package.</p><p>Transformations create new RDDs and actions to perform calculations. Lambda, Map, and Filter are some of the basic functions. Real datasets are generally key, value pairs similar to Python dictionaries but are represented like tuples. Please observe the code below for creating bigrams and word counts to see how similar to Python code it is. I would highly recommend <a href="https://learn.datacamp.com/courses/big-data-fundamentals-with-pyspark">this course</a> for further details.</p><pre>import re</pre><pre>sentences = sc.textFile(“ulyses.txt”) \<br> .glom() \<br> .map(lambda x: “ “.join(x)) \<br> .flatMap(lambda x: x.split(“.”)).map(lambda x: re.sub(‘[^a-zA-Z\s]’,’’,x))</pre><pre>bigrams = sentences.map(lambda x:x.split()) \<br> .flatMap(lambda x: [((x[i],x[i+1]),1) for i in range(0,len(x)-1) \<br> if all([x[i].lower() not in stop_words,x[i+1].lower() not in stop_words])])</pre><pre>freq_bigrams = bigrams.reduceByKey(lambda x,y:x+y) \<br> .map(lambda x:(x[1],x[0])) \<br> .sortByKey(False)</pre><pre>In [1]:freq_bigrams.take(10)</pre><pre>Out[1]:(1, (&#39;Author&#39;, &#39;James&#39;)),<br>       (1, (&#39;Joyce&#39;, &#39;Release&#39;)),<br>       (1, (&#39;EBook&#39;, &#39;Last&#39;)),<br>       (1, (&#39;Last&#39;, &#39;Updated&#39;)),<br>       (1, (&#39;December&#39;, &#39;Language&#39;)),<br>       (1, (&#39;English&#39;, &#39;Character&#39;)),<br>       (1, (&#39;Character&#39;, &#39;set&#39;)),<br>       (1, (&#39;set&#39;, &#39;encoding&#39;)),<br>       (1, (&#39;UTF&#39;, &#39;START&#39;)),<br>       (1, (&#39;II&#39;, &#39;III&#39;))]</pre><pre># word count example</pre><pre>text_file = sc.textFile(“example_text1.txt”)<br>counts_rdd = text_file.flatMap(lambda line: line.split(“ “)) \<br> .map(lambda word: (word, 1)) \<br> .reduceByKey(lambda a, b: a + b)</pre><pre># print the word frequencies in descending order</pre><pre>counts_rdd.map(lambda x: (x[1], x[0])) \<br> .sortByKey(ascending=False)\<br> .collect()[:10]</pre><p>Output</p><pre>Out [3]: [(988, &#39;the&#39;),<br>          (693, &#39;and&#39;),<br>          (623, &#39;of&#39;),<br>          (604, &#39;to&#39;),<br>          (513, &#39;I&#39;),<br>          (450, &#39;a&#39;),<br>          (441, &#39;my&#39;),<br>          (387, &#39;in&#39;),<br>          (378, &#39;HAMLET&#39;),<br>          (356, &#39;you&#39;)]</pre><p><strong>Day 3/4: Feature Engineering / Cleaning Data with Pyspark</strong></p><p><strong>Spark ML </strong>provides higher-level API built on top of DataFrames for constructing ML pipelines, standardizing APIs for machine learning algorithms to make it easier to combine multiple algorithms into a single workflow. Here we will cover the key concepts introduced by the Spark ML API.</p><p>Machine learning steps can be applied to from Spark SQL to support a variety of data types under a unified Dataset concept. A SchemaRDD can be created either implicitly or explicitly from a regular RDD.</p><p>A <a href="https://spark.apache.org/docs/1.2.2/api/scala/index.html#org.apache.spark.ml.Transformer">Transformer</a> is an abstraction which includes feature transformers and learned models, implementing a method transform() which converts one SchemaRDD into another, generally by appending one or more columns.</p><p>An <a href="https://spark.apache.org/docs/1.2.2/api/scala/index.html#org.apache.spark.ml.Estimator">Estimator</a> abstracts the concept of a learning algorithm or any algorithm which fits or trains on data, implementing a method fit() which accepts a SchemaRDD and produces a Transformer. For example, a learning algorithm such as LogisticRegression is an Estimator, and calling fit() trains a LogisticRegressionModel, which is a Transformer.</p><p>In machine learning, it is common to run a sequence of algorithms to process and learn from data such as splitting each document text into words, converting words into a numerical feature vector, and training a prediction model using the feature vectors and labels. Spark ML represents such workflows as <a href="https://spark.apache.org/docs/1.2.2/api/scala/index.html#org.apache.spark.ml.Pipeline">Pipeline</a>s, which consist of a sequence of Transformers and Estimators to be run in a specific order, calling transform() and fit() methods respectively, to produce a Transformer (which becomes part of the PipelineModel, or fitted Pipeline), and that Transformer’s transform() method is called on the dataset.</p><p>A Pipeline is an Estimator. Thus, after a Pipeline’s fit() method runs, it produces a PipelineModel which is a Transformer. Once the PipelineModel’s transform() method is called on a test dataset, the data passes through the Pipeline in order, updating the data set as each stage’s transform() method updates it and passes it to the next stage.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*mPopWiMdb6djZBz7.png" /></figure><p>Pipelines and PipelineModels help to ensure that training and test data go through identical feature processing steps. Below is an example of a pipeline model. Please note that data preprocessing is carried out on a Pandas Dataframe.</p><pre>#define regex pattern for preprocessing<br>pat1 = r’@[A-Za-z0–9_]+’<br>pat2 = r’https?://[^ ]+’<br>combined_pat = r’|’.join((pat1,pat2))<br>www_pat = r’<a href="http://www.[">www.[</a>^ ]+’<br>negations_dic = {“isn’t”:”is not”, “aren’t”:”are not”, “wasn’t”:”was not”, “weren’t”:”were not”,<br> “haven’t”:”have not”,”hasn’t”:”has not”,”hadn’t”:”had not”,”won’t”:”will not”,<br> “wouldn’t”:”would not”, “don’t”:”do not”, “doesn’t”:”does not”,”didn’t”:”did not”,<br> “can’t”:”can not”,”couldn’t”:”could not”,”shouldn’t”:”should not”,”mightn’t”:”might not”,<br> “mustn’t”:”must not”}<br>neg_pattern = re.compile(r’\b(‘ + ‘|’.join(negations_dic.keys()) + r’)\b’)</pre><pre>def pre_processing(column):<br>   first_process = re.sub(combined_pat, ‘’, column)<br>   second_process = re.sub(www_pat, ‘’, first_process)<br>   third_process = second_process.lower()<br>   fourth_process = neg_pattern.sub(lambda x:\<br>    negations_dic[x.group()], third_process)<br>   result = re.sub(r’[^A-Za-z ]’,’’,fourth_process)<br>   return result.strip()</pre><pre>df[‘text’]=df.iloc[:,:].text.apply(lambda x: pre_processing(x))<br>df=df[~(df[‘text’]==’’)]</pre><pre># shuffle the data<br>df = df.sample(frac=1).reset_index(drop=True)<br># set the random seed and split train and test with 99 to 1 ratio<br>np.random.seed(777)<br>msk = np.random.rand(len(df)) &lt; 0.99<br>train = df[msk].reset_index(drop=True)<br>test = df[~msk].reset_index(drop=True)<br># save both train and test as CSV files<br>train.to_csv(‘pyspark_train_data.csv’)<br>test.to_csv(‘pyspark_test_data.csv’)</pre><pre>tokenizer = [Tokenizer(inputCol=&#39;text&#39;,outputCol=&#39;words&#39;)]</pre><pre>ngrams = [NGram(n=i, inputCol=&#39;words&#39;, <br>outputCol=&#39;{}_grams&#39;.format(i)) for i in range(1,4)]</pre><pre>cv =[CountVectorizer(vocabSize=5460,inputCol=&#39;{}_grams&#39;.format(i),\ <br>          outputCol=&#39;{}_tf&#39;.format(i)) for i in range(1,4)]</pre><pre>idf = [IDF(inputCol=&#39;{}_tf&#39;.format(i),\<br>        outputCol=&#39;{}_tfidf&#39;.format(i), minDocFreq=5) \<br>        for i in range(1,4)]</pre><pre>assembler = [VectorAssembler(inputCols=[&#39;{}_tfidf&#39;.format(i)\<br>            for i  in range(1,4)], outputCol=&#39;features&#39;)]</pre><pre>label_stringIdx = [StringIndexer(inputCol=&#39;sentiment&#39;, \  <br>                  outputCol=&#39;label&#39;)]</pre><pre>lr = [LogisticRegression()]</pre><pre>pipeline = Pipeline(stages=tokenizer+ngrams+cv+idf+assembler+\<br>           label_stringIdx+lr)</pre><pre>model = pipeline.fit(train_set)</pre><pre>predictions = model.transform(test_set)</pre><p>In PySpark, interactions with SparkSQL through DataFrame API and SQL queries are priority subjects to learn. <a href="https://spark.apache.org/docs/latest/api/python/pyspark.sql.html">Pyspark.sql module</a> must be perused for a better understanding of basic operations. DataFrame transformations and actions are not too hard to construct programmatically and operations on DataFrames can also be done using SQL queries. If you have time to take only one course, please spend it on<a href="https://learn.datacamp.com/courses/feature-engineering-with-pyspark"> Feature Engineering with Pyspark</a> course by John Hogue. <a href="https://learn.datacamp.com/courses/cleaning-data-with-pyspark">This course</a> is also recommended, time permitting (Day 5).</p><p><strong>Day 6/7: Practice Makes Perfect</strong></p><p>The best way to learn is practice. You will need a hardened skill set for next week. Try to deep dive into Pyspark modules, since you will be extensively using them. To re-iterate <em>“First Steps With PySpark and Big Data Processing”</em>, please follow <a href="https://hackersandslackers.com/transforming-pyspark-dataframes/">these exercises</a>. It is a recommended practice to download some datasets from <a href="https://www.kaggle.com/">Kaggle</a> or <a href="https://www.gutenberg.org/">Project Gutenberg</a> and solve pre-processing problems, as well as others. Please follow this excellent <a href="https://github.com/vkocaman/PySpark_Essentials_March_2019/blob/master/PySpark%20-%20From%20Zero%20to%20Hero%20(Sept%202019).ipynb">notebook</a> from the creators of Spark NLP. For additional resources, please observe <a href="https://github.com/changhsinlee/changhsinlee.github.io/blob/master/notebook/2018-03-04-pyspark-dataframe-basics/dataframe-basics.ipynb">this notebook</a> and <a href="https://github.com/changhsinlee/changhsinlee.github.io/blob/master/notebook/2018-03-04-pyspark-dataframe-basics/dataframe-basics.ipynb">this article</a>.</p><p>Next Up: <a href="https://towardsdatascience.com/how-to-wrap-your-head-around-spark-nlp-a6f6a968b7e8">Part II — How to Wrap Your Head Around Spark NLP</a>. In this part, we will understand “Annotators /Transformers in Spark NLP” and emphasize “Text Preprocessing with Spark”, “Pretrained Models” and “Text Classifiers” using numerous notebooks. Moreover, we will run a complete pipeline with spaCy and SparkNLP and compare the results.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=cb47b2ba994d" width="1" height="1" alt=""><hr><p><a href="https://medium.com/data-science/how-to-get-started-with-sparknlp-in-2-weeks-cb47b2ba994d">How to Get Started with SparkNLP in 2 Weeks</a> was originally published in <a href="https://medium.com/data-science">TDS Archive</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
    </channel>
</rss>