<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Stefano Lori on Medium]]></title>
        <description><![CDATA[Stories by Stefano Lori on Medium]]></description>
        <link>https://medium.com/@wolliq?source=rss-2194f57e111a------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/1*kqnKWEOUIeHIz3c2TJOSaw.png</url>
            <title>Stories by Stefano Lori on Medium</title>
            <link>https://medium.com/@wolliq?source=rss-2194f57e111a------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Thu, 19 Nov 2026 04:15:25 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@wolliq/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[Spark NLP Document Similarity Ranker as-retriever for RAG tasks]]></title>
            <link>https://wolliq.medium.com/spark-nlp-document-similarity-ranker-as-retriever-for-rag-tasks-641b7ccd03b2?source=rss-2194f57e111a------2</link>
            <guid isPermaLink="false">https://medium.com/p/641b7ccd03b2</guid>
            <dc:creator><![CDATA[Stefano Lori]]></dc:creator>
            <pubDate>Mon, 18 Mar 2024 09:19:33 GMT</pubDate>
            <atom:updated>2024-03-18T09:19:33.149Z</atom:updated>
            <content:encoded><![CDATA[<p>Breaking news! Spark NLP (<a href="https://sparknlp.org/">https://sparknlp.org/</a>) gets enhanced with a new DocumentSimilarityRanker as-retriever interface for your RAG (Retrieval-Augmented Generation) applications at scale! :D</p><p>The Spark NLP project, the great popular open-source library for natural language processing, has recently been upgraded to provide a way to use its DocumentSimilarityRanker annotator in RAG tasks. This interface provides a powerful tool for creating vector databases and applying approximate similarity search on a very large scale.</p><p><strong>What is DocumentSimilarityRanker as-retriever interface?</strong></p><p>DocumentSimilarityRanker as-retriever interface is a new capability of the annotator that allows users to create a vector database of a large collection of text documents and then execute searches to retrieve documents that are similar to a given query. <br>This is done by first training a DocumentSimilarityRanker model (Spark ML LSH model) on a corpus of text documents. The model then learns to represent each document as a vector in a high-dimensional space using sentence embeddings. Once the model is trained, it can be used to retrieve documents that are similar to a given query by finding the documents that have the most similar vectors to the query vector.</p><pre>...<br>query = &quot;Fifth document, Florence in Italy, is among the most beautiful cities in Europe.&quot;<br>...<br>document_similarity_ranker = DocumentSimilarityRankerApproach() \<br>            .setInputCols(&quot;sentence_embeddings&quot;) \<br>            .setOutputCol(&quot;doc_similarity_rankings&quot;) \<br>            .setSimilarityMethod(&quot;brp&quot;) \<br>            .setNumberOfNeighbours(3) \<br>            .setBucketLength(2.0) \<br>            .setNumHashTables(3) \<br>            .setVisibleDistances(True) \<br>            .setIdentityRanking(True) \<br>            .asRetriever(query)            # &lt;--- HERE IT IS!<br>...</pre><p><strong>Why is DocumentSimilarityRanker as-retriever important?</strong></p><p>DocumentSimilarityRanker asRetriever(query) is an important capability for several reasons. First, it can be used to identify relevant documents from a very large corpus of text data quickly and efficiently. This is because the model can use approximate similarity search techniques to find documents that are similar to the query, even if the query is not a perfect match for any of the documents in the corpus. Second, the model can be used to rank retrieved documents based on their similarity to the query. This allows users to focus on the most relevant documents first.<br>Third, in the LLM era, having the capability of searching massive document repositories and ranking document similarity at scale is a great advantage for more accurate answers using RAG techniques.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*IWzoYdjb6KNJOEi2.png" /><figcaption>Credit by Anyscale blog (<a href="https://www.anyscale.com/blog/a-comprehensive-guide-for-building-rag-based-llm-applications-part-1">https://www.anyscale.com/blog/a-comprehensive-guide-for-building-rag-based-llm-applications-part-1</a>)</figcaption></figure><p><strong>How to use DocumentSimilarityRanker as-retriever ?</strong></p><p>Let’s have an hands-on!</p><p>After installing Spark with for instance conda,</p><pre>$ java -version<br># should be Java 8 (Oracle or OpenJDK)<br>$ conda create -n sparknlp python=3.8 -y<br>$ conda activate sparknlp<br>$ pip install spark-nlp==5.2.x pyspark==3.3.x</pre><p>we need to install Spark NLP (<a href="https://sparknlp.org/docs/en/install">https://sparknlp.org/docs/en/install</a>).<br>We can launch the session enriched with the Spark NLP dependencies as follow.</p><pre>from pyspark.sql import SparkSession<br><br>spark = SparkSession.builder \<br>    .appName(&quot;Spark NLP&quot;)\<br>    .master(&quot;local[*]&quot;)\<br>    .config(&quot;spark.driver.memory&quot;,&quot;16G&quot;)\<br>    .config(&quot;spark.driver.maxResultSize&quot;, &quot;0&quot;) \<br>    .config(&quot;spark.kryoserializer.buffer.max&quot;, &quot;2000M&quot;)\<br>    .config(&quot;spark.jars.packages&quot;, &quot;com.johnsnowlabs.nlp:spark-nlp_2.12:5.2.x&quot;) \<br>    .getOrCreate()</pre><p>Then, we need to define a set of documents (corpus), for instance:</p><pre>df = spark.createDataFrame([<br>            [&quot;First document, this is my first sentence. This is my second sentence.&quot;],<br>            [&quot;Second document, this is my second sentence. This is my second sentence.&quot;],<br>            [&quot;Third document, climate change is arguably one of the most pressing problems of our time.&quot;],<br>            [&quot;Fourth document, climate change is definitely one of the most pressing problems of our time.&quot;],<br>            [&quot;Fifth document, Florence in Italy, is among the most beautiful cities in Europe.&quot;],<br>            [&quot;Sixth document, Florence in Italy, is a very beautiful city in Europe like Lyon in France.&quot;],<br>            [&quot;Seventh document, the French Riviera is the Mediterranean coastline of the southeast corner of France.&quot;],<br>            [&quot;Eighth document, the warmest place in France is the French Riviera coast in Southern France.&quot;]<br>        ]).toDF(&quot;text&quot;)</pre><p>Then we can define the target query as the 5th document as follow:</p><pre>query = &quot;Fifth document, Florence in Italy, is among the most beautiful cities in Europe.&quot;<br><br>document_assembler = DocumentAssembler() \<br>    .setInputCol(&quot;text&quot;) \<br>    .setOutputCol(&quot;document&quot;)<br><br>sentence_embeddings = RoBertaSentenceEmbeddings.pretrained() \<br>    .setInputCols([&quot;document&quot;]) \<br>    .setOutputCol(&quot;sentence_embeddings&quot;)<br><br>document_similarity_ranker = DocumentSimilarityRankerApproach() \<br>    .setInputCols(&quot;sentence_embeddings&quot;) \<br>    .setOutputCol(&quot;doc_similarity_rankings&quot;) \<br>    .setSimilarityMethod(&quot;brp&quot;) \<br>    .setNumberOfNeighbours(3) \<br>    .setBucketLength(2.0) \<br>    .setNumHashTables(3) \<br>    .setVisibleDistances(True) \<br>    .setIdentityRanking(True) \<br>    .asRetriever(query)<br><br>document_similarity_ranker_finisher = DocumentSimilarityRankerFinisher() \<br>   .setInputCols(&quot;doc_similarity_rankings&quot;) \<br>   .setOutputCols(&quot;finished_doc_similarity_rankings_id&quot;, &quot;finished_doc_similarity_rankings_neighbors&quot;)<br><br>pipeline = Pipeline(stages=[<br>    document_assembler,<br>    sentence_embeddings,<br>    document_similarity_ranker,<br>    document_similarity_ranker_finisher<br>])<br><br>model = pipeline.fit(df)<br><br>(<br>    model<br>    .transform(df)<br>    .select(&quot;text&quot;, &quot;nearest_neighbor_id&quot;, &quot;nearest_neighbor_distance&quot;)<br>    .orderBy(&#39;nearest_neighbor_distance&#39;, ascending=True)<br>    .show(10, False)<br>)</pre><p>After running the code, we can observe and rank the results!</p><pre>sent_roberta_base download started this may take some time.<br>Approximate size to download 284.8 MB<br>[OK!]<br>[Stage 71:========================&gt;                                (7 + 9) / 16]<br>+--------------------------------------------------------------------------------------------+-------------------+-------------------------+<br>|text                                                                                        |nearest_neighbor_id|nearest_neighbor_distance|<br>+--------------------------------------------------------------------------------------------+-------------------+-------------------------+<br>|Fifth document, Florence in Italy, is among the most beautiful cities in Europe.            |-1320876223        |0.0                      |<br>|Sixth document, Florence in Italy, is a very beautiful city in Europe like Lyon in France.  |1293373212         |0.17848861258809434      |<br>|Eighth document, the warmest place in France is the French Riviera coast in Southern France.|-1719102856        |0.2761524746260818       |<br>+--------------------------------------------------------------------------------------------+-------------------+-------------------------+</pre><p>The most similar documents among the corpus are provided with their ranking distance from the query ! Internally the annotator has used:</p><ul><li>RoBertaSentenceEmbeddings <br>(<a href="https://sparknlp.org/api/python/reference/autosummary/sparknlp/annotator/embeddings/roberta_sentence_embeddings/index.html">https://sparknlp.org/api/python/reference/autosummary/sparknlp/annotator/embeddings/roberta_sentence_embeddings/index.html</a>)</li><li>LSH model<br>(<a href="https://spark.apache.org/docs/3.3.2/ml-features.html#approximate-nearest-neighbor-search">https://spark.apache.org/docs/3.3.2/ml-features.html#approximate-nearest-neighbor-search</a>)</li></ul><p>To note that as we used the identity flag to True</p><pre>...<br>.setIdentityRanking(True)<br>...</pre><p>we can observe that the first result is the target query at distance 0.0 .</p><p><strong>Conclusions</strong></p><p>The new DocumentSimilarityRanker as-retriever interface is a powerful addition to the Spark NLP project. It allows users to create LSH vector databases of text documents and then use them to retrieve documents that are similar to a given query. This is an important tool for several RAG based tasks, including information retrieval, question answering, and summarization. Enjoy the power of Spark NLP! :D</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=641b7ccd03b2" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Polars vs DuckDB for Delta Lake ops]]></title>
            <link>https://wolliq.medium.com/polars-vs-duckdb-for-delta-lake-ops-54353107a032?source=rss-2194f57e111a------2</link>
            <guid isPermaLink="false">https://medium.com/p/54353107a032</guid>
            <category><![CDATA[delta-lake]]></category>
            <category><![CDATA[duckdb]]></category>
            <category><![CDATA[polar]]></category>
            <dc:creator><![CDATA[Stefano Lori]]></dc:creator>
            <pubDate>Wed, 13 Sep 2023 14:01:21 GMT</pubDate>
            <atom:updated>2023-09-13T14:02:04.100Z</atom:updated>
            <content:encoded><![CDATA[<h3><strong>Introduction</strong></h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*2kqvcq1kdED1DhfnhsWR0g.png" /></figure><p>Delta Lake is a unified lakehouse storage format that combines the reliability of a data warehouse with the flexibility of a data lake. It is a popular choice for storing large amounts of data that needs to be analyzed frequently. Lately it has introduced a lot of very interesting features.</p><p>Python is a popular programming language for data engineering and machine learning. It is also a good choice for developing REST APIs.</p><p>I found myself building a Python microservice in which there was the need to consume (read) a Delta Lake table using SQL inputs. I had the perfect chance to test some of my best friends Delta Lake integration APIs:<br>- Pola-rs (<a href="https://www.pola.rs/">https://www.pola.rs/</a>)<br>- Duck DB (<a href="https://duckdb.org/">https://duckdb.org/</a>)</p><h3>Requirements and installation</h3><p>I used Conda:</p><pre># create env and activate it<br>$ conda create -n delta-ops python=3.9<br><br># install deps<br>$ conda install -c conda-forge polars deltalake notebook python-duckdb<br><br># launch notebook<br>$ jupyter notebook</pre><h3>Dateset</h3><p>You can create a test Delta Table using Pola-rs as follow:</p><pre>import polars as pl<br>import datetime<br><br># Create a DataFrame with the desired columns<br>df = pl.DataFrame({<br>    &quot;id&quot;: [1, 5, 8],<br>    &quot;timestamp&quot;: [datetime.datetime.now(), datetime.datetime.now(), datetime.datetime.now()],<br>    &quot;xyz_value&quot;: [1.2, 3.4, 5.6]<br>})<br><br># Write the DataFrame to a Delta Lake table<br>df.write_delta(&quot;delta-table-test&quot;)</pre><h3>Processors creation</h3><p>Let’s start defyning some useful functions to process the Delta Lake data.</p><p>Let’s use a <strong>decorator</strong> to benchmark the functions:</p><pre>from functools import wraps<br>import time<br><br>def timeit(func):<br>    @wraps(func)<br>    def timeit_wrapper(*args, **kwargs):<br>        start_time = time.perf_counter()<br>        result = func(*args, **kwargs)<br>        end_time = time.perf_counter()<br>        total_time = end_time - start_time<br>        # print(f&#39;Function {func.__name__}{args} {kwargs} Took {total_time:.4f} secs&#39;)<br>        return result, total_time<br>    return timeit_wrapper</pre><p><strong>Process Delta Lake data using DuckDB API:</strong></p><pre>from deltalake import DeltaTable<br>import duckdb<br>import pyarrow<br><br>@timeit<br>def proc_duck_test(query, path) -&gt; pyarrow.dataset.Dataset:<br><br>    dt = DeltaTable(path)<br>    pyarrow_dataset = dt.to_pyarrow_dataset()<br>    sample = duckdb.arrow(pyarrow_dataset)<br>    <br>    res = duckdb.query(query)<br><br>    return res</pre><p><strong>NB</strong>: DuckDB quacks Arrow! :D<br>DuckDB uses PyArrow to store an intermediate dataset to be queried. using SQL queries. For more info see: <a href="https://arrow.apache.org/blog/2021/12/03/arrow-duckdb/">https://arrow.apache.org/blog/2021/12/03/arrow-duckdb/</a></p><p><strong>Process Delta Lake data using Pola-rs API:</strong></p><pre>import polars as pl<br><br>@timeit<br>def proc_polars_test(query, path) -&gt; pl.DataFrame:<br>    dt = pl.read_delta(path)<br>    <br>    res = pl.SQLContext(sample=dt).execute(<br>        query,<br>        eager=True<br>    )<br>    <br>    return res</pre><p>NB: SQLContext it’s used to apply SQL on the dataset.</p><h3>Tessssssts time!</h3><pre>query =  &quot;&quot;&quot;SELECT id, timestamp FROM sample WHERE id = 8&quot;&quot;&quot; <br>path = &quot;PATH-TO/delta-table-test&quot;<br><br>res = [proc_polars_test(query, path) for i in range(10000)]<br>res = [proc_duck_test(query, path) for i in range(10000)]</pre><h3>Results</h3><p>This is what I got:</p><pre>[POLARS] Average time: 0.0045 seconds<br><br>[DUCKDB] Average time: 0.0030 seconds</pre><h3>Conclusions</h3><p>DuckDB proved to be a little faster in accessing table reading than Pola-rs in local.<br>But I must say that in general both libraries show excellent performances.</p><p>In terms of usability both accept SQL queries and respond with dataframes. In this case I must say I found Pola-rs interface to be more usable, having the ability to define the view attribution on the call instead of defining the PyArrow intermediary step.</p><p>Of course to fully evaluate scalability and limits of the libraries need to be tested at much greater scale with full features such as lazyness, but this is already a very good start. I hope you enjoyed ! ;)</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=54353107a032" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Ranking document similarity at scale with Spark NLP]]></title>
            <link>https://wolliq.medium.com/ranking-document-similarity-at-scale-with-spark-nlp-4b7a23f32694?source=rss-2194f57e111a------2</link>
            <guid isPermaLink="false">https://medium.com/p/4b7a23f32694</guid>
            <category><![CDATA[nlp]]></category>
            <category><![CDATA[lsh]]></category>
            <category><![CDATA[document-similarity]]></category>
            <category><![CDATA[spark-nlp]]></category>
            <category><![CDATA[sentence-embedding]]></category>
            <dc:creator><![CDATA[Stefano Lori]]></dc:creator>
            <pubDate>Sun, 02 Jul 2023 20:06:40 GMT</pubDate>
            <atom:updated>2023-07-06T18:49:40.724Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/830/1*8Ug-qYreGkpJc_mIDiUYpg.png" /></figure><p>Combining the power of Spark NLP sentence embeddings and LSH approximate nearest neighbors search pipelines to catch contextual and semantic meanings in large textual document datasets</p><p><strong>Context</strong></p><p>Are you tired of manually sifting through mountains of text documents to find relevant similarity? Say goodbye to the tedious process! Spark NLP, the cutting-edge natural language processing library, has just added a new useful annotator: the Document Similarity Ranker.</p><p>As the name suggests this annotator can effortlessly compare and rank the similarity between different documents. Whether you’re dealing with articles, reports, customer reviews, or any other textual data, this powerful tool will save you countless hours and boost your productivity.</p><p><strong>Core architecture</strong></p><p>At its core, the Document Similarity Ranker leverages the power of various components to provide its impressive functionality.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/818/1*AODpB_OUJVLn_6yb-vokjg.png" /></figure><p>Let’s break it down:</p><ol><li>Firstly, the Document Similarity Ranker relies on a sentence embeddings pipeline. This pipeline processes individual sentences and transforms them into fixed-dimensional vectors called embeddings. These embeddings capture the contextual and semantic information of each sentence, enabling a more nuanced understanding of the text.<br>Spark NLP implements it in many ways, for instance <a href="https://sparknlp.org/api/com/johnsnowlabs/nlp/embeddings/RoBertaSentenceEmbeddings$.html">RoBertaSentenceEmbeddings</a> or <a href="https://sparknlp.org/api/com/johnsnowlabs/nlp/embeddings/E5Embeddings.html">E5Embeddings</a> .</li><li>Once the sentence embeddings are generated, the Document Similarity Ranker connects them to a Spark ML <a href="https://spark.apache.org/docs/2.2.3/ml-features.html#locality-sensitive-hashing">Locality Sensitive Hashing</a> (LSH) pipeline. Locality-Sensitive Hashing is a technique used for efficient nearest neighbor search in high-dimensional spaces. It maps similar vectors to the same or nearby hash buckets, allowing for fast retrieval of similar items.</li></ol><p>In summary, the Document Similarity Ranker seamlessly integrates the strength of Spark NLP’s sentence embeddings pipelines with the power of the Spark built-in Approximate Nearest Neighbors (ANN) search engine.<br>Awesome! Coding time!</p><p><strong>Hands-on using the Document Similarity Ranker !</strong></p><p>I’ll code in Python, but remember you can use the great Scala APIs too! :)</p><p>Spark NLP library provides many sentence level embeddings models such as Bert, RoBerta or XlmRoBerta (multilingual), which leverages pretrained transformer models. <br>Please find more info here: <a href="https://sparknlp.org/docs/en/transformers">https://sparknlp.org/docs/en/transformers</a> .</p><p>There exists many well-written Spark NLP articles to quick start with sentence embeddings pipelines, for instance you can find one here:<br><a href="https://www.johnsnowlabs.com/understanding-the-power-of-transformers-a-guide-to-sentence-embeddings-in-spark-nlp/">https://www.johnsnowlabs.com/understanding-the-power-of-transformers-a-guide-to-sentence-embeddings-in-spark-nlp/</a> .</p><p><strong>Install Spark NLP (≥ 5.x)</strong></p><p>Please refer to <a href="https://sparknlp.org/docs/en/install">https://sparknlp.org/docs/en/install</a> to install the library.</p><p><strong>Code time!</strong></p><ol><li>Load the necessary classes</li></ol><pre>from sparknlp.base import *<br>from sparknlp.annotator import *<br>from sparknlp.pretrained import PretrainedPipeline</pre><p>2. Initialise the PySpark session</p><pre>from pyspark.sql import SparkSession<br><br>spark = SparkSession.builder \<br>    .appName(&quot;Spark-NLP-doc-sim-ranker&quot;)\<br>    .master(&quot;local[*]&quot;)\<br>    .config(&quot;spark.driver.memory&quot;,&quot;16G&quot;)\<br>    .config(&quot;spark.driver.maxResultSize&quot;, &quot;0&quot;) \<br>    .config(&quot;spark.kryoserializer.buffer.max&quot;, &quot;2000M&quot;)\<br>    .config(&quot;spark.jars.packages&quot;, &quot;com.johnsnowlabs.nlp:spark-nlp_2.12:5.x.x&quot;)\<br>    .getOrCreate()</pre><p>3. Let’s create a test dataset with quite evident similarity in pair, i.e. first and second, third and fouth, fifth ad sixth, seventh and eigth sentences.</p><pre>data = spark.createDataFrame(<br>        [<br>            [&quot;First document, this is my first sentence. This is my second sentence.&quot;],<br>            [&quot;Second document, this is my second sentence. This is my second sentence.&quot;],<br>            [&quot;Third document, climate change is arguably one of the most pressing problems of our time.&quot;],<br>            [&quot;Fourth document, climate change is definitely one of the most pressing problems of our time.&quot;],<br>            [&quot;Fifth document, Florence in Italy, is among the most beautiful cities in Europe.&quot;],<br>            [&quot;Sixth document, Florence in Italy, is a very beautiful city in Europe like Lyon in France.&quot;],<br>            [&quot;Seventh document, the French Riviera is the Mediterranean coastline of the southeast corner of France.&quot;],<br>            [&quot;Eighth document, the warmest place in France is the French Riviera coast in Southern France.&quot;]<br>        ]<br>    ).toDF(&quot;text&quot;)</pre><p>4. Let’s build a RoBerta Sentence Embeddings pipeline, i.e. <a href="https://sparknlp.org/api/com/johnsnowlabs/nlp/embeddings/RoBertaSentenceEmbeddings.html">https://sparknlp.org/api/com/johnsnowlabs/nlp/embeddings/RoBertaSentenceEmbeddings.html</a></p><pre>document_assembler = DocumentAssembler() \<br>            .setInputCol(&quot;text&quot;) \<br>            .setOutputCol(&quot;document&quot;)<br><br>sentence_embeddings = RoBertaSentenceEmbeddings.pretrained() \<br>            .setInputCols([&quot;document&quot;]) \<br>            .setOutputCol(&quot;sentence_embeddings&quot;)<br><br>pipeline = Pipeline(stages=[<br>            document_assembler,<br>            sentence_embeddings<br>        ])<br><br>roberta_pipeline = pipeline.fit(data).transform(data)<br><br>roberta_pipeline.show()</pre><p>This will download the default pretrained model sent_roberta_base: <a href="https://sparknlp.org/2021/09/01/sent_roberta_base_en.html">https://sparknlp.org/2021/09/01/sent_roberta_base_en.html</a> .</p><p>When executing in the preferred notebook of your choise you’ll see the following</p><pre>[ / ]sent_roberta_base download started this may take some time.<br>Approximate size to download 284.8 MB<br>[ — ]Download done! Loading the resource.</pre><p>and if you make a show of the processed pipeline model.</p><pre>+--------------------+--------------------+--------------------+<br>|                text|            document| sentence_embeddings|<br>+--------------------+--------------------+--------------------+<br>|First document, t...|[{document, 0, 69...|[{sentence_embedd...|<br>|Second document, ...|[{document, 0, 71...|[{sentence_embedd...|<br>|Third document, c...|[{document, 0, 88...|[{sentence_embedd...|<br>|Fourth document, ...|[{document, 0, 91...|[{sentence_embedd...|<br>|Fifth document, F...|[{document, 0, 79...|[{sentence_embedd...|<br>|Sixth document, F...|[{document, 0, 89...|[{sentence_embedd...|<br>|Seventh document,...|[{document, 0, 10...|[{sentence_embedd...|<br>|Eighth document, ...|[{document, 0, 91...|[{sentence_embedd...|<br>+--------------------+--------------------+--------------------+</pre><p>We can now zoom into the sentence embeddings schema</p><pre>root<br> |-- sentence_embeddings: array (nullable = true)<br> |    |-- element: struct (containsNull = true)<br> |    |    |-- annotatorType: string (nullable = true)<br> |    |    |-- begin: integer (nullable = false)<br> |    |    |-- end: integer (nullable = false)<br> |    |    |-- result: string (nullable = true)<br> |    |    |-- metadata: map (nullable = true)<br> |    |    |    |-- key: string<br> |    |    |    |-- value: string (valueContainsNull = true)<br> |    |    |-- embeddings: array (nullable = true)<br> |    |    |    |-- element: float (containsNull = false)</pre><p>and struct result (truncated…), showing at last the embeddings array containing the final representations of our text dataset.</p><pre>+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------<br>|sentence_embeddings                                                                                                                                                                                                                                                                                                           <br>+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------<br>|[{sentence_embeddings, 0, 69, First document, this is my first sentence. This is my second sentence., {sentence -&gt; 0, token -&gt; First document, this is my first sentence. This is my second sentence., pieceId -&gt; -1, isWordStart -&gt; true}, [-0.0016509573, -0.20525712, -0.21965097, -0.059896577, 0.1287623, 0.19543253, 0.2<br>|[{sentence_embeddings, 0, 71, Second document, this is my second sentence. This is my second sentence., {sentence -&gt; 0, token -&gt; Second document, this is my second sentence. This is my second sentence., pieceId -&gt; -1, isWordStart -&gt; true}, [-7.9203E-4, -0.19994189, -0.21818015, -0.068899736, 0.12664562, 0.1954791, 0.<br>|[{sentence_embeddings, 0, 88, Third document, climate change is arguably one of the most pressing problems of our time., {sentence -&gt; 0, token -&gt; Third document, climate change is arguably one of the most pressing problems of our time., pieceId -&gt; -1, isWordStart -&gt; true}, [-0.008408386, -0.20978682, -0.21336395, -0.<br>|[{sentence_embeddings, 0, 91, Fourth document, climate change is definitely one of the most pressing problems of our time., {sentence -&gt; 0, token -&gt; Fourth document, climate change is definitely one of the most pressing problems of our time., pieceId -&gt; -1, isWordStart -&gt; true}, [-0.003835852, -0.20862408, -0.2177316<br>|[{sentence_embeddings, 0, 79, Fifth document, Florence in Italy, is among the most beautiful cities in Europe., {sentence -&gt; 0, token -&gt; Fifth document, Florence in Italy, is among the most beautiful cities in Europe., pieceId -&gt; -1, isWordStart -&gt; true}, [-0.017528567, -0.20887347, -0.21519172, -0.03888638, 0.133574<br>|[{sentence_embeddings, 0, 89, Sixth document, Florence in Italy, is a very beautiful city in Europe like Lyon in France., {sentence -&gt; 0, token -&gt; Sixth document, Florence in Italy, is a very beautiful city in Europe like Lyon in France., pieceId -&gt; -1, isWordStart -&gt; true}, [-0.023446266, -0.20165044, -0.21878108, -<br>|[{sentence_embeddings, 0, 101, Seventh document, the French Riviera is the Mediterranean coastline of the southeast corner of France., {sentence -&gt; 0, token -&gt; Seventh document, the French Riviera is the Mediterranean coastline of the southeast corner of France., pieceId -&gt; -1, isWordStart -&gt; true}, [-0.008019538, -0<br>|[{sentence_embeddings, 0, 91, Eighth document, the warmest place in France is the French Riviera coast in Southern France., {sentence -&gt; 0, token -&gt; Eighth document, the warmest place in France is the French Riviera coast in Southern France., pieceId -&gt; -1, isWordStart -&gt; true}, [-0.016810948, -0.2047661, -0.2230267,<br>+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ ... </pre><p>5. Now that we have produced the transformers representation, let’s add the Document Similarity Ranker to create a LSH based ANN search engine.</p><p>To do so, let’s rewrite the pipeline adding the necessary steps:</p><pre>from sparknlp.annotator.similarity.document_similarity_ranker import *<br><br>document_assembler = DocumentAssembler() \<br>            .setInputCol(&quot;text&quot;) \<br>            .setOutputCol(&quot;document&quot;)<br><br>sentence_embeddings = RoBertaSentenceEmbeddings.pretrained() \<br>            .setInputCols([&quot;document&quot;]) \<br>            .setOutputCol(&quot;sentence_embeddings&quot;)<br><br># Let&#39;s set the DocumentSimilarityRankerApproach instance<br># necessary to create the LSH model engine<br>document_similarity_ranker = DocumentSimilarityRankerApproach() \<br>            .setInputCols(&quot;sentence_embeddings&quot;) \<br>            .setOutputCol(&quot;doc_similarity_rankings&quot;) \<br>            .setSimilarityMethod(&quot;brp&quot;) \<br>            .setNumberOfNeighbours(1) \<br>            .setBucketLength(2.0) \<br>            .setNumHashTables(3) \<br>            .setVisibleDistances(True) \<br>            .setIdentityRanking(False)<br><br># Let&#39;s extract the information we need in the columns of interest<br>document_similarity_ranker_finisher = DocumentSimilarityRankerFinisher() \<br>        .setInputCols(&quot;doc_similarity_rankings&quot;) \<br>        .setOutputCols(<br>            &quot;finished_doc_similarity_rankings_id&quot;,<br>            &quot;finished_doc_similarity_rankings_neighbors&quot;) \<br>        .setExtractNearestNeighbor(True)<br><br>pipeline = Pipeline(stages=[<br>            document_assembler,<br>            sentence_embeddings,<br>            document_similarity_ranker,<br>            document_similarity_ranker_finisher<br>        ])<br><br>docSimRankerPipeline = pipeline.fit(data).transform(data)<br>(<br>    docSimRankerPipeline<br>        .select(<br>               &quot;finished_doc_similarity_rankings_id&quot;,<br>               &quot;finished_doc_similarity_rankings_neighbors&quot;<br>        ).show(10, False)<br>)</pre><p>The run of the pipeline will compute the RoBerta Sentence Embeddings and will propagate the embeddings to the LSH Approximate Nearest Neighbors search engine (ANN) that will compute for each domcument a result such as the following:</p><pre>+-----------------------------------+------------------------------------------+<br>|finished_doc_similarity_rankings_id|finished_doc_similarity_rankings_neighbors|<br>+-----------------------------------+------------------------------------------+<br>|1510101612                         |[(1634839239,0.12448559273510636)]        |<br>|1634839239                         |[(1510101612,0.12448559273510636)]        |<br>|-612640902                         |[(1274183715,0.12201215887654807)]        |<br>|1274183715                         |[(-612640902,0.12201215887654807)]        |<br>|-1320876223                        |[(1293373212,0.17848861258809434)]        |<br>|1293373212                         |[(-1320876223,0.17848861258809434)]       |<br>|-1548374770                        |[(-1719102856,0.2329717161223739)]        |<br>|-1719102856                        |[(-1548374770,0.2329717161223739)]        |<br>+-----------------------------------+------------------------------------------+</pre><p>Alternatively we can use the brand new E5Embeddings</p><pre>from sparknlp.annotator.similarity.document_similarity_ranker import *<br><br>document_assembler = DocumentAssembler() \<br>            .setInputCol(&quot;text&quot;) \<br>            .setOutputCol(&quot;document&quot;)<br><br>sentence_embeddings = E5Embeddings.pretrained() \<br>            .setInputCols([&quot;document&quot;]) \<br>            .setOutputCol(&quot;sentence_embeddings&quot;)<br><br>document_similarity_ranker = DocumentSimilarityRankerApproach() \<br>            .setInputCols(&quot;sentence_embeddings&quot;) \<br>            .setOutputCol(&quot;doc_similarity_rankings&quot;) \<br>            .setSimilarityMethod(&quot;brp&quot;) \<br>            .setNumberOfNeighbours(1) \<br>            .setBucketLength(2.0) \<br>            .setNumHashTables(3) \<br>            .setVisibleDistances(True) \<br>            .setIdentityRanking(False)<br><br>document_similarity_ranker_finisher = DocumentSimilarityRankerFinisher() \<br>        .setInputCols(&quot;doc_similarity_rankings&quot;) \<br>        .setOutputCols(<br>            &quot;finished_doc_similarity_rankings_id&quot;,<br>            &quot;finished_doc_similarity_rankings_neighbors&quot;) \<br>        .setExtractNearestNeighbor(True)<br><br>pipeline = Pipeline(stages=[<br>            document_assembler,<br>            sentence_embeddings,<br>            document_similarity_ranker,<br>            document_similarity_ranker_finisher<br>        ])<br><br>docSimRankerPipeline = pipeline.fit(data).transform(data)<br><br>(<br>    docSimRankerPipeline<br>        .select(<br>               &quot;finished_doc_similarity_rankings_id&quot;,<br>               &quot;finished_doc_similarity_rankings_neighbors&quot;<br>        ).show(10, False)<br>)</pre><p>The results are consistent with our dataset construction, where the document pairs 1–2, 3–4, 5–6 and 7–8 are retrieved as best neighbors.</p><p>Now, let’s inspect the input parameters we set and the results we obtained.</p><p><strong>Input setters for parameters</strong></p><ul><li>setInputCols(“sentence_embeddings”) : this setter will address input column</li><li>setOutputCol(“doc_similarity_rankings”) : this setter will address ouput column</li><li>setSimilarityMethod(“brp”) : this setter will select the LSH method (lsh|mh) used to apply approximate nearest neigbours search</li><li>setNumberOfNeighbours(10) : this setter will address the desired number of similar documents for a given document in the set</li><li>setBucketLength(2.0) : LSH parameter used to control the average size of hash buckets and improve recall</li><li>setNumHashTables(3) : LSH parameter used to control number of hash tables used in LSH OR-amplification and improve recall</li><li>setVisibleDistances(True) : this setter will make distances visible in the result, useful for debugging level information</li><li>setIdentityRanking(False) : this setter will make identity distance (0.0) visible, useful for debugging level information</li></ul><p><strong>Output parameters and results</strong></p><p>During pipeline execution the DocumentSimilarityRankerApproach will train a DocumentSimilarityRankerModel and the resulting model will compute 2 columns:</p><ul><li>finished_doc_similarity_rankings_id: this column represents the ID of the given document</li><li>finished_doc_similarity_rankings_neighbors: this column represents the result of the ANN model, providing the ranked list of similar documents to the corresponding ID, ordered by LSH distance</li></ul><p><strong>A simple similarity self-test</strong></p><p>Are the document reprensentation and similarity search effective?</p><p>A simple similarity test can be done by setting the identity ranking to True, observing that each document will appaear at distance 0.0 froim itself.</p><pre>+-----------------------------------+------------------------------------------+<br>|finished_doc_similarity_rankings_id|finished_doc_similarity_rankings_neighbors|<br>+-----------------------------------+------------------------------------------+<br>|1510101612                         |[(1510101612,0.0)]                        |<br>|1634839239                         |[(1634839239,0.0)]                        |<br>|-612640902                         |[(-612640902,0.0)]                        |<br>|1274183715                         |[(1274183715,0.0)]                        |<br>|-1320876223                        |[(-1320876223,0.0)]                       |<br>|1293373212                         |[(1293373212,0.0)]                        |<br>|-1548374770                        |[(-1548374770,0.0)]                       |<br>|-1719102856                        |[(-1719102856,0.0)]                       |<br>+-----------------------------------+------------------------------------------+</pre><p><strong>Ranking nearest neighbors of a specific document</strong></p><p>In the previous pipeline we can just set tha visible distance parameter to False</p><pre>...<br>document_similarity_ranker = DocumentSimilarityRankerApproach() \<br>            .setInputCols(&quot;sentence_embeddings&quot;) \<br>            .setOutputCol(&quot;doc_similarity_rankings&quot;) \<br>            .setSimilarityMethod(&quot;brp&quot;) \<br>            .setNumberOfNeighbours(3) \<br>            .setBucketLength(2.0) \<br>            .setNumHashTables(3) \<br>            .setVisibleDistances(False) \      &lt;-- HERE<br>            .setIdentityRanking(False)<br>...</pre><p>obtaining a string column containing the ranked IDs:</p><pre>+-----------------------------------+------------------------------------------+<br>|finished_doc_similarity_rankings_id|finished_doc_similarity_rankings_neighbors|<br>+-----------------------------------+------------------------------------------+<br>|1510101612                         |[1634839239,1274183715,-612640902]        |<br>|1634839239                         |[1510101612,1274183715,-612640902]        |<br>|-612640902                         |[1274183715,-1719102856,-1548374770]      |<br>|1274183715                         |[-612640902,-1719102856,-1320876223]      |<br>|-1320876223                        |[1293373212,-1719102856,1274183715]       |<br>|1293373212                         |[-1320876223,-1719102856,-1548374770]     |<br>|-1548374770                        |[-1719102856,1274183715,-612640902]       |<br>|-1719102856                        |[-1548374770,-1320876223,1274183715]      |<br>+-----------------------------------+------------------------------------------+</pre><p>The step to parse and explode the column is straight forward:</p><pre>from pyspark.sql.functions import *<br><br># Parse the string column into an array of integers and explode with filter on id<br>target_id = 1510101612<br><br>(<br>    res<br>    .withColumn(&quot;finished_doc_similarity_rankings_neighbors&quot;, regexp_replace(&quot;finished_doc_similarity_rankings_neighbors&quot;, &quot;[\\[\\]]&quot;, &quot;&quot;))<br>    .withColumn(&quot;col_array&quot;, split(col(&quot;finished_doc_similarity_rankings_neighbors&quot;), &quot;,&quot;).cast(&quot;array&lt;int&gt;&quot;))<br>    .select(&quot;finished_doc_similarity_rankings_id&quot;, explode(&quot;col_array&quot;).alias(&quot;exploded_value&quot;))<br>    .filter(col(&quot;finished_doc_similarity_rankings_id&quot;) == target_id)<br>    .show()<br>)</pre><p>finally obtaining the N nearest neighbours documents of target_id.</p><pre>+-----------------------------------+--------------+<br>|finished_doc_similarity_rankings_id|exploded_value|<br>+-----------------------------------+--------------+<br>|                         1510101612|    1634839239|<br>|                         1510101612|    1274183715|<br>|                         1510101612|    -612640902|<br>+-----------------------------------+--------------+</pre><p><strong>Conclusion</strong></p><p>In conclusion, the document similarity ranker showcases the exceptional usability of the library. Spark NLP’s pipelines offer a seamless and intuitive way to orchestrate the entire document similarity analysis process. <br>With the ability to incorporate custom components and fine-tune parameters, these pipelines empower users to tailor the document similarity analysis to specific requirements and domains. This emphasis on usability makes Spark NLP’s pipelines a valuable tool for both experienced data scientists and NLP practitioners, fostering efficiency and productivity in large scale NLP projects.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=4b7a23f32694" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Polars is all you need: SQL chapter]]></title>
            <link>https://wolliq.medium.com/polars-is-all-you-need-sql-chapter-2e4385dd75c2?source=rss-2194f57e111a------2</link>
            <guid isPermaLink="false">https://medium.com/p/2e4385dd75c2</guid>
            <category><![CDATA[pyspark]]></category>
            <category><![CDATA[rust-programming-language]]></category>
            <category><![CDATA[performance]]></category>
            <category><![CDATA[sql]]></category>
            <category><![CDATA[python-data-analysis]]></category>
            <dc:creator><![CDATA[Stefano Lori]]></dc:creator>
            <pubDate>Sat, 20 May 2023 14:23:19 GMT</pubDate>
            <atom:updated>2023-05-21T15:11:13.095Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*cakvvdiEk9RZpe9qWal98w.png" /></figure><h3>Intro</h3><p>Let’s do some (quick) data analysis in Python. No doubts, Pandas!</p><p>Wait, I’m a Spark (Scala/Python) developer…syntax is so different…what are these indexes coming up all the times…do I have an alternative tool that adds an arrow in my quiver? <br>Yeah! Polars [pola-rs] is the brand new DataFrame library there for you.</p><p>It’s written in Rust..ok so what? Do I have to learn a new language?<br>No don’t worry, you just have to use the great Python API with so many advantages..let’s have an overview.</p><h3>Rust brief history</h3><p>Rust programming language project started in 2006 (Mozilla Research, Graydon Hoare) and it was announced later in 2010. Since then, many core aspects of the language changed. <br>Version 1.0 was announced on May 15, 2015. <br>During August 2020 COVID-19 pandemic , the team behind Servo, an Rust based experimental browser engine, was disbanded by Mozilla. Because of some members of the team were active Rust contributors, the fact raised concerns about the future of the language.<br>In February 2021, formation of the Rust Foundation was announced by its five founding companies (AWS, Huawei, Google, Microsoft, and Mozilla).<br>In December 2022, Linux 6.1 officially added support for Rust in the Kernel. The future looks bright!</p><h3>Why Rust is so cool (for data analysis too) ?</h3><ol><li>fast and memory-efficient: Rust facilitates easy <strong>scalability</strong> and <strong>concurrency</strong></li><li><strong>memory-safety and thread-safety</strong> with an enhanced memory management without having a garbage collector (automatic memory management feature)</li><li><strong>Polars</strong>: here we go!<br>It is a DataFrame interface on top of an OLAP Query Engine implemented in Rust using Apache Arrow Columnar Format as the memory model, implemented in Rust, Python, Node.js, R and SQL!</li></ol><ul><li><strong>Fast</strong>: Polars is written from the ground up, designed close to the machine and without external dependencies.</li><li><strong>I/O</strong>: First class support for all common data storage layers: local, cloud storage &amp; databases.</li><li><strong>Easy to use</strong>: Write your queries the way they were intended. Polars, internally, will determine the most efficient way to execute using its query optimizer.</li><li><strong>Out of Core</strong>: Polars supports out of core data transformation with its streaming API. Allowing you to process your results without requiring all your data to be in memory at the same time</li><li><strong>Parallel</strong>: Polars fully utilises the power of your machine by dividing the workload among the available CPU cores without any additional configuration.</li><li><strong>Vectorized Query Engine</strong>: Polars uses Apache Arrow, a columnar data format, to process your queries in a vectorized manner. It uses SIMD to optimize CPU usage.</li></ul><h3>Now..why Polars is so cool (for data analysis too) ?</h3><p><a href="https://pola-rs.github.io/polars-book/">Polars</a> is written in Rust, but don’t worry you don’t have to learn a new language. It exposes a complete Python API, including the full set of features to manipulate DataFrames.<br>If you are a Spark developer, Scala or Python APIs, the violins will start playing.</p><p><strong>Installation</strong></p><p>It has never been simpler than that:</p><pre>$ pip install polars</pre><p>Eventually you may want to install the <a href="https://pola-rs.github.io/polars-book/user-guide/installation/#feature-flags">additional dependencies</a>.</p><p><strong>Getting Started with DataFrames</strong></p><p>Let’s create a DataFrame.<br>Syntax recalls pandas, so you will not find much difference.<br>You have utility methods to output information such as print and head.</p><pre>import polars as pl<br>from datetime import datetime<br><br>df = pl.DataFrame(<br>    {<br>        &quot;index&quot;: [1, 2, 3, 4, 5],<br>        &quot;date&quot;: [<br>            datetime(2023, 1, 1),<br>            datetime(2023, 1, 2),<br>            datetime(2023, 1, 3),<br>            datetime(2023, 1, 4),<br>            datetime(2023, 1, 5),<br>        ],<br>        &quot;value&quot;: [4.0, 5.0, 6.0, 7.0, 8.0],<br>    }<br>)<br><br>print(df)<br>---------<br><br>shape: (5, 3)<br>┌───────┬─────────────────────┬───────┐<br>│ index ┆ date                ┆ value │<br>│ ---   ┆ ---                 ┆ ---   │<br>│ i64   ┆ datetime[μs]        ┆ f64   │<br>╞═══════╪═════════════════════╪═══════╡<br>│ 1     ┆ 2023-01-01 00:00:00 ┆ 4.0   │<br>│ 2     ┆ 2023-01-02 00:00:00 ┆ 5.0   │<br>│ 3     ┆ 2023-01-03 00:00:00 ┆ 6.0   │<br>│ 4     ┆ 2023-01-04 00:00:00 ┆ 7.0   │<br>│ 5     ┆ 2023-01-05 00:00:00 ┆ 8.0   │<br>└───────┴─────────────────────┴───────┘<br><br><br>df.head()<br>---------<br># similar to above just prettier</pre><p>You can see visualize you DataFrame with column types and shape!</p><p><strong>Make some quick data analysis</strong></p><p>Let’s do some descriptive analysis in one command:</p><pre>print(df.describe())<br>--------------------<br>shape: (7, 4)<br>┌────────────┬──────────┬────────────────────────────┬──────────┐<br>│ describe   ┆ index    ┆ date                       ┆ value    │<br>│ ---        ┆ ---      ┆ ---                        ┆ ---      │<br>│ str        ┆ f64      ┆ str                        ┆ f64      │<br>╞════════════╪══════════╪════════════════════════════╪══════════╡<br>│ count      ┆ 5.0      ┆ 5                          ┆ 5.0      │<br>│ null_count ┆ 0.0      ┆ 0                          ┆ 0.0      │<br>│ mean       ┆ 3.0      ┆ null                       ┆ 6.0      │<br>│ std        ┆ 1.581139 ┆ null                       ┆ 1.581139 │<br>│ min        ┆ 1.0      ┆ 2023-01-01 00:00:00.000000 ┆ 4.0      │<br>│ max        ┆ 5.0      ┆ 2023-01-05 00:00:00.000000 ┆ 8.0      │<br>│ median     ┆ 3.0      ┆ null                       ┆ 6.0      │<br>└────────────┴──────────┴────────────────────────────┴──────────┘</pre><p>That will give you a summary statistics for the DataFrame. Cool!</p><p><strong>Index or not to index? This is the question..or not.</strong></p><p>If you are new to Pandas you will find this “get an index” thing kind of bizarre. I asked myself many times why should you get an index column..? O&gt;O I still dunno.<br>The more I worked with Pandas the more I found myself using reset_index() or set_index() at every operational step in the pipeline.<br>IMPO this index labelling feature in Pandas is somehow not very usefull.</p><p>If you have the time to get through the Polars doc you can find the following explanation: “<em>In Polars a DataFrame will always be a 2D table with heterogeneous data-types. The data-types may have nesting, but the table itself will not. Operations like resampling will be done by specialized functions or methods that act like ‘verbs’ on a table explicitly stating columns that ‘verb’ operates on. As such, it is our conviction that not having indices make things simpler, more explicit, more readable and less error-prone.</em>”</p><p>Very convincing to me.</p><p><strong>Apache Arrow (Polars) vs Numpy (Pandas)</strong></p><p>Apache Arrow is an emerging standard for in-memory columnar analytics that can accelerate data load times, reduce memory usage and accelerate calculations.</p><p>Pandas was initially built on Numpy; however, it was never built as a backend for dataframe libraries and it has some limitations. <br>Pandas 2.x has adopted Arrow, Polars was born with it representing data in memory with Arrow arrays instead of Numpy arrays. To maintain porting capability Polars offers a to_numpy method.</p><p><strong>Selection operation</strong></p><p>This is where your (Spark) SQL senses will wake up!</p><p>No index so no more</p><pre>df[‘a’] <br># and/or <br>df.loc[:,’a’]</pre><p>so you need to use a real SQL expression</p><pre>df.select([&#39;a&#39;])<br># or<br>df.select(pl.col(&#39;a&#39;))</pre><p><strong>Filtering</strong> <strong>operation</strong></p><p>If you want to select rows based on the values then in Polars you use the .filter method:</p><pre>df.filter(pl.col(&#39;a&#39;) &lt; 10)</pre><p><strong>Aggregation operation</strong></p><p>Another usefull expression greatly close to Spark syntax is the aggregation that we can chain with a casting on datetime to date</p><pre>df_count = (<br>    df<br>    .with_columns(<br>        pl.col(&#39;date&#39;).cast(pl.Date).alias(&#39;date&#39;)<br>    )<br>    .groupby([&#39;date&#39;, &#39;index&#39;]).agg(pl.count().alias(&quot;items&quot;))<br>    .sort(by=&quot;date&quot;, descending=False)<br>)<br><br>print(df_count)<br>---------------<br>shape: (5, 3)<br>┌────────────┬───────┬───────┐<br>│ date       ┆ index ┆ items │<br>│ ---        ┆ ---   ┆ ---   │<br>│ date       ┆ i64   ┆ u32   │<br>╞════════════╪═══════╪═══════╡<br>│ 2023-01-01 ┆ 1     ┆ 1     │<br>│ 2023-01-02 ┆ 2     ┆ 1     │<br>│ 2023-01-03 ┆ 3     ┆ 1     │<br>│ 2023-01-04 ┆ 4     ┆ 1     │<br>│ 2023-01-05 ┆ 5     ┆ 1     │<br>└────────────┴───────┴───────┘</pre><p><strong>SQL query integration</strong></p><p>Similarly to Spark, Polars uses the SQLContext to manage SQL queries .<br>While it’s not the recommended approach this allows plain SQL portability in your applications, which is kind of useful. SQL is then translated to expressions to be evaluated by the execution engine…but this is material for another article! ;)</p><pre>import polars as pl<br><br><br># prep for sql execution<br>sql = pl.SQLContext()<br>sql.register(&#39;df&#39;, df)<br><br>result_df = sql.execute(<br>    &quot;&quot;&quot;<br>      SELECT *<br>      FROM df<br>    &quot;&quot;&quot;<br>).collect()<br><br>print(result_df)<br>----------------shape: (5, 3)<br>┌───────┬─────────────────────┬───────┐<br>│ index ┆ date                ┆ value │<br>│ ---   ┆ ---                 ┆ ---   │<br>│ i64   ┆ datetime[μs]        ┆ f64   │<br>╞═══════╪═════════════════════╪═══════╡<br>│ 1     ┆ 2023-01-01 00:00:00 ┆ 4.0   │<br>│ 2     ┆ 2023-01-02 00:00:00 ┆ 5.0   │<br>│ 3     ┆ 2023-01-03 00:00:00 ┆ 6.0   │<br>│ 4     ┆ 2023-01-04 00:00:00 ┆ 7.0   │<br>│ 5     ┆ 2023-01-05 00:00:00 ┆ 8.0   │<br>└───────┴─────────────────────┴───────┘</pre><p><strong>Conclusion</strong></p><p>I found in Polars a powerful and promising data processing tool.<br>Syntax is easy, elegant, flexible and winks at Spark.<br>If you — like me — come from a development background scaling application over Spark and encounter some difficulties in Pandas’ phylosophy, Polars is all you need to quickly implement and deploy your safer and more performant applications.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=2e4385dd75c2" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Cleaning and extracting content from HTML/XML documents using Spark NLP]]></title>
            <link>https://medium.com/spark-nlp/cleaning-and-extracting-content-from-html-xml-documents-using-spark-nlp-documentnormalizer-913d96b2ee34?source=rss-2194f57e111a------2</link>
            <guid isPermaLink="false">https://medium.com/p/913d96b2ee34</guid>
            <category><![CDATA[html]]></category>
            <category><![CDATA[spark-nlp]]></category>
            <category><![CDATA[spark]]></category>
            <category><![CDATA[nlp]]></category>
            <category><![CDATA[xml]]></category>
            <dc:creator><![CDATA[Stefano Lori]]></dc:creator>
            <pubDate>Wed, 13 Jan 2021 19:26:23 GMT</pubDate>
            <atom:updated>2021-01-18T21:35:48.412Z</atom:updated>
            <content:encoded><![CDATA[<h3>Cleaning and extracting text from HTML/XML documents by using Spark NLP</h3><p><strong>Spark NLP</strong> is an open-source text processing library for advanced natural language processing for the <strong>Python, Java, and Scala</strong> programming languages. The library obtained today the best performing academic peer-reviewed results for two years in a row with an important growing community (2.5M Downloads and <strong>9x growth in 2020</strong>).</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*Dpk2nFzUJlhKmwNR" /><figcaption>Photo by <a href="https://unsplash.com/@florianolv?utm_source=medium&amp;utm_medium=referral">Florian Olivo</a> on <a href="https://unsplash.com/?utm_source=medium&amp;utm_medium=referral">Unsplash</a></figcaption></figure><p>Some more impressive numbers from the latest <strong>2.7.x </strong>release:</p><ul><li>more accurate, faster, and support up to <strong>375 languages.</strong></li><li>support to state-of-the-art <strong>Seq2Seq </strong>and <strong>Text2Text transformers</strong>. This includes new annotators for Google T5 (Text-To-Text Transfer Transformer) and MarianMNT for Neural Machine Translation — with over 646 new pre-trained models and pipelines.</li><li><strong>720+ new pretrained models and pipelines</strong> while extending our support of multi-lingual models to 192+ languages such as Chinese, Japanese, Korean, Arabic, Persian, Urdu, and Hebrew.</li></ul><p>Today I’m going to talk about a new annotator that was added in the latest release: the <strong>DocumentNormalizer</strong>.</p><p><strong>Why do we need a document normalizer?</strong></p><p>Spark NLP community expressed the need for an annotator capable of directly processing input HTML/XML documents to clean or extract specific contents.</p><p>Imagine you aggregate a collection of raw HTML documents you just collected from a given data source with your preferred crawler library and you want to remove all the tags to focus on the tag contents.</p><p>Please don’t call the ghostbusters, just use the brand new Spark NLP DocumentNormalizer annotator! :D</p><p>But wait, what is an annotator? o.O<br>Let’s see the definition to have an idea.</p><p>In Spark NLP, all <strong>Annotators</strong> are either Estimators or Transformers as we see in Spark ML. An Estimator in Spark ML is an algorithm that can be fit on a DataFrame to produce a Transformer. E.g., a learning algorithm is an Estimator that trains on a DataFrame and produces a model. A Transformer is an algorithm that can transform one DataFrame into another DataFrame. E.g., an ML model is a Transformer that transforms a DataFrame with features into a DataFrame with predictions.</p><p>Let’s load some data to a text column in your input Spark SQL DataFrame:</p><pre>path = &quot;html-docs&quot;<br><br>data = spark.sparkContext.wholeTextFiles(path)<br>df = data.toDF(schema=[&quot;filename&quot;, &quot;text&quot;]).select(&quot;text&quot;)<br><br>df.show()</pre><pre>...</pre><pre>+--------------------+<br>|                text|<br>+--------------------+<br>|&lt;div class=&#39;w3-co...|<br>|&lt;span style=&quot;font...|<br>|&lt;!DOCTYPE html&gt; &lt;...|<br>+--------------------+</pre><p>Once your input DataFrame is loaded you can define your next pipeline stages:</p><pre>documentAssembler = DocumentAssembler() \<br>    .setInputCol(&#39;text&#39;) \<br>    .setOutputCol(&#39;document&#39;)<br><br>inpuColName = &quot;document&quot;<br>outputColName = &quot;normalizedDocument&quot;<br><br>action = &quot;clean&quot;<br>cleanUpPatterns = [&quot;&lt;[^&gt;]*&gt;&quot;]<br>replacement = &quot; &quot;<br>removalPolicy = &quot;pretty_all&quot;<br>encoding = &quot;UTF-8&quot;<br><br>documentNormalizer = DocumentNormalizer() \<br>    .setInputCols(inpuColName) \<br>    .setOutputCol(outputColName) \<br>    .setAction(action) \<br>    .setPatterns(cleanUpPatterns) \<br>    .setReplacement(replacement) \<br>    .setPolicy(removalPolicy) \<br>    .setLowercase(<strong>True</strong>) \<br>    .setEncoding(encoding)</pre><p>Let’s make a pass on the different parameters we are going to set in our example.</p><ul><li><strong>inpuColName</strong>: input column name string which targets a column of type Array(AnnotatorType.DOCUMENT).</li><li><strong>outputColName</strong>: output column name string which targets a column of type AnnotatorType.DOCUMENT.</li><li><strong>action</strong>: action string to perform applying regex patterns, i.e. (clean | extract). Default is “clean”.</li><li><strong>cleanUpPatterns</strong>: normalization regex patterns that match will be removed from the document.<br>Default is “&lt;[^&gt;]*&gt;” (e.g., it removes all HTML tags).</li><li><strong>replacement</strong>: replacement string to apply when regexes match.<br>Default is “ “.</li><li><strong>removalPolicy</strong>: parameter to remove patterns from text with a given policy. Valid policy values are: “all”, “pretty_all”, “first”, “pretty_first”.<br>Defaults are “pretty_all”.</li><li><strong>lowercase</strong> boolean: whether to convert strings to lowercase. <br>Default is False.</li><li><strong>encoding</strong>: file encoding to apply on normalized documents. Supported encodings are: UTF_8, UTF_16, US_ASCII, ISO-8859–1, UTF-16BE, UTF-16LE. <br>Default is “UTF-8”.</li></ul><p>Thanks to the new DocumentNormalizer annotator, we can now apply the regular expressions we have chosen in order to normalize the document and prepare it for the following stages.</p><p>In this example, our goal is therefore to extract the text that is contained within the HTML tags.</p><p>Having the DataFrame input column “text” containing the document,</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/794/1*2xmnTvZAv590oSsWeV8qnw.png" /></figure><p>let’s build and execute a simple pipeline with the following code:</p><pre>...<br>documentAssembler = DocumentAssembler() \<br>    .setInputCol(&#39;text&#39;) \<br>    .setOutputCol(&#39;document&#39;)<br><br>cleanUpPatterns = [&quot;&lt;[^&gt;]*&gt;&quot;]<br><br>documentNormalizer = DocumentNormalizer() \<br>    .setInputCols(&quot;document&quot;) \<br>    .setOutputCol(&quot;normalizedDocument&quot;) \<br>    .setAction(&quot;clean&quot;) \<br>    .setPatterns(cleanUpPatterns) \<br>    .setReplacement(&quot; &quot;) \<br>    .setPolicy(&quot;pretty_all&quot;) \<br>    .setLowercase(<strong>True</strong>)</pre><pre>docPatternRemoverPipeline = \<br>  Pipeline() \<br>    .setStages([<br>        documentAssembler,<br>        documentNormalizer<br>    ])<br><br>ds = docPatternRemoverPipeline.fit(df).transform(df)<br><br>ds.select(&quot;normalizedDocument&quot;).show(1, False)</pre><p>to visualize the annotator action.</p><p>As we can see in the result below, the HTML tags content has been <strong>extracted, lowercased and formatted</strong> in the output column “normalizedDocument” and we can now use it as input of the next stages in a Spark NLP pipeline.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/811/1*Bw4qClC2URFYucVYvnvgyA.png" /></figure><p>To provide a more complex example, the DocumentNormalizer annotator can be used as a text preparation step before the SentenceDetector followed by the Tokenizer as show in the following pipeline:</p><pre>documentAssembler = DocumentAssembler() \<br>    .setInputCol(&#39;text&#39;) \<br>    .setOutputCol(&#39;document&#39;)<br><br>cleanUpPatterns = [&quot;&lt;[^&gt;]*&gt;&quot;]<br><br>documentNormalizer = DocumentNormalizer() \<br>    .setInputCols(&quot;document&quot;) \<br>    .setOutputCol(&quot;normalizedDocument&quot;) \<br>    .setAction(&quot;clean&quot;) \<br>    .setPatterns(cleanUpPatterns) \<br>    .setReplacement(&quot; &quot;) \<br>    .setPolicy(&quot;pretty_all&quot;) \<br>    .setLowercase(<strong>True</strong>)<br><br>sentenceDetector = SentenceDetector() \<br>      .setInputCols([&quot;normalizedDocument&quot;]) \<br>      .setOutputCol(&quot;sentence&quot;)<br><br>regexTokenizer = Tokenizer() \<br>      .setInputCols([&quot;sentence&quot;]) \<br>      .setOutputCol(&quot;token&quot;) \<br>      .fit(df)<br><br>docPatternRemoverPipeline = \<br>  Pipeline() \<br>    .setStages([<br>        documentAssembler,<br>        documentNormalizer,<br>        sentenceDetector,<br>        regexTokenizer])<br><br>ds = docPatternRemoverPipeline.fit(df).transform(df)<br><br>ds.select(&quot;normalizedDocument&quot;).show(10)</pre><pre>...<br></pre><p>This pipeline is processing your HTML documents, applying the document normalization following your parameter settings, and chaining the cleaning action with sentence detector and regex tokenizer to provide in output clean tokens to further process.</p><p>Other interesting use cases in which the DocumentNormalizer can be useful:</p><ul><li>extract content from a specif HTML tag, e.g. &lt;p&gt;</li></ul><pre>action = &quot;clean&quot;<br>tag = &quot;p&quot;<br>patterns = [&quot;&lt;&quot;+tag+&quot;(.+?)&gt;(.+?)&lt;\\/&quot;+tag+&quot;&gt;&quot;]</pre><ul><li>obfuscate PII such as emails in HTML content</li></ul><pre>action = &quot;clean&quot;<br>tag = &quot;p&quot;<br>patterns = [&quot;([^.@\\s]+)(\\.[^.@\\s]+)*@([^.@\\s]+\\.)+([^.@\\s]+)&quot;]<br>replacement = &quot;***OBFUSCATED PII***&quot;</pre><ul><li>extract XML name tag contents</li></ul><pre>action = &quot;extract&quot;<br>tag = &quot;name&quot;<br>patterns = [tag]</pre><p>Hope this article was useful! Have fun with the Spark NLP brand new release!</p><p>Further reference:</p><ul><li>Intro and further article series <a href="https://towardsdatascience.com/introduction-to-spark-nlp-foundations-and-basic-components-part-i-c83b7629ed59">https://towardsdatascience.com/introduction-to-spark-nlp-foundations-and-basic-components-part-i-c83b7629ed59</a></li><li>Spark NLP Web site: <a href="https://nlp.johnsnowlabs.com/">https://nlp.johnsnowlabs.com/</a></li><li>Notebook: <a href="https://github.com/JohnSnowLabs/spark-nlp-workshop/blob/master/jupyter/annotation/english/document-normalizer/document_normalizer_notebook.ipynb">https://github.com/JohnSnowLabs/spark-nlp-workshop/blob/master/jupyter/annotation/english/document-normalizer/document_normalizer_notebook.ipynb</a></li></ul><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=913d96b2ee34" width="1" height="1" alt=""><hr><p><a href="https://medium.com/spark-nlp/cleaning-and-extracting-content-from-html-xml-documents-using-spark-nlp-documentnormalizer-913d96b2ee34">Cleaning and extracting content from HTML/XML documents using Spark NLP</a> was originally published in <a href="https://medium.com/spark-nlp">spark-nlp</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Reliable and serverless data ingestion using Delta Lake on AWS Glue]]></title>
            <link>https://wolliq.medium.com/reliable-and-serverless-data-ingestion-using-delta-lake-on-aws-glue-ba16a9ba128f?source=rss-2194f57e111a------2</link>
            <guid isPermaLink="false">https://medium.com/p/ba16a9ba128f</guid>
            <category><![CDATA[delta-lake]]></category>
            <category><![CDATA[aws]]></category>
            <category><![CDATA[spark]]></category>
            <category><![CDATA[data-ingestion]]></category>
            <category><![CDATA[data-warehouse]]></category>
            <dc:creator><![CDATA[Stefano Lori]]></dc:creator>
            <pubDate>Tue, 21 Jul 2020 18:59:12 GMT</pubDate>
            <atom:updated>2020-07-21T18:59:12.246Z</atom:updated>
            <content:encoded><![CDATA[<p><strong>The Big Data scenario</strong></p><p>Imagine a new requirement suddenly comes out: it has become necessary to re-process your last 2 years of data to come out with some specific reports for the BI team. Wait, your production database is taking 15 hours to compute the reports and the clients need the outcomes tomorrow? And actually every week?</p><p>One possible solution is to create a data warehouse on <a href="https://aws.amazon.com/s3">S3</a> ingesting data from your database and scale the data processing using some of the most powerful parallel and distributed engines such as <a href="https://spark.apache.org/">Apache Spark</a>.</p><p><strong>Delta Lake</strong></p><p>Last April 2019 <a href="https://databricks.com/">Databricks</a> open sourced a very interesting project called Delta Lake. As per creators definition “Delta Lake is a storage layer that brings scalable, ACID transactions to Apache Spark and other big-data engines”.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/858/1*Y06u3J57nVqzcyCQFreHTw.png" /><figcaption>The Delta Lake overview</figcaption></figure><p>You can follow the open source project on GitHub: <a href="https://github.com/delta-io/delta">https://github.com/delta-io/delta</a> .</p><p><strong>Core concepts</strong></p><p><strong>Data consistency</strong>: one of Delta Lake core concepts is the <strong>transaction log</strong>, an ordered record of every transaction that has ever been performed on a Delta Lake table since its creation.<br>Technically, Delta Lake extends Apache Spark SQL module adding the<strong> single source of truth </strong>to track all the changes made on a Delta table, so that multiple readers and writers of the given table can work with a consistent and correct view of the data.<br>For every new access to a Delta table or new query on a modified open table, Spark uses the transaction log protocol to check the changes and ensure data consistency for the users. Transactions that fully execute are recorded and used as base for the guarantee of atomicity. In such a way, batch and streaming data ingestions can be unified consistently.</p><p><strong>Upserts and deletes</strong>: bringing ACID transaction guarantees between reads and writes, Delta Lake now supports merge, delete and update operations to solve complex scenarios.</p><p><strong>Schema enforcement and evolution</strong>: somebody added 3 columns in 2 different tables without notifying you and you didn’t have the time to prepare the ingestion workflow in order to process them properly? No worries, even if such as thing should never happen, Delta Lake provides useful options such as mergeSchema and overwriteSchema to handle schema variations and prevent errors during ingestion. These features can literally save the day.</p><p><strong>Going serverless using AWS Glue</strong></p><p>Behind the scenes AWS Glue, the fully managed ETL (extract, transform, and load) service, uses a Spark YARN cluster but it can be seen as an auto-scale “serverless Spark” solution.<br>Main differences with its self-managed counterpart, Amazon EMR, are of course the serverless service which can be configured to auto-scale with higher flexibility but at a higher price, and the possibility to take advantage of the Glue Catalogue to store metadata about data sources, transforms, and targets.<br>At the same time, it is possible to use the Spark session object in the Glue session, ignoring the AWS Glue SDK and the Glue Catalog, and replacing the auto-generated script with regular and customized Spark code. You can find more information in the developer guide: <a href="https://docs.aws.amazon.com/glue/latest/dg/what-is-glue.html">https://docs.aws.amazon.com/glue/latest/dg/what-is-glue.html</a> . <br>Regarding performance, AWS Glue has just entered a new phase to fill the gap with with the faster EMR. For more info <a href="https://pages.awscloud.com/glue-reduced-spark-times-preview-2020.html">https://pages.awscloud.com/glue-reduced-spark-times-preview-2020.html</a> .</p><p><strong>Some AWS Scala SDK hints</strong></p><p>AWS Scala SDK is not published on Maven so you need to “complete some prerequisite steps and then issue a Maven command to run your Scala ETL script locally” as mentioned on the AWS doc (<a href="https://docs.aws.amazon.com/glue/latest/dg/aws-glue-programming-etl-libraries.html">https://docs.aws.amazon.com/glue/latest/dg/aws-glue-programming-etl-libraries.html</a>).</p><p>Alternatively you can create a uber jar with all dependencies (not the best solution) or download the necessary jars from the artifact repository, <a href="https://aws-glue-etl-artifacts.s3.amazonaws.com/">https://aws-glue-etl-artifacts.s3.amazonaws.com/</a> .</p><p>In particular, if you want to locally use Glue 1.0, which is built on top of Spark 2.4.3 and Hadoop 2.8.x, you have to handle a couple of jars:</p><ul><li>AWSGlueETL-1.0.0.jar</li><li>AWSGlueReaders-1.0.0.jar</li></ul><p>and add them to your classpath using an sbt integration, for instance</p><pre>unmanagedBase <strong>:=</strong> baseDirectory<strong>.</strong>value <strong>/</strong> &quot;aws-glue-lib&quot;</pre><p><strong>Creating the uber jar</strong></p><p>After your development is finished you can assemble a jar to load your dependencies in Glue.</p><p>There are many ways to do it. If you are using sbt, you can create it by adding the Delta Lake dependency in the build.sbt</p><pre>// <a href="https://mvnrepository.com/artifact/io.delta/delta-core">https://mvnrepository.com/artifact/io.delta/delta-core</a><br>libraryDependencies += &quot;io.delta&quot; %% &quot;delta-core&quot; % &quot;0.6.1&quot;</pre><p>and then run</p><pre>$ sbt assembly</pre><p><strong>Loading Delta Lake dependency</strong></p><p>AWS Glue is providing many libraries to execute ETL jobs but sometimes, as presented in this article, we may want to add other functionalities to our applications such as the ones introduced above in the Delta lake library.<br>In such a case, AWS Glue is providing a nice way to create powerful and serverless data ingestion applications.<br>In fact dependencies can be packaged in a custom jar and pushed to a S3 location where Glue’s configuration could load it passing the path at execution time.</p><p>By assembling your customized jar which includes Delta Lake dependency, it is possible to run AWS Glue taking advantage of the library features.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*XOteudzWrvEf2ZpBoyqAZw.png" /><figcaption>Add your uber jar dependencies into AWS Glue configuration panel</figcaption></figure><p><strong>Consuming the ingested Delta Lake data</strong></p><p>Once the data has been ingested on S3 using the Delta format, it can be consumed by other Spark applications packaged with Delta Lake library, or can be registered and queried using serverless SQL services such Amazon Athena (performing a certain number of manual operations).</p><ol><li>From EC2 using a Spark shell</li></ol><pre>bin/spark-shell --packages io.delta:delta-core_2.11:0.6.1<br><strong>...<br>// load the delta table<br>val</strong> deltaTable <strong>=</strong> <strong>DeltaTable.</strong>forPath<strong>(</strong>pathToDeltaTable<strong>)</strong></pre><ol><li>From Athena</li></ol><p>In such direction, starting from Delta Lake version 0.5, the library introduced the symlink_format_manifest generation to add the capability of being consumed by the Amazon Athena service. For more info see <a href="https://docs.delta.io/0.5.0/presto-integration.html">https://docs.delta.io/0.5.0/presto-integration.html</a> .</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*2OkRs3qronuY5Bt6jZBVlQ.png" /><figcaption>Generate simlinks directly from APIs</figcaption></figure><p>The disadvantage is represented by the fact that skipping the AWS Glue Catalogue the table have to be registered in the Athena metastore manually using external tables creation scripts and partitions repair commands for every new data ingestion.</p><p><strong>Conclusion</strong></p><p>In this article is presented a relatively simple procedure to add Delta Lake features to AWS Glue framework in order to create a reliable and serveless data ingestion layer to your Data Warehouse or Data Lake.</p><p>This solution is flexible and allows you to create a custom jar which will bring very nice features to your serverless ingestion process and will make your Ops team very happy! :)</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=ba16a9ba128f" width="1" height="1" alt="">]]></content:encoded>
        </item>
    </channel>
</rss>