<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Mageswaran D on Medium]]></title>
        <description><![CDATA[Stories by Mageswaran D on Medium]]></description>
        <link>https://medium.com/@mageswaran1989?source=rss-f442f089254f------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/1*HgaN9EXxyBhtBRvW7HMuLw.jpeg</url>
            <title>Stories by Mageswaran D on Medium</title>
            <link>https://medium.com/@mageswaran1989?source=rss-f442f089254f------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Mon, 16 Nov 2026 21:39:35 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@mageswaran1989/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[How to keep Cloud cost under control?]]></title>
            <link>https://mageswaran1989.medium.com/how-to-keep-cloud-cost-under-control-6d7b52076eb3?source=rss-f442f089254f------2</link>
            <guid isPermaLink="false">https://medium.com/p/6d7b52076eb3</guid>
            <category><![CDATA[control]]></category>
            <category><![CDATA[cost]]></category>
            <category><![CDATA[cloud]]></category>
            <dc:creator><![CDATA[Mageswaran D]]></dc:creator>
            <pubDate>Sat, 16 Apr 2022 14:38:17 GMT</pubDate>
            <atom:updated>2022-04-16T14:38:17.659Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/638/0*SNpgFkJpuPt5zHbR.jpg" /></figure><ol><li>Collect the <strong>hardware metrics</strong> (CPU/memory utilization, disk I/O)</li></ol><figure><img alt="" src="https://cdn-images-1.medium.com/max/641/0*6XYy3sKhdZt9Wm_B.jpg" /></figure><p>2. Collect the <strong>app/web server metrics</strong> — The ones like Tomcat, Apache HTTP, Flask, Django etc running on the hardware (Collect metrics like top processes that consume more CPU memory, available heap memory, thread usage, request throughput, sessions, connection pooling etc)</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/928/0*canYQ4PVeP_6ZTm2" /></figure><p>3. Collect all the<strong> DB metrics</strong> (connection pool stats, memory/connection leaks, low performing queries/SP’s etc)</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*aNggie_lNa0cOzAk" /></figure><p>4. Collect Real usage metrics (concurrent connections, load on a particular service/feature etc)</p><p>5. Network metrics (latency, throughput, packet loss etc)</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*vHpMujFbeXD4aJRU.png" /></figure><p>6. UI metrics — page response, DNS lookup, latency with connecting to third party API’s etc</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*dfTVYdH4F0ihg8yG.png" /></figure><p>Once all these are collected, wear the hat of a performance tuning engineer (maybe along with a solution architect who knows how the full stack works) and then analyze what causes the bottleneck.</p><p>As an e.g, if you see the CPU of your VM peaking up, then check which are the processes that consume the CPU. If it is a app/web server process that eats CPU, then check what causes that spike in that app/web server (maybe a complex calculation or poorly written code)</p><p>Similarly, with the metrics that we collected in the 6 steps, we should be able to identify the issue exactly.</p><p>Tools like <a href="https://www.zabbix.com/">zabbix</a>, <a href="https://www.site24x7.com/">zoho site 24/7</a>, <a href="https://prometheus.io/">Prometheus</a>, <a href="https://aws.amazon.com/cloudwatch/">Cloudwatch</a>, <a href="https://www.idera.com/">Idera</a>, <a href="https://www.dynatrace.com/">dynatrace</a>, <a href="https://www.appdynamics.com/">appdynamics</a>, <a href="https://www.datadoghq.com/">datadog</a>, <a href="https://developer.chrome.com/docs/devtools/">Google developer toolbar</a>, <a href="https://www.telerik.com/fiddler">Fiddler</a> etc can provide you with the required metrics</p><p>All these data ingestion can be automated by tools like <a href="https://www.elastic.co/what-is/elk-stack">ELK</a>, <a href="https://www.splunk.com/">Splunk</a>, A<a href="https://aws.amazon.com/opensearch-service/the-elk-stack/what-is-opensearch/">WS Open Search</a> etc.</p><p>Now, I leave it to your imagination on what can be automated, how patterns can be identified and fed into machine learning and AI for self healing systems etc.</p><p>Hope you got the concept — This will help in having efficient applications that won’t hog all your hardware resources and ultimately you won’t end up over provisioning your resources.</p><p>Always remember cloud (any vendor for that matter) is a double edged sword — If not used in the right way, the expenses will exponentially skyrocket and when you really want to bring it down, you will face a nightmare!</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=6d7b52076eb3" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[2022 : No FileSystem for scheme: s3?]]></title>
            <link>https://mageswaran1989.medium.com/2022-no-filesystem-for-scheme-s3-cbd72c99a50c?source=rss-f442f089254f------2</link>
            <guid isPermaLink="false">https://medium.com/p/cbd72c99a50c</guid>
            <category><![CDATA[error]]></category>
            <category><![CDATA[apache-spark]]></category>
            <category><![CDATA[aws-s3]]></category>
            <category><![CDATA[filesystem]]></category>
            <category><![CDATA[delta-lake]]></category>
            <dc:creator><![CDATA[Mageswaran D]]></dc:creator>
            <pubDate>Thu, 24 Mar 2022 08:23:23 GMT</pubDate>
            <atom:updated>2022-03-24T08:23:23.395Z</atom:updated>
            <content:encoded><![CDATA[<h3>2022 : No FileSystem for scheme: s3? How to read/write data from AWS S3 with a Standalone Apache Spark 3.2.1 setup? and use Delta as format?</h3><p>How many times did you face this <a href="https://stackoverflow.com/questions/46740670/no-filesystem-for-scheme-s3-with-pyspark">error</a>? S3 or S3a schema not found error while working with AWS S3 + Spark outside the EMR?</p><pre>An error occurred while calling o25.partitions.<br>: java.io.IOException: No FileSystem for scheme: s3<br>    at org.apache.hadoop.fs.FileSystem.getFileSystemClass(FileSystem.java:2660)<br>    at org.apache.hadoop.fs.FileSystem.createFileSystem(FileSystem.java:2667)<br>    at org.apache.hadoop.fs.FileSystem.access$200(FileSystem.java:94)<br>    at org.apache.hadoop.fs.FileSystem$Cache.getInternal(FileSystem.java:2703)<br>    at org.apache.hadoop.fs.FileSystem$Cache.get(FileSystem.java:2685)<br>    at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:373)<br>    at org.apache.hadoop.fs.Path.getFileSystem(Path.java:295)<br>    at org.apache.hadoop.mapred.FileInputFormat.singleThreadedListStatus(FileInputFormat.java:258)<br>    at org.apache.hadoop.mapred.FileInputFormat.listStatus(FileInputFormat.java:229)<br>    at org.apache.hadoop.mapred.FileInputFormat.getSplits(FileInputFormat.java:315)<br>    at org.apache.spark.rdd.HadoopRDD.getPartitions(HadoopRDD.scala:194)<br>    at org.apache.spark.rdd.RDD$$anonfun$partitions$2.apply(RDD.scala:252)<br>    at org.apache.spark.rdd.RDD$$anonfun$partitions$2.apply(RDD.scala:250)<br>    at scala.Option.getOrElse(Option.scala:121)<br>    at org.apache.spark.rdd.RDD.partitions(RDD.scala:250)<br>    at org.apache.spark.rdd.MapPartitionsRDD.getPartitions(MapPartitionsRDD.scala:35)<br>    at org.apache.spark.rdd.RDD$$anonfun$partitions$2.apply(RDD.scala:252)<br>    at org.apache.spark.rdd.RDD$$anonfun$partitions$2.apply(RDD.scala:250)<br>    at scala.Option.getOrElse(Option.scala:121)<br>    at org.apache.spark.rdd.RDD.partitions(RDD.scala:250)<br>    at org.apache.spark.api.java.JavaRDDLike$class.partitions(JavaRDDLike.scala:61)<br>    at org.apache.spark.api.java.AbstractJavaRDDLike.partitions(JavaRDDLike.scala:45)<br>    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)<br>    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)<br>    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)<br>    at java.lang.reflect.Method.invoke(Method.java:498)<br>    at py4j.reflection.MethodInvoker.invoke(MethodInvoker.java:244)<br>    at py4j.reflection.ReflectionEngine.invoke(ReflectionEngine.java:357)<br>    at py4j.Gateway.invoke(Gateway.java:280)<br>    at py4j.commands.AbstractCommand.invokeMethod(AbstractCommand.java:132)<br>    at py4j.commands.CallCommand.execute(CallCommand.java:79)<br>    at py4j.GatewayConnection.run(GatewayConnection.java:214)<br>    at java.lang.Thread.run(Thread.java:748)</pre><p>Most of the time I get approched to do Spark based proof of concepts from scratch, and I always have to take care of this setup process for the latest Apache Spark version before anything and its not always straight-forward.</p><p>In my experience writing Spark code is easy comparitively, the tough job comes when we wanted it to integrate it into other systems. One such pain is to use AWS S3.</p><p>Most of the time I have to start with local machine, sometimes with AWS EC2 and if I am lucky I can spin up my own EMR where all the dependcies are already taken care and I am good to do my Proof of Concepts.</p><p>Be it local machine or EC2 the setup remains more or less the same. Reading and writing to AWS S3 from a standalone Spark needs some extra Java libraries plugged into Spark.</p><p>Java Jar dependencies are very finicky.</p><p>To read S3 from a standalone Spark setup, we need hadoop-aws and aws-java-sdk-bundle</p><ul><li><a href="https://mvnrepository.com/artifact/org.apache.hadoop/hadoop-aws">Maven Repository: org.apache.hadoop &quot; hadoop-aws</a></li><li><a href="https://mvnrepository.com/artifact/com.amazonaws/aws-java-sdk-bundle">Maven Repository: com.amazonaws &quot; aws-java-sdk-bundle</a></li></ul><p>With Spark 3.2.1, after a days effort I figured out the working combination and here is the screenshot of the versions.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*1CHaZ7H6aTYpkgwpqdkDGA.png" /></figure><pre>from pyspark.sql import SparkSession<br></pre><pre>def get_spark():<br>    spark = SparkSession.builder.master(&quot;local[4]&quot;).appName(&#39;SparkDelta&#39;) \<br>        .config(&quot;spark.sql.extensions&quot;, &quot;io.delta.sql.DeltaSparkSessionExtension&quot;) \<br>        .config(&quot;spark.sql.catalog.spark_catalog&quot;, &quot;org.apache.spark.sql.delta.catalog.DeltaCatalog&quot;) \<br>        .config(&quot;spark.hadoop.fs.s3a.impl&quot;, &quot;org.apache.hadoop.fs.s3a.S3AFileSystem&quot;) \<br>        .config(&quot;<strong>spark.jars.packages</strong>&quot;, <br>                &quot;io.delta:delta-core_2.12:1.1.0,&quot;<br>                &quot;org.apache.hadoop:<strong>hadoop-aws:3.2.2</strong>,&quot;<br>                &quot;com.amazonaws:<strong>aws-java-sdk-bundle</strong>:<strong>1.12.180</strong>&quot;) \<br>        .getOrCreate()</pre><pre>return spark</pre><pre># This is mandate config on spark session to use AWS S3<br>       spark._jsc.hadoopConfiguration().set(&quot;com.amazonaws.services.s3.enableV4&quot;, &quot;true&quot;)</pre><pre>spark._jsc.hadoopConfiguration().set(&quot;fs.s3a.impl&quot;, &quot;org.apache.hadoop.fs.s3a.S3AFileSystem&quot;)<br>    spark._jsc.hadoopConfiguration().set(&quot;fs.s3a.aws.credentials.provider&quot;,&quot;com.amazonaws.auth.InstanceProfileCredentialsProvider,com.amazonaws.auth.DefaultAWSCredentialsProviderChain&quot;)<br>    spark._jsc.hadoopConfiguration().set(&quot;fs.AbstractFileSystem.s3a.impl&quot;, &quot;org.apache.hadoop.fs.s3a.S3A&quot;)</pre><pre># spark.sparkContext.setLogLevel(&quot;DEBUG&quot;)</pre><blockquote>Remeber to give all dependencies as comma seperated in one shot for: <em>spark.jars.pacakges</em></blockquote><p>When you use Pyspark shell, use this command to download the dependencies prior to auto SparkSession initialization</p><pre>bin/pyspark --packages io.delta:delta-core_2.12:1.1.0,org.apache.hadoop:hadoop-aws:3.2.2,com.amazonaws:aws-java-sdk-bundle:1.12.180</pre><p>And in PySpark shell</p><pre>spark._jsc.hadoopConfiguration().set(&quot;<strong>com.amazonaws.services.s3.enableV4</strong>&quot;, &quot;true&quot;)</pre><pre>spark._jsc.hadoopConfiguration().set(&quot;<strong>fs.s3a.impl</strong>&quot;,<br>                           &quot;org.apache.hadoop.fs.s3a.S3AFileSystem&quot;)</pre><pre>spark._jsc.hadoopConfiguration().set(&quot;<strong>fs.s3a.aws.credentials.provider</strong>&quot;, &quot;com.amazonaws.auth.InstanceProfileCredentialsProvider,&quot;                                   &quot;com.amazonaws.auth.DefaultAWSCredentialsProviderChain&quot;)</pre><pre>spark._jsc.hadoopConfiguration().set(&quot;<strong>fs.AbstractFileSystem.s3a.impl</strong>&quot;, &quot;org.apache.hadoop.fs.s3a.S3A&quot;)</pre><h4>Issues faced:</h4><ul><li>AWS S3 support with Spark standalone setup : <a href="https://community.cloudera.com/t5/Support-Questions/Spark-on-S3/m-p/103435">https://community.cloudera.com/t5/Support-Questions/Spark-on-S3/m-p/103435</a> and <a href="https://stackoverflow.com/questions/30851244/spark-read-file-from-s3-using-sc-textfile-s3n/50276200#50276200">https://stackoverflow.com/questions/30851244/spark-read-file-from-s3-using-sc-textfile-s3n/50276200#50276200</a></li><li>Spark version mismatch with hadoop-aws library. <br> Official thread @ <a href="https://issues.apache.org/jira/browse/HADOOP-16080">https://issues.apache.org/jira/browse/HADOOP-16080</a> and <a href="https://issues.apache.org/jira/browse/SPARK-7442">https://issues.apache.org/jira/browse/SPARK-7442</a></li></ul><p>Now store data in <a href="http://delta.io/">delta.io</a> format:</p><pre>def test_spark():<br>    spark = get_spark()<br>    data = {&quot;col1&quot;: [1, 2, 3, 4, 5], &quot;col2&quot;: [6, 7, 8, 9, 0]}<br>    pdf = pd.DataFrame(data)<br>    print(pdf)<br>    sdf = spark.createDataFrame(pdf)<br>    sdf.show()<br>    sdf.write.format(&quot;delta&quot;).mode(&#39;overwrite&#39;).save(&quot;&quot;/tmp/delta/&quot;)<br>    return &quot;Done&quot;</pre><p>output:</p><pre>% ls /tmp/delta<br>_delta_log<br>part-00000-403015b4-2686-4198-8d4a-ac647a8f2e1c-c000.snappy.parquet<br>part-00001-1f3eb0ca-bacb-4315-9676-2e1faea57ad6-c000.snappy.parquet<br>part-00002-c9e55dc6-7e4d-4f45-83cc-494b2b5f4a42-c000.snappy.parquet<br>part-00003-d2930828-8aaa-4f8b-b62f-95b5c7404c7f-c000.snappy.parquet</pre><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=cbd72c99a50c" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[PySpark Practice Problems]]></title>
            <link>https://mageswaran1989.medium.com/pyspark-practice-problems-7154bdfc5cb?source=rss-f442f089254f------2</link>
            <guid isPermaLink="false">https://medium.com/p/7154bdfc5cb</guid>
            <category><![CDATA[window-functions]]></category>
            <category><![CDATA[pyspark]]></category>
            <category><![CDATA[interview]]></category>
            <dc:creator><![CDATA[Mageswaran D]]></dc:creator>
            <pubDate>Sun, 02 Jan 2022 22:06:07 GMT</pubDate>
            <atom:updated>2022-01-02T22:06:07.761Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*4n97lCkYBe8uydij.jpg" /></figure><ol><li>How to transform array of arrays into columns?</li></ol><pre>import pyspark.sql.functions as F<br><br>df = spark.createDataFrame(<br>    [([[&quot;a&quot;,&quot;b&quot;,&quot;c&quot;], [&quot;d&quot;,&quot;e&quot;,&quot;f&quot;], [&quot;g&quot;,&quot;h&quot;,&quot;i&quot;, &quot;j&quot;]],)],<br>    [&quot;data&quot;]<br>)<br><br>df.show(20, False)<br><br>df = df.withColumn(&quot;data1&quot;, F.explode(&quot;data&quot;))<br>df.select(&#39;data1&#39;).show()<br><br># Row(max(size(data1))=4) ---&gt; 4<br>max_size = df.select(F.max(F.size(&#39;data1&#39;))).collect()[0][0]<br><br><br>df.select(<br>    *[F.col(&quot;data1&quot;)[i].alias(f&quot;col_{i}&quot;) for i in range(max_size)]<br>).show()<br><br><br><br>+------------------------------------+<br>|data                                |<br>+------------------------------------+<br>|[[a, b, c], [d, e, f], [g, h, i, j]]|<br>+------------------------------------+<br><br>+------------+<br>|       data1|<br>+------------+<br>|   [a, b, c]|<br>|   [d, e, f]|<br>|[g, h, i, j]|<br>+------------+<br><br>+-----+-----+-----+-----+<br>|col_0|col_1|col_2|col_3|<br>+-----+-----+-----+-----+<br>|    a|    b|    c| null|<br>|    d|    e|    f| null|<br>|    g|    h|    i|    j|<br>+-----+-----+-----+-----+</pre><p>2. How to create a new dataframe from two time series dataframes?</p><pre># Dataset 1</pre><pre>cust_id     pt_dt</pre><pre>985XFT82Y4  20200824<br>985XFT82Y4  20200826<br>985XFT82Y4  20200902<br>985XFT82Y4  20200918<br>985XFT82Y4  20200930<br>985XFT82Y4  20201016<br>985XFT82Y4  20201021<br>985XFT82Y4  20201102<br>985XFT82Y4  20201111<br>985XFT82Y4  20201112<br>985XFT82Y4  20201208<br>985XFT82Y4  20210111<br>985XFT82Y4  20210202<br>985XFT82Y4  20210303<br>985XFT82Y4  20210309<br>985XFT82Y4  20210311</pre><pre><br># Dataset 2</pre><pre>cust_id     chg_date                       ins_status<br>985XFT82Y4  2020-08-24 22:12:34.332000     subscribed<br>985XFT82Y4  2020-11-11 14:45:31.152000     installed<br>985XFT82Y4  2021-02-02 01:26:34.500000     migration<br>985XFT82Y4  2021-03-09 08:11:57.790000     setup done</pre><pre><br># Expected output</pre><pre>cust_id     pt_dt           ins_status<br>985XFT82Y4  20200824        subscribed<br>985XFT82Y4  20200826        subscribed<br>985XFT82Y4  20200902        subscribed<br>985XFT82Y4  20200918        subscribed<br>985XFT82Y4  20200930        subscribed<br>985XFT82Y4  20201016        subscribed<br>985XFT82Y4  20201021        subscribed<br>985XFT82Y4  20201102        subscribed<br>985XFT82Y4  20201111        installed<br>985XFT82Y4  20201112        installed<br>985XFT82Y4  20201208        installed<br>985XFT82Y4  20210111        installed<br>985XFT82Y4  20210202        migration<br>985XFT82Y4  20210303        migration<br>985XFT82Y4  20210309        setup done<br>985XFT82Y4  20210311        setup done</pre><pre>import pyspark.sql.functions as F<br>from pyspark.sql import Window<br>from datetime import datetime</pre><pre>data = [(&quot;985XFT82Y4&quot;, &quot;20200824&quot;),<br>(&quot;985XFT82Y4&quot;, &quot;20200826&quot;), <br>(&quot;985XFT82Y4&quot;, &quot;20200902&quot;), <br>(&quot;985XFT82Y4&quot;, &quot;20200918&quot;), <br>(&quot;985XFT82Y4&quot;, &quot;20200930&quot;), <br>(&quot;985XFT82Y4&quot;, &quot;20201016&quot;), <br>(&quot;985XFT82Y4&quot;, &quot;20201021&quot;), <br>(&quot;985XFT82Y4&quot;, &quot;20201102&quot;), <br>(&quot;985XFT82Y4&quot;, &quot;20201111&quot;), <br>(&quot;985XFT82Y4&quot;, &quot;20201112&quot;), <br>(&quot;985XFT82Y4&quot;, &quot;20201208&quot;), <br>(&quot;985XFT82Y4&quot;, &quot;20210111&quot;), <br>(&quot;985XFT82Y4&quot;, &quot;20210202&quot;), <br>(&quot;985XFT82Y4&quot;, &quot;20210303&quot;), <br>(&quot;985XFT82Y4&quot;, &quot;20210309&quot;), <br>(&quot;985XFT82Y4&quot;, &quot;20210311&quot;)]</pre><pre>df1 = spark.createDataFrame(data, [&quot;cust_id&quot;, &quot;pt_dt&quot;]).withColumn(&quot;pt_dt&quot;, F.to_timestamp(&quot;pt_dt&quot;, &quot;yyyyMMdd&quot;)).withColumn(&quot;pt_dt&quot;, F.date_format(F.col(&#39;pt_dt&#39;),&quot;yyyy-MM-dd&quot;))<br>df1.show()</pre><pre>data1 = [(&quot;985XFT82Y4&quot;, &quot;2020-08-24 22:12:34.332000&quot;,  &quot;subscribed&quot;),   <br>(&quot;985XFT82Y4&quot;, &quot;2020-11-11 14:45:31.152000&quot;,  &quot;installed&quot;),     <br>(&quot;985XFT82Y4&quot;, &quot;2021-02-02 01:26:34.500000&quot;,  &quot;migration&quot;),     <br>(&quot;985XFT82Y4&quot;, &quot;2021-03-09 08:11:57.790000&quot;,  &quot;setup done&quot;)]    <br>ts_pattern = &quot;yyyy-MM-dd HH:mm:ss.SSSSSS&quot;</pre><pre>df2 = spark.createDataFrame(data1, [&quot;cust_id&quot;, &quot;chg_date&quot;, &quot;ins_status&quot;]).withColumn(&quot;chg_date&quot;, F.to_timestamp(&quot;chg_date&quot;, ts_pattern)).withColumn(&quot;chg_date&quot;, F.date_format(F.col(&#39;chg_date&#39;),&quot;yyyy-MM-dd&quot;))<br>df2.show()</pre><pre>window_spec = Window.partitionBy(&quot;cust_id&quot;).orderBy(&quot;chg_date&quot;)<br>df2 = df2.withColumn(&quot;end_chg_date&quot;, F.lead(&quot;chg_date&quot;).over(window_spec))<br>df2.show()</pre><pre>cond = [df1[&quot;cust_id&quot;] == df2[&quot;cust_id&quot;], df1[&quot;pt_dt&quot;] &gt;= df2[&quot;chg_date&quot;], df2[&quot;end_chg_date&quot;].isNull() | (df1[&quot;pt_dt&quot;] &lt; df2[&quot;end_chg_date&quot;])]<br>df3 = df1.join(df2, cond, &quot;left&quot;).select(df1[&quot;cust_id&quot;], df1[&quot;pt_dt&quot;], &quot;ins_status&quot;).orderBy(&quot;pt_dt&quot;)<br># use df1 in select to resolve same column name conflict<br>df3.show()</pre><p>3. Group DataFrame by hour of the Day:</p><p><a href="https://stackoverflow.com/questions/66647892/group-df-by-hour-of-day/66649736#66649736">https://stackoverflow.com/questions/66647892/group-df-by-hour-of-day/66649736#66649736</a></p><pre>import org.apache.spark.sql.{functions =&gt; F}<br>import org.apache.spark.sql.functions.udf<br>import org.apache.spark.sql.expressions.Window<br><br>val df = Seq(                                                                <br>(75,  &quot;2019-01-19 02:13:00&quot;,  5 ,   &quot;Brooklyn&quot;, &quot;Williamsburg&quot;),        <br>(255, &quot;2019-01-19 12:05:00&quot;,  8 ,   &quot;Brooklyn&quot;, &quot;Williamsburg&quot;),        <br>(99,  &quot;2019-01-20 12:05:00&quot;,  3 ,   &quot;Brooklyn&quot;, &quot;DUMBO&quot;),               <br>(102, &quot;2019-01-01 02:05:00&quot;,  1 ,   &quot;Brooklyn&quot;, &quot;DUBMO&quot;),               <br>(10,  &quot;2019-01-07 11:05:00&quot;,  13,   &quot;Brooklyn&quot;, &quot;Park Slope&quot;),          <br>(75,  &quot;2019-01-01 11:05:00&quot;,  2 ,   &quot;Brooklyn&quot;, &quot;Williamsburg&quot;),        <br>(12,  &quot;2019-01-11 01:05:00&quot;,  1 ,   &quot;Brooklyn&quot;, &quot;Park Slope&quot;),          <br>(98,  &quot;2019-01-28 01:05:00&quot;,  8 ,   &quot;Brooklyn&quot;, &quot;DUMBO&quot;),               <br>(75,  &quot;2019-01-10 00:05:00&quot;,  8 ,   &quot;Brooklyn&quot;, &quot;Williamsburg&quot;),        <br>(255, &quot;2019-01-11 12:05:00&quot;,  12,   &quot;Brooklyn&quot;, &quot;DUMBO&quot;),               <br>).toDF(&quot;PULocationID&quot;, &quot;pickup_datetime&quot;, &quot;number_of_pickups&quot;, &quot;Borough&quot;, &quot;Zone&quot;)<br>df.show()<br><br>val df1 = df.<br>            withColumn(&quot;pickup_datetime&quot;, F.to_timestamp(F.col(&quot;pickup_datetime&quot;),&quot;yyyy-MM-dd HH:mm:ss&quot;)).<br>            withColumn(&quot;hour&quot;, F.hour(F.col(&quot;pickup_datetime&quot;)))<br>df1.show()<br>df1.printSchema()<br><br>val windowSpec = Window.partitionBy(&quot;hour&quot;).orderBy(F.desc(&quot;number_of_pickups&quot;))<br>val df2 = df1.withColumn(&quot;rn&quot;, F.row_number.over(windowSpec))<br><br>df2.filter(F.col(&quot;rn&quot;) === 1).drop(F.col(&quot;rn&quot;)).select(&quot;hour&quot;, &quot;Zone&quot;, &quot;number_of_pickups&quot;).show()</pre><p>`</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=7154bdfc5cb" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Dynamic Programming Patterns to Ace Interviews]]></title>
            <link>https://mageswaran1989.medium.com/a-mind-map-of-dynamic-programming-patterns-to-ace-interviews-72ff0370bb27?source=rss-f442f089254f------2</link>
            <guid isPermaLink="false">https://medium.com/p/72ff0370bb27</guid>
            <category><![CDATA[inteviews]]></category>
            <category><![CDATA[data-structures]]></category>
            <category><![CDATA[trees]]></category>
            <category><![CDATA[dynamic-programming]]></category>
            <category><![CDATA[recursion]]></category>
            <dc:creator><![CDATA[Mageswaran D]]></dc:creator>
            <pubDate>Wed, 15 Dec 2021 08:53:05 GMT</pubDate>
            <atom:updated>2021-12-21T01:43:48.350Z</atom:updated>
            <content:encoded><![CDATA[<p>Following materials are taken from freecodeCamp.org Youtube video course <a href="https://www.youtube.com/watch?v=oBt53YbR9Kk">https://www.youtube.com/watch?v=oBt53YbR9Kk</a> strongly recommend to watch the video!</p><p>Python version Colab Notebook @ <a href="https://colab.research.google.com/gist/Mageswaran1989/c09f931e91b2b634d30ff83ec4264b66/dynamicprogramming.ipynb">https://colab.research.google.com/gist/Mageswaran1989/658eb75fed28f5953ac5167b3a14ff6d/dynamicprogramming.ipynb</a></p><p>Online Visualisation Tools:</p><ul><li><a href="https://visualgo.net/en/recursion">https://visualgo.net/en/recursion</a></li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/958/1*D9flUiX2FexpQ_fdNXVY3A.png" /></figure><p>Dynamic Programming is useful when a problem as repetitive computation that can be cached. When cached the repetitive computation is inferred from the memory and thus reduces overall time taken to compute the solution.</p><h4>Recursion Memoization</h4><ul><li>Visualise the problem as a tree</li><li>Implement the tree using recursion</li><li>Test it</li><li>Add memo object</li><li>Add a base case to return memo values</li></ul><h4>Array Tabulation</h4><ul><li>TODO for later time :)</li></ul><p>Once the underlying patterns in DP problems is recognised, with practice its easy to get a hang of them.</p><p>Types:</p><ul><li>Return one final value</li><li>Decision Problem (True/False)</li><li>Find a combination that matches given requirements</li><li>Finding a optimised solution in the combination</li></ul><p>Tricks:</p><ul><li>Remember the base case return type and final return should be the same</li><li>When one of the inputs happened to be list of values, then there is gonna be a for...loopin the recursion</li><li>When for...loop is involved, visualise what happens the last leaf nodes and build the result backwards</li></ul><p><strong>1One final value</strong></p><p>Find a final result value for a given problem which can have repetitive intermediate computation.</p><h3>Fibonacci Sequence</h3><pre>Base cases:<br>n &lt; 0: Return n<br>fib(n) : fib(n-1) + fib(n-2)</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/913/1*YH743ZrTRZvPz7ysidNXxQ.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/738/1*W4KTQA3vBSOZ4JLG8acGAw.png" /></figure><blockquote>Runtime: O(2^n)</blockquote><blockquote>Space: O(n)</blockquote><figure><img alt="" src="https://cdn-images-1.medium.com/max/800/1*ihViJG8E4df512tIMcDtkA.png" /></figure><pre>def fib(n, memo = {}):<br><strong>    if n in memo:<br>        return memo[n]</strong><br>    if n &lt;= 1:<br>        return n<br>    else:<br>        # calculate for first time<br>        <strong>memo[n] = fib(n-1, memo) + fib(n-2, memo)</strong><br>        return memo[n]</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/440/1*m0rXfz0wsH4FD2B1jebHJw.png" /></figure><blockquote>Runtime: O(n)</blockquote><blockquote>Space: O(n)</blockquote><p>Find a final result value for a given problem with multiple inputs, again which can have repetitive intermediate computations.</p><h3>Grid Traveller</h3><p>Say that you are a traveller on a grid. You begin in the top-left corner and your goal is to travel to the bottom-right corner. You may only move down or right. In how many ways can you travel to the goal on a grid with dimension m * n?</p><p>Write a function <strong>grid_traveller(m, n)</strong> that calculates this.</p><pre>Base cases:<br>m or n ≤ 0 : When there is no cell to move return 0<br>m and n = 1 : When there is only one cell return 1<br>memo(m,n) = memo(m-1, n) + memo(m, n-1) </pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/904/1*wuMyHAi6GLxpxarXLHsQ_A.png" /></figure><blockquote>Time: O(2^n+m)</blockquote><blockquote>Space: O(n+m)</blockquote><figure><img alt="" src="https://cdn-images-1.medium.com/max/961/1*c7AxtG-GNfP4Og87uCONHA.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/773/1*qzZPDvkhng-75XpPSBnOFA.png" /></figure><pre>def grid_travel(m,n, memo={}):<br>    <em>&quot;&quot;&quot;<br><br>    </em><strong><em>:param</em></strong><em> m: Total number of rows<br>    </em><strong><em>:param</em></strong><em> n: Total number of columns<br>    </em><strong><em>:return</em></strong><em>:<br>    &quot;&quot;&quot;<br>    </em></pre><pre><em>    # make the key with m &amp; n, to differentiate rows and cols<br>    key = </em>f&quot;m{m}n{n}&quot;</pre><pre><strong><em>    </em>if key in memo:<br>        return memo[key]</strong><br><br>    if (m == 1 and n==1):<br>        return 1<br>    elif (m ==0 or n ==0):<br>        return 0<br>    else:<br>        <strong>memo[key] = grid_travel(m-1, n, memo) + grid_travel(m, n-1, memo)</strong><br>        return memo[key]</pre><blockquote>Time: O(n * m)</blockquote><blockquote>Space: O(n+m)</blockquote><p><strong>2 Decision Problem</strong></p><p>Return a true or false after exploring all possible solution space</p><h4>canSum</h4><p>Write a function canSum(target_sum, numbers) that takes in a target sum and an array of numbers as arguments.</p><p>The function should return a boolean indicating whether or not it is possible to generate the target sum using numbers from the array.</p><p>You may use an element of the array as many times as needed and al inputs are non negative</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/749/1*_BDlfA6LqJENuCucwolKNA.png" /></figure><blockquote>m = target sum , n = array length</blockquote><blockquote>Time: O(n^m)</blockquote><blockquote>Space: O(m)</blockquote><pre>def can_sum(target_sum, values, memo={}):<br>    &quot;&quot;&quot;</pre><pre>:param target_sum:<br>    :param values: list<br>    :return:<br>    &quot;&quot;&quot;<br>    # print(memo, values)<br>    if target_sum in memo:<br>        return memo[target_sum]</pre><pre>    if target_sum == 0:<br>        return True<br>    if target_sum &lt; 0:<br>        return False</pre><pre>    <strong>for value in values:</strong><br>        <strong>reminder = target_sum - value</strong><br>        <strong>if can_sum(reminder, values, memo):<br>            memo[target_sum] = True<br>            return True</strong></pre><pre>     # If no way leads to zero then there is no possible values leads to sum<br>    memo[target_sum] = False<br>    return False</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/929/1*jkS5H24fSUQd1d35uZFvLw.png" /></figure><blockquote>m = target sum , n = array length</blockquote><blockquote>Time: O(n*m) branch for every choice of number in the list of length n</blockquote><blockquote>Space: O(m)</blockquote><h4>Can Construct</h4><p>Write function can_construct(target, word_bank) that accepts a target string and an array of strings</p><p>The function should return a boolean indicating whether or not the target can be constructed by concatenating elements of the word_bank array.</p><p>You may reuse elements of word_bank as many times as needed</p><blockquote>m = target word length , n = word bank length</blockquote><blockquote>Time: O(n^m * m) <em>n</em> branches for each element in the list and <em>m</em> subtract operation in each branch and <em>m</em> operation to slice array</blockquote><blockquote>Space: O(m*m) m for stack depth and m for to store sliced array</blockquote><figure><img alt="" src="https://cdn-images-1.medium.com/max/898/1*JUjpiAK4HhFYWzNaZjq6bA.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/867/1*ddxKfliL_f0kPqtBmsZ-Ew.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/820/1*s2DN2JE4dfjVzies6DN0GQ.png" /></figure><pre>def can_construct(target, word_bank, memo={}):</pre><pre>   if target in memo:<br>        return memo[target]</pre><pre>   if target == &#39;&#39;:<br>        return True # When we found a way reconstruct the string</pre><pre>   # Return true if any of branch returns true<br>   <strong> for word in word_bank:<br>        # check for matching prefix<br>        if target.startswith(word):<br>            suffix = target[len(word):]<br>            if can_construct(suffix, word_bank):<br>                memo[suffix] = True<br>                return True</strong></pre><pre>    # If no branch leads to positive case, return false<br>    memo[target] = False<br>    return False</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*P_6pJhmNPf1XpiZ5WvbPcg.png" /></figure><blockquote>m = target word length , n = word bank length</blockquote><blockquote>Time: O(n * m * m) <em>n</em> branches for each element in the list and <em>m</em> subtract operation in each branch and <em>m</em> operation to slice array</blockquote><blockquote>Space: O(m*m) <em>m</em> for stack depth and <em>m</em> for to store sliced array</blockquote><p><strong>3 Combination Problem</strong></p><p>Find possible combination for given target and list of possible values.</p><h4>How Sum</h4><p>Write a function how_much(target_sum, numbers), that takes in a target_sum and an array of numbers as arguments.</p><p>The function should return an array containing any combination of elements that add up to exactly the target sum. If there is no combination that adds up to the target sum, return null.</p><p>If there are multiple combinations possible, you may return any single one.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/747/1*hzeQFLqK0n-_S8F-bOjCeg.png" /></figure><blockquote>m = target sum , n = array length</blockquote><blockquote>Time: O(n^m * m) <em>n</em> branches for each element in the list and <em>m</em> subtract operation in each branch and <em>m</em> operation to copy the array into result array</blockquote><blockquote>Space: O(m)</blockquote><pre><br>def how_sum(target_sum, values, memo={}):<br>    if target_sum in memo:<br>        return memo[target_sum]</pre><pre>    if target_sum == 0:<br>        return []<br>    if target_sum &lt; 0:<br>        return None</pre><pre>  <strong>  for value in values:<br>        ret = how_sum(target_sum - value, values, memo)<br>        if ret is not None:<br>            ret.append(value)<br>            memo[target_sum] = ret<br>            return ret</strong></pre><pre>    memo[target_sum] = None<br>    return None</pre><blockquote>m = target sum , n = array length</blockquote><blockquote>Time: O(n*m²)</blockquote><blockquote>Space: O(m²)</blockquote><h4>Count Construct</h4><p>Write a function count_construct(target, word_bank) taht accepts a target string and an array of strings.</p><p>The function should return the number of ways that the target can be constructed by concatenating elements of the word_bank array.</p><p>You may reuse elements of word_bank as many time as needed.</p><blockquote>m = target word length , n = word bank length</blockquote><blockquote>Time: O(n^m * m) <em>n</em> branches for each element in the list and <em>m</em> subtract operation in each branch and <em>m</em> operation to slice array</blockquote><blockquote>Space: O(m*m) <em>m</em> for stack depth and <em>m</em> for to store sliced array</blockquote><figure><img alt="" src="https://cdn-images-1.medium.com/max/810/1*63CEHHoSvNwS4lk7D0PmGQ.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/809/1*zGqtNsu3FyAYNXWZ10Dq4w.png" /></figure><pre>def count_contruct(target, word_bank, memo={}):</pre><pre>  if target in memo:<br>    return memo[target]</pre><pre>  if len(target) == 0 :<br>    return True</pre><pre>  total_ways = 0<br>  for word in word_bank:<br>    if target.startswith(word):<br>      suffix = target[len(word):]<br>      total_ways += count_contruct(suffix, word_bank)<br>      <br>  memo[target] = total_ways<br>  return total_ways</pre><blockquote>m = target word length , n = word bank length</blockquote><blockquote>Time: O(n * m * m) <em>n</em> branches for each element in the list and <em>m</em> subtract operation in each branch and <em>m</em> operation to slice array</blockquote><blockquote>Space: O(m*m) <em>m</em> for stack depth and <em>m</em> for to store sliced array</blockquote><p>4 Optimisation Problem</p><p>Find all possible combination for given target and list of possible values and return the best one</p><h4>Best Sum</h4><p>Write a function best_sum(target_sum, numbers) that takes in a target sum and an array of numbers as arguments.</p><p>The function should return an array containing the shortest combination of numbers that add up to exactly the target sum.</p><p>If there is a tie for the shortest combination, you may return any on of the shortest</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/481/1*1r_LQzfWnr6IaGLhCumOJw.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/901/1*OoiuU2qWVhOr2xGDQmpE4g.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/885/1*-MnKC79yzmf4qKtnywcTXw.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/896/1*1E8Ts7CrxsWEGJeEH19n6g.png" /></figure><blockquote>m = target sum , n = array length</blockquote><blockquote>Time: O(n^m * m) <em>n</em> branches for each element in the list and <em>m</em> subtract operation in each branch and <em>m</em> operation to copy the array into result array</blockquote><blockquote>Space: O(m²)</blockquote><pre>def best_sum(target_sum, values, memo):</pre><pre>  if target_sum in memo:<br>        return memo[target_sum]</pre><pre>  if target_sum == 0:<br>        return []</pre><pre>  if target_sum &lt; 0:<br>        return None</pre><pre>  best_pair = None</pre><pre>  <strong>for value in values:<br>        ret = best_sum(target_sum=target_sum - value, values=values, memo=memo)<br>        if ret is not None:<br>            res = ret + [value]<br>            if best_pair is None or len(res) &lt; len(best_pair):<br>                best_pair = res<br>                memo[target_sum] = best_pair<br>                # return best_pair # Dont return the value until all possible combinations are evaluated for child tree</strong></pre><pre>  memo[target_sum] = best_pair<br>    return memo[target_sum]</pre><blockquote>m = target sum , n = array length</blockquote><blockquote>Time: O(n*m²)</blockquote><blockquote>Space: O(m²)</blockquote><h4>All Construct</h4><p>Write a function all_construct(target, word_bank) that accepts a target string and an array of strings</p><p>The function should return a 2D array containing all of the ways that the target can be constructed by concatenating elements of the word bank array. Each element of the 2D array should represent one combination that constructs the target.</p><p>You may reuse elements of the word bank as many times as needed.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/805/1*9IXS5Tu1oqOdCq0c5BUvkg.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/784/1*CT-p8nAgPr9wLafMEonbOg.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/829/1*CHrow3J14pbnAKlqBj5oPA.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/741/1*dNCO4HxUb4cwOpo8sT_hFw.png" /></figure><pre>def all_construct(target, word_bank, memo={}):</pre><pre>    if target in memo:<br>        return memo[target]</pre><pre>    if target in memo:<br>        return memo[target]</pre><pre>    if target == &#39;&#39;:<br>        return [[]]</pre><pre>    res = []<br><strong>    for word in word_bank:<br>        if target.startswith(word):<br>            suffix = target[len(word):]<br>            suffix_ways = all_construct(suffix, word_bank)<br>            target_ways = list(map(lambda l: l + [word], suffix_ways))<br>            res.extend(target_ways)</strong></pre><pre>     memo[target] = res</pre><pre>     return res</pre><blockquote>m = target word length , n = number of words in word bank length</blockquote><blockquote>Time: O(n^m ) If complexity is exponential i.e m^n then we can ignore additional complexity like slicing the array</blockquote><blockquote>Space: O(m)</blockquote><figure><img alt="" src="https://cdn-images-1.medium.com/max/917/1*C7nHE7BrFf6vWSQh0XixAA.png" /></figure><p>Test your understanding….</p><p><a href="https://medium.com/techie-delight/top-50-dynamic-programming-practice-problems-4208fed71aa3">Top 50 Dynamic Programming Practice Problems</a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=72ff0370bb27" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[What if I wanted to submit remote PySpark jobs to AWS EMR without worrying about library dependen…]]></title>
            <link>https://mageswaran1989.medium.com/what-if-i-wanted-to-submit-remote-pyspark-jobs-to-aws-emr-without-worrying-about-library-dependen-8f282adfe7be?source=rss-f442f089254f------2</link>
            <guid isPermaLink="false">https://medium.com/p/8f282adfe7be</guid>
            <category><![CDATA[yarn]]></category>
            <category><![CDATA[pyspark]]></category>
            <category><![CDATA[aws-emr]]></category>
            <category><![CDATA[apache-livy]]></category>
            <category><![CDATA[docker]]></category>
            <dc:creator><![CDATA[Mageswaran D]]></dc:creator>
            <pubDate>Wed, 08 Dec 2021 11:49:06 GMT</pubDate>
            <atom:updated>2021-12-08T11:49:06.197Z</atom:updated>
            <content:encoded><![CDATA[<h3>What if I wanted to submit remote PySpark jobs to AWS EMR without worrying about library dependency versions ?</h3><p>Typically Spark cluster is used to run ETL jobs or some streaming jobs, where everything is managed by developers for developers, where end user only access the processed data.</p><p>I worked for nearly two years in such setup, and I can replicate the same with ease.</p><p>How many times we happen to get what we are experienced? I am not exception here!</p><p>My new task is that I wanted to prepare the Spark cluster to run end user jobs thats gets triggered from Web UI. Yes you read it correct, not the typical spark-submit or Airflow operators. I have to come up with a plan to run Spark job from Web UI and read the results when it is ready and show them in UI.</p><p>Basically opening up the entire cluster to end users to run the jobs against available dataset with some custom configurations from a click of a button, fancy way to start Spark jobs isn’t ? :)</p><p>Two things to take care:</p><ol><li>Remote job submission</li><li>Manage Python library dependencies</li></ol><p><strong>Obviously the possible solution is to submit Spark jobs over REST API endpoints. It can be</strong><a href="https://hadoop.apache.org/docs/stable/hadoop-yarn/hadoop-yarn-site/ResourceManagerRest.html"><strong> YARN REST end point </strong></a><strong>or the famous </strong><a href="https://livy.incubator.apache.org/"><strong>Apache Livy</strong></a></p><p>So the next thing to handle is <strong>application’s 3rd party library dependencies</strong>. In a typical ETL pipeline all the requirements are captured in requirements file and it gets installed during bootstrap action on EMR nodes.</p><p>Usually for each job category I end up in creating a cluster or re-create a new cluster for every other updates that the code under goes and it’s not a big deal as I can spin up new cluster for each ETL batch processing.</p><p>Now it becomes problematic when I wanted to use the same cluster to run different jobs which are having different dependencies and versions. It’s a night mare to manage different clusters for different projects.</p><p>So whats the alternative? <strong>Docker containers. </strong>Pack what I need in an image and through it to runtime.</p><p>Ok, how to run Spark on Docker containers? Do I need to adopt Kubernetes instead of YARN? Luckily with EMR I can live with YARN Docker containers, before the real need comes for Kubernetes.</p><h3>YARN Docker Container</h3><p>Welcome to YARN Docker support.</p><p><strong>In one sentence: </strong>Instead of JVM application containers, now use Docker containers.</p><p>Must read blog from AWS on setting up YARN Docker Containers:</p><p><a href="https://docs.aws.amazon.com/emr/latest/ReleaseGuide/emr-spark-docker.html">Run Spark applications with Docker using Amazon EMR 6.x</a></p><p>I would say adding Docker support to YARN is a blessing for this kind of scenario as it made my life easy to architect whole pipeline.</p><p>AWS has a base image called <strong>amazoncorretto:8 </strong>which comes with preloaded Java which can be used to run Spark jobs.</p><p>Example Docker File:</p><pre><strong>FROM amazoncorretto:8</strong></pre><pre>RUN yum -y update<br>RUN yum -y install yum-utils<br>RUN yum -y groupinstall development</pre><pre>RUN yum list python3*<br>RUN yum -y install python3 python3-dev python3-pip python3-virtualenv</pre><pre>RUN python -V<br>RUN python3 -V</pre><pre>ENV PYSPARK_DRIVER_PYTHON python3<br>ENV PYSPARK_PYTHON python3</pre><pre>RUN pip3 install --upgrade pip<br>RUN pip3 install numpy pandas</pre><pre>RUN python3 -c &quot;import numpy as np&quot;</pre><p>Ok now I can use Dockerfile to package what I need into the image. Next, where to push/version the image ? Dockerhub!</p><p>Yes Dockerhub is good if I use open source tools, but what if I have to use proprietary code? Use AWS EKS?</p><p>Nah…my client uses Dockerhub with private repos. Okay… next hurdle how to pass the credentials to YARN ? without which Docker image pull fails.</p><p>Create a config.json, something like this, better to involve DevOps on this to get auth:</p><pre>cat ~/.docker/config.json <br>{<br> &quot;auths&quot;: {<br>  &quot;<a href="https://index.docker.io/v1/">https://index.docker.io/v1/</a>&quot;: {<br>   &quot;auth&quot;: &quot;123456789abcdefghijklm==&quot;<br>  }<br> }<br>}</pre><p>Now copy the config file to HDFS and point the path like below:</p><pre>--conf <strong>spark.executorEnv.YARN_CONTAINER_RUNTIME_DOCKER_CLIENT_CONFIG</strong>=&quot;hdfs:///path/to/hadoop/config.json&quot;<br>--conf <strong>spark.yarn.appMasterEnv.YARN_CONTAINER_RUNTIME_DOCKER_CLIENT_CONFIG</strong>=&quot;hdfs:///path/to/hadoop/config.json&quot;</pre><p>How to make Spark to use Docker container? Use following YARN Docker Spark configuration:</p><pre>--conf <strong>spark.executorEnv.PYSPARK_PYTHON</strong>=python3<br>--conf <strong>spark.executorEnv.PYSPARK_DRIVER_PYTHON</strong>=python3<br>--conf <strong>spark.executorEnv.YARN_CONTAINER_RUNTIME_TYPE</strong>=docker <br>--conf <strong>spark.executorEnv.YARN_CONTAINER_RUNTIME_DOCKER_IMAGE</strong>=$DOCKER_IMAGE_NAME<br>--conf <strong>spark.yarn.appMasterEnv.YARN_CONTAINER_RUNTIME_TYPE</strong>=docker<br>--conf <strong>spark.yarn.appMasterEnv.YARN_CONTAINER_RUNTIME_DOCKER_IMAGE</strong>=$DOCKER_IMAGE_NAME<br>--conf <strong>spark.yarn.appMasterEnv.PYSPARK_DRIVER_PYTHON</strong>=python3<br>--conf <strong>spark.yarn.appMasterEnv.PYSPARK_PYTHON</strong>=python3</pre><h3>Livy</h3><p>Livy enables programmatic, fault-tolerant, multi-tenant submission of Spark jobs from web/mobile apps (no Spark client needed). So, multiple users can interact with your Spark cluster concurrently and reliably.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/800/0*--NyATvi-8FXCDT5.png" /></figure><p>Submitting to Livy URL hosted in EMR needs the URL first to be exposed to outside world or VPN. Let’s leave that to DevOps team to take care for us!</p><p>Check here for Livy port details on EMR:</p><p><a href="https://docs.aws.amazon.com/emr/latest/ManagementGuide/emr-web-interfaces.html">View web interfaces hosted on Amazon EMR clusters</a></p><p>Tunnel the Livy port to local machine as follows, for REST endpoint. In production the URL and port needs to be mapped to a VPN.</p><pre>export USER=hadoop<br>export EMR_SPARK_MASTER=ip-99-33-222-111.us-west-2.compute.internal</pre><pre>sudo ssh -i ~/emrkey.pem -N -L 8998:$EMR_SPARK_MASTER:8998 hadoop@$EMR_SPARK_MASTER</pre><p>Check Livy Docs @</p><p><a href="https://livy.incubator.apache.org/docs/latest/rest-api.html">Livy Docs - REST API</a></p><p>Test the connection with curl :</p><pre>export LIVY_URL=localhost:8998<br>curl $LIVY_URL/sessions/ | python -m json.tool</pre><p>Move <a href="https://github.com/apache/spark/blob/master/examples/src/main/python/pi.py">pi.py</a> example to S3 to test run it with Livy:</p><pre>curl \<br>    -X POST \<br>    --data &#39;{<br>            &quot;name&quot;: &quot;UDCTestLivy1&quot;, <br>            &quot;file&quot;: &quot;s3://users-dev-bucket/mageswaran/livy/pi.py&quot;, <br>            &quot;driverMemory&quot;: &quot;4G&quot;, <br>            &quot;driverCores&quot;: 4, <br>            &quot;executorMemory&quot;: &quot;32G&quot;, <br>            &quot;executorCores&quot;: 5, <br>            &quot;numExecutors&quot;: 2, <br>            &quot;conf&quot;: {&quot;spark.scheduler.mode&quot;: &quot;FAIR&quot;}<br>            }&#39;\<br>    -H &quot;Content-Type: application/json&quot; \<br>    $LIVY_URL/batches</pre><p>Check YARN UI for logs and if everything seems working we are set to use Python!</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/e2553a254a96803aaa7db6fbb266eb4b/href">https://medium.com/media/e2553a254a96803aaa7db6fbb266eb4b/href</a></iframe><pre>python3 livy_Spark.py \<br>--url http://$LIVY_URL \<br>--name SparkDockerLivyTest \<br>--file s3://bucket/mageswaran/livy/main.py \<br>--pyFiles s3://bucket/mageswaran/livy/project.zip \<br>--args &#39;--config_file s3://bucket/mageswarand/livy/job1_config.json&#39;</pre><p>Note: project.zip contains the zipped version of your source packages thats needs to be distributed to the cluster.</p><p>Next step is to integrate this into backend services and re-route some of the jobs to Spark cluster!</p><p><strong>References:</strong></p><ul><li><a href="https://docs.aws.amazon.com/emr/latest/ReleaseGuide/emr-spark-docker.html">Run Spark applications with Docker using Amazon EMR 6.x</a></li><li><a href="https://dzone.com/articles/execute-spark-applications-with-apache-livy">Execute Spark Applications With Apache Livy - DZone Big Data</a></li><li><a href="https://www.statworx.com/at/blog/access-your-spark-cluster-from-everywhere-with-apache-livy/">Access your Spark Cluster from Everywhere with Apache Livy</a></li><li><a href="https://livy.incubator.apache.org/docs/latest/rest-api.html">Livy Docs - REST API</a></li></ul><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=8f282adfe7be" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[NLP: What it takes to design a full stack DeepLearning based Receipts form filling system using NER?]]></title>
            <link>https://mageswaran1989.medium.com/nlp-what-it-takes-to-design-a-full-stack-deeplearning-based-receipts-form-filling-system-using-ner-5dbab09a4c6d?source=rss-f442f089254f------2</link>
            <guid isPermaLink="false">https://medium.com/p/5dbab09a4c6d</guid>
            <category><![CDATA[vue]]></category>
            <category><![CDATA[fastapi]]></category>
            <category><![CDATA[full-stack]]></category>
            <category><![CDATA[nlp]]></category>
            <category><![CDATA[transformers]]></category>
            <dc:creator><![CDATA[Mageswaran D]]></dc:creator>
            <pubDate>Mon, 29 Nov 2021 02:26:13 GMT</pubDate>
            <atom:updated>2021-11-30T12:46:29.077Z</atom:updated>
            <content:encoded><![CDATA[<p><a href="https://gitlab.com/gyan42/receipts-form-filling">gyan42 / receipts-form-filling</a></p><h4><a href="https://colab.research.google.com/gist/Mageswaran1989/442d575d8f5ca11b7ae12b1b061a04d3/receiptsautoformfilling.ipynb">Online Colab Notebook </a>for model training.</h4><p><a href="https://gitlab.com/gyan42/receipts-form-filling/-/tree/main#docker-compose"><strong>Use docker compose to launch the demo.</strong></a></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*lFVnkaBipH0o_zp2jXS34w.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*mqacAhbyyRuG5G3U9bS_BA.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/621/1*MQICswwfiq7Cf13aXwliVg.png" /></figure><p>If we are lucky getting a dataset and building a model and evaluating it in development environment is pretty easy with tools we have to throw at the problem. The real pain comes when we wanted to move the model to production.</p><p>Three years back for a client we wanted to do a POC to retrieve13 tags from client documents with a limited training set around 2.5K documents.</p><p>Initially CRF models were used which where not giving satisfactory results, so we moved towards Deep Learning models using Bidirectional LSTM networks. By using both character and word level embeddings we were able to get some amazing results, where we were able to retrieve 8 high occurring tags with F1 score around 70 to 90 and the rest of the tags were not up to the mark because of their low frequency and individualities in them.</p><p>In the age of Transformers, I was thinking why not revisit the topic and see how it performs with receipts datasets which is highly noisy and very hard problem to tackle as the images can have random noise in it along with different camera angles to it.</p><p>Here I have outlined the components that are brought together to build a NER system, which uses scalable tools and frameworks but focused to run on developer machine to get a hands on experience.</p><p>It’s not a complete system however it gives a glimpse of designing and building a Deep Learning systems for production.</p><p>There are lot of external resources, its up the reader to decide how much to dive in them or skip them if they are familiar :)</p><h3>1. Dataset</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/463/0*I2DWmANYcDJxJmrv.jpeg" /></figure><pre>TAN O<br>WOON O<br>YANN O<br><strong>BOOK company<br>TA company<br>.K(TAMAN company<br>DAYA) company<br>SDN company<br>BND company</strong><br>789417-W O<br><strong>NO.53 address<br>55 address<br>57 address<br>&amp; address<br>59 address<br> address<br>JALAN address<br>SAGU address<br>18 address<br>TAMAN address<br>DAYA address<br>81100 address<br>JOHOR address<br>BAHRU address<br>JOHOR. address</strong><br>DOCUMENT O<br>NO O<br>: O<br>TD01167104 O<br>DATE: O<br><strong>25/12/2018 date<br>8:13:39 date</strong><br>PM date<br>CASHIER: O<br>MANIS O<br>MEMBER: O<br>CASH O<br>BILL O<br>CODE/DESC O<br>PRICE O<br>DISC O<br>AMOUNT O<br>QTY O<br>RM O<br>9556939040116 O<br>KF O<br>MODELLING O<br>CLAY O<br>KIDDY O<br>FISH O<br>1 O<br>PC O<br>* O<br><strong>9.000 total</strong><br>0.00 O<br><strong>9.00 total</strong><br>TOTAL: O<br>ROUR O<br>DING O<br>ADJUSTMENT: O<br>ROUND O<br>D O<br>TOTAL O<br>(RM): O<br>CASH O<br>10.00 O<br>CHANGE O<br>1.00 O<br>GOODS O<br>SOLD O<br>ARE O<br>NOT O<br>RETURNABLE O<br>OR O<br>EXCHANGEABLE O<br>*** O<br>THANK O<br>YOU O<br>PLEASE O<br>COME O<br>AGAIN O<br>! O</pre><p>The dataset we are going to use today is <a href="https://rrc.cvc.uab.es/?ch=13">ICDAR 2019 Robust Reading Challenge on Scanned Receipts OCR and Information Extraction</a></p><p>Preparing dataset is paramount work of itself, which I covered in my previous blog @ <a href="https://mageswaran1989.medium.com/how-to-build-custom-ner-huggingface-dataset-for-receipts-and-train-with-huggingface-transformers-6c954b84473c">https://mageswaran1989.medium.com/how-to-build-custom-ner-huggingface-dataset-for-receipts-and-train-with-huggingface-transformers-6c954b84473c</a></p><p>It is highly recommended to read it.</p><ul><li><a href="https://gitlab.com/gyan42/receipts-form-filling/-/blob/main/ner-model/src/gyan42/dataset/sroie2019_dataset.py">HFSREIO2019Dataset</a> class is a custom HuggingFace dataset wrapper, which downloads the SREIO2019 data from my Github repo and prepares list of (token, tag) pairs.</li><li>The CoNLL formated dataset is then loaded and batches are created with <a href="https://huggingface.co/transformers/main_classes/data_collator.html#datacollatorfortokenclassification">DataCollatorForTokenClassification</a> after preprocessing with <a href="https://gitlab.com/gyan42/receipts-form-filling/-/blob/main/ner-model/src/gyan42/dataset/hf_tokenize.py">HFTokenizer</a></li></ul><h3>2. DeepLearning Model with HuggingFace Transformers</h3><p><a href="https://huggingface.co/distilbert-base-uncased">distilbert-base-uncased</a> is used as the Transformer model loaded with <a href="https://huggingface.co/transformers/model_doc/auto.html#automodelfortokenclassification">AutoModelForTokenClassification</a> class of HuggingFace.</p><p>Training the model follows the usual routine, so I am gonna skip it.</p><p>There is a <strong>online Colab version</strong> which can be used to build the model @ <a href="https://colab.research.google.com/gist/Mageswaran1989/442d575d8f5ca11b7ae12b1b061a04d3/receiptsautoformfilling.ipynb">https://colab.research.google.com/gist/Mageswaran1989/442d575d8f5ca11b7ae12b1b061a04d3/receiptsautoformfilling.ipynb</a></p><h3>3. OCR</h3><p>Google Tesseract with PyTesseract package is used to convert the image into text for its simplicity and ease of use.</p><p>Tesseract is an easy option for most of the OCR task, though there are other DL options, such as following ones which can be trained for specific datasets.</p><ul><li><a href="https://github.com/clovaai/CRAFT-pytorch">https://github.com/clovaai/CRAFT-pytorch</a></li><li><a href="https://github.com/Calamari-OCR/calamari">https://github.com/Calamari-OCR/calamari</a></li></ul><p><a href="https://tesseract-ocr.github.io/tessdoc/tess4/TrainingTesseract-4.00.html">Remember Tesseract also comes with LSTM module which can also be trained on custom datasets.</a></p><pre>import pytesseract<br>import cv2</pre><pre>custom_config = r&#39;--oem 3 --psm 6&#39;<br>img = cv2.imread(file_path)<br>text = pytesseract.image_to_string(img, lang=&#39;eng&#39;, config=custom_config)</pre><ul><li>The --oem argument, or OCR Engine Mode, controls the type of algorithm used by Tesseract.</li></ul><pre>tesseract --help-oem # for oem. </pre><pre>OCR Engine modes:<br>  0    Legacy engine only.<br>  1    Neural nets LSTM engine only.<br>  2    Legacy + LSTM engines.<br>  3    Default, based on what is available.</pre><ul><li>The --psm controls the automatic Page Segmentation Mode used by Tesseract.</li></ul><pre>tesseract --help-psm # for psm.</pre><pre>Page segmentation modes:<br>  0    Orientation and script detection (OSD) only.<br>  1    Automatic page segmentation with OSD.<br>  2    Automatic page segmentation, but no OSD, or OCR. (not implemented)<br>  3    Fully automatic page segmentation, but no OSD. (Default)<br>  4    Assume a single column of text of variable sizes.<br>  5    Assume a single uniform block of vertically aligned text.<br>  6    Assume a single uniform block of text.<br>  7    Treat the image as a single text line.<br>  8    Treat the image as a single word.<br>  9    Treat the image as a single word in a circle.<br> 10    Treat the image as a single character.<br> 11    Sparse text. Find as much text as possible in no particular order.<br> 12    Sparse text with OSD.<br> 13    Raw line. Treat the image as a single text line,<br>       bypassing hacks that are Tesseract-specific.</pre><h3>4.FastAPI Backend</h3><p>When user uploads the image it needs to be converted to text right? So the image is opened in the browser and the data is send to a python backend process to OCR it.</p><p><a href="https://fastapi.tiangolo.com/">FastAPI - FastAPI</a></p><p>FastAPI is used as the REST API backend for its <a href="https://fastapi.tiangolo.com/tutorial/first-steps/">type safety features and ease of testing the endpoints with in-build features.</a></p><ul><li>Setting up CORS is important to allow local testing, <a href="https://developer.mozilla.org/en-US/docs/Web/HTTP/CORS">however this needs to be changed to meet production requirements.</a></li></ul><pre>ALLOWED_ORIGINS = [&quot;*&quot;]</pre><pre>app.add_middleware(<br>    CORSMiddleware,<br>    allow_origins=ALLOWED_ORIGINS,<br>    allow_credentials=True,<br>    allow_methods=[&quot;*&quot;],<br>    allow_headers=[&quot;*&quot;],<br>)</pre><ul><li><a href="https://pydantic-docs.helpmanual.io/">pydantic</a> is used to <a href="https://gitlab.com/gyan42/receipts-form-filling/-/blob/main/api/internal/json_models.py">model the JSON requests</a>, and this is where FastAPI shines over Flask.</li><li>Keeping all endpoints in one file is not a way to handle big projects, then whats the better way? <a href="https://fastapi.tiangolo.com/tutorial/bigger-applications/">FastAPI router</a></li><li><a href="https://fastapi.tiangolo.com/tutorial/bigger-applications/">FastAPI router</a> is used to keep the endpoints files as a separate module. This design greatly helps in maintaining and adding new features without altering the main file after initial setup. Code @ <a href="https://gitlab.com/gyan42/receipts-form-filling/-/blob/main/api/routers/ocr/tesseract/tesseract.py">https://gitlab.com/gyan42/receipts-form-filling/-/blob/main/api/routers/ocr/tesseract/tesseract.py</a></li></ul><h3>5. TorchServe</h3><p>Ok, we have trained and got a model to use? How to use the model to predict in real world? How to put it behind a REST API endpoint ? How to scale the model ? How to update the model after deploying?</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*-QvClVBikyojVh5m.jpg" /></figure><p><a href="https://pytorch.org/serve/">TorchServe - PyTorch/Serve master documentation</a></p><p>I will leave you to explore TorchServe and its benefits ;)</p><p>To serve a NLP model all we need is the <strong>model weights, vocabulary used </strong>and<strong> the tokenizer, </strong>luckily HuggingFace model saves all these for us:</p><pre>! ls ~/.gyan42/models/hf/sroie2019v1/</pre><pre>config.json    special_tokens_map.json  training_args.bin pytorch_model.bin  tokenizer_config.json    vocab.txt</pre><p>Torch serve has a tool called <a href="https://github.com/pytorch/serve/blob/master/model-archiver/README.md"><strong>torch model archiver</strong></a><strong> </strong>which packages the model as self contained archive which then can be invoked independently.</p><pre>%set_env SERIALIZED_MODEL_FILE=/root/.gyan42/models/hf/sroie2019v1/pytorch_model.bin</pre><pre>!torch-model-archiver --force \<br>--model-name sroie2019v1 \<br>--version 1.0 \<br>--serialized-file  $SERIALIZED_MODEL_FILE \<br>--handler gyan42/serving/handler/<strong>hf_transformer_handler.py</strong> \<br>--extra-files /root/.gyan42/models/hf/sroie2019v1/config.json,/root/.gyan42/models/hf/sroie2019v1/special_tokens_map.json,/root/.gyan42/models/hf/sroie2019v1/training_args.bin,/root/.gyan42/models/hf/sroie2019v1/tokenizer_config.json,/root/.gyan42/models/hf/sroie2019v1/vocab.txt \<br>--export-path /root/.gyan42/model-store/</pre><p>Wait what is <a href="https://gitlab.com/gyan42/receipts-form-filling/-/blob/main/ner-model/src/gyan42/serving/handler/hf_transformer_handler.py"><strong>hf_transformer_handler.py</strong></a><strong>?</strong></p><p>The best part I like about Torch serve is its ease of packaging model unlike Tensorflow (no comments!)</p><p>So what needs to be done typically to handle an incoming HTTP request ?</p><ul><li>Load the model part of initialization</li><li>Extract the text from incoming request</li><li>Preprocess the text</li><li>Run inference through the model</li><li>Post process the results</li><li>Send back the reply</li></ul><p>This is what done in <a href="https://gitlab.com/gyan42/receipts-form-filling/-/blob/main/ner-model/src/gyan42/serving/handler/hf_transformer_handler.py"><strong>hf_transformer_handler.py</strong></a></p><p>When torch serve serves the model, it loads the handler which has the knowledge of loading the artifacts for our model and how to process the incoming request. Neat and simple isn’t?</p><p>torch-model-archiver creates a file with extension .mar which stand for Model Archive File, which then can be served with :</p><pre>torchserve --start --model-store data/model-store --models all --ts-config configs/torch_serve_config.properties --foreground</pre><p>Above command loads all the model under the directory data/model-store</p><p>Torch Serve can be configured for three URLs, namely:</p><pre>inference_address = <a href="http://0.0.0.0:6543/">http://0.0.0.0:6543</a><br>management_address = <a href="http://0.0.0.0:6544/">http://0.0.0.0:6544</a><br>metrics_address = <a href="http://0.0.0.0:6545/">http://0.0.0.0:6545</a></pre><p>Model predictions are done at <strong>http://0.0.0.0:6543/predictions/{model_name}</strong></p><p>List of models can be found with <a href="http://0.0.0.0:6544/models"><strong>http://0.0.0.0:6544/models</strong></a></p><p>Management URL has features to load, scale and delete the model over the REST endpoints, comes in handy when we wanted to manage models remotely.</p><p>And don’t forget to <a href="https://gitlab.com/gyan42/receipts-form-filling/-/blob/main/configs/torch_serve_config.properties">configure CORS for serving REST API endpoints</a></p><pre># cors_allowed_origin is required to enable CORS, use &#39;*&#39; or your domain name<br>cors_allowed_origin=*<br># required if you want to use preflight request<br>cors_allowed_methods=GET,POST,PUT,OPTIONS<br># required if the request has an Access-Control-Request-Headers header<br>cors_allowed_headers=X-Custom-Header,content-type</pre><p>Part of our demo we are loading the model statically, however more refined way is to load the model over the management URL. By storing the model in S3 kind of storage, it can be loaded dynamically.</p><pre>curl -X POST  &quot;http://localhost:6544/models?initial_workers=1&amp;synchronous=true&amp;url=s3://some_bucket/gyna42/model-store/sroie2019v1.mar&quot;</pre><p><a href="https://github.com/pytorch/serve/blob/master/kubernetes/README.md">Torch Serve comes with support to work with Kubernetes.</a></p><h3>6. Vue3 WebUI</h3><p>Without an UI the whole model showcasing becomes command line oriented which is less intuitive.</p><p>Though Streamlit seems to be attractive option it lacks the power of customisation what we need sometimes.</p><p><a href="https://streamlit.io/">Streamlit * A faster way to build and share data apps</a></p><p>I chose to learn Vue compared to other Web framework, as usual for its simplicity and ease of use.</p><p><a href="https://v3.vuejs.org/">Vue.js</a></p><ul><li>Bulma CSS framework is used for HTML styling. Read about<a href="https://bulma.io/documentation/columns/basics/"> column styling here</a> which is used to split the web page into different column segments</li></ul><p><a href="https://bulma.io/">Bulma: Free, open source, and modern CSS framework based on Flexbox</a></p><ul><li>Axios is used for backend communications.</li></ul><p><a href="https://github.com/axios/axios">GitHub - axios/axios: Promise based HTTP client for the browser and node.js</a></p><ul><li>Nginx for reverse proxy and as web server.</li></ul><p><a href="https://www.nginx.com/">Advanced Load Balancer, Web Server, &amp; Reverse Proxy - NGINX</a></p><p>Environment files are used to load different configurations during build time. I have used following env files, namely linux, mac, maclocal and heroku to setup URLs accordingly.</p><pre>VUE_APP_API_BASE_URL=http://localhost:8088<br>VUE_APP_TORCH_PRED_BASE_URL=http://0.0.0.0:6543<br>VUE_APP_TORCH_MGMNT_BASE_URL=http://0.0.0.0:6544<br><br>#End points<br>VUE_APP_API_OCR_TESSERACT=/gyan42/ocr/engine/pytesseract/file</pre><p>Each Vue component consists of:</p><ul><li>HTML</li><li>CSS : Which plays major role in decorating the web pages. One good thing is there are well defined CSS framework like <a href="https://bulma.io/">https://bulma.io/</a> that gives all essential CSS classes for free</li><li>JavaScript with some predefined layout</li></ul><p>To understand the VueUI following Vue 3 concepts are necessary:</p><ul><li><a href="https://v3.vuejs.org/guide/data-methods.html">data</a></li><li><a href="https://v3.vuejs.org/guide/data-methods.html#data-properties">methods</a></li><li><a href="https://vuejs.org/v2/guide/class-and-style.html">v-bind</a></li><li><a href="https://vuejs.org/v2/guide/forms.html">v-model</a></li><li><a href="https://vuejs.org/v2/guide/list.html">v-for</a></li><li><a href="https://vuejs.org/v2/guide/conditional.html">v-if-else</a> / v-show</li><li><a href="https://vueschool.io/lessons/understanding-the-vuejs-lifecycle-hooks">Vue lifecycle hooks</a></li></ul><p>We use following HTML component:</p><ul><li><a href="https://vuejs.org/v2/guide/forms.html">Input</a></li><li><a href="https://www.w3schools.com/tags/tag_label.asp">Label</a></li><li><a href="https://vuejs.org/v2/guide/events.html">Button</a></li><li><a href="https://www.w3schools.com/tags/tag_img.asp">Img</a></li><li><a href="https://vuejs.org/v2/guide/list.html">List</a></li></ul><p>Once you get knowledge of all these components, understanding the Vue file becomes comfortable but it takes time and practice to digest HTML and JS interactions.</p><p><a href="https://gitlab.com/gyan42/receipts-form-filling/-/blob/main/ui/src/components/ReceiptsPage.vue">ui/src/components/ReceiptsPage.vue · main · gyan42 / receipts-form-filling</a></p><ul><li>Sample Button</li></ul><pre>&lt;button class=&quot;button is-link ml-5&quot; v-on:click=&quot;onImageSample&quot; &gt; Sample a Test Image &lt;/button&gt;</pre><ul><li>Once the sample button is clicked, it calls the onImageSample JS function, which samples the image from static path and converts the blob data into a file data which then passed to backend for OCR</li></ul><pre>onImageSample() {<br><br>      console.info(&quot;onImageSample&quot;)<br>      this.startTime = performance.now()<br>      this.timeElapsed = 0<br>      this.modelTimeElapsed = 0<br>      this.ocrTimeElapsed = 0<br><br>      this.predictions = []<br><br>      // Number of test iamges are 138!<br>      const rndInt = Math.floor(Math.random() * 138) + 1<br>      console.log(rndInt)<br><br>      const blobUrlToFile = (blobUrl)  =&gt; new Promise((resolve) =&gt; {<br>          fetch(blobUrl).then((res) =&gt; {<br>            res.blob().then((blob) =&gt; {<br>            const fileName = blobUrl.split(&quot;/&quot;)[2]<br>            console.info(fileName)<br>            const file = new File([blob], fileName, {type: blob.type})<br>            resolve(file)<br>            })<br>          })<br>       })<br><br>      blobUrlToFile(require(&quot;@/assets/images/test/&quot;+rndInt+&quot;.jpg&quot;)).then( f =&gt; {<br>          this.imageFileName = f<br>          console.info(this.imageFileName )<br>          this.createImage(this.imageFileName);<br>          this.onRunOCR()<br>        }<br>      )<br>    },<br>    createImage(file) {<br>      var reader = new FileReader();<br>      reader.onload = (e) =&gt; {<br>        this.imageFile = e.target.result;<br>      };<br>      reader.readAsDataURL(file);<br>    },<br>  }</pre><ul><li>Image holder : Once the file is available through the variable imageFile the image content is displayed on the UI</li></ul><pre>&lt;div class=&quot;box&quot; style=&quot;background-color:transparent;&quot;&gt;<br>&lt;p v-if=&quot;imageFile.length &gt; 0&quot;&gt; &lt;img v-bind:src=&quot;imageFile&quot; /&gt;&lt;/p&gt;<br>&lt;/div&gt;</pre><ul><li>The image file data is then given to OCR backend API, which returns the text data that gets stored in inTextData</li></ul><pre>async onRunOCR() {<br>      this.status = &quot;Running Tesseract&quot;<br>      this.ocrStartTime = performance.now()<br>      console.info(&quot;Running Tesseract&quot;)<br>      console.info(process.env.VUE_APP_API_BASE_URL,  process.env.VUE_APP_API_OCR_TESSERACT)<br>      let formData = new FormData();<br>      formData.append(&#39;file&#39;, this.imageFileName);<br>      console.info(this.imageFileName)<br>      console.info(formData)<br>      let headers =   {<br>        headers: {<br>          &#39;Content-Type&#39;: &#39;multipart/form-data&#39;<br>        },<br>        timeout: 30000<br>      }<br><br><strong>      api.fastapi<br>          .post(process.env.VUE_APP_API_OCR_TESSERACT, formData, headers)</strong><br>          .then(res =&gt; {<br>            console.info(res);<br>            this.inTextData = res[&quot;data&quot;][&#39;text&#39;]<br>            this.ocrTimeElapsed = performance.now() - this.ocrStartTime<br>            this.predict()<br>          })<br>          .catch((err) =&gt; alert(err));<br>    },</pre><ul><li>The text data is then passed to model prediction API, which returns list of tuples of (token, tag) pairs.</li></ul><pre>predict(){<br>      this.status = &quot;Running Transformer Model&quot;<br>      this.modelStartTime = performance.now()<br><strong>      api.torchserve.post(&quot;predictions/&quot; + this.selectedTorchModel, {&quot;text&quot;: this.inTextData}, {timeout: 20000})</strong><br>          .then(value =&gt; {<br>            console.info(value[&quot;data&quot;])<br>            this.extractTags(value[&quot;data&quot;])<br>          })<br>    },</pre><ul><li>extractTags function converts the list of tuples into list of map objects</li><li>Once the predictions are available in predictions data variable, iterate through it and create the label and input text boxes</li></ul><pre>&lt;li <strong>v-for=&quot;prediction in predictions&quot; :key=&quot;prediction.id&quot;</strong>&gt;<br>  &lt;div class=&quot;field is-horizontal&quot;&gt;<br>  &lt;div class=&quot;field-label is-normal&quot;&gt;<br>     <strong>&lt;label class=&quot;label&quot;&gt;{{prediction.tag}}&lt;/label&gt;</strong><br>  &lt;/div&gt;<br>  &lt;div class=&quot;field-body mr-5&quot;&gt;<br>    &lt;div class=&quot;field&quot;&gt;<br>      &lt;div class=&quot;control&quot;&gt;<br>        <strong>&lt;input class=&quot;input&quot; type=&quot;text&quot; v-model=prediction.token&gt;</strong><br>      &lt;/div&gt;<br>    &lt;/div&gt;<br>  &lt;/div&gt;<br>  &lt;/div&gt;<br>  &lt;br&gt;<br>&lt;/li&gt;</pre><ul><li>Predictions are list of map object as follows:</li></ul><pre>predictions: [ {id: 1, tag: &quot;TAG1&quot;, token: &quot;TOKEN1&quot;}, <br>               {id:2, tag: &quot;TAG2&quot;, token: &quot;TOKEN2&quot;}]</pre><h3>7. Docker Images</h3><p><a href="https://docs.docker.com/develop/develop-images/multistage-build/">Multi stage build is used</a></p><p>All docker files are found here:</p><p><a href="https://gitlab.com/gyan42/receipts-form-filling/-/tree/main/ops">ops · main · gyan42 / receipts-form-filling</a></p><ul><li>Multistage build id used to reduce the size of the images</li><li><a href="https://gitlab.com/gyan42/receipts-form-filling/-/blob/main/ui/nginx.conf">Nginx is configured inside the Docker as reverse proxy, which is a must for Vue UI to work out of Docker</a></li></ul><pre>COPY ui/nginx.conf /etc/nginx/nginx.conf</pre><ul><li>.env files are used with Vue builder for environment variables like URLs</li><li><a href="https://gunicorn.org/">https://gunicorn.org/</a> is used to launch the FastAPI backend as a daemon service</li><li>torchserve to launch Torch Serve REST API endpoints from a static model file</li></ul><blockquote><strong>With that comes to our demo time : </strong><a href="https://gitlab.com/gyan42/receipts-form-filling/-/tree/main#docker-compose"><strong>https://gitlab.com/gyan42/receipts-form-filling/-/tree/main#docker-compose</strong></a></blockquote><p>An attempt was made to run the app in Heroku platform, however as suspected the RAM needs are greater than 500MB as we put Tesseract and Transformer under one roof.</p><p>On our next POC lets build a model and serve them under 500MB, as we have PyTorch Mobile <a href="https://pytorch.org/mobile/home/">https://pytorch.org/mobile/home/</a> or <a href="https://pytorch.org/tutorials/recipes/recipes/dynamic_quantization.html">quantization</a></p><p>Thanks for reading and raise issues if something doesn’t work, happy to fix them any time!</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=5dbab09a4c6d" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[NLP: What it takes to model Google search suggestions or auto complete?]]></title>
            <link>https://mageswaran1989.medium.com/nlp-what-it-takes-to-model-google-search-suggestions-or-auto-complete-8b94d3671d79?source=rss-f442f089254f------2</link>
            <guid isPermaLink="false">https://medium.com/p/8b94d3671d79</guid>
            <category><![CDATA[heroku]]></category>
            <category><![CDATA[spark]]></category>
            <category><![CDATA[nlp]]></category>
            <category><![CDATA[vue-3]]></category>
            <category><![CDATA[fastapi]]></category>
            <dc:creator><![CDATA[Mageswaran D]]></dc:creator>
            <pubDate>Wed, 10 Nov 2021 08:10:03 GMT</pubDate>
            <atom:updated>2021-12-13T03:48:44.157Z</atom:updated>
            <content:encoded><![CDATA[<h3>NLP: How to predict next word in a search query? Full stack N-Gram Model withVue, FastAPI &amp; Heroku</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*oyokXXsKHjAG6RKv.png" /></figure><ol><li>Dataset : <a href="https://huggingface.co/datasets/google_wellformed_query">google_wellformed_query</a> 25K examples</li></ol><p>2. Model: Naive N-Gram probability model : Single machine version and PySpark version</p><p>3. Web UI : <a href="https://v3.vuejs.org/">https://v3.vuejs.org/</a></p><p>4. Backend API: <a href="https://fastapi.tiangolo.com/">https://fastapi.tiangolo.com/</a></p><p>5. Docker and Docker Compose</p><p>6. Deployment on Heroku</p><p><strong>Git: </strong><a href="https://github.com/gyan42/autocomplete-ngram-model">https://github.com/gyan42/autocomplete-ngram-model</a></p><p><strong>Colab Notebook for Model training:</strong> <a href="https://colab.research.google.com/gist/Mageswaran1989/e49e043f09c1de89c6f433967be118a2/autocorrectmodel.ipynb">https://colab.research.google.com/gist/Mageswaran1989/e49e043f09c1de89c6f433967be118a2/autocorrectmodel.ipynb</a></p><blockquote>Live Demo: <a href="https://gyan42-autocompleter.herokuapp.com/">https://gyan42-autocompleter.herokuapp.com/</a></blockquote><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*17VkLEePCjhr9ekoVQzQ7Q.png" /></figure><p>I hope I don’t have set any context here, as the above picture depicts what we wanna achieve.</p><pre>Who have won the _____</pre><p>To fill the above sentence we need to know the current search trend and whats happening in the locality of the search is happening. It can be anything from local election result to international game results.</p><p>Like any other ML model the result is subjective to the dataset we train them on. Luckily we have <a href="https://huggingface.co/datasets/google_wellformed_query">google_wellformed_query</a> which we are gonna use for our exploration. We are not going to use any fancy ML algorithm here, a simple bag of words probability model. We also see how to use Apache Spark to produce the model artefacts that is needed for our predictions for large datasets.</p><p>And finally we are going to see how to use large dataset and train using Spark and use the artefacts with our model.</p><h4>1. Dataset</h4><p>To keep data handling simple we are going to use Datasets package from <a href="https://huggingface.co/datasets">https://huggingface.co/datasets</a></p><p>Here is the online view linker for `google_wellformed_query` which can be used for exploring the dataset</p><p><a href="https://huggingface.co/datasets/viewer/?dataset=google_wellformed_query">Streamlit</a></p><pre>from datasets import load_dataset<br>dataset = load_dataset(&#39;google_wellformed_query&#39;)<br>dataset[&#39;train&#39;][&#39;content][:2]<br>&gt;&gt; The European Union includes how many ?<br>   What are Mia Hamms accomplishment ?</pre><h4>2. N-Gram Model</h4><iframe src="https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Fwww.youtube.com%2Fembed%2FhM49MPmakNI%3Flist%3DPLaZQkZp6WhWyvdiP49JG-rjyTPck_hvEu&amp;display_name=YouTube&amp;url=https%3A%2F%2Fwww.youtube.com%2Fwatch%3Fv%3DhM49MPmakNI&amp;image=https%3A%2F%2Fi.ytimg.com%2Fvi%2FhM49MPmakNI%2Fhqdefault.jpg&amp;key=a19fcc184b9711e1b4764040d3dc5c07&amp;type=text%2Fhtml&amp;schema=youtube" width="854" height="480" frameborder="0" scrolling="no"><a href="https://medium.com/media/dee5aa7df77429209ebe7faf85bf2b56/href">https://medium.com/media/dee5aa7df77429209ebe7faf85bf2b56/href</a></iframe><p>N-Gram models are Statistical(Probabilistic) Language models that aim to assign probabilities to a given sequence of words. Any N-gram is just a sequence of “n” words. For example, “Saurav” is a unigram and “Hi There” is a bigram.</p><blockquote>Sentences -&gt; Tokenisers (splitting sentences into words) -&gt; Grouping tokens into tuples</blockquote><p>So what it does basically? it estimates the probability of next word given previous set of words.</p><iframe src="https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Fwww.youtube.com%2Fembed%2FUyC0bBiZY-A%3Flist%3DPLaZQkZp6WhWyvdiP49JG-rjyTPck_hvEu&amp;display_name=YouTube&amp;url=https%3A%2F%2Fwww.youtube.com%2Fwatch%3Fv%3DUyC0bBiZY-A&amp;image=https%3A%2F%2Fi.ytimg.com%2Fvi%2FUyC0bBiZY-A%2Fhqdefault.jpg&amp;key=a19fcc184b9711e1b4764040d3dc5c07&amp;type=text%2Fhtml&amp;schema=youtube" width="854" height="480" frameborder="0" scrolling="no"><a href="https://medium.com/media/cd130523fb0f38761eca1057b8eefddf/href">https://medium.com/media/cd130523fb0f38761eca1057b8eefddf/href</a></iframe><p>What is word or unigram probability ?</p><iframe src="https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Fmath.embed.fun%2Fembed%2FwYf63LzD56HhjuHzeTF9de&amp;display_name=Embed+Fun&amp;url=https%3A%2F%2Fmath.embed.fun%2Fembed%2FwYf63LzD56HhjuHzeTF9de&amp;key=a19fcc184b9711e1b4764040d3dc5c07&amp;type=text%2Fhtml&amp;schema=embed" width="200" height="157" frameborder="0" scrolling="no"><a href="https://medium.com/media/ccb5bca9143c5d831d548375d60d2c50/href">https://medium.com/media/ccb5bca9143c5d831d548375d60d2c50/href</a></iframe><p>What is bi-gram probability?</p><iframe src="https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Fmath.embed.fun%2Fembed%2FsU7YURp5Bdc5ea3dCXTnP1&amp;display_name=Embed+Fun&amp;url=https%3A%2F%2Fmath.embed.fun%2FsU7YURp5Bdc5ea3dCXTnP1&amp;key=a19fcc184b9711e1b4764040d3dc5c07&amp;type=text%2Fhtml&amp;schema=embed" width="611" height="229" frameborder="0" scrolling="no"><a href="https://medium.com/media/5bf5783f8cc614c401a96bc008109bc5/href">https://medium.com/media/5bf5783f8cc614c401a96bc008109bc5/href</a></iframe><p>What is n-gram probability?</p><iframe src="https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Fmath.embed.fun%2Fembed%2F5MPPmoPrdpgSxcknCmfWsn&amp;display_name=Embed+Fun&amp;url=https%3A%2F%2Fmath.embed.fun%2F5MPPmoPrdpgSxcknCmfWsn&amp;key=a19fcc184b9711e1b4764040d3dc5c07&amp;type=text%2Fhtml&amp;schema=embed" width="822" height="231" frameborder="0" scrolling="no"><a href="https://medium.com/media/7dd76423a2db115123bb85ff00012ed3/href">https://medium.com/media/7dd76423a2db115123bb85ff00012ed3/href</a></iframe><pre>def _count_n_grams(self, tokenized_sentences, ngram):<br>    &#39;&#39;&#39;<br>    Creates n-gram from tokenized sentence and counts the same<br>    &#39;&#39;&#39;<br>    freq = defaultdict(lambda: 0)<br>    for sentence in tqdm(tokenized_sentences, desc=&quot;NGrams&quot;):<br>      sentence = [self._start_token] * ngram + sentence + [self._end_token]<br>      m = len(sentence) if ngram == 1 else len(sentence) - 1<br>      for i in range(m):<br>        ngram_token = sentence[i:i+ngram]<br>        #freq[tuple(ngram_token)] += 1<br>        # tuples can&#39;t be used as key in JSON<br>        freq[&quot; &quot;.join(ngram_token)] += 1<br>    return freq</pre><p>Now comes the next question how to predict next word in my search query sentence? simple, use n-gram of your choice</p><ul><li>Calculate the counts for n-gram <strong>Eg: bi-gram</strong></li><li>Calculate the counts for n-gram + 1 <strong>Eg: tri-gram . </strong>By +1 it means we are considering next n-gram.</li></ul><iframe src="https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Fwww.youtube.com%2Fembed%2FHr6UWRwAIEQ%3Flist%3DPLaZQkZp6WhWyvdiP49JG-rjyTPck_hvEu&amp;display_name=YouTube&amp;url=https%3A%2F%2Fwww.youtube.com%2Fwatch%3Fv%3DHr6UWRwAIEQ&amp;image=https%3A%2F%2Fi.ytimg.com%2Fvi%2FHr6UWRwAIEQ%2Fhqdefault.jpg&amp;key=a19fcc184b9711e1b4764040d3dc5c07&amp;type=text%2Fhtml&amp;schema=youtube" width="854" height="480" frameborder="0" scrolling="no"><a href="https://medium.com/media/bcafd9e8ec5739ec4ddf3dd9101e90a4/href">https://medium.com/media/bcafd9e8ec5739ec4ddf3dd9101e90a4/href</a></iframe><ul><li>Apply the n-gram probability formula with K-Smoothing, to avoid out of vocab and skewed words</li></ul><blockquote>Try <a href="https://medium.com/@dennyc/a-simple-numerical-example-for-kneser-ney-smoothing-nlp-4600addf38b8)">Kneser Ney </a>smoothening instead of plain Laplace smoothening!</blockquote><figure><img alt="" src="https://cdn-images-1.medium.com/max/878/0*a_8Jwz4X0OurcUkU.png" /></figure><pre>start_tokens = [&quot;how&quot;, &quot;many&quot;, <strong>&quot;pairs&quot;, &quot;of&quot;</strong>]</pre><pre>model.suggestions(start_tokens)</pre><pre>&gt;&gt; [&#39;chromosomes&#39;, &#39;how&#39;, &#39;do&#39;, &#39;you&#39;, &#39;change&#39;]</pre><p>So basically we have two counts n-gram and n-gram+1, as an example lets take bi-gram and tri-gram. As the user types in we consider previous bi-gram, here [“<strong>pairs”, “of”] </strong>and iterate through all words in the vocabulary and calculate the probability of each word combination with our n-gram tuple.</p><p>What is the probability of a word from the vocabulary given the previous bi-gram (pairs, of)?</p><iframe src="https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Fmath.embed.fun%2Fembed%2FjVXwMg276GZJ7YUEWTpea5&amp;display_name=Embed+Fun&amp;url=https%3A%2F%2Fmath.embed.fun%2FjVXwMg276GZJ7YUEWTpea5&amp;key=a19fcc184b9711e1b4764040d3dc5c07&amp;type=text%2Fhtml&amp;schema=embed" width="622" height="157" frameborder="0" scrolling="no"><a href="https://medium.com/media/a022337f5656afeffe3e5af12892b9f3/href">https://medium.com/media/a022337f5656afeffe3e5af12892b9f3/href</a></iframe><pre>def _estimate_probability(self, word, previous_ngram):<br>    vocab_size = len(self._word_frequency)<br>    #previous_ngram = tuple(previous_ngram)<br>    if type(previous_ngram) != list:<br>      previous_ngram = [previous_ngram]<br>    previous_ngram = &quot; &quot;.join(previous_ngram)<br>    previous_ngram_count = self._ngram_word_frequency.get(previous_ngram, 0)<br>    if previous_ngram_count == 0:<br>      # print(&quot;Warning no match found for entered words!&quot;)<br>      return 0<br>  <strong>  denominator = previous_ngram_count + self._k * len(self._vocab)</strong><br>    n_plus1_gram = previous_ngram + &quot; &quot; + word<br>    n_plus1_gram_count =  self._ngram_plus1_word_frequency.get(n_plus1_gram, 0)<br>    <strong>numerator = n_plus1_gram_count + self._k</strong><br>    probability = numerator / denominator<br>    return probability</pre><p>Iterate through all words in vocabulary and calculate the probabilities, this is needed as weed figure out which word has highest probability for given n-gram.</p><pre>def _estimate_probabilities(self, previous_ngram):<br>    probabilities = {}<br>    # previous_n_gram = tuple(previous_n_gram)<br>    if type(previous_ngram) != list:<br>      previous_ngram = [previous_ngram]<br>    previous_ngram = &quot; &quot;.join(previous_ngram).lower()<br>    for word in self._vocab:<br>      probabilities[word] = self._estimate_probability(word, previous_ngram)<br>    return probabilities</pre><h4><strong><em>Apache Spark</em></strong></h4><p>Ok cool! we managed to build our model with some understanding of its maths (at-least for now!). We are able to train on small corpus and build the model quickly.</p><p>For our to model to suggest next words all we need is the <strong>vocab</strong>, <strong>counts of n-gram and counts of n-gram+1</strong>. Now, lets see how to use Apache Spark to get the counts for n-gram and n-gram +1 on a bigger corpus.</p><pre>class SparkAutoCorrectModel(object):<br>  def __init__(self, <br>               spark,<br>               dataset,<br>               ngram=2):<br>    self._spark = spark<br>    self._df = spark.createDataFrame(pd.DataFrame({&quot;text&quot;: ds.lines}))<br>    self._ngram = ngram</pre><pre>self._tokenizer = Tokenizer(inputCol=&quot;text&quot;, outputCol=&quot;words&quot;)<br>    self._bigram = NGram(n=2, inputCol=&quot;words&quot;, outputCol=&quot;ngrams&quot;)<br>    self._trigram = NGram(n=ngram+1, inputCol=&quot;words&quot;, outputCol=&quot;ngram_plus_one&quot;)</pre><pre>def transform(self):<br>    df_tokenized = self._tokenizer.transform(self._df)<br>    ngram_df = self._bigram.transform(df_tokenized)<br>    ngram_df = self._trigram.transform(ngram_df)<br>    ngram_df.show()<br>    self._ngram_df = ngram_df</pre><pre>def save_as_json(self, file_path):<br>    vocab = self._ngram_df.select(F.explode(&quot;words&quot;).alias(&quot;vocab&quot;)).collect()<br>    vocab = {row[&#39;vocab&#39;] for row in vocab}<br>    vocab = list(vocab)</pre><pre>ngram = self._ngram_df.select(F.explode(&quot;ngrams&quot;).alias(&quot;ngram&quot;)).groupBy(&quot;ngram&quot;).count().collect()<br>    ngram_word_frequency = {row[&#39;ngram&#39;]: row[&#39;count&#39;] for row in ngram}</pre><pre>ngram_plus_one = self._ngram_df.select(F.explode(&quot;ngram_plus_one&quot;).alias(&quot;ngram_plus_one_&quot;)).groupBy(&quot;ngram_plus_one_&quot;).count().collect()<br>    ngram_plus1_word_frequency = {row[&#39;ngram_plus_one_&#39;]: row[&#39;count&#39;] for row in ngram_plus_one}</pre><pre>data = {}<br>    data[&#39;ngram&#39;] = self._ngram<br>    data[&#39;vocab&#39;] = vocab <br>    data[&#39;ngram_word_frequency&#39;] = ngram_word_frequency<br>    data[&#39;ngram_plus1_word_frequency&#39;] = ngram_plus1_word_frequency</pre><pre>with open(file_path, &quot;w&quot;, encoding=&#39;utf-8&#39;) as file:<br>      json.dump(data, file, ensure_ascii=False, indent=4)</pre><p>So whats happening here?</p><ul><li>Convert the list of dataset lines into Spark DataFrame, with lines in text column</li><li>Tokenizer the text column with Tokenizer</li><li>With NGram create two new column ngramand ngram_plus_one</li><li>With explode SQL in-build function convert array of tokens into rows and aggregate the tuple counts</li><li>Collect the vocabulary, n-grm counts and n-gram + 1 counts and create a JSON file, that can be loaded into out model class.</li></ul><h4>3. Vue Web UI</h4><p>In recent time <a href="https://streamlit.io/">https://streamlit.io/</a> library has become the go to option for Data Science UI as it only takes an hour to build a UI that you wanna for your proof of projects all in Python of-course, nevertheless it comes with cost of loosing customisation.</p><p>In my Googling I found Vue to match my object oriented programming experience and a framework that has an easy learning curve to kick start. It has this nice way of modeling data variables, user methods, signal emits &amp; props for data sharing and HTML handling.</p><p>Each Vue component consists of:</p><ul><li>HTML</li><li>CSS : Which plays major role in decorating the web pages. One good thing is there are well defined CSS framework like <a href="https://bulma.io/">https://bulma.io/</a> that gives all essential CSS classes for free</li><li>JavaScript with some predefined layout</li></ul><p>To understand the VueUI following Vue 3 concepts are necessary:</p><ul><li><a href="https://v3.vuejs.org/guide/data-methods.html">data</a></li><li><a href="https://v3.vuejs.org/guide/data-methods.html#data-properties">methods</a></li><li><a href="https://vuejs.org/v2/guide/class-and-style.html">v-bind</a></li><li><a href="https://vuejs.org/v2/guide/forms.html">v-model</a></li><li><a href="https://vuejs.org/v2/guide/list.html">v-for</a></li><li><a href="https://vuejs.org/v2/guide/conditional.html">v-if-else</a> / v-show</li><li><a href="https://vueschool.io/lessons/understanding-the-vuejs-lifecycle-hooks">Vue lifecycle hooks</a></li></ul><p>We need a search box and a list to display the suggestions.</p><pre>&lt;input class=&quot;input is-rounded&quot; <br>          v-model=&quot;search&quot; <br>          <a href="http://twitter.com/input">@input</a>=&quot;onChange&quot;<br>          <a href="http://twitter.com/keyup">@keyup</a>.down=&quot;onArrowDown&quot; <br>          <a href="http://twitter.com/keyup">@keyup</a>.up=&quot;onArrowUp&quot; <br>          <a href="http://twitter.com/keyup">@keyup</a>.enter=&quot;onEnter&quot;<br>          autocomplete=&quot;off&quot;<br>          placeholder=&quot;Enter your text here for auto suggestion...&quot;/&gt;</pre><pre>&lt;ul id=&quot;autocomplete-results&quot; v-show=&quot;isOpen&quot; class=&quot;autocomplete-results&quot;&gt;<br>        &lt;li class=&quot;loading&quot; v-if=&quot;isLoading&quot;&gt;<br>          Loading Results...<br>        &lt;/li&gt;<br>        &lt;li v-else v-for=&quot;(result, i) in results&quot; :key=&quot;i&quot; <a href="http://twitter.com/click">@click</a>=&quot;setResult(result)&quot;<br>                  class=&quot;autocomplete-result&quot; :class=&quot;{ &#39;is-active&#39;: i === arrowCounter }&quot;&gt;<br>        {{result}}<br>        &lt;/li&gt;<br>      &lt;/ul&gt;</pre><p>Whats happening?</p><ul><li>Bind search data variable to input box</li><li>On user input call onChange method, similarly on arrow key press call respective methods</li><li>List is used to display the suggestions, and show the list only we have suggestion from backend with data variable isOpen</li><li>If back end is busy doing the computation, show Loading Results based on the value of data variable isLoading</li><li>When suggestions are available iterate through it and display it with a style for active selection</li></ul><p>When user inputs the search query, call backend API:</p><pre>       // Backend <br>      axios.post(path, {text: cleandedQuery})<br>      .then((res) =&gt; {<br>        this.isLoading = false<br>        this.results = res.data.tokens<br>        console.info(&quot;suggestions&quot;, this.results)<br>      })</pre><p>When user navigates the suggestions in the list and presses the enter button, add the current selection the our search query</p><pre>onEnter() {<br>      console.info(&quot;enter&quot;, this.arrowCounter)<br>      if (this.arrowCounter != -1) {<br>        if (this.search.split(&quot; &quot;).length &gt; 1) {<br>          this.search = this.search.split(&quot; &quot;).slice(0, -1).join(&quot; &quot;)<br>        }<br>        // based on the selection from suggestions add the suggested word to <br>        this.search = this.search + &quot; &quot; + this.results[this.arrowCounter];<br>      }<br>      this.isOpen = false<br>      this.arrowCounter = -1<br>    },</pre><p>Thats all to it for our intelligent search box 😉</p><h4>4. FastAPI Backend</h4><p>I have picked <a href="https://fastapi.tiangolo.com/">https://fastapi.tiangolo.com/</a> instead of traditional frameworks as it makes testing the API very easy and has all the batteries that is needed for quick prototyping.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*vm7lY3oqRysZ6WLq3NLrEg.png" /></figure><ul><li>Setup<a href="https://github.com/gyan42/autocomplete-ngram-model/blob/main/main.py"> FastAPI main file</a> with cross origin policies</li></ul><pre>app.add_middleware(<br>    CORSMiddleware,<br>    allow_origins=ALLOWED_ORIGINS,<br>    allow_credentials=True,<br>    allow_methods=[&quot;*&quot;],<br>    allow_headers=[&quot;*&quot;],<br>)</pre><ul><li>Initialise AutoCorrectModel and load one of the json which we got as result of our training</li></ul><pre>auto_correct_model = AutoCorrectModel()<br>auto_correct_model.load_from_json(&quot;data/trigram-autocompleter.json&quot;)</pre><ul><li>Our end point is at “<strong>/suggestions</strong>”, which can receive a string, split it on spaces and returns a max of 10 words fro next word</li></ul><pre><a href="http://twitter.com/app">@app</a>.post(&quot;/suggestions&quot;)<br>def suggestions(data: Text):<br>    start_with = None<br>    tokens = data.text.split(&quot; &quot;)<br>    if len(tokens) &lt; auto_correct_model._ngram:<br>        tokens = [&#39;&lt;s&gt;&#39;] * (auto_correct_model._ngram - len(tokens)) + tokens<br>    print(&quot;Inputs&quot;, tokens)<br>    res = auto_correct_model.suggestions(tokens, num_suggestions=10,   start_with=start_with)<br>    print(&quot;Sugestions&quot;, res)<br>    return {&quot;tokens&quot;: res}</pre><h4>5. Docker</h4><p>Multistage build are used to speed up the docker builds and keep the size of the images considerably low.</p><p>Check the README <a href="https://github.com/gyan42/autocomplete-ngram-model">here</a> for all build commands.</p><p><strong>API : </strong><a href="https://github.com/gyan42/autocomplete-ngram-model/blob/main/ops/api/Dockerfile">https://github.com/gyan42/autocomplete-ngram-model/blob/main/ops/api/Dockerfile</a></p><ul><li>In my trials, on Mac the docker IP address is different from the host even if we bridge the network. So I had to come up with two Docker images one for Linux and one for Mac. Each having its own backend API host address.</li><li>On Mac it defaults to 192.168.99.100 , whereas in Linux it is 0.0.0.0 or localhost</li><li>This is handled in Vue .env file @ <a href="https://github.com/gyan42/autocomplete-ngram-model/tree/main/ui/autocomplete">https://github.com/gyan42/autocomplete-ngram-model/tree/main/ui/autocomplete</a></li></ul><p><strong>UI: </strong><a href="https://github.com/gyan42/autocomplete-ngram-model/tree/main/ops/ui">https://github.com/gyan42/autocomplete-ngram-model/tree/main/ops/ui</a></p><ul><li>Basic understanding of <a href="https://www.nginx.com/">https://www.nginx.com/</a> is required as we use it as our webserver.</li><li>Beginners guide @ <a href="http://nginx.org/en/docs/beginners_guide.html">http://nginx.org/en/docs/beginners_guide.html</a></li><li>To run on local machine we use this ngnix.conf @ <a href="https://github.com/gyan42/autocomplete-ngram-model/blob/main/ui/autocomplete/nginx.conf">https://github.com/gyan42/autocomplete-ngram-model/blob/main/ui/autocomplete/nginx.conf</a></li><li>It listens on port 80 and loads the index file from our Vue build directory</li></ul><p><strong>DockerCompose: </strong><a href="https://github.com/gyan42/autocomplete-ngram-model/tree/main/ops">https://github.com/gyan42/autocomplete-ngram-model/tree/main/ops</a></p><ul><li>Since we have two platform to handle we have two Docker compose files.</li></ul><h4><strong>6. Heroku</strong></h4><p>Lets host our front end and back end along with our model on <a href="https://www.heroku.com/">https://www.heroku.com/</a></p><ul><li>Create an account</li><li>Create an app</li><li>Login into Heroku and Heroku container on command line</li><li>Build an Docker image with both Vue and FastaPI on single docker image @ <a href="https://github.com/gyan42/autocomplete-ngram-model/blob/main/ops/heroku/Dockerfile">https://github.com/gyan42/autocomplete-ngram-model/blob/main/ops/heroku/Dockerfile</a></li><li>Heroku has different <a href="https://github.com/gyan42/autocomplete-ngram-model/blob/main/ops/heroku/default.conf">ngnix file</a> that takes in Heroku supplied port number for UI and we have configured proxy for redirect backend API calls to FastAPI endpoints</li><li>Part of Docker command the $PORT number is replaced during the launch time : <strong>sed -i -e ‘s/$PORT/’”$PORT”’/g’ /etc/nginx/conf.d/default.conf</strong></li></ul><pre>server {<br>  listen      $PORT;</pre><pre>root /usr/share/nginx/html;<br>  index index.html index.html;</pre><pre>location / {<br>    client_max_body_size 200M;<br>    root   /app;<br>    index  index.html;<br>    try_files $uri $uri/ /index.html;<br>  }</pre><pre>location /suggestions {<br>      proxy_connect_timeout 6000;<br>      proxy_read_timeout 6000;<br>      proxy_pass <a href="http://127.0.0.1:5000/">http://127.0.0.1:5000</a>;<br>  }</pre><pre>error_page   500 502 503 504  /50x.html;<br>  location = /50x.html {<br>    root   /usr/share/nginx/html;<br>  }<br>}</pre><ul><li>Push the docker image to the Heroku registry</li><li>Release the app</li></ul><p>Hope this post gives some pointers for hosting Vue + Fast API application on Heroku and as well on local machine.</p><p><strong>References:</strong></p><ul><li><a href="https://towardsdatascience.com/index-48563e4c1572">https://towardsdatascience.com/index-48563e4c1572</a></li><li><a href="https://www.kaggle.com/sauravmaheshkar/auto-completion-using-n-gram-models">https://www.kaggle.com/sauravmaheshkar/auto-completion-using-n-gram-models</a></li><li><a href="https://testdriven.io/blog/developing-a-single-page-app-with-fastapi-and-vuejs/">https://testdriven.io/blog/developing-a-single-page-app-with-fastapi-and-vuejs/</a></li><li><a href="https://testdriven.io/blog/deploying-flask-to-heroku-with-docker-and-gitlab/">https://testdriven.io/blog/deploying-flask-to-heroku-with-docker-and-gitlab/</a></li><li><a href="https://www.tutlinks.com/create-and-deploy-fastapi-app-to-heroku/">https://www.tutlinks.com/create-and-deploy-fastapi-app-to-heroku/</a></li></ul><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=8b94d3671d79" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[How to build custom NER HuggingFace dataset for receipts and train with HuggingFace Transformers…]]></title>
            <link>https://mageswaran1989.medium.com/how-to-build-custom-ner-huggingface-dataset-for-receipts-and-train-with-huggingface-transformers-6c954b84473c?source=rss-f442f089254f------2</link>
            <guid isPermaLink="false">https://medium.com/p/6c954b84473c</guid>
            <category><![CDATA[hugging-face]]></category>
            <category><![CDATA[transformers]]></category>
            <category><![CDATA[dataset]]></category>
            <category><![CDATA[ner]]></category>
            <category><![CDATA[digital-receipts]]></category>
            <dc:creator><![CDATA[Mageswaran D]]></dc:creator>
            <pubDate>Thu, 28 Oct 2021 06:42:09 GMT</pubDate>
            <atom:updated>2021-10-28T06:49:56.738Z</atom:updated>
            <content:encoded><![CDATA[<h3>How to build custom NER HuggingFace dataset for receipts and train with HuggingFace Transformers library?</h3><p>Disclaimer: It is assumed that you have some working knowledge in Hugging face library and datasets library, to begin with.</p><p>Git link:</p><p><a href="https://github.com/gyan42/mozhi-datasets/tree/main/sroie2019">mozhi-datasets/sroie2019 at main · gyan42/mozhi-datasets</a></p><p>Named Entity Recognition the back bone of extracting information from text documents, often less visited topic on NLP.</p><p>Let’s not get into what is NER topic for today’s discussion.</p><p>Rather lets see how to prepare data so that we can train using the famous NLP library <a href="https://github.com/huggingface/transformers">Transformers from Hugging Face.</a></p><p>The dataset we are going to use today is <a href="https://rrc.cvc.uab.es/?ch=13">ICDAR 2019 Robust Reading Challenge on Scanned Receipts OCR and Information Extraction</a></p><p>It consists of three tasks Text localisation , OCR and Information extraction.</p><p>We are going to use OCR and Information extraction task data to prepare NER dataset for our training.</p><p>Following is a sample image from the train dataset:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/463/1*yI8zWi3uo1ZDfDXiJWmMvw.jpeg" /></figure><p>OCR text file:</p><pre><br>72,25,326,25,326,64,72,64,TAN WOON YANN<br>50,82,440,82,440,121,50,121,<strong>BOOK TA .K(TAMAN DAYA) SDN BND</strong><br>205,121,285,121,285,139,205,139,789417-W<br>110,144,383,144,383,163,110,163,<strong>NO.53 55,57 &amp; 59, JALAN SAGU 18,</strong><br>192,169,299,169,299,187,192,187,<strong>TAMAN DAYA,</strong><br>162,193,334,193,334,211,162,211,<strong>81100 JOHOR BAHRU,</strong><br>217,216,275,216,275,233,217,233,<strong>JOHOR.</strong><br>50,342,279,342,279,359,50,359,DOCUMENT NO : TD01167104<br>50,372,96,372,96,390,50,390,DATE:<br>165,372,342,372,342,389,165,389,<strong>25/12/2018</strong> 8:13:39 PM<br>48,396,117,396,117,415,48,415,CASHIER:<br>164,397,215,397,215,413,164,413,MANIS<br>49,423,122,423,122,440,49,440,MEMBER:<br>191,460,298,460,298,476,191,476,CASH BILL<br>30,508,121,508,121,523,30,523,CODE/DESC<br>200,507,247,507,247,521,200,521,PRICE<br>276,506,306,506,306,522,276,522,DISC<br>374,507,441,507,441,521,374,521,AMOUNT<br>69,531,102,531,102,550,69,550,QTY<br>221,531,247,531,247,545,221,545,RM<br>420,529,443,529,443,547,420,547,RM<br>27,570,137,570,137,583,27,583,9556939040116<br>159,570,396,570,396,584,159,584,KF MODELLING CLAY KIDDY FISH<br>77,598,113,598,113,613,77,613,1 PC<br>138,597,148,597,148,607,138,607,*<br>202,597,245,597,245,612,202,612,9.000<br>275,598,309,598,309,612,275,612,0.00<br>411,596,443,596,443,613,411,613,9.00<br>245,639,293,639,293,658,245,658,TOTAL:<br>118,671,291,671,291,687,118,687,ROUR DING ADJUSTMENT:<br>408,669,443,669,443,684,408,684,0.00<br>86,704,292,704,292,723,86,723,ROUND D TOTAL (RM):<br>401,703,443,703,443,719,401,719,<strong>9.00</strong><br>205,744,243,744,243,765,205,765,CASH<br>402,748,441,748,441,763,402,763,10.00<br>205,770,271,770,271,788,205,788,CHANGE<br>412,772,443,772,443,786,412,786,1.00<br>97,845,401,845,401,860,97,860,GOODS SOLD ARE NOT RETURNABLE OR<br>190,864,309,864,309,880,190,880,EXCHANGEABLE<br>142,883,353,883,353,901,142,901,***<br>137,903,351,903,351,920,137,920,***<br>202,942,292,942,292,959,202,959,THANK YOU<br>163,962,330,962,330,977,163,977,PLEASE COME AGAIN !<br>412,639,442,639,442,654,412,654,<strong>9.00</strong></pre><p>IE text file:</p><pre>{<br>    &quot;company&quot;: &quot;BOOK TA .K (TAMAN DAYA) SDN BHD&quot;,<br>    &quot;date&quot;: &quot;25/12/2018&quot;,<br>    &quot;address&quot;: &quot;NO.53 55,57 &amp; 59, JALAN SAGU 18, TAMAN DAYA, 81100 JOHOR BAHRU, JOHOR.&quot;,<br>    &quot;total&quot;: &quot;9.00&quot;<br>}</pre><p>First task is to merge IE text file data and OCR text file data and make (word, tag) pair.</p><pre>TAN O<br>WOON O<br>YANN O<br>BOOK company<br>TA company<br>.K(TAMAN company<br>DAYA) company<br>SDN company<br>BND company<br>789417-W O<br>NO.53 address<br>55 address<br>57 address<br>&amp; address<br>59 address<br> address<br>JALAN address<br>SAGU address<br>18 address<br>TAMAN address<br>DAYA address<br>81100 address<br>JOHOR address<br>BAHRU address<br>JOHOR. address<br>DOCUMENT O<br>NO O<br>: O<br>TD01167104 O<br>DATE: O<br>25/12/2018 date<br>8:13:39 date<br>PM date<br>CASHIER: O<br>MANIS O<br>MEMBER: O<br>CASH O<br>BILL O<br>CODE/DESC O<br>PRICE O<br>DISC O<br>AMOUNT O<br>QTY O<br>RM O<br>9556939040116 O<br>KF O<br>MODELLING O<br>CLAY O<br>KIDDY O<br>FISH O<br>1 O<br>PC O<br>* O<br>9.000 total<br>0.00 O<br>9.00 total<br>TOTAL: O<br>ROUR O<br>DING O<br>ADJUSTMENT: O<br>ROUND O<br>D O<br>TOTAL O<br>(RM): O<br>CASH O<br>10.00 O<br>CHANGE O<br>1.00 O<br>GOODS O<br>SOLD O<br>ARE O<br>NOT O<br>RETURNABLE O<br>OR O<br>EXCHANGEABLE O<br>*** O<br>THANK O<br>YOU O<br>PLEASE O<br>COME O<br>AGAIN O<br>! O</pre><p>Above one is very simple one with (word, tag) pair, if you wanted to be more informative and complicated, this can be converted into IOB/BOI format. Where <strong>B</strong> stands for <strong>beginning</strong>, <strong>O</strong> stands for <strong>other tag </strong>and <strong>I</strong> stands for <strong>inside</strong>.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*orJf_jf3HK8XqqU0.png" /></figure><blockquote>Yet another commonly used format is CoNLL more details @ <a href="https://universaldependencies.org/format.html">https://universaldependencies.org/format.html</a></blockquote><blockquote>The de-facto dataset for NER tasks is CoNLL-2003 dataset</blockquote><blockquote><a href="https://www.clips.uantwerpen.be/conll2003/ner/">https://www.clips.uantwerpen.be/conll2003/ner/</a></blockquote><blockquote><a href="https://paperswithcode.com/dataset/conll-2003">https://paperswithcode.com/dataset/conll-2003</a></blockquote><blockquote>Here is the Git link for CoNLL 2003 dataset : <a href="https://github.com/davidsbatista/NER-datasets/tree/master/CONLL2003">https://github.com/davidsbatista/NER-datasets/tree/master/CONLL2003</a></blockquote><p>Check this notebook for the code that prepares SROIE 2019 NER dataset :</p><p><a href="https://github.com/gyan42/mozhi-datasets/blob/main/sroie2019/SroieNERDataset.ipynb">mozhi-datasets/SroieNERDataset.ipynb at main · gyan42/mozhi-datasets</a></p><p>Now comes the part where we wanted to use the data for transformer model training.</p><p>Dataset library from Huggingface has become a good choice to use for many model experimentation. However it has only support for some of well established NER datasets like CoNLL 2003</p><ul><li>Online version @ <a href="https://huggingface.co/datasets/viewer/?dataset=conll2003">https://huggingface.co/datasets/conll2003</a></li><li>Source Code @ <a href="https://github.com/huggingface/datasets/blob/master/datasets/conll2003/conll2003.py">https://github.com/huggingface/datasets/blob/master/datasets/conll2003/conll2003.py</a></li></ul><p>What if we need to use our own dataset like the one we created above?</p><p>One easy way is to check existing setup, so I was checking how the existing CoNLL 2003 dataset is working in HF datasets library and came up with following class, with minor additions.</p><p>Below class takes in raw url path to Git hub directory, file names and list of NER tags. With that it downloads the data and prepare the data for training.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/fccc787c1a2b54f3edec4924a75dea0a/href">https://medium.com/media/fccc787c1a2b54f3edec4924a75dea0a/href</a></iframe><p>_info() is mandatory where we need to specify the columns of the dataset. In our case it is three columns id, ner_tags, tokens , where id and tokens are values from the dataset, ner_tags is for names of the NER tags which needs to be set manually.</p><p>_generate_examples(file_path) reads our IOB formatted text file and creates list of (word, tag) for each sentence. Our dataset will then a list of list i.e outer list os for each sentence and inner list for word tag pairs. Open the file, collect id, token, ner_tag till we encounter new line and append it to outer list and repeat till the end of the file.</p><p>We also need a tokeniser which can take our list of examples and converts it into integers</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/8bb40479b239e7e9b6688656df8b63e3/href">https://medium.com/media/8bb40479b239e7e9b6688656df8b63e3/href</a></iframe><p>`tokenize_and_align_labels` function act as a data transformer that takes raw examples and convers it into compatible format for model training.</p><p>Now that we have a dataset class that is compatible with HuggingFace transformer, its a cake walk to integrate with a <strong>AutoModelForTokenClassification</strong> model named <strong>distilbert-base-uncased</strong></p><p>Initialise dataset class</p><pre># HFSREIO2019Dataset is wrapper around SROIE2019 with some extra methods like id2word word2id methods<br>hf_dataset = HFSREIO2019Dataset()</pre><pre>hf_model = AutoModelForTokenClassification.from_pretrained(&quot;<strong>distilbert-base-uncased&quot;</strong>,num_labels=len(hf_dataset.labels))</pre><p>dataset info:</p><pre>Dataset({<br>    features: [&#39;id&#39;, &#39;tokens&#39;, &#39;ner_tags&#39;],<br>    num_rows: 588<br>})<br>Dataset({<br>    features: [&#39;id&#39;, &#39;tokens&#39;, &#39;ner_tags&#39;],<br>    num_rows: 82<br>})<br>Dataset({<br>    features: [&#39;id&#39;, &#39;tokens&#39;, &#39;ner_tags&#39;],<br>    num_rows: 49<br>})</pre><pre>num_rows: 49</pre><pre>})</pre><p>toenized dataset info:</p><pre>DatasetDict({<br>    train: Dataset({<br>        features: [&#39;attention_mask&#39;, &#39;id&#39;, &#39;input_ids&#39;, &#39;labels&#39;, &#39;ner_tags&#39;, &#39;tokens&#39;],<br>        num_rows: 588<br>    })<br>    validation: Dataset({<br>        features: [&#39;attention_mask&#39;, &#39;id&#39;, &#39;input_ids&#39;, &#39;labels&#39;, &#39;ner_tags&#39;, &#39;tokens&#39;],<br>        num_rows: 49<br>    })<br>    test: Dataset({<br>        features: [&#39;attention_mask&#39;, &#39;id&#39;, &#39;input_ids&#39;, &#39;labels&#39;, &#39;ner_tags&#39;, &#39;tokens&#39;],<br>        num_rows: 82<br>    })<br>})</pre><p>Now prepare the data collector for training</p><pre>hf_preprocessor = HFTokenizer.init_vf(hf_pretrained_tokenizer_checkpoint=hf_pretrained_tokenizer_checkpoint)</pre><p>Now prepare the data collector for training</p><pre>from transformers import DataCollatorForTokenClassification<br>data_collator = DataCollatorForTokenClassification(hf_preprocessor.tokenizer)</pre><p>Finally training</p><pre>args = TrainingArguments(<br>    f&quot;test-ner&quot;,<br>    evaluation_strategy=&quot;epoch&quot;,<br>    learning_rate=learning_rate,<br>    per_device_train_batch_size=batch_size,<br>    per_device_eval_batch_size=batch_size,<br>    num_train_epochs=max_epochs,<br>    weight_decay=0.01,<br>)</pre><pre>trainer = Trainer(<br>    hf_model,<br>    args,<br>    train_dataset=tokenized_datasets[&quot;train&quot;],<br>    eval_dataset=tokenized_datasets[&quot;validation&quot;],<br>    data_collator=data_collator,<br>    tokenizer=hf_preprocessor.tokenizer,<br>    compute_metrics=lambda p: compute_metrics(p=p, label_list=hf_dataset.labels)<br>)<br><br>trainer.train()<br>trainer.evaluate()</pre><p>Result</p><pre>{<br>&#39;date&#39;: {&#39;precision&#39;: 0.6068376068376068, &#39;recall&#39;: 0.6893203883495146, &#39;f1&#39;: 0.6454545454545455, &#39;number&#39;: 103}, </pre><pre>&#39;address&#39;: {&#39;precision&#39;: 0.782608695652174, &#39;recall&#39;: 0.8, &#39;f1&#39;: 0.7912087912087912, &#39;number&#39;: 90}, </pre><pre>&#39;company&#39;: {&#39;precision&#39;: 0.7088607594936709, &#39;recall&#39;: 0.691358024691358, &#39;f1&#39;: 0.7, &#39;number&#39;: 81}, </pre><pre>&#39;total&#39;: {&#39;precision&#39;: 0.6605504587155964, &#39;recall&#39;: 0.7659574468085106, &#39;f1&#39;: 0.7093596059113301, &#39;number&#39;: 94}, </pre><pre>&#39;overall_precision&#39;: 0.6826196473551638, <br>&#39;overall_recall&#39;: 0.7364130434782609, <br>&#39;overall_f1&#39;: 0.7084967320261438, <br>&#39;overall_accuracy&#39;: 0.969463275079296<br>}</pre><p>Check this Google Colab notebook for end to end training on out receipt dataset</p><p><a href="https://colab.research.google.com/gist/Mageswaran1989/87a04a5064dbc9d59799cad5e9fa8fdd/hftransformerwithcustom-ner-datset.ipynb#scrollTo=9Eq6T2HuTa4h">Google Colaboratory</a></p><p>On left side of the notebook in colab, you can navigate to the files.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*7ilCfhCHt8dO6LvFdZrHOg.png" /></figure><p>You could modify the train params in `hf_model_train.py` as it fits for you ;)</p><p>Thanks for reading!</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=6c954b84473c" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[A dive into Apache Spark Parquet Reader for small size files]]></title>
            <link>https://mageswaran1989.medium.com/a-dive-into-apache-spark-parquet-reader-for-small-file-sizes-fabb9c35f64e?source=rss-f442f089254f------2</link>
            <guid isPermaLink="false">https://medium.com/p/fabb9c35f64e</guid>
            <category><![CDATA[pyspark]]></category>
            <category><![CDATA[parquet]]></category>
            <category><![CDATA[spark]]></category>
            <category><![CDATA[big-data]]></category>
            <dc:creator><![CDATA[Mageswaran D]]></dc:creator>
            <pubDate>Tue, 26 Oct 2021 12:16:44 GMT</pubDate>
            <atom:updated>2021-11-29T11:36:37.935Z</atom:updated>
            <content:encoded><![CDATA[<p>While working part of my current project, I was asked how Spark reads Parquet files and how does it achieves the parallelisation. Very simple question indeed.</p><p>Part of my current project I had to load lot of small files and do some filtering and aggregations to get counts for given user config.</p><p>With existing Pandas + Kubernetes setup, the solution is to list the files for the user config and then load them in parallel for each time range split, process them, merge them and get the counts with all nuts and bolts in place.</p><p>When Spark was brought into picture it seems to process faster than the existing solution in the range of 2X (with test config, however it is subjective to user config). The very first question that was brought up was, how Spark reads parquet data ?</p><p>I have worked with Spark for years now, tasks like reading Parquet files are natural and it doesn’t make me to question them. Did you? Come on!</p><p>How many time we questioned ourself on how file open works in python? We may dig in if someone ask it in part of an interview 😉 If my memory goes correct, it involves more or like:</p><pre>Python -&gt; C API -&gt; Unix File descriptor in user Space -&gt; Kernel Space -&gt; Hard disk Driver -&gt; Physical location</pre><p>And to be frank I was blank on the spot, what “HOW” ? there is an API and bunch of configs we use them to load data and Spark loads it for us.</p><p>For the team knowing “HOW” part is important for them to make a decision on how Spark gonna handle their work load on peak hours? Moving a solution to Spark simply doesn’t guarantee its worthiness unless all edge cases are evaluated and sometimes the basic ones too!</p><p>With lots and lots of small files in the picture, it doesn’t matter how big the cluster gonna be, the read operation is going to be a big bottle neck or a show stopper, indeed. Imagine what happens if the user config leads to reading 1000 files, 10,000 files or even 100,000 files? How Spark reads the file and creates partitions when multiple files are loaded ?</p><p>Join with me to investigate them!</p><p>If you have patience to read thesis paper, you may hit this <a href="http://Predicate Pushdown in Parquet and Apache Spark"><strong>link</strong></a> to read “<strong>Predicate Pushdown in Parquet and Apache Spark</strong>”</p><h3>Parquet Format</h3><p>I have been working with Parquet for say 5 years now and all I know is it is columnar format good for Big Data! No idea on how or why part of it? even though I wanted to do this exploration for long time now, there was no bottle neck I faced till this point as I was dealing large files most of the time.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*X9wL6xlATJsAru7N5u-IEA.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*qkCt8ihtUfAl02OEYiy-Hw.png" /><figcaption>Source: <a href="https://databricks.com/glossary/what-is-parquet">https://databricks.com/glossary/what-is-parquet</a></figcaption></figure><p>Googling for Parquet format lead me to official Git repo @ <a href="https://github.com/apache/parquet-format">https://github.com/apache/parquet-format</a> and I spend few mins to read and understand the high level and few follow ups form Google search.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/601/0*fVY7dlFfBuskBJtP" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/584/0*VcVpekIgtTntO5rf.gif" /><figcaption>Take few minutes to read…</figcaption></figure><p><strong>From official Git…</strong></p><p>In Parquet, a data set comprising of rows and columns is partition into one or multiple files. Each individual file contains one or multiple horizontal partitions of rows called <em>row groups </em>(by default 128MB in size). Each row group subsequently contains a <em>column chunk </em>(i.e. vertical partition) for each of the columns of the horizontal partition. Within a column chunk, the actual values are stored in <em>data pages </em>(by default storing 1MB of values). Each data page stores either a fixed or variable amount of (compressed) values, depending on the data type of the column values and the encoding scheme used.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/300/0*s0MQPWen9P5IDB9Q.jpg" /></figure><p>In summary</p><ul><li>Data is stored in columns instead of rows, helping aggregation queries to run faster, by selecting only the columns needs for the query</li><li>Parquet is binary format, thus enabling storing encoded data types for effective serialising and deserialising of the data on the disk (compression)</li><li>Support fo nested schema</li><li>Metadata is stored after writing the data at the footer capturing the summary statistics of the column chunks.</li></ul><p>Parquet is binary format, so to do any experiment on raw files needs considerable amount of code to read without any library support or we have to use some community projects like Apache Arrow for quick experiments.</p><p>Since our focus is with Spark, I am gonna skip this and move on, with one task for you…</p><p>Can we read Parquet files in chunks if so how to read them (any library is fine) ? Please do post in the comments when you come up with one</p><p>For example metadata can be read with arrow like :</p><p><a href="https://arrow.apache.org/docs/python/generated/pyarrow.parquet.read_metadata.html">pyarrow.parquet.read_metadata - Apache Arrow v5.0.0</a></p><p>Some good reads on similar topic…</p><p><a href="https://medium.com/@songkunjump/spark-understand-the-basic-of-pushed-filter-and-partition-filter-using-parquet-file-3e5789e260bd">Spark: Understand the Basic of Pushed Filter and Partition Filter Using Parquet File</a></p><h3>Parquet file in Spark</h3><p>Reading file in Spark is straight forward, knowing all the configurations is pain, and even more pain is when new configuration gets added and we don’t know its existence.</p><pre>- <strong>spark.sql.files.maxPartitionBytes</strong>: 128MB (The maximum number of bytes to pack into a single partition when reading files. This configuration is effective only when using file-based sources such as Parquet, JSON and ORC.)</pre><pre>- <strong>spark.sql.parquet.enableVectorizedReader</strong>: Enables vectorized parquet decoding.</pre><pre>- <strong>spark.sql.parquet.filterPushdown</strong> : Enables Parquet filter push-down optimization when set to true.</pre><pre>- <strong>spark.sql.parquet.mergeSchema </strong>: When true, the Parquet data source merges schemas collected from all data files, otherwise the schema is picked from the summary file or a random data file if no summary file is available.</pre><pre>- <strong>spark.sql.parquet.compression.codec</strong>: ets the compression codec use when writing Parquet files. Acceptable values include: uncompressed, snappy, gzip, lzo.</pre><pre>- <strong>spark.sql.parquet.cacheMetadata</strong>: Turns on caching of Parquet schema metadata. Can speed up querying of static data.</pre><pre>- <strong>spark.sql.files.openCostInBytes</strong>: The estimated cost to open a file, measured by the number of bytes could be scanned in the same time. This is used when putting multiple files into a partition. It is better to over-estimated, then the partitions with small files will be faster than partitions with bigger files (which is scheduled first). This configuration is effective only when using file-based sources such as Parquet, JSON and ORC.</pre><ul><li>Partitions in Spark are determined by type of file system and then by file type</li><li>In HDFS files are written in blocks so when reading each block will be read as a partition, neat and simple!</li><li>In S3, as it is object store where there is no such information available to partition the data, so Spark depends on some user configs</li></ul><p>Following two links gives good information on how the number of partitions are determined while reading data files</p><ul><li><a href="https://medium.com/swlh/building-partitions-for-processing-data-files-in-apache-spark-2ca40209c9b7">Building Partitions For Processing Data Files in Apache Spark</a></li><li><a href="https://jaceklaskowski.gitbooks.io/mastering-spark-sql/content/spark-sql-SparkPlan-FileSourceScanExec.html">FileSourceScanExec</a></li></ul><p>The main configs that determines the number of partitions while reading Parquet files are:</p><pre>spark.default.parallelism (default: Total No. of CPU cores)<br>spark.sql.files.maxPartitionBytes (default: 128 MB)<br>spark.sql.files.openCostInBytes (default: 4 MB)</pre><p>And the following calculation:</p><pre>maxSplitBytes = Minimum(maxPartitionBytes,  Maximum(openCostInBytes, bytesPerCore))</pre><pre>bytesPerCore = (Sum of all data file sizes or size of a given file) / default.parallelism</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/751/1*QaLThl6-3ieWgXyf-Cq3Xw.png" /></figure><p>Spark Scala Source Code Reference @ <a href="https://github.com/apache/spark/blob/master/sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/FilePartition.scala">https://github.com/apache/spark/blob/master/sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/FilePartition.scala</a></p><h3>Demo Time</h3><p>Lets do two experiments</p><ul><li>Loading 128 files all in one shot as list of files for the read API: Where number of partitions is controlled by number of cores, file sizes and maxPartitionBytes</li><li>Load 288 files one by one : Based on <strong>individual file size</strong>, `maxPartitionBytes` and `openCostInBytes` partitions are created</li></ul><p><strong>Cluster Info: </strong>6 executors with 5 cores each</p><h4><strong>Case 1:</strong></h4><p>If we load <strong>128 S3 files which are less than 10MB each and a total of 392MB, </strong>with spark.read.parquet(paths), the 128 files will be distributed across the nodes, with each core creating one partition, compressing 4 or so files into one partition.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*b5VvN4i1axjNaoet3oJivQ.png" /></figure><p>You could see a <strong>count operation @ row number 0:</strong> reads 128 files into 31 partitions @ row number 1(need to investigate why one extra partition, may be sometime later!). We do two such count operation so you can see second read and partitioning @ row number 7</p><h4>Case 2:</h4><p>If we load <strong>288 S3 files</strong> with file size ranging form <strong>0.5MB to 4.3MB</strong> one after another, with 12 file sizes are greater than 4 MB marginally, the total number of partitions will be <strong>300 @ row number 13</strong>.</p><p>Wondering how?</p><pre>maxSplitBytes = Minimum(maxPartitionBytes,  Maximum(openCostInBytes, bytesPerCore))</pre><pre>bytesPerCore = (Sum of all data file sizes  or size of a given file ) / default.parallelism</pre><pre># say one of the file size is 4282731 bytes</pre><pre>bytesPerCore = 4282731 / 30 ~= 0.13MB</pre><pre>Therefore, maxSplitBytes = min(128MB, max(4MB, 0.13MB))</pre><p>So for those 12 files whose file sizes are greater than 4MB ends up creating two partitions, totalling 288 + 12 = 300 partitions</p><p>When I changed <strong>`spark.sql.files.openCostInBytes`</strong> to 5MB, the number of partitions dropped to <strong>288</strong> with one partition for each file.</p><p>So next time when you see more number of partitions for small files, you know what to tweak around in Spark.</p><p>With big files, <strong>maxPartitionBytes </strong>would be used as the value for partitioning Parquet files.</p><p>Once the Parquet file small or big read into Spark Dataframe with certain partitions, rest of the processing is assured by Spark engine.</p><p>Code snippets that were used while playing around loading small files in PySpark.</p><pre><strong># Code snippet to get number of records in each partition</strong></pre><pre><strong>from</strong> pyspark.sql.functions <strong>import</strong> spark_partition_id, asc, desc</pre><pre>df.withColumn(&quot;partitionId&quot;, spark_partition_id()).groupBy(&quot;partitionId&quot;).count().orderBy(asc(&quot;count&quot;)).show(False, 350)</pre><pre># ------------------------------------------------------------------</pre><pre><strong># boto3 code to get S3 file size:</strong></pre><pre>import boto3<br>from botocore.errorfactory import ClientError</pre><pre>s3 = boto3.client(&#39;s3&#39;)</pre><pre>s3_resource = boto3.resource(&#39;s3&#39;)</pre><pre>def s3_file_size(s3_url):<br>    s3_bucket = s3_url.split(&quot;/&quot;)[2]<br>    key = &quot;/&quot;.join(s3_url.split(&quot;/&quot;)[3:])<br>    file_size = 0<br>    try:<br>        object = s3_resource.ObjectSummary(s3_bucket, key)<br>        file_size = object.size<br>        if int(file_size)/1024/1024 &gt; 4:<br>            print(f&quot;\ns3://{s3_bucket}/{key}&quot;, int(file_size))<br>    except ClientError:<br>        # Not found<br>        file_size = 0<br>    return file_size</pre><p>Thanks, for reading!</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=fabb9c35f64e" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Some interesting interview Q&A around “randomness”]]></title>
            <link>https://mageswaran1989.medium.com/some-interesting-interview-q-a-around-randomness-2b92cc7c1022?source=rss-f442f089254f------2</link>
            <guid isPermaLink="false">https://medium.com/p/2b92cc7c1022</guid>
            <category><![CDATA[interview]]></category>
            <category><![CDATA[randomness]]></category>
            <dc:creator><![CDATA[Mageswaran D]]></dc:creator>
            <pubDate>Mon, 24 May 2021 16:33:00 GMT</pubDate>
            <atom:updated>2021-05-24T16:33:00.126Z</atom:updated>
            <content:encoded><![CDATA[<p>1 How to sample a random Element from an infinite stream?</p><p>For an average developed this is a insane question! or at-least it was for me!!!</p><p>If we know the length of the stream we can use in-build or numpy random function to generate a number.</p><pre>import numpy as np<br>np.random.randint(0, len(stream_data)) </pre><pre>import random<br>random.randint(0, len(stream_data))</pre><p>What is random probability ? Its picking an element an element from N numbers i.e <strong>1/N</strong></p><p>How to sample from a stream where we don’t know the <strong>N </strong>? Interesting right!</p><p>It’s true we don’t know the what <strong>N</strong> is actually, however can we exploit the formula 1/<strong>N</strong> by replacing N with <strong>i</strong>, where <strong>i</strong> is the current index of the element in the stream?</p><ul><li>Case 1: When there is only one element in the stream</li></ul><pre>def select_rand(big_stream):<br>    rand_element = None<br>    for i, e in enumerate(big_stream):<br>        if i == 0: # could have been if(len(big_Stream)==0),hold on!<br>           rand_element = e<br>    return rand_element</pre><ul><li>Case 2: Consider stream length greater than 1 i.e the length varying from (1, i+1), starting from the index 1 to i+1. +1 since enumerate start from 0 and we wanted to consider the last element also for sampling</li></ul><p>— Ok good! but how to sample a random element from given index <strong>i </strong>? Lets see what is the probability of getting <strong>1</strong> in the range of <strong>[1, i+1]</strong>?</p><p><strong>1/i+1</strong></p><pre>random.randint(1, i+1) == 1 </pre><p>1 is selected because the length of the stream starts from index 1, use this property to select the element.</p><pre>import random<br><br>def select_rand(big_stream):<br>    rand_element = None<br><br>    for i, e in enumerate(big_stream):<br>        if i == 0:<br>            rand_element = e<br>        if random.randint(1, i + 1) == 1:<br>            rand_element = e<br>    return rand_element</pre><p>In more technical terms:</p><p>For i = 0, we would&#39;ve picked uniformly from [0, 0].</p><p>For i &gt; 0, before the loop began, any element K in [0, i - 1] had 1 / i chance of being chosen as the random element. We want K to have 1 / (i + 1) chance of being chosen after the iteration. This is the case since the chance of having being chosen already but not getting swapped with the ith element is 1 / i * (1 - (1 / (i + 1))) which is 1 / i * i / (i + 1) or 1 / (i + 1)</p><p>This is solved using <a href="https://www.geeksforgeeks.org/loop-invariant-condition-examples-sorting-algorithms/">loop invariant </a>the <a href="http://www.dailycodingproblem.com/">dailycodingproblem.com</a> mail says :), I have no idea what that means! Good one to get subscribed to.</p><p>2 Using a function rand5() that returns an integer from 1 to 5 (inclusive) with uniform probability, implement a function rand7() that returns an integer from 1 to 7 (inclusive)</p><pre>import numpy as np<br>from collections import Counter<br>N = 1000<br>Counter(np.random.randint(1,6,N) <strong>+</strong> <strong>np.random.randint(1,6,N)//2</strong>)<br># Counter({1: 392, 7: 847, 3: 2007, 2: 1209, 6: 1570, 5: 1973, 4: 2002})</pre><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=2b92cc7c1022" width="1" height="1" alt="">]]></content:encoded>
        </item>
    </channel>
</rss>