<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Bhavin Tandel on Medium]]></title>
        <description><![CDATA[Stories by Bhavin Tandel on Medium]]></description>
        <link>https://medium.com/@tandelbhavin?source=rss-230dc92fa3cc------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/0*Fcjr3uSyZ_A30rAs.jpg</url>
            <title>Stories by Bhavin Tandel on Medium</title>
            <link>https://medium.com/@tandelbhavin?source=rss-230dc92fa3cc------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Sat, 21 Nov 2026 19:35:37 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@tandelbhavin/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[Algorithm for Eggless Madeira Cake]]></title>
            <link>https://tandelbhavin.medium.com/algorithm-for-eggless-madeira-cake-793d514dae81?source=rss-230dc92fa3cc------2</link>
            <guid isPermaLink="false">https://medium.com/p/793d514dae81</guid>
            <category><![CDATA[recipe]]></category>
            <category><![CDATA[cake]]></category>
            <category><![CDATA[algorithms]]></category>
            <dc:creator><![CDATA[Bhavin Tandel]]></dc:creator>
            <pubDate>Sat, 17 Apr 2021 16:56:41 GMT</pubDate>
            <atom:updated>2021-04-17T16:56:41.298Z</atom:updated>
            <content:encoded><![CDATA[<p>Algorithm is list of instruction which helps you achieve the desired result. Similarly following are steps which I follow to make eggless madeira cake. However, I have following disclaimers.</p><blockquote>Disclaimer : Every human is different and everyone have different kind of taste buds. So forgive if the recipe doesn’t turn out good for your taste.</blockquote><h3>tl;dr</h3><blockquote>Madeira Cake = 200 gm F + 250 ml Y + 150 gm S + 125 ml O + 1/2 tsp BS + 1 tsp BP+ 1 tsp VE + P</blockquote><p>F = Flour</p><p>Y = Yogurt</p><p>S = Sugar</p><p>O = Oil / Butter</p><p>BS = Baking Soda</p><p>BP = Baking Powder</p><p>VE = Vanilla Essence</p><p>P = Patience</p><h3>Assumptions</h3><ul><li>1 tsp = 5 ml</li></ul><h3>Ingredients</h3><ul><li>Flour 200 gm</li><li>Plain curd (plain yogurt) 250 ml</li><li>Sugar 150 gm</li><li>Baking soda 1/2 tsp</li><li>Baking powder 1 tsp</li><li>Cooking oil or butter 125 ml</li><li>Vanilla essence 1 tsp</li></ul><h3>Algorithm (Instruction)</h3><p><strong>Step 1: Add Yogurt in a bowl</strong></p><p>Nothing special in it. 😐</p><p><strong>Step 2: Add that sweetness</strong></p><p>Add the sugar in it and mix it so that sugar completely dissolves.</p><p><strong>Step 3: Make it fluffy</strong></p><p>Add baking power and baking soda in it, so that cake becomes fluffy.</p><p><strong>Step 4: Sleep 5 mins</strong></p><p>As the title says, mix and leave it for 5 mins so that ingredients reacts and small bubbles appear on top.</p><p><strong>Step 5: Make it greasy</strong></p><p>Add oil to it and mix it properly.</p><p><strong>Step 6: Add that fragnance</strong></p><p>Add the vanilla essence and mix it. This will add the flavour to your cake.</p><p><strong>Step 7: Flour it up</strong></p><p>Add the flour in parts and mix it along so that there is no flour lumps. Mix it to make a consistent batter.</p><p><strong>Step 8: Heat 🔥</strong></p><p>Finally bake the cake in oven at 180* C for 35-40 mins.</p><p><strong>Step 9: Wating ⏲</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*DWZVaFospwjUZ2KmVtzt1g.jpeg" /></figure><p>Finally, when it’s ready, its very tempting be attacked. But its best to leave it for 1 hours for best taste.</p><p>Enjoy !</p><h3>Bibliography</h3><p><a href="https://foodviva.com/dessert-recipes/eggless-vanilla-sponge-cake/">https://foodviva.com/dessert-recipes/eggless-vanilla-sponge-cake/</a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=793d514dae81" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Algorithm for Good Masala Chai]]></title>
            <link>https://tandelbhavin.medium.com/algorithm-for-good-masala-chai-fc092e3b1d50?source=rss-230dc92fa3cc------2</link>
            <guid isPermaLink="false">https://medium.com/p/fc092e3b1d50</guid>
            <category><![CDATA[tea]]></category>
            <category><![CDATA[recipies]]></category>
            <category><![CDATA[chai]]></category>
            <category><![CDATA[indian]]></category>
            <category><![CDATA[algorithms]]></category>
            <dc:creator><![CDATA[Bhavin Tandel]]></dc:creator>
            <pubDate>Sun, 13 Sep 2020 22:54:05 GMT</pubDate>
            <atom:updated>2021-04-11T21:57:44.804Z</atom:updated>
            <content:encoded><![CDATA[<p>Algorithm in layman term are series of steps which helps you achieve the desired result. Similarly following are steps which I follow to make a perfect blend of Indian Masala tea. However, I have two following disclaimers.</p><blockquote>Disclaimer 1: I apologies if I misguide anyone with the title, don’t expect some polynomial function of ingredients to make tea.</blockquote><blockquote>Disclaimer 2: Every human is different and everyone have different kind of taste buds. So forgive if the recipe doesn’t turn out good for your taste.</blockquote><h3><strong>tl;dr</strong></h3><p>Masala tea = 1/3 cup W + 1/4 tsp CM+ 1/2 tsp T + 2/3 cup M + 1 tsp S + C</p><ul><li>W = Water</li><li>CM = Chai Masala/Ginger</li><li>T = Tea Leaves</li><li>M = Milk</li><li>S = Sugar</li><li>C = Constant but very dynamic</li></ul><h3>Assumptions</h3><ul><li>1 Cup = 200 ml</li><li>1 Tea spoon = 5 ml</li><li>Number of cups to make = 2</li></ul><h3>Ingredients</h3><ul><li>Milk</li><li>Water</li><li>Tea Leaves</li><li>Sugar</li><li>Tea Masala/Ginger</li></ul><p>With just above 5 items, you can get rolling.</p><h3>Algorithm</h3><h4>Step 1: Boil water</h4><p>One can make fully milk based chai, however having some water to boil that spices will definitely enhance the flavour.</p><p><strong>Total cups of water = 1/3 of total cups</strong>. In our case that will be 2/3 cup.</p><h4>Step 2: Spice it up</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/374/1*07NfuXJSDGKY7QJBKMi4sA.gif" /><figcaption>Tea Masala</figcaption></figure><p>This is what makes Indian Masala Tea so unique and it can be bought from any Indian Groceries Stores or can make yourself. It mostly consists of:</p><ul><li>Ginger Powder</li><li>Cardamom Powder</li><li>Cinnamon Powder</li><li>Grounded Cloves</li></ul><p>However, If you don’t have it in hand then you swap it with <strong>crushed ginger </strong>and <strong>cardamom.</strong></p><p>Let is boil it for 2–3 minutes until you can that feel of spices in the air.</p><p><strong>Spices = 1/4 teaspoon per cup. </strong>In our case that will be 1/2 tsp.</p><h4>Step 3: Color it up</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/270/1*XqoCP9YvwJxM3vvYFpdwUQ.gif" /><figcaption>Tea Leaves</figcaption></figure><p>This will give that black color and obviously the flavour you want in your drink. One can use any <strong>black tea </strong>of his/her choice.</p><p>Leave it to boil for 2–3 minutes to get that perfect blend. More you boil, stronger it gets.</p><p><strong>Tea Leaves = 1/2 teaspoon per cup. </strong>In our case that will be 1 tsp.</p><h4>Step 4: Fade that color</h4><p>Now comes the milk to add richness to your tea. For some people its quite odd to so much milk but Indian masala tea is suppose to be milky and full of flavours.</p><p><strong>Milk = 2/3 of Total cups. </strong>In our case that will be 4/3 cup = 1 + 1/3 cup.</p><h4>Step 5: Bit of sugar won’t hurt</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/270/1*0JKKiPeSPHA6_8N4ILVEyQ.gif" /></figure><p>Finally, losen some of your calories and add some sweetness to your chai. You can definitely avoid adding sugar, but I believe its hard to get that complete taste that your are looking in an Masala chai.</p><p>To get the best taste let tea shimmer for 4–5 minutes on low flames. This will heavily enhance the taste and you won’t get that milky taste in your tea. The more you boil, better it gets.</p><p><strong>Sugar = 1 teaspoon per cup. </strong>In our case that will be 2 tsp.</p><h4>Final Step: Secret Ingredients</h4><p>I know, I haven’t mentioned about the 6th step but I feel this is most important step for this recipe or in any activity of your life that is <strong>love</strong>. You need to have that love to make the tea, only then you will perfect it. And when you will achieve that you will not need this Algorithm.</p><p><strong>Secret Ingredients = Love</strong>. In our case its immeasurable.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=fc092e3b1d50" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Connect Jupyter Notebook to AWS Glue Endpoint]]></title>
            <link>https://medium.com/explore-ml/connect-jupyter-notebook-to-aws-glue-endpoint-f5fc78b4e00?source=rss-230dc92fa3cc------2</link>
            <guid isPermaLink="false">https://medium.com/p/f5fc78b4e00</guid>
            <category><![CDATA[aws-glue]]></category>
            <category><![CDATA[python]]></category>
            <category><![CDATA[jupyter-notebook]]></category>
            <category><![CDATA[spark]]></category>
            <category><![CDATA[data-engineering]]></category>
            <dc:creator><![CDATA[Bhavin Tandel]]></dc:creator>
            <pubDate>Sat, 04 Jul 2020 23:32:35 GMT</pubDate>
            <atom:updated>2020-07-04T23:42:30.797Z</atom:updated>
            <content:encoded><![CDATA[<p>If I am not wrong, then almost everyone in data engineering industry have heard of Apache Spark and if not (highly unlikely) then you are just one google search away for ample number resources. This post will revolve around Spark, AWS Glue, notebook and binding these tools for optimal results.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*UoOwz2cwTHzUSkZa2lTUqA.png" /><figcaption>The jupyter pipeline</figcaption></figure><h3>Introduction</h3><p>As Spark has been here for more than 10 years and industry have recognized its potential, thus many companies have orchestrated this framework and started providing it as service. Amazon is of it with offerings like <strong>AWS EMR</strong> and <strong>AWS Glue (Fully managed)</strong>. AWS has done most of the heavy lifting for us and provided fully managed spark service which can be scaled to more than TB of memory. It also provides <strong>Glue Endpoint</strong>, which is long running spark cluster and you can connect to its REPL or<strong> launch a zeppelin or jupyter notebook deployed in cloud</strong>.</p><p>However, as a developer wants to retain all the notebook he/she creates for future(I think it make sense to save anyway, losing something is hurtful), one has to implement some custom logic like saving in S3 or Github or downloading locally(not a good decision at all). It makes sense if we can just leverage our locally installed jupyter and connect to spark, we dont need to change much and plus more importantly it saves $$ (For running notebook in cloud).</p><h3>Set up</h3><p>We will need a:</p><ul><li><a href="https://docs.aws.amazon.com/glue/latest/dg/dev-endpoint-tutorial-prerequisites.html">Glue Endpoint</a></li><li><a href="https://jupyter.readthedocs.io/en/latest/install.html">Jupyter Notebook</a></li><li>sparkmagic</li></ul><pre>pip install jupyter<br>pip install sparkmagic<br>python -m jupyter nbextension enable --py --sys-prefix widgetsnbextension</pre><h4>Install spark kernels</h4><p>Get the spark magic location and ‘cd’ to that directory</p><pre># Get the sparkmagic location <br>pip show sparkmagic</pre><pre><strong># The location will be different for you<br></strong>cd ~/.local/share/virtualenvs/jupyter_to_glue-5i9d0kFW/lib/python3.6/site-packages</pre><pre># Install spark kernel for scala spark<br>jupyter-kernelspec install sparkmagic/kernels/sparkkernel</pre><pre># Install pyspark kernel for python spark<br>jupyter-kernelspec install sparkmagic/kernels/pysparkkernel</pre><p>Copy the <a href="https://github.com/jupyter-incubator/sparkmagic/blob/master/sparkmagic/example_config.json">config.json</a> into ~/.sparkmagic/config.json</p><h3>Connection to AWS Glue Endpoint</h3><p>I believe you would have launched glue endpoint with your <strong>public ssh key, </strong>it will take nearly 10 mins for it be available to connect.</p><p>After provisioning, get the details to connect to Glue Endpoint, for example something like</p><blockquote>ssh -i &lt;private-key.pem&gt; -vnNT -L :9007:169.254.76.1:9007 glue@ec2–3–70–179–133.compute-1.amazonaws.com</blockquote><p>However, we need to change the SSH tunnelling port to get tunnel the <a href="https://livy.apache.org/">Livy server</a> port (<strong>8998</strong>) which is a rest client to connect to spark cluster. And thus the final tunnelling looks like</p><pre>ssh -i ~/.ssh/id_rsa -vnNT -L 8998:169.254.76.1:8998 <a href="/service/mailto:glue@ec2-3-80-176-139.compute-1.amazonaws.com">glue@ec2–3–70–179–133.compute-1.amazonaws.com</a></pre><p>You can verify that livy server UI is available at <a href="http://localhost:8998/">localhost:8998</a></p><p>Leave the shell open for connection to be maintained or you can detach it using nohup or any tool of your choice.</p><h3>Launching Jupyter Server</h3><p>Now we launch jupyter server using command: <strong><em>jupyter notebook</em></strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*5mdRxpHeRJkHbNH9K23Oyw.png" /><figcaption>Jupyter Notebook</figcaption></figure><p>As you can see now you can find two new kernels <strong>PySpark</strong> (for python) and <strong>Spark</strong> (for scala).</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*NGdrq7mXZReyzQ_ch77pdw.png" /><figcaption>Spark Scala</figcaption></figure><p>As you can see we can access Glue Spark and also save the notebook when the endpoint is stopped.</p><p>Also, you can access livy server UI to monitor multiple sessions by going to:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*B33CPdFk-62arSW0NhtvWg.png" /><figcaption>livy server</figcaption></figure><h3>Benefits</h3><ul><li>No need to keep endpoint running as all codes are saved to your notebook.</li><li>No need to extract data out of S3 to your notebook to perform feature engineering.</li><li>Can share the spark cluster among the team, resulting in cost saving.</li><li>Can perform ad hoc logic of TB’s of data.</li><li>Endpoint can scale to <strong>50 DPU</strong> which is approximately (16 x 50 = <strong>800 GB memory</strong> and 4 x 50 = <strong>200 cores</strong>) 100 executors.</li></ul><h3><strong>Pricing</strong></h3><p>Glue is priced per DPU which stands for Data Processing Unit</p><p>$0.44/Hour/DPU, billed per second, with a 10-minute minimum.</p><h3>Bibliography</h3><ul><li><a href="https://blog.chezo.uno/livy-jupyter-notebook-sparkmagic-powerful-easy-notebook-for-data-scientist-a8b72345ea2d">https://blog.chezo.uno/livy-jupyter-notebook-sparkmagic-powerful-easy-notebook-for-data-scientist-a8b72345ea2d</a></li></ul><h4>P.S. Please delete the endpoint after your development or be ready for big fat bill from AWS</h4><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=f5fc78b4e00" width="1" height="1" alt=""><hr><p><a href="https://medium.com/explore-ml/connect-jupyter-notebook-to-aws-glue-endpoint-f5fc78b4e00">Connect Jupyter Notebook to AWS Glue Endpoint</a> was originally published in <a href="https://medium.com/explore-ml">Explore ML</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Exploring ML Tools - AWS Translate]]></title>
            <link>https://medium.com/explore-ml/exploring-ml-tools-aws-translate-d93442fab069?source=rss-230dc92fa3cc------2</link>
            <guid isPermaLink="false">https://medium.com/p/d93442fab069</guid>
            <category><![CDATA[exploreml]]></category>
            <category><![CDATA[aws-translate]]></category>
            <category><![CDATA[aws]]></category>
            <category><![CDATA[machine-learning]]></category>
            <dc:creator><![CDATA[Bhavin Tandel]]></dc:creator>
            <pubDate>Wed, 24 Jun 2020 11:59:15 GMT</pubDate>
            <atom:updated>2020-06-24T11:59:15.178Z</atom:updated>
            <content:encoded><![CDATA[<p>According to one of the surveys there are roughly 6500 languages spoken in whole world. However, I am sure that actual number is definitely more. So, it 99.9% impossible to learn all the languages in the world, but we as human are always keen to know the unknowns. So ideally if we have a tool which can translate foreign language to the one that we know, then it would really take the pain away in understanding someone with foreign language. So, in this blog we will focus on an AWS tool names <strong>Translate</strong> that does what it says, that is, it translates one language to other.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*MiRPG-v9IJcVWzfyHP13yA.png" /><figcaption>AWS Translate accross multipe devices</figcaption></figure><h3>Introduction</h3><p>AWS Translate is the neural machine translation service which deliver fast and more accurate result than traditional rule-based approach. It supports wide number of languages along with custom terminology which allows you to specific names, organization and the way it gets translated. It supports batch translation for bulk/short text data and real time translation for short data like chats, messages etc.</p><h3><em>How it Works?</em></h3><ul><li>Model is made up to two components encoder and decoder which has source and target associated with each, it uses attention mechanism to understand the context.</li><li>For automatic source language detection, you specify <strong>auto</strong> in source language.</li><li>Following is an example which converts English into Hindi using AWS cli.</li></ul><pre><br>aws translate translate-text \<br>            --source-language-code &quot;en&quot; \<br>            --target-language-code &quot;hi&quot; \<br>            --text &quot;hello, world, How are you&quot;<br><br>{<br>    &quot;TranslatedText&quot;: &quot;नमस्ते, दुनिया, आप कैसे हैं&quot;,<br>    &quot;SourceLanguageCode&quot;: &quot;en&quot;,<br>    &quot;TargetLanguageCode&quot;: &quot;hi&quot;<br>}</pre><h3><em>Features</em></h3><ul><li>At the time of writing it supported <a href="https://docs.aws.amazon.com/translate/latest/dg/what-is.html#language-pairs">55 languages</a>.</li><li>Automatic language identification.</li><li>SDK for mobile development is available which means you can directly utilize AWS Translate capabilities into your mobile application.</li><li>Two type of processing modes:<br> * Real-Time translation: You send short text in each call and you get the output.<br> * Asynchronous batch processing: You put large text in S3 and trigger the job to process the text.</li></ul><h3><em>Usecase</em></h3><ul><li>Multilingual chatbot.<br>* Website which supports multiple languages based on region.<br>* Sentiment analysis on product review from world wide users.</li></ul><h3><em>Usage</em></h3><p>You can perform translation via:</p><ul><li>AWS cli</li><li>AWS Console</li><li>AWS SDK</li><li>SDK for mobile development</li></ul><h3><em>Input</em></h3><ul><li>For Real-Time translation we pass short text in the call itself.</li><li>For Asynchronous Batch Processing, the documents is stored in S3 and must be in `.txt or .html` format.</li><li>Each document must be 1 MB or less in size and must not exceed 5 GB or 1 million in total documents.</li></ul><h3><em>Process</em></h3><ul><li>Executing real time translation of hHndi language into English in python.</li></ul><pre>import boto3<br>translate_client = boto3.client(&#39;translate&#39;, use_ssl=True)<br><br>result = translate_client.translate_text(Text=&quot;नमस्ते, दुनिया, आप कैसे हैं&quot;, SourceLanguageCode=&#39;hi&#39;, TargetLanguageCode=&#39;en&#39;)<br><br>print(&#39;TranslatedText: &#39; + result.get(&#39;TranslatedText&#39;))<br>print(&#39;SourceLanguageCode: &#39; + result.get(&#39;SourceLanguageCode&#39;))<br>print(&#39;TargetLanguageCode: &#39; + result.get(&#39;TargetLanguageCode&#39;))</pre><ul><li>Output</li></ul><pre>TranslatedText: Hello, World, How are you<br>SourceLanguageCode: hi<br>TargetLanguageCode: en</pre><h3><em>Output</em></h3><ul><li>If the query is real time then a json response is received as response from the call which is of following format,</li></ul><pre>{<br>&#39;TranslatedText&#39;: &#39;Pouvez-vous aider avec la connexion&#39;,<br> &#39;SourceLanguageCode&#39;: &#39;en&#39;,<br> &#39;TargetLanguageCode&#39;: &#39;fr&#39;,<br> &#39;ResponseMetadata&#39;: {&#39;RequestId&#39;: &#39;8468by00-985d-4827-a266-15ece5fb8645&#39;,<br>  &#39;HTTPStatusCode&#39;: 200,<br>  &#39;HTTPHeaders&#39;: {&#39;x-amzn-requestid&#39;: &#39;8468by00-985d-4827-a266-15ece5fb8645&#39;,<br>   &#39;cache-control&#39;: &#39;no-cache&#39;,<br>   &#39;content-type&#39;: &#39;application/x-amz-json-1.1&#39;,<br>   &#39;content-length&#39;: &#39;108&#39;,<br>   &#39;date&#39;: &#39;Tue, 23 Jun 2020 20:13:13 GMT&#39;},<br>  &#39;RetryAttempts&#39;: 0}<br>}</pre><h3><em>Findings</em></h3><ul><li>If the input text couldn&#39;t convert to target language then it just returns the input text.</li></ul><h3><em>Errors</em></h3><ul><li>When trying to execute batch operation StartTextTranslationJob api, I received **NO_READ_ACCESS_TO_S3** error despite of role having full access to s3 on all the resources.</li></ul><h3><em>Pricing</em></h3><ul><li>No upfront cost</li><li>Free tier: <br> * 2 million characters per month for 12 months</li><li>On demand:<br> * $15.00 per million characters<br> * Detailed list can be found <a href="https://aws.amazon.com/translate/pricing/">here</a></li></ul><h3><em>Bibliography</em></h3><ul><li><a href="https://blog.busuu.com/most-spoken-languages-in-the-world/">https://blog.busuu.com/most-spoken-languages-in-the-world/</a></li><li><a href="https://docs.aws.amazon.com/translate/latest/dg/what-is.html">AWS Translate doc</a></li><li><a href="https://aws.amazon.com/blogs/machine-learning/amazon-translate-now-offers-113-new-language-pairs/">https://aws.amazon.com/blogs/machine-learning/amazon-translate-now-offers-113-new-language-pairs/</a></li></ul><blockquote>P.S. Will update the blog when the batch job testing is completed.</blockquote><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=d93442fab069" width="1" height="1" alt=""><hr><p><a href="https://medium.com/explore-ml/exploring-ml-tools-aws-translate-d93442fab069">Exploring ML Tools - AWS Translate</a> was originally published in <a href="https://medium.com/explore-ml">Explore ML</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Exploring ML Tools — AWS Transcribe]]></title>
            <link>https://medium.com/explore-ml/exploring-ml-tools-aws-transcribe-9887f2eb1b87?source=rss-230dc92fa3cc------2</link>
            <guid isPermaLink="false">https://medium.com/p/9887f2eb1b87</guid>
            <category><![CDATA[exploreml]]></category>
            <category><![CDATA[aws-transcribe]]></category>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[speech-analytics]]></category>
            <category><![CDATA[aws]]></category>
            <dc:creator><![CDATA[Bhavin Tandel]]></dc:creator>
            <pubDate>Fri, 29 May 2020 17:48:23 GMT</pubDate>
            <atom:updated>2020-05-29T17:48:23.459Z</atom:updated>
            <content:encoded><![CDATA[<h3>Exploring ML Tools — AWS Transcribe</h3><p>Audio is part of our life which exists in many forms like, voice messages, podcasts, songs, lectures, recorded conversations etc. These files are really hard to process by computer and so it just stays in magnetic disk or expensive disk and never actually used proactively unless its needed. However, machines have become very intelligent in process texts. And thus, so we can convert audio to text and put machine to work for getting insight.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/638/1*nEdZQeKy7dsLN1VxXzhZsQ.jpeg" /><figcaption>Pipeline for AWS Transcribe</figcaption></figure><h3>Introduction</h3><p>There have pretty good research in speech-to-text conversion with model like Hidden Markow Models(HMM), Dynamic Time Warping and many neural networks. AWS have provide fully managed service called AWS Transcribe which can be used to transcribe audio, video files and also transcribe medical files.</p><h3>How it Works?</h3><ul><li>Input can be provided to AWS transcribe in following ways:</li></ul><p><em>Files in audio format using StartTranscriptionJob.</em></p><p><em>Bi-directional HTTP/2 stream using StartStreamTranscription.</em></p><p><em>Websocket protocol.</em></p><ul><li>It returns transcription with highest level of confidence, however, you can additional transcribes by setting <em>ShowAlternatives</em> to true and MaxAlternatives to your required number.</li><li>For speaker identification(diarization), the result is best when the number of speaker set is same as actual number of speakers in the clip.</li><li>For Channel Identification, it split the file into multiple audio files and provide multiple transcribe along with combined file. The overlap audios are transcribe based on audio start time. This can be done by setting <em>ChannelIdentification</em> flag.</li><li>You can provide custom vocabularies which can be of upto 50 Kb size. The data can be in list format or table format.</li><li>Automatically redacts sensitive PII data which includes Bank Account Number, Credit Card details, Phone numbers, etc. It will replace it with <strong>[PII]</strong> tag. We can set <em>RedactionType</em> flag to <strong>PII</strong> under ContentRedaction.</li><li>You can also enable Queuing for the transcribing as there are limited slots available for job and when queuing is enabled then it will process as soon as slots are available in FIFO order.</li></ul><h3>Features</h3><ul><li>Support transcribing of many languages. Detail list can be found <a href="https://docs.aws.amazon.com/transcribe/latest/dg/what-is-transcribe.html">here</a>.</li><li>Identifies speakers in the audio.</li><li>Transcribe separate channels like in call centre recording, it can give two separate files, one for representative and other of customer.</li><li>Streaming audio transcribing.</li><li>Custom Vocabulary</li></ul><h3>Usecase</h3><ul><li>Generate captioning of video recording.</li><li>Generating subtitles of the video.</li><li>Call centre recording analysis.</li></ul><h3>Usage</h3><p>The AWS transcribe can be used via:</p><ul><li>AWS cli</li><li>AWS Console</li><li>AWS SDK</li></ul><h4>Input</h4><p>Accepted input format is FLAC, MP3, MP4, or WAV file format. Should be less than 4 hours in length or 2 GB in size. Latest updated limits can be found <a href="https://docs.aws.amazon.com/transcribe/latest/dg/input.html">here</a>.</p><p>Transcribing Numbers:</p><ul><li>There are set of rules for transcribing the numbers which can be found <a href="https://docs.aws.amazon.com/transcribe/latest/dg/how-numbers.html">here</a>.</li><li>In few languages, the numbers are transcribed into digits. In others, the numbers are converted into word.</li></ul><h4>Process</h4><p>Following steps will walk through transcribing steps via python SDK.</p><ul><li>Initialize aws transcribe client</li></ul><pre>import boto3<br>transcribe_client = boto3.client(&#39;transcribe&#39;, region_name=&#39;eu-west-1&#39;)</pre><ul><li>Start Transcribing job</li></ul><pre>short_job_uri = &quot;https://s3.eu-west-1.amazonaws.com/exploring-ml-tools/aws-transcribe/assets/VodaFoneCallCenter.mp3&quot;<br># Start Transcribing job<br>job_name = &quot;TEST_JOB_With_VodaFone&quot;<br>trascribe_response = transcribe_client.start_transcription_job(<br>    TranscriptionJobName=job_name,<br>    Media={&#39;MediaFileUri&#39;: short_job_uri},<br>    MediaFormat=&#39;mp3&#39;,<br>    LanguageCode=&#39;en-IN&#39;,<br>    Settings={<br>        &#39;ShowAlternatives&#39;: True,<br>        &#39;MaxAlternatives&#39;: 6<br>    }<br>)</pre><ul><li>Get Transcribing job details</li></ul><pre>import time<br><br># Get Transcribing job<br>while True:<br>    status = transcribe_client.get_transcription_job(TranscriptionJobName=job_name)<br>    if status[&#39;TranscriptionJob&#39;][&#39;TranscriptionJobStatus&#39;] in [&#39;COMPLETED&#39;, &#39;FAILED&#39;]:<br>        break<br>    print(&quot;Not ready yet...&quot;)<br>    time.sleep(5)<br>print(status)</pre><ul><li>Downloading transcript</li></ul><pre>if status[&#39;TranscriptionJob&#39;][&#39;TranscriptionJobStatus&#39;] == &#39;COMPLETED&#39;:<br>    uri = status[&#39;TranscriptionJob&#39;][&#39;Transcript&#39;][&#39;TranscriptFileUri&#39;]<br><br>response = requests.get(uri)<br><br>if response.status_code == 200:<br>    json_data = json.loads(response.text)<br><br>json_data[&#39;results&#39;][&#39;transcripts&#39;]<br># Output &gt;&gt;&gt;[{&#39;transcript&#39;: &#39;his taste for a phone call centre.&#39;}]</pre><p>If you dont specify output bucket then transcripts will be deleted after expiration of job that is in 90 days.</p><h4>Output</h4><p>The output files can be stored in your output bucket if specified in job or by default transcribing job stores the data up to 90 days. The output will contain alternatives as well which show other output with various confidence level.</p><p>Also, if we have enabled redaction then the redacted file will replace the PII value with [PII] tag.</p><h3>Cleanup</h3><ul><li>Deleting Transcribing job</li></ul><pre># Delete Transcribing job response = transcribe_client.delete_transcription_job( TranscriptionJobName = &#39;TEST_JOB&#39; )</pre><p>Transcribing job expires in 90 days.</p><h3>Findings</h3><ul><li>It takes approximately 1.5 minutes to transcribe 3-second-long audio and 3.5 mins to transcribe 27 mins audio.</li><li>Good transcribing of UK and US English.</li><li>When transcribing english, the only language supported to transcribe using redaction enabled was en-US, for other you will need to enable it on your AWS account.</li></ul><h3>Pricing</h3><ul><li>No upfront cost</li><li>Free tier: 60 mins/month for 12 months</li><li>On demand: Detailed list can be found <a href="https://aws.amazon.com/transcribe/pricing/">here</a></li></ul><h3>Bibliography</h3><ul><li><a href="https://boto3.amazonaws.com/v1/documentation/api/latest/reference/services/transcribe.html">AWS Python SDK Client</a></li><li><a href="https://www.kaggle.com/wiradkp/mini-speech-diarization">Kaggle Data</a></li><li><a href="https://docs.aws.amazon.com/transcribe/latest/dg/what-is-transcribe.html">AWS Docs</a></li><li><a href="https://www.slideshare.net/AmazonWebServices/automate-for-efficiency-with-amazon-transcribe-and-amazon-translate">AWS Transcribe slide</a></li></ul><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=9887f2eb1b87" width="1" height="1" alt=""><hr><p><a href="https://medium.com/explore-ml/exploring-ml-tools-aws-transcribe-9887f2eb1b87">Exploring ML Tools — AWS Transcribe</a> was originally published in <a href="https://medium.com/explore-ml">Explore ML</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Exploring ML Tools — Amazon Forecast]]></title>
            <link>https://medium.com/explore-ml/exploring-ml-tools-amazon-forecast-a1791b5c9f66?source=rss-230dc92fa3cc------2</link>
            <guid isPermaLink="false">https://medium.com/p/a1791b5c9f66</guid>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[covid19]]></category>
            <category><![CDATA[forecasting]]></category>
            <category><![CDATA[aws]]></category>
            <dc:creator><![CDATA[Bhavin Tandel]]></dc:creator>
            <pubDate>Thu, 21 May 2020 07:26:35 GMT</pubDate>
            <atom:updated>2020-05-21T07:38:12.720Z</atom:updated>
            <content:encoded><![CDATA[<h3>Exploring ML Tools — Amazon Forecast</h3><p>This post will focus on forecasting services offered by Amazon Web Service called as <strong>AWS Forecast</strong>. Companies have been performing forecast on business outcome since long time spanning from financial market forecast to retail product demand forecast. Earlier Statistical methods and Advanced Mathematics have been used to predict the future outcomes, however, with advancement of machine learning and available of big data, we are able to use deep learning techniques to make accurate predictions.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*GkQ9x50xwuUMRuUTMuRN_g.png" /><figcaption>Forecast Workflow</figcaption></figure><h3>Introduction</h3><p>In basic, forecasting is the method of predicting future based on past and present data. It consists of a <strong>model </strong>to which we feed historical time-based data <strong>y</strong> and we get future value of <strong>y</strong>. For example, we have model to predict future value of a stock then, we would feed historical data of stock to it and we will get the predicted value of the stock.</p><p>AWS Forecast is a managed service which provides the platform to users for running the forecasting on their data without the need to maintain the complex ML infrastructure. It is based on DeepAR+ algorithm which is supervised algorithm for forecasting one-dimensional time series using Recurrent Neural Networks. It involves datasets which is used to train predictors and generate forecasts.</p><h3><strong>How it Works ?</strong></h3><p>It consists of,</p><p><strong>Dataset and Dataset Groups</strong></p><ul><li>You create Dataset Group and a Forecasting domain(Retail/Inventory Planning/Custom/..) based on your usecase. The complete list can be found <a href="https://docs.aws.amazon.com/forecast/latest/dg/howitworks-domains-ds-types.html">here</a>.</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*rr3vPesYxlU84QFnG1TAyA.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*CVxN8KHzf1AB8AZN2PY63g.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*DirJ0j7CQtltJA7xuoEn-g.png" /><figcaption>Dataset Group Creation, Target Time Series Dataset Creation, Dataset import job</figcaption></figure><ul><li>Each dataset group can have three datasets, one of each type:</li></ul><p><em>* target time series(required)</em></p><p><em>* related time series(optional)</em></p><p><em>* item metadata(optional) — only used when algorithm used is DeepAR+</em></p><ul><li>You will have to select the frequency of your data. For example, in energy meter, we might take reading every 30 minutes.</li><li>Following columns have to be mandatory present in target and related time series:</li></ul><p><em>* `timestamp` (must be of timestamp type)</em></p><p><em>* `item_id` (must be of string type)</em></p><p><em>* Data type of all other columns must be of type string.</em></p><ul><li>Target time series must also contain `target_value` column, which is the column you are planning to forecast.</li></ul><p><strong>Predictors</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*EcZN3ElifoGdrd0NCRg5Vg.png" /><figcaption>creating predictor</figcaption></figure><ul><li>It creates trained model which is called as <strong>predictors</strong>.</li><li>Users select Forecast horizon, which indicates how far you want to predict.</li><li>One can manually select algorithm or select AutoML feature for automatic algorithm selection.</li><li>There are some additional features available for fine tuning the forecasts.</li></ul><p><strong>Forecasts</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*2pQ-Bily_3hAm7XT_4Pb0g.png" /><figcaption>Creating Forecasts</figcaption></figure><ul><li>We can create forecast on the predictor which is created above.</li><li>You can select up to 5 quantile value including mean. You can choose based on your business need which can be either under-forecasting or over-forecasting. For example, 0.05 quantile means, the true data is expected to be lower than predicted data 5% of time. If we take example of predicting pandemic and number of cases, we would want quantile to be 0.99 for predicting hospital bed, because we don’t want to under-forecast the resources in this scenario.</li></ul><p><strong>Forecast Lookup</strong></p><ul><li>We can lookup in the console itself as shown in following screen shot.</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*DEe6LfA68mhFJxbZEnLGuw.png" /><figcaption>Looking up the forecasts</figcaption></figure><h3>Features</h3><ul><li>Automatically identifies key attributes for forecasting.</li><li>Manual Algorithm selection: DeepAR+, ARIMA, Prophet, Exponential Smoothing (ETS), Non-Parametric Time Series (NPTS). The detail list can be found <a href="https://docs.aws.amazon.com/forecast/latest/dg/aws-forecast-choosing-recipes.html">here</a>.</li><li>Provide AutoML option for model training which automates complex tasks(algorithm selection, hyperparameter tuning, etc).</li><li>Automatically fill missing values for target and related time series.</li></ul><h3>Usecase</h3><p>Following are some usecases for AWS Forecast:</p><ul><li>Estimating product demand.</li><li>Forecasting weather.</li><li>Predicting web traffic.</li><li>Estimating pandemic cases.</li></ul><h3><strong>Usage</strong></h3><p>The forecast can be used via AWS cli, Console or SDK. Above we saw how one can perform forecast from console. Here we will see how we can use sdk to perform the forecast job</p><h4><strong>Input</strong></h4><p>We will be using data of covid19 cases available on <a href="https://www.kaggle.com/c/covid19-global-forecasting-week-5/data">kaggle</a>.</p><p>Following is the screenshot of sample data.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*YCRwL-VaJRY_XqBJ2HdRSQ.png" /><figcaption>sample-data</figcaption></figure><p>We will be forecasting Confirmed Cases for various countries which is named as <strong>item_id</strong>.</p><h4>Process</h4><p>Following steps will describe the forecasting using python sdk.</p><ol><li>Initializing AWS forecast client</li><li>Creating Dataset group, we will be creating dataset group with name as <strong>covid19_week5</strong>.</li><li>Creating Target time series dataset</li><li>Import data to the dataset, we will import the data from s3 bucket into the dataset which should match to schema of it. And role should have get object permission on the bucket.</li><li>Link dataset to dataset group</li><li>Creating predictor</li><li>Creating forecast</li><li>Export the forecast data to S3 bucket</li></ol><p>Following gist describes the above steps.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/6272fa61303c8d711d4b4a020108d2cf/href">https://medium.com/media/6272fa61303c8d711d4b4a020108d2cf/href</a></iframe><p>After completion of forecasting job, we can export the forecast value to S3 bucket and then utilize it based on our requirements like directly querying from Athena or loading it into data mart, etc.</p><h4><strong>Output</strong></h4><p>We can directly make a look on forecast in console as shown below. As you can see the forecast has been generated for our supplied quantile.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*r_mVIR2qCiJuku9XwnN5dw.png" /><figcaption>Forecast Lookup</figcaption></figure><p>The output is the csv files which can be exported to s3 bucket and then utilized.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*AEbIUMqi9iY7_Dq2UL1syA.png" /><figcaption>Generated forecast files</figcaption></figure><p><strong>For clean script, one can follow the snippet over </strong><a href="https://bhavintandel.github.io/resources/data_science/exploring-ml-tools/aws-forecast/exploring-ml-tools-aws-forecast.html#cleanup"><strong>here</strong></a><strong>.</strong></p><h3><strong>Findings</strong></h3><ul><li>Easy to use tool to generate forecast.</li><li>Can auto fill the missing values.</li></ul><h3><strong>Pricing</strong></h3><ul><li>No upfront cost</li><li>Forecast are billed in unit of 1000</li></ul><p>Free tier:</p><ul><li>Generate Forecasts — Up to 10K time series forecast/month for first 2 month</li><li>Data Storage — Up to 10/month for first 2 months</li><li>Training hours — Up to 10 hrs/month for first 2 months</li></ul><p>On demand:</p><ul><li>Generate Forecasts — $0.60 for 1000 forecasts</li><li>Data Storage — $0.088 per GB</li><li>Training hours — $0.24 per hour</li><li>Detailed example of pricing can be found and <a href="https://aws.amazon.com/forecast/pricing/">Amazon Forecast pricing page</a>.</li></ul><h3><strong>Bibliography</strong></h3><ul><li><a href="https://aws.amazon.com/">AWS</a></li><li><a href="https://boto3.amazonaws.com/v1/documentation/api/latest/reference/services/forecast.html">Boto3 forecast client</a></li></ul><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=a1791b5c9f66" width="1" height="1" alt=""><hr><p><a href="https://medium.com/explore-ml/exploring-ml-tools-amazon-forecast-a1791b5c9f66">Exploring ML Tools — Amazon Forecast</a> was originally published in <a href="https://medium.com/explore-ml">Explore ML</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Exploring ML Tools — Amazon Textract]]></title>
            <link>https://medium.com/explore-ml/exploring-ml-tools-amazon-textract-73495f310efc?source=rss-230dc92fa3cc------2</link>
            <guid isPermaLink="false">https://medium.com/p/73495f310efc</guid>
            <category><![CDATA[ocr]]></category>
            <category><![CDATA[aws]]></category>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[aws-textract]]></category>
            <dc:creator><![CDATA[Bhavin Tandel]]></dc:creator>
            <pubDate>Sat, 16 May 2020 08:29:47 GMT</pubDate>
            <atom:updated>2020-05-16T13:36:19.644Z</atom:updated>
            <content:encoded><![CDATA[<h3>Exploring ML Tools — <strong>Amazon Textract</strong></h3><p>Documents have been part of our lives since very long time, however, its not always so easy to find something in document. With the new advances in machine learning we can actually digitize our document and thus put it to good use.</p><h3>Introduction</h3><p>In this post, we will be exploring <strong>Amazon Textract </strong>which has the capabilities to extract text and data from documents. It uses Optical Character Recognition(OCR) and augmented AI to detects the document layout and key element from the it.</p><p>Amazon textract has been trained on tens of millions of documents including invoices, contracts, receipts, insurance claims, policy documents, etc.</p><h3>Features</h3><ul><li>Key-value pair extraction</li><li>Table extraction</li><li>Bounding Box for all the extracted result</li><li>Confidence score for every extracted result</li><li>It can handle multi page document</li></ul><h3>Usecase</h3><p>Following is one of the examples of how textract can be used to process large scale documents using complementary AWS services.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*pb7h8reZkG8MjV1e.png" /><figcaption>Large scale document processing with Amazon Textract</figcaption></figure><h3>Usage</h3><p>The service can be used directly from aws console or using api in your preferred language like java, python, javascript, etc. We will be using aws python sdk — boto3 to interact with the service.</p><h4>Input</h4><p>It accepts images of format (JPEG/PNG) and PDF’s.</p><p>For Images, we can perform synchronous or asynchronous call with reading it as bytes or passing s3 location.</p><p>To process PDF, only asynchronous call is supported via s3 location.</p><p>Initialize the textract client in python, if you are planning to run extract job on files under s3 bucket then, region_name should be same as the location on data.</p><figure><img alt="Input image for detecting and analyzing text in image. This is a receipt from a shop in switzerland" src="https://cdn-images-1.medium.com/max/1024/1*4y5VmId10MeLh2BqDx5Mkg.jpeg" /><figcaption>Input image for detecting and analyzing text in image</figcaption></figure><h4>Output</h4><p>Detecting text in Image</p><pre>import boto3</pre><pre>textract_client = boto3.client(&#39;textract&#39;, region_name=&#39;eu-west-1&#39;)</pre><pre># Reading image file<br>with open(&#39;assets/IMG_2093.JPG&#39;, &#39;rb&#39;) as f:<br>    image = f.read()</pre><pre># Detecting text in Image<br>img_response_1 = textract_client.detect_document_text(<br>    Document={<br>        &#39;Bytes&#39;: image<br>    })</pre><p>The response contains lot of information regarding the analysis. First of all we get <em>DocumentMetadata </em>which has information like page number of the analysis. And we get many Blocks which can be one of the following value ‘KEY_VALUE_SET’|’PAGE’|’LINE’|’WORD’|’TABLE’|’CELL’|’SELECTION_ELEMENT’.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/914/1*0_OgxP_X4TSnvJxQRQLKKw.png" /></figure><p>To extract text from the output:</p><pre>for block in img_response_1[&#39;Blocks&#39;]:<br>    if (block[&#39;BlockType&#39;] == &#39;LINE&#39;):<br>        print(block[&#39;Text&#39;])</pre><blockquote>Gate Gourmet Switzerland GmbH<br>(d/b/a Gate Retail Onboard)<br>Flight Flight Date 2072 24/05/2017 Sector MUC-LTN 19:25:00<br>Bar Set 7454<br>Transaction Type SALE<br>Transaction Date e:24/05/2017<br>Transaction Time 10:25 AM<br>Transaction 5b42b840b1c<br>Item<br>Price Oty<br>Water Still<br>1.80<br>3<br>Coffee Reg<br>2.50<br>I<br>SubTotal<br>4.30<br>Net Subtotal (GBP)<br>4.30<br>Cash (EUR)<br>5.50<br>Terinal 00-02-78-b0-e6-ef 00-00<br>CUSTOMER COPY<br>Please retain for Vour records</blockquote><p>With the above information we also get the actual location of that on a page with Bounding box information. Which can we used to create the indexed search engine of our documents and thus make it searchable on word level.</p><p>Following are some example on running textract on S3 object</p><pre># Analyzing text in Image<br>img_response_2 = textract_client.analyze_document(<br>    Document={<br>        &#39;Bytes&#39;: image<br>    },<br>    FeatureTypes=[<br>        &#39;TABLES&#39;,<br>    ])</pre><pre># Asynchronous call to extract text from pdf<br>asset_bucket = &quot;&lt;bucket_name&gt;&quot;<br>pdf_prefix = &quot;amazon-textract/assets/pdf/receipt_uber.pdf&quot;</pre><pre>pdf_response_1 = textract_client.start_document_text_detection(<br>        DocumentLocation={<br>            &#39;S3Object&#39;: {<br>                &#39;Bucket&#39;: asset_bucket,<br>                &#39;Name&#39;: pdf_prefix<br>            }<br>        })</pre><pre>job_id_1 = pdf_response_1[&#39;JobId&#39;]</pre><pre>## check the progress<br>response = textract_client.get_document_text_detection(<br>    JobId=job_id<br>)</pre><pre>if response[&#39;JobStatus&#39;] == &#39;IN_PROGRESS&#39;:<br>    print(&quot;Job is still in progress&quot;)<br>else:<br>    print(&quot;Extraction job completed&quot;)<br>    print(response)</pre><h3>Findings</h3><p>Some key findings are:</p><ul><li>It returns information of bounding box from where the text has been extracted.</li><li>It is not angle invariant, that is, document or image to be analyze must be aligned vertically.</li></ul><h3>Pricing</h3><ul><li>No upfront cost</li><li>Free tier: Detecting Text — 1000 pages/month for first 3 months, Analyzing Document — 100 pages/month for first 3 months</li><li>Detecting Text: First 1 Million pages — $0.0015/page — $1.5/1000 pages, Over 1 Million pages — $0.0006/page — $0.6/1000 pages</li><li>The other details for analyze document api can be found at: <a href="https://aws.amazon.com/textract/pricing/">https://aws.amazon.com/textract/pricing/</a></li></ul><h3>Bibliography</h3><ul><li><a href="https://github.com/aws-samples/amazon-textract-serverless-large-scale-document-processing">amazon-textract-serverless-large-scale-document-processing</a></li><li><a href="https://aws.amazon.com/">https://aws.amazon.com/</a></li><li><a href="https://aws.amazon.com/textract/pricing/">https://aws.amazon.com/textract/pricing/</a></li></ul><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=73495f310efc" width="1" height="1" alt=""><hr><p><a href="https://medium.com/explore-ml/exploring-ml-tools-amazon-textract-73495f310efc">Exploring ML Tools — Amazon Textract</a> was originally published in <a href="https://medium.com/explore-ml">Explore ML</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Packaging External libs for AWS Glue]]></title>
            <link>https://tandelbhavin.medium.com/packaging-external-libs-for-aws-glue-574963a0f02d?source=rss-230dc92fa3cc------2</link>
            <guid isPermaLink="false">https://medium.com/p/574963a0f02d</guid>
            <category><![CDATA[aws-glue]]></category>
            <category><![CDATA[pyspark]]></category>
            <category><![CDATA[package-management]]></category>
            <category><![CDATA[python]]></category>
            <dc:creator><![CDATA[Bhavin Tandel]]></dc:creator>
            <pubDate>Mon, 20 Apr 2020 09:01:10 GMT</pubDate>
            <atom:updated>2020-04-20T09:01:10.046Z</atom:updated>
            <content:encoded><![CDATA[<p>Cloud computing is warehouse for vast amount of computing power but sometimes unlocking the treasure requires us to solve some riddles. Here I will discuss one of such riddle, that is, using external libraries in AWS Glue.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/400/1*Q26gqWRhSGNO5LJIv22qiA.png" /></figure><p>AWS Glue is managed service which is tightly integrate with other AWS native tools. It allows us to build ETL pipelines using either python or spark framework. As you must be aware that python is popular just because if vast repository of libraries which a user can use to expidite the development. However when we deal with manage service, we something get limited in our creativity.</p><h3>tl;dr</h3><ul><li>Clone the repository <a href="https://github.com/bhavintandel/py-packager/tree/master">https://github.com/bhavintandel/py-packager/tree/master</a></li><li>Update the rquirements.txt and execute make build</li></ul><h3>PreRequisite</h3><p>We need to create a project to generate the dependency file. Following files need to be created:</p><ul><li>setup.py</li><li>requirements.txt</li></ul><p>Follow <a href="https://github.com/bhavintandel/py-packager/tree/master">py-packager</a> to generate the above files. Update the project name and version name for final dependency file name in <strong>setup.py</strong>. Also, add the required package name under <strong>requirements.txt</strong></p><h3>Python packaging</h3><p>Glue <strong>python-shell </strong>job provide with 1 to 16 GB of memory to execute your python code. If we need external libs then we need to provide egg files as external libraries to the Glue. Execute the following command to generate the dependencies file,</p><pre>python setup.py bdist_egg</pre><p>This will generate file under dist/ folder and one can use it in python-shell job</p><h3>Pyspark packaging</h3><p>Glue <strong>pyspark </strong>job provides vast amount of computing power and for external modules we need to provide dependencies in zip format. We need to execute following commands to generate the zip for pyspark,</p><pre>VERSION=$(shell grep &quot;VERSION = [0-9.]<em>*</em>&quot; setup.py | cut -d\&quot; -f2)<br>PROJECT_NAME=$(shell grep &quot;PACKAGE_NAME = <em>*</em>&quot; setup.py | cut -d\&quot; -f2)</pre><pre>mkdir -p libs dist<br>pip install -r requirements.txt -t libs<br>(cd libs; zip -r ../dist/${PROJECT_NAME}-${VERSION}.zip *)<br>rm -rf libs/</pre><p>This will generate file under dist/ folder which can be used for pyspark job.</p><p><strong>P.S. python module which depends on c will not be exeutable under pyspark environment.</strong></p><p>For using it AWS Glue, One need to upload the files to S3 and add the S3 path in <strong>Python library path </strong>under job details.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=574963a0f02d" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Running GPU like a swag on windows]]></title>
            <link>https://becominghuman.ai/running-gpu-like-a-swag-on-windows-6ae716ef42b7?source=rss-230dc92fa3cc------2</link>
            <guid isPermaLink="false">https://medium.com/p/6ae716ef42b7</guid>
            <category><![CDATA[artificial-intelligence]]></category>
            <category><![CDATA[deep-learning]]></category>
            <category><![CDATA[tensorflow]]></category>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[gpu]]></category>
            <dc:creator><![CDATA[Bhavin Tandel]]></dc:creator>
            <pubDate>Sun, 24 Sep 2017 12:52:37 GMT</pubDate>
            <atom:updated>2017-09-25T16:12:14.117Z</atom:updated>
            <content:encoded><![CDATA[<p>With rapid evolution of computing industry, it has become prime importance for data fanatics to be updated with latest technology. Graphic cards were only famous among gamers until innovator decided to use it for munging the data. There are various companies which provide GPU however, NVIDIA is the one who is ruling the game in making stories from data.</p><p>Cloud providers have made really easy an innovator to deploy GPU-enabled machine with a click. This will maybe small but definitely incur cost which can be disastrous if one forgets to shut the machine down (Been there !). So I decided to make good use of prebuild NVIDIA GPU drivers which comes along with some windows laptop.</p><p>Let me aware you that this was bit painful at first as lot of components are involve but if you stick in there then you will fly like a unicorn.</p><h3>Requirements</h3><p><strong>Hardware:</strong></p><ul><li>Windows PC</li><li>NVIDIA Graphics card</li></ul><p><strong>Software:</strong></p><ol><li>CUDA® Toolkit 8.0</li><li>cuDNN — deep learning drivers</li><li>Visual studio C++ drivers</li></ol><h3>Installation</h3><ol><li><strong>CUDA Toolkit</strong></li></ol><p>Following <a href="http://docs.nvidia.com/cuda/cuda-installation-guide-microsoft-windows/">link</a>, can be used to find is your machine is compatible.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/811/1*T_5opGBYzuYyz3JR2SXoIQ.png" /></figure><p>To download cuda driver go to following link: <a href="https://developer.nvidia.com/cuda-downloads">https://developer.nvidia.com/cuda-downloads</a></p><p>Select the version respective to your machine as shown in picture(left).</p><p><strong>2. cuDNN</strong></p><p>Please follow the link(<a href="https://developer.nvidia.com/cuda-gpus">https://developer.nvidia.com/cuda-gpus</a>) to find the compatible cuDNN with respective to your GPU. For instance, in my case its 5.0 due to NVIDIA GEFORCE 840M model.</p><p>Now, lets download the <a href="https://developer.nvidia.com/cudnn">deep learning library offered by CUDA</a>, However, you need to have NVIDIA account.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/907/1*llflVBvWOyeNCpepoM3QzQ.png" /></figure><p>Download the files and extract in CUDA toolkit location that is,</p><blockquote>`C:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v8.0`</blockquote><p>copy and replace the files</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*ZJ5rajzasJTezSo46ywMOw.png" /><figcaption>Extract the cuDNN library to CUDA driver</figcaption></figure><p><strong>3. Visual studio C++ drivers</strong></p><p>Visual studio library is also required and can be found at <a href="https://www.microsoft.com/en-gb/download/developer-tools.aspx">Microsoft developer-tools</a>.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/967/1*eoVmd20dssIR_3eD8jJjgA.png" /><figcaption>Installing visual studio C++</figcaption></figure><h3>Set environment variable</h3><p>Confirm that CUDA_HOME variable is set after installation of CUDA toolkit. If it’s not set the we need to do it manually,</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/609/1*Am9EbN0ro_7eCwJRUubLkQ.png" /><figcaption>set CUDA_HOME variable</figcaption></figure><p>And set the path variable as,</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/776/1*zq0wPPqntKCDrgSHFIP3nA.png" /><figcaption>set PATH variable</figcaption></figure><h3>Install tensorflow</h3><p>One can use “native” pip or “anaconda” to install tensorflow. However, it makes easy to manage libraries with anaconda.</p><p><strong>Download and install anaconda</strong></p><p><a href="https://www.anaconda.com/download/">Anaconda</a></p><p><strong>Create a conda environment</strong></p><blockquote>conda create -n tensorflow python=3.5</blockquote><p><strong>Activate environment</strong></p><blockquote>activate tensorflow</blockquote><p><strong>Install tensorflow-gpu</strong></p><blockquote>pip install — ignore-installed — upgrade tensorflow-gpu</blockquote><p><strong>Run tensorflow</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*fF5MtZnJB3SDt2tApBfIIg.png" /><figcaption>run tensorflow</figcaption></figure><h3>Conclusion</h3><p>Its not always straight forward to tweak you personal computer to achieve awesome stuff. However, if you could then definitely you will save some bucks.</p><iframe src="https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Fupscri.be%2F8f5f8b%3Fas_embed%3Dtrue&amp;dntp=1&amp;url=https%3A%2F%2Fupscri.be%2F8f5f8b&amp;key=a19fcc184b9711e1b4764040d3dc5c07&amp;type=text%2Fhtml&amp;schema=upscri" width="800" height="400" frameborder="0" scrolling="no"><a href="https://medium.com/media/c43026df6fee7cdb1aab8aaf916125ea/href">https://medium.com/media/c43026df6fee7cdb1aab8aaf916125ea/href</a></iframe><figure><a href="https://becominghuman.ai/artificial-intelligence-communities-c305f28e674c"><img alt="" src="https://cdn-images-1.medium.com/max/255/1*2f7OqE2AJK1KSrhkmD9ZMw.png" /></a></figure><figure><a href="https://upscri.be/8f5f8b"><img alt="" src="https://cdn-images-1.medium.com/max/255/1*v-PpfkSWHbvlWWamSVHHWg.png" /></a></figure><figure><a href="https://becominghuman.ai/write-for-us-48270209de63"><img alt="" src="https://cdn-images-1.medium.com/max/255/1*Wt2auqISiEAOZxJ-I7brDQ.png" /></a></figure><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=6ae716ef42b7" width="1" height="1" alt=""><hr><p><a href="https://becominghuman.ai/running-gpu-like-a-swag-on-windows-6ae716ef42b7">Running GPU like a swag on windows</a> was originally published in <a href="https://becominghuman.ai/">Becoming Human: Artificial Intelligence Magazine</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
    </channel>
</rss>