<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Avinash Lepakshi on Medium]]></title>
        <description><![CDATA[Stories by Avinash Lepakshi on Medium]]></description>
        <link>https://medium.com/@lepakshi.avinash?source=rss-da6911ab637b------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/1*y-NULb_5t5jRa7-zfF5nGg.jpeg</url>
            <title>Stories by Avinash Lepakshi on Medium</title>
            <link>https://medium.com/@lepakshi.avinash?source=rss-da6911ab637b------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Tue, 17 Nov 2026 18:19:30 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@lepakshi.avinash/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[Getting started with Apache-beam]]></title>
            <link>https://medium.com/@lepakshi.avinash/getting-started-with-apache-beam-49949d049ec2?source=rss-da6911ab637b------2</link>
            <guid isPermaLink="false">https://medium.com/p/49949d049ec2</guid>
            <category><![CDATA[etl-pipeline]]></category>
            <category><![CDATA[data-processing]]></category>
            <category><![CDATA[etl]]></category>
            <category><![CDATA[gcp]]></category>
            <category><![CDATA[apache-beam]]></category>
            <dc:creator><![CDATA[Avinash Lepakshi]]></dc:creator>
            <pubDate>Wed, 06 Jul 2022 11:41:23 GMT</pubDate>
            <atom:updated>2022-07-06T11:41:23.634Z</atom:updated>
            <content:encoded><![CDATA[<h4>Brief intro &amp; walkthrough of Apache-beam with code</h4><p>Welcome!</p><p>this article explains what?, why?, when? and how? to use Apache Beam..</p><p>input-&gt;transformation-&gt;output in a pipeline is ETL</p><p>What —</p><p>Apache beam in simpler words means its a framework built for ETL</p><p>Why —</p><p>It provides Unicode -&gt;multiformat -&gt; faster process[bigdata] -&gt;multi deploy</p><p>Confusing!, the above mapper says by a single chosen language out of many supported languages of apache beam we could process either batch or streaming input data faster by parallel processing on bigdata or local environments.</p><p>Below pic explains better</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*NfNwXF3BJ10Up5yAZhX5rg.png" /><figcaption>ref:-<a href="https://beam.apache.org/get-started/beam-overview/">https://beam.apache.org/get-started/beam-overview/</a></figcaption></figure><p>When —</p><p>This can be used almost in all scenarios of ETL processing either on-prem or on cloud most importantly useful if a development is new and looking out for a framework that could withstand and be used on longer term</p><p>How —</p><p>This is vast however I&#39;ll be covering few intro concepts along with code.</p><p>Exciting! lets get started.</p><p>Apache beam handles all data as Pcollection nothing but a term given in beam in short it can be treated as an input data passes to transformation eventually output is generated which is also called as Pcollection(collection of data)</p><p>Pcollection can be elements[lists, string etc..] or as a key-value pairs.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*pFfDlo6iNvPzh5XYAVI8pQ.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*VlZxr812k2NgU_yJFw-z6g.png" /></figure><p>Strongly suggest to understand concepts of beam terminologies from below link[ images taken from]</p><p><a href="https://beam.apache.org/documentation/programming-guide/">Beam Programming Guide</a></p><p>Similar to P-collection beam has variety of transforms as well</p><p>pardo( Map, Filter,Combiner.count.perkey() etc..)</p><ul><li>beam &lt;-&gt; sql ; names</li></ul><blockquote>beam.CombinePerKey(&lt;aggregate&gt;) &lt;-&gt; groupby()</blockquote><blockquote>beam.combiners.Count.PerKey() &lt;-&gt; groupby() &amp;count()</blockquote><blockquote>beam.Filter &lt;-&gt; where</blockquote><ul><li>apart from these we can use user defined functions as well via doFn</li></ul><pre>def fil(element):</pre><pre>if element[10]==&#39;Arts and Recreation Services&#39;:</pre><pre>return element</pre><pre>with beam.Pipeline() as pipe:</pre><pre>v1=(pipe</pre><pre>|beam.io.ReadFromText(&#39;input.csv&#39;,skip_header_lines=True)</pre><pre>|beam.Map(lambda x: x.split(&#39;,&#39;))</pre><pre>|beam.Filter(fil)</pre><pre># |beam.combiners.Count.Globally()</pre><pre>|beam.Map(lambda x:(x[4],x[10]))</pre><pre>|beam.combiners.Count.PerKey()</pre><pre>|beam.Map(print)</pre><pre># |beam.io.WriteToText(&#39;./op2.csv&#39;)</pre><pre>)</pre><ul><li>can pass user defined variables inside pipeline</li></ul><pre>v2=[1,2,3]</pre><pre>with beam.Pipeline() as p:</pre><pre>var=(p</pre><pre>|beam.Create(v2)</pre><pre>|beam.Map(print)</pre><pre>)</pre><ul><li>composite transform- combine more than one functions into a class and call the class instead</li></ul><pre>#### defining<br>class comptrans(beam.PTransform):</pre><pre>def comp(self,elem):</pre><pre>   ip1=(elem</pre><pre>   | beam.Map(func1)</pre><pre>   |beam.Map(func2)</pre><pre>   )</pre><pre>return ip1</pre><pre>with beam.Pipeline() as p:</pre><pre>v1=(p</pre><pre>      |beam.io.ReadFromText(&#39;ip.csv&#39;,skip_header_lines=True)</pre><pre>     |beam.Map(lambda x: x.split(&#39;,&#39;))</pre><pre>)</pre><pre>v2=(v1</pre><pre>     |comptrans()## calling </pre><pre>)</pre><ul><li>PardoTranform — All above tranforms discussed are part of pardo we could directly invoke pardo</li></ul><pre>from apache_beam.transforms.core import ParDo</pre><pre>class pardocheck(beam.DoFn): # pass dofn object to class</pre><pre>    def process(self,element):   # overwrite process method</pre><pre>    sp=element.split(&#39;,&#39;)</pre><pre>    return <strong>[sp]</strong>      # must be an iterator as yield or list or any</pre><pre>with beam.Pipeline() as p:</pre><pre>v1=(p</pre><pre>|beam.io.ReadFromText(&#39;business-financial-data-mar-2022-quarter-csv.csv&#39;,skip_header_lines=True)</pre><pre>|beam.ParDo(pardocheck())</pre><pre>|beam.Map(print)</pre><pre>)</pre><ul><li>side input , side output</li></ul><p>Multiple inputs/outputs can be derived or passed to a pipeline</p><pre>###### side o/p</pre><pre>class fil(beam.DoFn):</pre><pre>    def process(self,element):</pre><pre>        if element[10]==&#39;Arts and Recreation Services&#39;:</pre><pre>        yield beam.pvalue.TaggedOutput(&#39;Art&#39;,element) <br># TAg o/p so it can be used later</pre><pre>       else:</pre><pre>       yield element</pre><pre>with beam.Pipeline() as p:</pre><pre>ip=p|beam.io.ReadFromText(&#39;business-financial-data-mar-2022-quarter-csv.csv&#39;,skip_header_lines=True)</pre><pre>out=ip |beam.ParDo(fil()).with_outputs(&#39;Art&#39;,main=&#39;NotArt&#39;) # main is default tag if not tagged by its function</pre><pre>art=out.Art</pre><pre>nart=out.NotArt</pre><pre>art|&quot;write arts to o/p&quot;&gt;&gt;beam.io.WriteToText(&#39;Arts.txt&#39;)</pre><pre>nart|&quot;write non-arts to o/p&quot;&gt;&gt;beam.io.WriteToText(&#39;NonArts.txt&#39;)</pre><pre>art|beam.Map(print)</pre><pre>############### side i/p</pre><pre>def combine(ele,max): # passing max argument as extra i/p(side i/p)</pre><pre>if sum(ele)&gt;max:</pre><pre>return 1</pre><pre>else:</pre><pre>return 0</pre><pre><br>beam.CombinePerKey(combine,200)  <br>## Applying side i/p in beam pipeline</pre><h4>Schema</h4><p>Structured data can be processed via Dataframes, can alter changes from databases via sql such as SqlTransform, GroupBy .</p><p>As a mandate before these functionalities schema defining metadata has to be provided that can be done by 3 ways</p><ol><li>Register and yield instances of named tuple</li></ol><pre>## create class<br>class <strong>veggies</strong>(typing.NamedTuple):</pre><pre>Series_reference:str</pre><pre>Data_value:int</pre><pre>status:str</pre><pre>##Register</pre><pre>beam.coders.registry.register_coder(<strong>veggies</strong>,beam.coders.RowCoder)</pre><pre>with beam.Pipeline() as p:</pre><pre>pc=(p|beam.Create([veggies(&quot;BDCQ.SF1AA2CA&quot;, 1116.386,&quot;  F&quot;,&quot;  Industry by financial variable (NZSIOC Level 2) &quot;,</pre><pre>&quot;Sales (operating income)&quot;,&quot; Forestry and Logging&quot;)]).with_output_types(veggies)</pre><pre>);</pre><p>2. Yield beam.row instance</p><pre>p|beam.Create([veggies(&quot;BDCQ.SF1AA2CA&quot;])<br>|beam.Map(lambda x: <strong>beam.Row</strong>(id:x[0]))<br>#type of columns are inferred automatically</pre><p>3. by PTransform to produce schema</p><p>Now we look into use of sql and dataframes(for structured table like data)</p><blockquote>SqlTransform</blockquote><p>P collection as input , transform by sql then reprocess by python function if necessary</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/882/1*O8TNeZnlPH6I-A_v5m0RGw.png" /><figcaption>ref:-<a href="https://www.youtube.com/watch?v=zx4p-UNSmrA">https://www.youtube.com/watch?v=zx4p-UNSmrA</a></figcaption></figure><p>Tagged Pcollection as input-</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/887/1*CG3aqta7TLtA12lrAEwm6w.png" /><figcaption>ref:-<a href="https://www.youtube.com/watch?v=zx4p-UNSmrA">https://www.youtube.com/watch?v=zx4p-UNSmrA</a></figcaption></figure><p>Windowing function-</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/844/1*YXjbYY_zOz_EmvE-Pkqffg.png" /><figcaption>ref:-<a href="https://www.youtube.com/watch?v=zx4p-UNSmrA">https://www.youtube.com/watch?v=zx4p-UNSmrA</a></figcaption></figure><p>Limitations- As of september 2020</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/831/1*AIGSoA4z8Kh7QwvmUrApdw.png" /><figcaption>ref:-<a href="https://www.youtube.com/watch?v=zx4p-UNSmrA">https://www.youtube.com/watch?v=zx4p-UNSmrA</a></figcaption></figure><blockquote>Group By</blockquote><p>Group elements</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/846/1*BHMZSk-dAUEP5uGOViAlig.png" /><figcaption>ref:-<a href="https://www.youtube.com/watch?v=zx4p-UNSmrA">https://www.youtube.com/watch?v=zx4p-UNSmrA</a></figcaption></figure><p>Group and aggregate</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/884/1*Mts20wQa0e52kr3jaAAYVw.png" /><figcaption>ref:-<a href="https://www.youtube.com/watch?v=zx4p-UNSmrA">https://www.youtube.com/watch?v=zx4p-UNSmrA</a></figcaption></figure><blockquote>DataFrame Transform</blockquote><p><strong><em>Currently this is not recommended for Production use but planned to be in-active by 2.25 release</em></strong></p><p>Pass pcollection(table)-&gt;convert to df -&gt;make necessary changed by dataframe logic</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/847/1*RfijlFjh5ROYye-dnpJbkA.png" /><figcaption>ref:-<a href="https://www.youtube.com/watch?v=zx4p-UNSmrA">https://www.youtube.com/watch?v=zx4p-UNSmrA</a></figcaption></figure><p>Can convert Pcollection → data frames( make all pandas transforms) →Pcollection to o/p</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/620/1*V3V4wIaAzHspmZ5ckBgk6w.png" /><figcaption>ref:-<a href="https://www.youtube.com/watch?v=zx4p-UNSmrA">https://www.youtube.com/watch?v=zx4p-UNSmrA</a></figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/731/1*9AHNofWYTdrESyNUEMu4aQ.png" /><figcaption>ref:-<a href="https://www.youtube.com/watch?v=zx4p-UNSmrA">https://www.youtube.com/watch?v=zx4p-UNSmrA</a></figcaption></figure><p>Found a very good example showcasing beam wrap around over regular python code , check out</p><p><a href="https://medium.com/analytics-vidhya/transform-json-to-csv-from-google-bucket-using-a-dataflow-python-pipeline-92906c87cc96">Transform JSON to CSV from Google bucket using a Dataflow Python pipeline</a></p><p>!!Stay tuned more such information will be updated.!!</p><p>Happy Learning…</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=49949d049ec2" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Getting started with Networking’s on cloud (AWS)]]></title>
            <link>https://medium.com/@lepakshi.avinash/getting-started-with-networkings-on-cloud-aws-6dce3ef7e0c2?source=rss-da6911ab637b------2</link>
            <guid isPermaLink="false">https://medium.com/p/6dce3ef7e0c2</guid>
            <category><![CDATA[cloud-networking]]></category>
            <category><![CDATA[networking]]></category>
            <category><![CDATA[gcp]]></category>
            <category><![CDATA[aws]]></category>
            <category><![CDATA[cloud]]></category>
            <dc:creator><![CDATA[Avinash Lepakshi]]></dc:creator>
            <pubDate>Sun, 29 Dec 2019 14:30:48 GMT</pubDate>
            <atom:updated>2019-12-29T14:30:48.238Z</atom:updated>
            <content:encoded><![CDATA[<h4>This is for beginners to understand networking on AWS</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/810/0*lMrBWV7nNToA1368.jpg" /><figcaption>image source: <a href="https://uk.pcmag.com/networking-communications-software/16824/what-is-cloud-computing">https://uk.pcmag.com/networking-communications-software/16824/what-is-cloud-computing</a></figcaption></figure><p>In today’s world as data increase exponentially and major or all of the companies moving towards cloud for an easier and faster way to maintain data it is much needed to understand the networking on cloud and how can connect/establish connection to cloud from on-premise system.</p><p>For these we start with understanding of VPC</p><h3>VPC(virtual private cloud):</h3><p>This is a logical establishment of a group of things(machines, with ip address ranges network rules etc..) together or like a closed network similar to VPN outside of cloud.</p><p>The picture below is a VPC with two subnets private and public and virtual machines with in the subnet range ,with public subnet having access to internet and private subnet accessing internet via public subnet’s NAT gateway.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/738/1*D-Shs9sxIz22_1F5X5U4Cw.png" /></figure><p><strong>component description:</strong></p><p><strong>Internet gateway:</strong> the gateway that connects instances with in private vpc to internet and rules can be given on routing table i.e. in bound and out bound access to be allowed form and to internet.</p><p><strong>Routing table:</strong> Table which defines rules associated to gateway</p><p><strong>NAT gateway: </strong>This allows only traffic with in vpc , helpful for the 172.31.2.0/24 private subnet which can get responses from internet via NAT gateway i.e. NAT gateway which is in public sub-net access internet via internet gateway and and passes on responses to private subnet</p><h3>Connection between VPC’s :</h3><p>The question arises now, like i have machine on VPC in cloud which is secured and now wants to communicate with other machine on a different VPC ?</p><p>Below is answer for that either vpc peering or transit gateway</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/738/1*tcbsREeJvAbIgzp2_HkhWA.png" /></figure><ol><li><strong>VPC peering:</strong></li></ol><p>connection between two vpc’s can’t bypass information to third vpc from first vpc like below</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/732/1*QCWFrBkscw3GOYApg_kSHA.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/725/1*9a-U9owGAM0BySez5sT3IA.png" /></figure><p><strong>2. Transit Gateway:</strong></p><p>This is like an connection layer between many to many vpc’s</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/736/1*lOgQ8wmaItPCdzfAdJWcWQ.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/741/1*Rh1E8H7_grRrZIyZQAQg7w.png" /></figure><h3>Shared VPC / VPC sharing:</h3><p>This is a concept called shared vpc with host and service projects were shared vpc admin maintains all of these.</p><p>This is much helpful for an organisation they have one host project with all the network connectivity that had been done and they pass on subnets of this vpc to service projects where the particular team can directly deploy or use for data / app /storage etc.. without have to worry about anything else all of these will be controlled by host admin.</p><p>Shared VPC allows an <a href="https://cloud.google.com/resource-manager/docs/cloud-platform-resource-hierarchy">organization</a> to connect resources from multiple projects to a common <a href="https://cloud.google.com/vpc/docs/vpc">VPC network</a>, so that they can communicate with each other securely and efficiently using internal IPs from that network. When you use Shared VPC, you designate a project as a <em>host project</em> and attach one or more other <em>service projects</em> to it. The VPC networks in the host project are called <em>Shared VPC networks</em>. <a href="https://cloud.google.com/vpc/docs/shared-vpc#resources_that_can_be_attached_to_shared_vpc_networks_from_a_service_project">Eligible resources</a> from service projects can use subnets in the Shared VPC network.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/716/1*e3bSklhDq03CJUcgj8vqxA.png" /><figcaption>GCP shared vpc connectvity</figcaption></figure><p>(for more info:: <a href="https://cloud.google.com/vpc/docs/shared-vpc">https://cloud.google.com/vpc/docs/shared-vpc</a>)</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/730/1*-0dagTG3YaGWSdnxLylKtg.png" /></figure><h4>VPC end points:</h4><p>End points are the resources that a vpc can talk to others outside the vpc via theses mechanisms.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/674/1*xZ0NMqgtZ48tMi3ItNKtlw.png" /><figcaption>all three vpc endpoints</figcaption></figure><p><strong>1.Gateway vpc end points</strong></p><p>This is an secured way of connection between vpc to aws services on internet alternative for internet gateway</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/718/1*XceSwrw-ILc9g4aqFn92Dg.png" /><figcaption>accessing AWS services by gateway vpc endpoints</figcaption></figure><p><strong>2. Interface VPC endpoints</strong></p><p>TO connect services api from vpc in a secured way</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/729/1*Y1gD2YDFV9F7mIm4Jb1EZg.png" /></figure><p><strong>3. Private link</strong></p><p>Connection via private link could restrict connection only to the particular service .</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/731/1*3QITHTYnG5OhLVryBxfR-Q.png" /></figure><h3>Connection between on-premise to Cloud</h3><ol><li>AWS direct connect</li><li>AWS VPN</li></ol><p>Below two images show connection between on-premise to cloud data-centers</p><p><strong>fig1:</strong> layout of connection with multiple vpc’s on cloud by AWS direct connect gateway</p><p><strong>fig2:</strong> connetivity to aws services and layout of conetion</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/732/1*qNXT2T3ERnc_oIqDxMhF0w.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/730/1*0bDW4Emu86snOgyMkIScug.png" /></figure><p><strong>Customer network:</strong> On-premise datacentre to be connected to cloud</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/878/1*nm1yiXTAHuO3a3mx5nIsmw.png" /></figure><p>configuration for routing via transit gateway and vpn connect</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/719/1*Q1GPdsDkL9B-gixLKNcGmg.png" /></figure><p>. . .</p><p>This is an initial version would be updating more on this until then</p><p>!!happy learning!!</p><h4>References:</h4><ul><li><a href="https://youtu.be/hiKPPy584Mg">https://youtu.be/hiKPPy584Mg</a></li></ul><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=6dce3ef7e0c2" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Machine Learning & A.I on GCP]]></title>
            <link>https://medium.com/@lepakshi.avinash/machine-learning-a-i-on-gcp-5c9a1c00384b?source=rss-da6911ab637b------2</link>
            <guid isPermaLink="false">https://medium.com/p/5c9a1c00384b</guid>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[artificial-intelligence]]></category>
            <category><![CDATA[google-cloud]]></category>
            <category><![CDATA[gcp]]></category>
            <category><![CDATA[cloud-computing]]></category>
            <dc:creator><![CDATA[Avinash Lepakshi]]></dc:creator>
            <pubDate>Tue, 24 Dec 2019 16:42:23 GMT</pubDate>
            <atom:updated>2019-12-24T16:42:23.568Z</atom:updated>
            <content:encoded><![CDATA[<h4>For developing &amp; deploying M.L &amp; A.I on GCP</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/600/1*M_FVKZ9L-q1BrS804kkK3g.png" /><figcaption>source : <a href="https://www.whizlabs.com/blog/google-cloud-machine-learning/">https://www.whizlabs.com/blog/google-cloud-machine-learning/</a></figcaption></figure><h4>Intro::</h4><p>As the time changes the platform of execution of machine learning is being changed, and this is one of it i.e. with the world moving towards cloud many of us would be wondering how to implement data-science on cloud and this helps for such people..</p><h4>Data Ingestion:</h4><p>In GCP we could use tfrecord csv file instead of traditional csv file which is faster and scalable way to process data.</p><p>Basically there are three types of data ingestion in cloud i.e. getting data into cloud shown below</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/567/1*C_KG4-h_h9UYDMsYZpylgg.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/457/1*Ru-MjihN1vRN8It6sAXcaQ.png" /><figcaption>one of the possible data flow model</figcaption></figure><p><strong>Pubsub-</strong>For streaing data (ex: gaming experience change)</p><p><strong>Bigquery-</strong>For structured data directly to M.L model (can be a batch data like from databases)</p><p><strong>Cloud Storage-</strong>Storing transformed data for later trainings</p><h4>Data Analysis:</h4><p>With data in cloud we need to explore data/study data that is shown below</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/559/1*nqJblqSHZ1qwZr8KSppqLg.png" /></figure><p><strong>Data Studio:</strong> Dashboarding tool which can be visualised from bigquery ( for more info : <a href="https://cloud.google.com/bigquery/docs/visualize-data-studio">https://cloud.google.com/bigquery/docs/visualize-data-studio</a>)</p><p><strong>Cloud reliability:</strong> Alerting and Monitoring system</p><p><strong>Datalab: </strong>Scripting tool with jupyter notebooks( for more info : <a href="https://cloud.google.com/datalab/docs/quickstart">https://cloud.google.com/datalab/docs/quickstart</a> )</p><h4>Data Transformation:</h4><p>So with known we would also like to transform /change data as per our needs that could be easily done with below components</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/500/1*-tzYGI7ASf-1dV1eIqBMoQ.png" /></figure><p><strong>Dataflow:</strong> For streaming/batch processing data this is good( more info : <a href="https://cloud.google.com/dataflow/docs/">https://cloud.google.com/dataflow/docs/</a>)</p><p><strong>Dataproc:</strong>This is an Big data based platform that is good for transformation ( <a href="https://cloud.google.com/dataproc/docs/">https://cloud.google.com/dataproc/docs/</a>)</p><p><strong>Dataprep:</strong> Use Cloud Dataprep to explore and transform raw data from disparate and/or large datasets into clean and structured data for further analysis and processing.( <a href="https://cloud.google.com/dataprep/docs/">https://cloud.google.com/dataprep/docs/</a>)</p><h4>Model Training:</h4><p>With data ready for model building below pic shows ways to do it .</p><p>These are for faster model trainings with data parallelism and tensor operations</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/557/1*hdI10GVbWZo0TCHw78OLhQ.png" /></figure><p><strong>ML Engine:</strong> One stop solution for all Machine learning models to train with built in tensor flow and auto scaling , versioning models</p><p>( <a href="https://cloud.google.com/ml-engine/docs/technical-overview">https://cloud.google.com/ml-engine/docs/technical-overview</a>)</p><p><strong>GKE kube flow: </strong>For Hybrid M.L flow in kube models</p><p>( <a href="https://www.kubeflow.org/docs/pipelines/overview/pipelines-overview/">https://www.kubeflow.org/docs/pipelines/overview/pipelines-overview/</a>)</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/513/1*8U4DhasezZ5n8Jy93FTG3Q.png" /><figcaption>to train model faster</figcaption></figure><h4>Model evaluation:</h4><p>With model trained and tested with the data that we are aware of but google’s TFX model analysis had a variety of components and sets which lets to evaluate and judge our model</p><p>more details ( <a href="https://www.tensorflow.org/tfx/model_analysis/get_started">https://www.tensorflow.org/tfx/model_analysis/get_started</a>)</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/504/1*-MMfWo2--vhUEzFJhwoHsQ.png" /></figure><p>All the above process we’ve seen are the available options that GCP provides , not necessary to use all of them for machine learning development whereas selection of tool and process is entirely dependent on individual requirements.</p><h4>Develop [build train test M.L models]</h4><ol><li>Static</li></ol><p>Here the data is not changed very frequently it’ll be mostly constant below fig shows data and code being resided in storage bucket and called as and when needed</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/313/1*jkVoCNGeveDeC87ZSMbnSA.png" /></figure><p>2.Dynamic</p><p>Here the data tends to change very frequently and model being executed with arrival of new data followed by cloud functions and M.L engine</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/561/1*QC_oAPNwZKyyYcc4bBCmtg.png" /></figure><h4>Deployment [prediction on M.L model ]</h4><p>A simple example of deployed model with streaming and batch data being processed for prediction</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/565/1*bcx8lMWNuBqlHpKcYowXuQ.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/524/1*2BlZPkVQCj2UwycUsvUkxg.png" /><figcaption>Batch &amp; streaming deployment</figcaption></figure><p><strong>Orchestration:</strong></p><p>This is an end-end pipeline of data flow from ingestion till model building and prediction with new data as well , this can be given as a step by step process to cloud composer</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/571/1*srUKcUiLOWlP7k6FoBlplQ.png" /></figure><p><strong>Cloud Composer: </strong>Traditional GCP orchestration which manages everything</p><p><strong>Argo :</strong> This is an GKE based orchestartion</p><h4>Visualisation</h4><p>Tensor board is an one stop visualised GCP tool for all types of graphs that we need in data-science</p><p>more info :(<a href="https://www.tensorflow.org/tensorboard/get_started">https://www.tensorflow.org/tensorboard/get_started</a>)</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/313/1*wh7jIOfKBKbMXjOMOs-Ysg.png" /></figure><h4>References/Credits:</h4><blockquote><a href="https://www.coursera.org/learn/gcp-production-ml-systems/home/welcome">https://www.coursera.org/learn/gcp-production-ml-systems/home/welcome</a> (coursera course)</blockquote><p>. . .</p><p>Would be updating this story with more details on deployments , feel free to add/change the story with comments</p><p>!!!thanks!!!</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=5c9a1c00384b" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Analytics over years on different datasize]]></title>
            <link>https://medium.com/@lepakshi.avinash/analytics-over-years-on-different-datasize-1b1a3a5c3a28?source=rss-da6911ab637b------2</link>
            <guid isPermaLink="false">https://medium.com/p/1b1a3a5c3a28</guid>
            <category><![CDATA[big-data]]></category>
            <category><![CDATA[analytics]]></category>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[cloud-storage]]></category>
            <category><![CDATA[hadoop]]></category>
            <dc:creator><![CDATA[Avinash Lepakshi]]></dc:creator>
            <pubDate>Sun, 11 Aug 2019 19:03:12 GMT</pubDate>
            <atom:updated>2019-08-11T19:03:12.374Z</atom:updated>
            <content:encoded><![CDATA[<h3>Analytics over years and datasize</h3><blockquote>This blog explains analytics/data science point of view to handle data from byte -&gt; Exabyte</blockquote><figure><img alt="" src="https://cdn-images-1.medium.com/max/768/1*I0ZnlxxxQu1rxj-4YrN7EQ.jpeg" /></figure><p>The main problem faced by the data science community or any other data engineer is how can we use a large amount of data in a faster manner.</p><p>The reason behind writing this blog is due to struggles faced in handling large amount of data, now let’s see multiple ways and evolution of data and analytics.</p><h4>1.Megabytes</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/960/1*cY5L82rwlG93P5IK-wgcgA.gif" /><figcaption>source: <a href="https://machinate.com/">machinate.com</a></figcaption></figure><blockquote>Storage: Personal storage (csv, xl etc…)</blockquote><blockquote>Analytics: Python,R[open source]</blockquote><p>With lesser data i.e. anything within Megabytes can be handled on excel files and on prem installation of python(which is easy and powerful) OR R(stats designed tool) could be used for analytics on top of data by retrieving data within python/R environment.</p><h4>2.Petabytes[Big data]</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/412/1*e3eA8kjj9x_8vk18AjBCoQ@2x.jpeg" /></figure><blockquote>Storage: Hadoop(HDFS)</blockquote><blockquote>Analytics: Pyspark,R &amp; Python (using package Spark MLlib package)</blockquote><p>Once the data exceeds Gigabytes it’ll be really hard to work on python or R as either local on prem systems should have a large amount of RAM which is expensive or move to BIG DATA environment which is expensive but worth as this enables not only analytics but entire B.I. environments will be enriched, large organizations have already been adopted by using Hadoop (HDFS ) environment.</p><p>Storage will be taken care by HDFS which can handle any type of data and ingestion to this could be via kafka, flume,sqoop etc…</p><p>Data retrieval could be done by HIVE which Hadoop uses two ways MapReduce or spark, comparatively spark is faster and this is a framework which can accommodate any software on top of it like python, R, etc.. Python on spark i.e. Pyspark which is a best suitable option in these environments.</p><h4>3.Exabytes[Cloud]</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/806/1*v3JBjsKq8h88w2yU33zrug@2x.jpeg" /></figure><blockquote>Storage: AWS , AZURE ,GCP etc…</blockquote><blockquote>Analytics: GCP (google collab, google datalab ,Auto M.L , inbuilt images )</blockquote><blockquote>AWS(amazon sagemaker, textract, translate etc..)</blockquote><blockquote>AZURE(Azure M.L studio ,Azure machine learning etc..)</blockquote><p>When the data is extreme either we can scale up our Hadoop clusters which are expensive of infrastructure management and money else we can move on to cloud which is zero infra management maintained by cloud providers but expensive as per usage.</p><p>Cloud providers have different machine learning frameworks with code zero environment (Azure M.L studio, Google Auto M.L) or M.L flexible with faster data processing and built-in libraries ready to use Deep Learning and Machine learning code.</p><p>Help me to add/modify content of this topic with your contribution in comment section, Thanks!!</p><blockquote>Will be updating more on this blog stay tuned!!</blockquote><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=1b1a3a5c3a28" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Getting started with Terraform]]></title>
            <link>https://medium.datadriveninvestor.com/getting-started-with-terraform-eada2cbb442d?source=rss-da6911ab637b------2</link>
            <guid isPermaLink="false">https://medium.com/p/eada2cbb442d</guid>
            <category><![CDATA[terraform]]></category>
            <category><![CDATA[aws]]></category>
            <category><![CDATA[automation]]></category>
            <category><![CDATA[google-cloud-platform]]></category>
            <category><![CDATA[infrastructure]]></category>
            <dc:creator><![CDATA[Avinash Lepakshi]]></dc:creator>
            <pubDate>Mon, 05 Aug 2019 15:21:04 GMT</pubDate>
            <atom:updated>2019-08-16T12:48:25.996Z</atom:updated>
            <content:encoded><![CDATA[<figure><a href="http://www.track.datadriveninvestor.com/1B9E"><img alt="" src="https://cdn-images-1.medium.com/max/700/0*W6LBdC2ZvoSZUlZw" /></a></figure><blockquote>Check this out for learning Terraform…….</blockquote><figure><img alt="" src="https://cdn-images-1.medium.com/max/332/1*DO9YtvSEfmBb8YHeJ0QW2w.png" /></figure><h4>Why Terraform??</h4><p>there are two ways we can manage the cloud environment</p><ol><li>Direct login to respective U.I and make changes</li><li>Pass commands via console to the cloud(GCP/AWS/AZURE)</li></ol><p>These are the cloud service providers options that is been given to user , apart from these options there is one more way i.e. terraform which is an Hashicorp Language specifically built for cloud infrastructure management</p><h4>Advantages of using Terraform</h4><ul><li>Terraform maintains .tf state file which keeps track of all changes that have been done via terraform to cloud infra</li></ul><h4>What is Terraform??</h4><p>Terraform is an HCL language script for infrastructure management, it works on API based system to create/modify infrastructures like GCP, AWS, DIGITAL OCEAN, etc…</p><p><a href="https://www.datadriveninvestor.com/2019/02/12/ready-or-not-the-revolution-is-upon-us/">Ready or not, the revolution is upon us | Data Driven Investor</a></p><p>Terraform commands::</p><p>— — -These are few basic commands to start with a complete list of commands have been mentioned at the end of this article. — — —</p><pre>#INITIALIZE BY SET UP ENVIRONMENT<br>terraform init</pre><pre>#PREVIEW CHANGES ON EXECUTE AND STORE INTO A FILE(OPTIONAL)<br>terraform plan -out changes.terraform</pre><pre>#EXECUTE TERRAFORM SCRIPTS ON THE CURRENT FOLDER FROM ##changes.terraform file(OPTIONAL ELSE ALL FILES WILL BE EXECUTED)</pre><pre>terraform apply changes.terraform </pre><pre>#REMOVE FILE FROM CURRENT LOCATION<br>rm changes.terraform</pre><pre>#COMPLETELY DESTROYS ALL THE TERRAFORM CHANGES HAPPENED <br>terraform destroy</pre><h4>Syntax types:</h4><p>Terraform is a Hashi corp Language (HCL) scripted language this expects two different file types</p><p>1 .tf.json — This is a json language file which is creating resource aws_instance like shown below</p><pre>{<br>  &quot;resource&quot;: {<br>    &quot;aws_instance&quot;: {<br>      &quot;example&quot;: {<br>        &quot;instance_type&quot;: &quot;t2.micro&quot;,<br>        &quot;ami&quot;: &quot;ami-abc123&quot;<br>      }<br>    }<br>  }<br>}</pre><p>2 .tf — This is a regular HCL language type shown below for the same type shown above and all our syntax in this article will be on this syntax</p><pre>resource &quot;aws_instance&quot; &quot;example&quot; {<br>  instance_type <strong>=</strong> &quot;t2.micro&quot;<br>  ami           <strong>=</strong> &quot;ami-abc123&quot;<br>}</pre><p>Now let us dive through various topics which will be used in terraform</p><h4>1. Variables:</h4><p>Any non-fixed values such as region, images, etc.. which are flexible can be called from vars.tf which will be dynamic</p><p>There are 3 types of variables we can pass on</p><ol><li>String</li></ol><pre>//declare<br>variable &quot;template&quot; {<br>    type = &quot;string&quot;<br>    default = &quot;01000000-0000-4000-8000-000030080200&quot;<br>}</pre><pre>//assign<br>storage = &quot;${var.template}&quot;</pre><p>2.List</p><pre>//declare<br>variable &quot;users&quot; {<br>    type    = &quot;list&quot;<br>    default = [&quot;root&quot;, &quot;user1&quot;, &quot;user2&quot;]<br>}</pre><pre>//assign<br>username = &quot;${var.users[0]}&quot;</pre><p>3. Map</p><pre>variable &quot;plans&quot; {<br>    type = &quot;map&quot;<br>    default = {<br>        &quot;5USD&quot;  = &quot;1xCPU-1GB&quot;<br>        &quot;10USD&quot; = &quot;1xCPU-2GB&quot;<br>        &quot;20USD&quot; = &quot;2xCPU-4GB&quot;<br>    }<br>}</pre><pre>//assign<br>plan = &quot;${var.plans[&quot;5USD&quot;]}&quot;</pre><p>4.Boolean</p><pre>variable &quot;set_password&quot; {<br>    default = false<br>}</pre><pre>create_password = &quot;${var.set_password}&quot;</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/872/1*dfjyeYuRzzo8avNisInYFA.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/899/1*lcAuYD9rY2m9Z8MiNg2h7A.png" /></figure><p>.tf —Store main terraform codes with this suffix</p><p>.tfvars —all secret files to be placed which can be ignored while saving into git or any external providers</p><p>vars.tf — to store all variables</p><h4>2. File Uploads</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/547/1*KnZMEbgnW71_J5if1cVUTg.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/542/1*XCpaMjYdI0TdZwvC3OtbSg.png" /></figure><h4>3. Executing the script::</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/551/1*BZ8hIuYQEO1fcrri0GJe3w.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/555/1*BVMkIUm3dccaSIjAuvayLw.png" /></figure><h4>4. Attributes:</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/624/1*cVw37ufviJASY4lw1bSC8Q.png" /></figure><h4>5. Data resource</h4><p>A <em>data source</em> represents a piece of read-only information that is fetched from the provider (in this case, AWS) every time you run Terraform. Adding a data source to your Terraform configurations does not create anything new; it’s just a way to query the provider’s APIs for data and to make that data available to the rest of your Terraform code.</p><pre>data &quot;&lt;PROVIDER&gt;_&lt;TYPE&gt;&quot; &quot;&lt;NAME&gt;&quot; {<br> [CONFIG ...]<br>}</pre><pre>#  PROVIDER is the name of a provider (e.g., aws), TYPE is the type of data source you want to use (e.g., vpc</pre><h4>6. Data Sources</h4><p>Data sources on AWS are something which we can retrieve all dynamic information from AWS through API&#39;s like list of ami types, availability zones, etc…</p><p>Below is an AWS code allowing ip ranges only from eu zone, like below we have “eu-west-1” and “eu-central-1” for ec2 services is input.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/431/1*680ZZW4sXhwIjLsJR3id2A.png" /></figure><p>The main difference between data source and variable, variable-static &amp; datasource-dynamic pulls latest values just before executing</p><h4>7. Modules</h4><p>Modules are like packages in Terraform for code reusability</p><p>2 types:-</p><ol><li>Local Module</li></ol><p>This is something that we write on local system like below, terraform starts looking from /module-example folder for terraform scripts and the following values act as parameters for terraform scripts</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/553/1*iyJDli0jgy_PrX7pH5Rpyw.png" /></figure><p>2.External module</p><p>We can connect to an external system like git etc…</p><pre>terraform get #This downloads files from git <br>terraform apply #executes terraform files downloaded from git</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/566/1*GDptuonm_o7xlR78MMGC0A.png" /></figure><h4>8. Import</h4><p>This is an excellent feature that terraform is providing, this lets us to import existing cloud infrastructure into our tf state one at a time</p><p>more info::- <a href="https://www.terraform.io/docs/import/">https://www.terraform.io/docs/import/</a></p><p>steps to do:</p><ol><li>create a blank/filled resource block like below</li></ol><pre>resource &quot;google_compute_instance&quot; &quot;default&quot; {<br>  # ...instance configuration...<br>}</pre><p>2.run terraform import like below</p><pre>//  Instances can be imported using the project, zone and name, e.g.<br>// terraform import &lt;type&gt;.&lt;name&gt;</pre><pre>$ terraform import google_compute_instance.default gcp-project/us-central1-a/test</pre><p>This extracts and adds the particular instance details into .tfstate, same can be seen by “terraform plan”</p><h4>9.Output</h4><p>This block is useful to showcase outputs to a user once execution is finished</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/629/1*bhJ8Nm0u7ntg8U3iDHqRtQ.png" /></figure><h4>Terraform State:</h4><p>.tfstate and .tf statebackup are created and stores all the terraform scripts on terraform execution, thus it keeps tracks of all the activities done by terraform scripts .</p><p>ex:- If we have created an instance and deleted manually on GCP, when a terraform is executed it creates instance automatically.</p><p>We can monitor these changes by storing on GIT or on remote we can manage these states executing backend.tf file and storing backend on GCS scripts can be found here</p><p><a href="https://www.terraform.io/docs/backends/types/gcs.html">Backend Type: gcs - Terraform by HashiCorp</a></p><p>Below for AWS-S3</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/612/1*8Wu1hcx6aqGSMpVz0EBpPw.png" /></figure><p>If .tfstate gets deleted, use terraform import to re-create .tfstate else it doesn’t create automatically</p><h4>10. Interpolation</h4><p>[ credits:: <a href="https://blog.gruntwork.io/terraform-tips-tricks-loops-if-statements-and-gotchas-f739bbae55f9">https://blog.gruntwork.io/terraform-tips-tricks-loops-if-statements-and-gotchas-f739bbae55f9</a>]</p><p>Like a traditional language we can’t use IF-ELSE condition/FOR loops, instead, we can use this concept of interpolation.</p><p><strong>[i] For-loop</strong></p><ul><li><strong>count parameter</strong>: loop over resources.</li><li><strong>for_each expressions</strong>: loop over inline blocks within a resource.</li><li><strong>for expressions</strong>: loop over lists and maps.</li></ul><p><strong>i.i) count</strong></p><p>parameter is useful if you want to “loop” over an entire resource</p><pre>--Traditional for loop for creating 3 users<br>  for (i = 0; i &lt; 3; i++) {<br>  resource &quot;aws_iam_user&quot; &quot;example&quot; {<br>    name = &quot;neo.${i}&quot;<br>  }<br>}</pre><pre>---TERRAFORM way</pre><pre>resource &quot;aws_iam_user&quot; &quot;example&quot; {<br>  count = 3<br>  name  = &quot;neo.${count.index}&quot;<br>}</pre><p>if we want to pass a list instead of numeric</p><pre>--TRADITIONAL WAY--<br>for (i = 0; i &lt; 3; i++) {<br>  resource &quot;aws_iam_user&quot; &quot;example&quot; {<br>    name = vars.user_names[i]<br>  }<br>}</pre><pre>---TERRAFORM---</pre><pre>variable &quot;user_names&quot; {<br>  description = &quot;Create IAM users with these names&quot;<br>  type        = list(string)<br>  default     = [&quot;neo&quot;, &quot;trinity&quot;, &quot;morpheus&quot;]<br>}</pre><pre>resource &quot;aws_iam_user&quot; &quot;example&quot; {<br>  count = length(var.user_names)<br>  name  = var.user_names[count.index]<br>}</pre><p><strong>i.ii) for_each</strong></p><p>We know a way to loop over resources, now we’ll see loop inside resources</p><pre>resource &quot;aws_autoscaling_group&quot; &quot;example&quot; {<br>  # (...)  # This is just pseudo code. It won&#39;t actually work in Terraform.<br>  for (tag in var.custom_tags) {<br>    tag {<br>      key                 = tag.key<br>      value               = tag.value<br>      propagate_at_launch = true<br>    }<br>  }<br>}</pre><pre>---TERRAFORM----<br>  variable &quot;custom_tags&quot; {<br>  description = &quot;Custom tags to set on the Instances in the ASG&quot;<br>  type        = map(string)<br>}<br>--</pre><pre>resource &quot;aws_autoscaling_group&quot; &quot;example&quot; {<br>  # (...)<br>  <br>  dynamic &quot;tag&quot; {<br>    for_each = var.custom_tags    content {<br>      key                 = tag.key<br>      value               = tag.value<br>      propagate_at_launch = true<br>    }<br>  }<br>}</pre><p><strong>i.iii) for expression</strong></p><p>This is mainly useful if we are doing a list comprehensive operation in python</p><pre>---PYTHON-----<br>names = [&quot;neo&quot;, &quot;trinity&quot;, &quot;morpheus&quot;]short_upper_case_names = [name.upper() for name in names if len(name) &lt; 5]</pre><pre>----TERRAFORM----<br>                     ---SYNTAX----<br>            [for &lt;ITEM&gt; in &lt;LIST&gt; : &lt;OUTPUT&gt;]<br>            [for &lt;KEY&gt;, &lt;VALUE&gt; in &lt;MAP&gt; : &lt;OUTPUT&gt;]</pre><pre>variable &quot;names&quot; {<br>  description = &quot;A list of names&quot;<br>  type        = list(string)<br>  default     = [&quot;neo&quot;, &quot;trinity&quot;, &quot;morpheus&quot;]<br>}<br>output &quot;upper_names&quot; {<br>  <strong>value = [for name in var.names : upper(name)if length(name)&lt;5]</strong><br>}<br>--<br> <strong> output &quot;upper_roles&quot; {<br>  value = {for name, role in var.hero_thousand_faces : upper(name) =&gt; upper(role)}<br>}</strong></pre><p><strong>[II] IF-ELSE</strong></p><p>Below sample code shows the usage of conditions in terraform</p><pre>//set variable</pre><pre>variable &quot;give_neo_cloudwatch_full_access&quot; {<br>  description = &quot;If true, neo gets full access to CloudWatch&quot;<br>  type        = bool<br>}<br></pre><pre>//IF BLOCK<br>resource &quot;aws_iam_user_policy_attachment&quot; &quot;neo_cloudwatch_full&quot; {<br>  count = var.give_neo_cloudwatch_full_access ? 1 : 0  </pre><pre>  user       = aws_iam_user.example[0].name<br>  policy_arn = aws_iam_policy.cloudwatch_full_access.arn}</pre><pre>//ELSE BLOCK</pre><pre>resource &quot;aws_iam_user_policy_attachment&quot; &quot;neo_cloudwatch_read&quot; {<br>  count = var.give_neo_cloudwatch_full_access ? 0 : 1 </pre><pre>   user       = aws_iam_user.example[0].name<br>  policy_arn = aws_iam_policy.cloudwatch_read_only.arn<br>}</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/674/1*prFz3W13-Y6mnnXHKTBS0w.png" /></figure><p>Count Limitations::</p><ol><li>You cannot reference any resource outputs in count.</li><li>You cannot use count within a module configuration.</li><li>You cannot (easily) change count.</li></ol><h4>11. Functions</h4><p>Terraform has some built-in functions like,</p><p>i) ${file(“myfile.pub)} ← reads content inside the file</p><p>ii) basename(“abc/test.txt”) ← returns filename end file of path(test.txt)</p><p>iii) coalesce(str1,str2..) ← returns first non-empty string</p><p>iv) element(module.vpc_subnet.count.index) ← returns single element from list at a given index</p><p>v) format(“server-%03d,count.index+1”) :: o/p: server-001,002 ← formats string/list to a given format</p><p>vi)slice(str/list,from index,to index)</p><p>some more functions below…….|</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/621/1*F96j70R-nG7nGNxxWmORwA.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/620/1*lZVBlXMptDrYHjrDTjCblg.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/624/1*zakYPG7liZ2solNVxHNwTA.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/620/1*MXoZpQfM7lay0v5zrQsEog.png" /></figure><h4>Overall Terraform commands</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/697/1*HA2RJ4t-reVLhFQy69FC_w.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/668/1*CH8IDX07yN_8IznpJH3C-w.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/678/1*2lUs8uXFtB28-ddHSyUwaw.png" /></figure><h4>Examples::</h4><blockquote>VPC</blockquote><p>We can make GCP/AWS completely private via virtual private networks(vpc) i.e. create a vpc network on GCP so that we can connect to GCP from that network and by making use of G-SUITE we can make all logins from G-SUITE organisations make use of this by integrating g-suite with LDAP so that only their employees can log in</p><blockquote>GCP-compute instance creation</blockquote><p>Below sample code helps in the creation of compute instance</p><pre>/////////////////////////main.tf/////////////////////////</pre><pre>resource &quot;google_compute_instance&quot; &quot;example&quot; {</pre><pre>name         = &quot;${var.NAME}&quot;</pre><pre>machine_type = &quot;${var.MACHINETYPE}&quot;</pre><pre>zone=&quot;${var.ZONE}&quot;</pre><pre>#firewall rules to appply to this instance</pre><pre>tags = &quot;${var.TAGS}&quot;</pre><pre>boot_disk {</pre><pre>initialize_params {</pre><pre>image = &quot;${var.IMAGE}&quot;</pre><pre>size=&quot;30&quot;</pre><pre>//type=pd-ssd/pd-standard</pre><pre>type=&quot;pd-standard&quot;</pre><pre>}</pre><pre>}</pre><pre>network_interface {</pre><pre>network = &quot;default&quot;</pre><pre>//network_ip=&quot;1.1.3.4&quot; //this is internal ip</pre><pre>//subnetwork=&quot;10.132.0.2&quot;</pre><pre>access_config {</pre><pre>// Include this section to give the VM an external ip address</pre><pre>}</pre><pre>}</pre><pre>scheduling {</pre><pre>automatic_restart=true</pre><pre>}</pre><pre>metadata ={</pre><pre>foo=&quot;bar&quot;</pre><pre>}</pre><pre>metadata_startup_script=&quot;echo hi&quot;</pre><pre>labels = {</pre><pre>&quot;${var.LABELSKEY}&quot; = &quot;${var.LABELSVALUE}&quot;</pre><pre>}</pre><pre>}</pre><pre>//////////////////variables .tf////////////////////////////<br>variable &quot;GOOGLE_REGION&quot; {</pre><pre>default = &quot;europe-west1&quot;</pre><pre>}</pre><pre>variable &quot;IMAGE&quot; {</pre><pre>default = &quot;ubuntu-minimal-1804-bionic-v20190617&quot;</pre><pre>}</pre><pre>variable &quot;NAME&quot; {</pre><pre>default = &quot;test-terraform&quot;</pre><pre>}</pre><pre>variable &quot;MACHINETYPE&quot; {</pre><pre>default = &quot;n1-standard-1&quot;</pre><pre>}</pre><pre>variable &quot;ZONE&quot; {</pre><pre>default = &quot;europe-west1-c&quot;</pre><pre>}</pre><pre>variable &quot;TCP&quot; {</pre><pre>default = &quot;80&quot;</pre><pre>variable &quot;TAGS&quot; {</pre><pre>type=&quot;list&quot;</pre><pre>default = [&quot;https-server&quot;]</pre><pre>}</pre><pre>variable &quot;LABELSKEY&quot; {</pre><pre>default = &quot;keyname&quot;</pre><pre>}</pre><pre>variable &quot;LABELSVALUE&quot; {</pre><pre>default = &quot;value&quot;</pre><pre>}</pre><pre>variable &quot;auto_start&quot; {</pre><pre>default=true</pre><pre>}</pre><h4>ERRORS:</h4><ol><li>re-run from U.I filling all the details for compute create, for below error</li></ol><figure><img alt="" src="https://cdn-images-1.medium.com/max/855/1*SbeM7qV3EclitbVQhUvcsA.png" /></figure><h4>Command help:</h4><pre>terraform --help       //lists all options of terraform<br>terraform apply --help //lists all options of terraform apply<br>etc.. similarly any commands could be found with this.</pre><blockquote>Packer</blockquote><ol><li>This is used for creating images and can be utilised for multiple environments(any cloud provider)</li><li>Process: Packer picks images from source providers creates instances with the following images on it load images in target cloud seever and delete instance</li><li>There are two parts in this script builders-source location of image provisioner- process of things to do along with necessary image like extra s/w installation etc..</li><li><a href="https://www.packer.io/docs/provisioners/index.html">https://www.packer.io/docs/provisioners/index.html</a> — for help</li></ol><blockquote>Terra test</blockquote><h4>References:</h4><ol><li>below link shows all approved modules built by respective cloud providers</li></ol><p><a href="https://registry.terraform.io/modules/">Terraform Module Registry</a></p><h4>— — — — — Will be updating more on this topic stay tuned! — — — —</h4><iframe src="https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Fupscri.be%2Fb2a0d6%3Fas_embed%3Dtrue&amp;dntp=1&amp;display_name=Upscribe&amp;url=https%3A%2F%2Fupscri.be%2Fb2a0d6%2F&amp;key=a19fcc184b9711e1b4764040d3dc5c07&amp;type=text%2Fhtml&amp;schema=upscri" width="800" height="400" frameborder="0" scrolling="no"><a href="https://medium.com/media/0707f5c806284d01a4a13c7b13a91ce3/href">https://medium.com/media/0707f5c806284d01a4a13c7b13a91ce3/href</a></iframe><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=eada2cbb442d" width="1" height="1" alt=""><hr><p><a href="https://medium.datadriveninvestor.com/getting-started-with-terraform-eada2cbb442d">Getting started with Terraform</a> was originally published in <a href="https://medium.datadriveninvestor.com/">DataDrivenInvestor</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[AI in minutes using Google Api]]></title>
            <link>https://medium.com/@lepakshi.avinash/ai-in-minutes-using-google-api-b6436eeb029c?source=rss-da6911ab637b------2</link>
            <guid isPermaLink="false">https://medium.com/p/b6436eeb029c</guid>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[api]]></category>
            <category><![CDATA[google-cloud-platform]]></category>
            <category><![CDATA[ai]]></category>
            <dc:creator><![CDATA[Avinash Lepakshi]]></dc:creator>
            <pubDate>Fri, 19 Jul 2019 13:57:07 GMT</pubDate>
            <atom:updated>2019-07-19T13:57:07.550Z</atom:updated>
            <content:encoded><![CDATA[<p>This blog is about building A.I. using Google built algorithms in backend via api call.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/529/1*XOYCFunV9IWV24vNMjfkGQ.png" /></figure><p>we can build multiple features like image/video classification , translation, speech -&gt;text etc..</p><p><strong>Pre-requisites::</strong></p><ol><li>GCP(Google cloud platform) account</li><li>Jupyter/any python installed framework</li></ol><p><strong>Step:1</strong></p><p>install and import google packages</p><pre>pip install google-cloud-&lt;vision etc...&gt;<br></pre><p><strong>step:2</strong></p><p>From GCP download a json key into our local/cloud environment where the jupyter is running ,this lets our script to connect GCP via these credentials .Check out below link to download service account key</p><p><a href="https://cloud.google.com/iam/docs/creating-managing-service-accounts">Create service accounts | IAM Documentation | Google Cloud</a></p><p>Enable the particular Api from GCP account as well to make use of it.</p><p>Note::Usage of Api’s incur billing have a look on the billed amount as well</p><p><strong>step:3</strong></p><p>Let python know about these credentials by giving below command in console like below</p><pre>export GOOGLE_CREDENTIALS=~/&lt;path&gt;/key.json</pre><pre>vi ~/.profile <br>#add &quot;export GOOGLE_CREDENTIALS=~/&lt;path&gt;/key.json&quot; at end of file</pre><pre>#re-execute<br>source ~/.bashrc<br></pre><p><strong>step:4</strong></p><p>Configuration is done and now comes the fun part “coding!!!”</p><p><strong>#Code for Vision-Api</strong></p><pre>import io<br>from google.cloud import vision</pre><pre>client=vision.ImageAnnotatorClient()</pre><pre>file_name=&#39;./name.jpeg&#39;</pre><pre>with io.open(file_name,&#39;rb&#39;) as image_file:<br>        content =image_file.read()<br>        image = vision.types.Image(content=content)</pre><pre>response = client.image_properties(image=image)<br>props = response.image_properties_annotation<br>print(&#39;Properties of the image:&#39;)</pre><pre>for color in props.dominant_colors.colors:<br>    print(&#39;Fraction: {}&#39;.format(color.pixel_fraction))<br>    print(&#39;\tr: {}&#39;.format(color.color.red))<br>    print(&#39;\tg: {}&#39;.format(color.color.green))<br>    print(&#39;\tb: {}&#39;.format(color.color.blue))</pre><pre>from PIL import Image<br>jpgfile=Image.open(file_name)<br>Image.open(file_name)</pre><p><strong>#code for Translate -Api</strong></p><pre>from google.cloud import translate<br>trans_client=translate.Client()<br>trans_client.get_languages()</pre><pre>def translate(input_text,lang):<br>    target=trans_client.translate(input_text,lang)<br>    return target[&#39;translatedText&#39;]</pre><pre>translate(&#39;happy birthday!&#39;,&#39;Te&#39;)</pre><p>and many more Api’s can be called like above..</p><p>Congrats!!! now you know usage of most advanced google built algorithms for our build from local jupyter notebook with in minutes…..</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=b6436eeb029c" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Getting started with NLP part 4 (Text Classification)]]></title>
            <link>https://medium.com/@lepakshi.avinash/getting-started-with-nlp-part-4-text-classification-610ce6b089a3?source=rss-da6911ab637b------2</link>
            <guid isPermaLink="false">https://medium.com/p/610ce6b089a3</guid>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[clustering]]></category>
            <category><![CDATA[nlp]]></category>
            <dc:creator><![CDATA[Avinash Lepakshi]]></dc:creator>
            <pubDate>Tue, 12 Mar 2019 08:25:34 GMT</pubDate>
            <atom:updated>2019-03-12T08:25:34.656Z</atom:updated>
            <content:encoded><![CDATA[<h4>Split the docs into an meaningful insight</h4><p>In <a href="https://medium.com/@lepakshi.avinash/getting-started-with-nlp-part3-2e00446517e">our previous blogs</a> we have seen the vectorisation i.e. making machine to read meaningful insights out of a group of texts.</p><p>Here our objective is splitting documents into different categories i.e. classification, there are various approaches to achieve this.</p><p>Supervised Learning: When we have documents, with the label available for each document then we follow this approach</p><p>Unsupervised Learning: To classify the documents without having any prior knowledge about types and how to classify</p><h4>1. Unsupervised :</h4><p>i)K-Means clustering</p><p>This is one of the best and popular method to begin with the clustering ..</p><p>Let’s say for example we have some sample of data like below</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/436/1*vpwTDaKitvy5EN2SW0GrpQ.png" /><figcaption>Source: <a href="https://www.r-bloggers.com/hybrid-hierarchical-k-means-clustering-for-optimizing-clustering-outputs-unsupervised-machine-learning/">https://www.r-bloggers.com/hybrid-hierarchical-k-means-clustering-for-optimizing-clustering-outputs-unsupervised-machine-learning/</a></figcaption></figure><p>Here all the dots represents data points in a dimension space and each shape/colour group is a cluster.</p><p>Before going to classification we have to specify how many number of clusters we would be needing, this can be found by some standard procedures like below</p><p>a)silhouette score</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/520/1*9A0-PIRhiU4ZUzxwsL3Wjw.png" /><figcaption>SIlhouette score for 2–18 cluster groups</figcaption></figure><p>The silhouette value is a measure of how similar an object is to its own cluster (cohesion) compared to other clusters (separation). The silhouette ranges from −1 to +1, where a high value indicates that the object is well matched to its own cluster and poorly matched to neighboring clusters. From the above values it shows good to consider 2 clusters</p><p>b)elbow method</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/540/1*OJ9FDQzgvL6dNfi69NC4XA.png" /><figcaption>Error vs Cluster number</figcaption></figure><p>This is the plot of error value towards each cluster size selection as highlighted here 3 is the optimal clusters to be considered as low error for low number of clusters.</p><p>For example our number of clusters==4;</p><p>Logic of K-Means::</p><ol><li>4 random points as centroids and calculate</li><li>distance between them the centroid to every other point and accordingly the nearest distance point goes to that particular cluster , this gives first iterated result,</li><li>Now the centroid would be changed with in each cluster by calculating with centroid formula</li><li>with new centroid points second iteration begins like steps 2&amp;3.</li><li>Thus with multiple iterations we’d be left with the best clustered output as per the distances between them.</li></ol><p>There are multiple distances measures that can be taken care for this i.e.</p><ol><li>Euclidean distance</li><li>Cosine distance…</li></ol><h4>2.Supervised :</h4><p>This is used when the labels are available for the documents below are some of the popular methods to start with,</p><p>i)Naive bayes</p><p>ii)SVM</p><p>iii)Random forest</p><p>iv) using neural networks like ULMFIT, RNN etc..</p><h4>Thanks for reading!!</h4><h4>Clap if you like it /comment for improvements…</h4><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=610ce6b089a3" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Getting started with NLP-part3]]></title>
            <link>https://medium.com/@lepakshi.avinash/getting-started-with-nlp-part3-2e00446517e?source=rss-da6911ab637b------2</link>
            <guid isPermaLink="false">https://medium.com/p/2e00446517e</guid>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[word-embeddings]]></category>
            <category><![CDATA[artificial-neural-network]]></category>
            <category><![CDATA[nlp]]></category>
            <dc:creator><![CDATA[Avinash Lepakshi]]></dc:creator>
            <pubDate>Mon, 10 Dec 2018 14:17:16 GMT</pubDate>
            <atom:updated>2019-01-17T08:58:20.897Z</atom:updated>
            <content:encoded><![CDATA[<h3>Getting started with NLP-part3 (Word Embeddings)</h3><h4>Word →Number</h4><p><a href="https://medium.com/@lepakshi.avinash/getting-started-with-nlp-part2-b2103dbf0c65">In our previous blogs</a> we have seen Intro to NLP along with Terms which helps in data preprocessing, Now we’re one step away from modelling ,Let&#39;s get started then..</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/797/1*gtr9_YNqjuOj3iImhh02-A.png" /><figcaption>Source:Tech Scouter</figcaption></figure><h4>Word Embeddings [Vectorisation]</h4><p>This says making human readable text to-&gt;Machine understandable numbers through which we can do wonders.</p><p>These vectors capture hidden information about a language, like word analogies or semantic. It is also used to improve performance of text classifiers.</p><p>There are many techniques to achieve one of the Traditional ways is</p><p><strong>One-hot encoding</strong> i.e. arranging words in ascending order with vector size=vocabulary size and encoding lower index 1 for lower order words higher index 1 for higher order words</p><pre>ex:- words:[apple,bat,cat,zoo]</pre><pre>apple:[1,0,0,0]....................zoo[0,0,0,1]</pre><p>we all know this is memory consuming and this approach isn’t considering semantics[relationship between words and context]</p><p>To overcome this below are two approaches and we’ll go through Predictive methods..</p><p><strong>i) <em>Count-based methods</em></strong> like <strong><em>Latent Semantic Analysis (LSA) </em></strong>which can be used to compute some statistical measures of how often words occur with their neighboring words in a corpus and then building out dense word vectors for each word from these measures</p><p><strong><em>ii)Predictive methods</em></strong> like <strong><em>Neural Network based language models</em></strong> try to predict words from its neighboring words looking at word sequences in the corpus and in the process it learns distributed representations giving us dense word embeddings such as…</p><h4>1. word 2 vec:</h4><p>This is next level of traditional one-hot encoding which takes context into consideration with the help of neighbouring words and deriving set of numerical values from it.</p><p>There are two types in word 2 vec ;</p><blockquote>i)Skip Gram</blockquote><p>The input word is passed on to an neural network where hidden layer weights are the vector values for the input word.</p><p>ex::</p><p>1. [‘Avinash’, ‘is’, ‘a’ , ‘good’, ‘man’ , ‘and’ , ‘a’ , ‘great’ , ‘king’]</p><p>2.Stop words(a,is,and)→[‘Avinash’, ‘good’, ‘man’ , ‘great’ , ‘king’]</p><p>3.One-hot -encoding →[10000,01000,00100,00010,00001]</p><p>4.Neighbouring words -Considering window-size =2 for skip gram</p><p>i)Avinash &lt;-&gt;[good,man]</p><p>ii)good-&gt;[Avinash,man,great] ,….etc…</p><p>These input and target(neighbouring words) are passed on to neural network as below and the weights derived out of hidden layer will be the vector values for word Avinash(input)</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/800/1*TuzNUQY7tcOxwrX7xzrmrw.png" /><figcaption>Flow chart representation of word2vec 2 dimension Hidden Neural Network</figcaption></figure><p>here the input Layer is the word Avinash[10000] passed on to one hidden layer with two dimensions(embed) user specified , which gives probabilistic values for multiple target words where loss is calculated by softmax activation function and back propagate through hidden layers for optimisation(reduce loss value) which finally picks the best word as output and weights assigned to it accordingly in hidden layer as vector values.</p><p>The nearest word will have similar value of input and farthest will have more difference accordingly.</p><p>Below image shows the advantage of context and neighboring word similarities</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/795/1*CZpu7CyBamqLvLY1doEciw.png" /><figcaption>Word vector Visualisation [<a href="https://www.tensorflow.org/images/linear-relationships.png">https://www.tensorflow.org/images/linear-relationships.png</a>]</figcaption></figure><blockquote>ii)CBOW(continuous bag of words model)</blockquote><p>This is same as Skip Gram except here we pass all target words along with input word one after other and the vector values are defined by taking average of all these target words , this is good for predicting word based on context Flow chart is shown below for reference.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/789/1*ZU9YA-9NXQdX_DSXQMicHQ.png" /><figcaption>Flow chart representation of CBOW with 2 dimension Hidden Neural Network</figcaption></figure><p>Below is another pic which explains flow of data</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/542/1*fe0tERnr-3sRwmR-fARLCg.png" /><figcaption>source:(<a href="https://towardsdatascience.com/understanding-feature-engineering-part-4-deep-learning-methods-for-text-data-96c44370bbfa">https://towardsdatascience.com/understanding-feature-engineering-part-4-deep-learning-methods-for-text-data-96c44370bbfa</a>)</figcaption></figure><p>Here the input word Avinash is passed and target neighbouring words are also passed , for every target word the weights would be updated with multiple back propagations in process of reducing loss and finally the average of all these weights is taken as vector value for word-Avinash.</p><p>It is said that generally “Skip Gram” is slightly better than “CBOW” even though both are almost similar ,I say the result is entirely dependent on the text.</p><p>In either case, it’s just a simple supervised learning problem that we’re training our network on. The stuff we’re given are the features, and the word(s) we’re predicting are the labels</p><p>Below is one more example taken out from net which shows</p><p>words-&gt;one-hot encoding -&gt; embedding value and graph with distance similarity</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/511/1*nFiz6rm274Sst7XU8e6xFQ.png" /><figcaption>Word2Vec (Youtube:-introduce and tensorflow implementation)</figcaption></figure><p><strong>Implementation:</strong></p><p>#For word2vec</p><pre>from gensim.models.word2vec import Word2Vec<br>model=Word2Vec(....)</pre><p>#For doc2vec ~word2vec which gives vector value for each document by #taking average of all vector values of words in a doc</p><pre>model = Doc2Vec(docs, vector_size = 50, window = 300, min_count = 3, workers = 4)<br>#vector_size-dimensionality of vector<br>#window-distance b/w current to predicted word to consider i.e. semantic<br>#min_count-ignore words below this freq <br>#workers-no. of threads in m/c to be utilised for running model<br><em>#sg-</em> whether to use skip-gram or CBOW</pre><p>#output vector values</p><pre>model.docvecs.doctag_syn0</pre><p><strong>Difference between skip-gram &amp; CBOW::</strong></p><p>In skip-gram, you’re <strong>calculating the context word(s)</strong> from the word at the current position in the sentence.</p><p>In CBOW, you’re <strong>calculating the current word</strong> from the context word(s)</p><h4>2. Fast text</h4><p>Word 2 vec has a disadvantage it takes care of only the training corpus words ignoring less spoken words this is overcome by FastText.</p><p>FastText is an extension to Word2Vec proposed by Facebook in 2016. Here each individual word is split into multiple words by character gram</p><p>ex:- apple -&gt;app,ppl,ple [by 3 char gram] which increases the number of words passed onto neural network .</p><p>Now all these n-grams passed as input and summation of all these n-grams constitutes vector value for apple, we also keep vector values of all n-grams which has greater probability that rare words would match to n-gram words and helps in deriving context of rare words from it.</p><p><strong>Implementation:</strong></p><pre>from gensim.models.fasttext import FastText</pre><p>#Convert all text data(from a column) into a list</p><pre>docs_fast = []<br>for i, text in enumerate(df[&#39;col.name&#39;]):<br>    words = text<br>    docs_fast.append(text)</pre><p>#fasttext model</p><pre>fast_model=FastText(docs_fast,min_count=2,sg=0,window=30,workers=4,size=50)</pre><p>#vectors</p><pre>fast_model.wv.syn0</pre><h4>3. Glove</h4><p>Even though the above models are a huge success to model the semantic knowledge hidden in the words, it missed the distance between the words , Word2Vec starts with local individual examples of word co-occurrence pairs whereas GloVe starts with global aggregated co-occurrence statistics across all words in the corpus and other intrinsic statistical properties.</p><p>The basic methodology of the GloVe model is to first create a huge word-context co-occurrence matrix consisting of (word, context) pairs such that each element in this matrix represents how often a word occurs with the context (which can be a sequence of words). The idea then is to apply matrix factorization to approximate this matrix as depicted in the following figure</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/800/1*8UEaG6Bm9krv8N7IDnjvxg.png" /><figcaption>Source:(<a href="https://towardsdatascience.com/understanding-feature-engineering-part-4-deep-learning-methods-for-text-data-96c44370bbfa">https://towardsdatascience.com/understanding-feature-engineering-part-4-deep-learning-methods-for-text-data-96c44370bbfa</a>)</figcaption></figure><p><strong>WC = WF x FC;</strong></p><p>The vector matrix is build at word-feature matrix where this will be propagated multiple time i.e update weights to decrease the error rate</p><p><strong>The GloVe algorithm consists of following steps:</strong></p><ol><li>Collect word co-occurrence statistics in a form of word co-occurrence matrix X. Each element Xij Xij of such matrix represents how often word i appears in context of word j. Usually we scan our corpus in the following manner: for each term we look for context terms within some area defined by a window_size before the term and a window_size after the term. Also we give less weight for more distant words, usually using this formula:</li></ol><p>decay=1/offset</p><p>2. Define soft constraints for each word pair <br> wTiwj+bi+bj=log(Xij) <br> Here wiwi — vector for the main word, wjwj — vector for the context word, bibi, bjbj are scalar biases for the main and context words.</p><p>3. J=∑vi=1∑v j=1f(Xij)(wTiwj+bi+bj−logXij)2</p><p>Refer (<a href="https://nlp.stanford.edu/projects/glove/">https://nlp.stanford.edu/projects/glove/</a>) for official Glove algorithm..</p><p><strong>Implementation:</strong></p><pre><em>!pip install glove==1.0.0<br>from glove import Glove<br>Glove()--Give necessary parameters</em></pre><p>Thanks for reading !!</p><p>Is this helpful!? If yes let me hear by a Clap👏🏻😊</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=2e00446517e" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Grafana for beginners]]></title>
            <link>https://medium.com/@lepakshi.avinash/grafana-for-beginners-1756dae76f9f?source=rss-da6911ab637b------2</link>
            <guid isPermaLink="false">https://medium.com/p/1756dae76f9f</guid>
            <category><![CDATA[visualization]]></category>
            <category><![CDATA[grafana]]></category>
            <category><![CDATA[open-source]]></category>
            <dc:creator><![CDATA[Avinash Lepakshi]]></dc:creator>
            <pubDate>Thu, 29 Nov 2018 07:25:39 GMT</pubDate>
            <atom:updated>2018-11-29T07:25:39.501Z</atom:updated>
            <content:encoded><![CDATA[<p>Grafana is an open source tool for Data query-&gt;Visualise</p><p>Especially good for Time series analysis</p><p><strong>1.Download and installation for Windows:</strong></p><p><a href="https://dl.grafana.com/oss/release/grafana-5.3.4.windows-amd64.zip">click here</a> for installing grafana for a 64 bit machine</p><p>Unzip the folder</p><p>Go to conf folder duplicate sample.ini and rename to custom.ini</p><p>Edit custom.ini, under “# The http port to use” remove “;” which is a comment line and give any port like 8080 below , This is to change default port of grafana 3000 to 8080 because 3000 requires admin access and some restrictions</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/909/1*QGTplsNmyxjIJZARBhwlJQ.png" /></figure><p><strong>2. Execution:</strong></p><p>Go to bin folder and run<strong> grafana-server.exe</strong> from command line</p><p>To run Grafana open your browser and go to the port you configured above, e.g. <a href="http://localhost:8080/">http://localhost:8080/</a></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/929/1*i6M8OiLkaVPZYo-pvYH2_g.png" /><figcaption>Login page for grafana</figcaption></figure><p>For the first time username= admin and password =admin , change password after login</p><p><strong>3. Data source addition</strong></p><p>After logging the page would look like below , The first step for any visualisation is Data ,so lets add data source</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*8QzHH7pnTmYPpQx5NNpZNg.png" /><figcaption>Add data source</figcaption></figure><p>Click on above icon and provide credentials of Database to connect as show below!</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*JCL8-26Q9AkKXLnlJLkqvg.png" /></figure><p>Grafana supports many different storage backends for your time series data (Data Source) like,</p><p>Graphite, InfluxDB, OpenTSDB, Prometheus, Elasticsearch, CloudWatch,MySQL,Microsoft Sql server,PostgreSQL,stackdriver,AWSCloudWatch etc…</p><p><strong>4.Dashboard built</strong></p><p>With Data available we’re ready for visualisations</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*eLmm_UemTEwkod1i-4FQ0g.png" /></figure><p>The above shown are available built in graphs to be selected from.</p><p>Just click on and the graphs are ready, each graph built is called as panels and one dashboard can contain multiple panels i.e. panels from different data sources</p><p><strong>5.Adding Users</strong></p><p>Right now only the admin i.e. the creator the dashboards has access but to let others in the team to view /make changes send them invite to connect like below shown</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*uL_drMcaa6Ct7IcqDPK7eQ.png" /></figure><p><strong>6.Install Apps &amp; plugins</strong></p><p>Right now we have few panels and data sources available to use but the usage is enormous and limitless with the additional plugins available .</p><p>Just click Apps &amp;plugins which opens below page view</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Ho0PhM-WR3gQVmsBE221DA.png" /></figure><p>This has option to select panel/Datasource/App which displays all available items to download and use as per requirement..</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=1756dae76f9f" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Getting Started with NLP -Part2]]></title>
            <link>https://medium.com/@lepakshi.avinash/getting-started-with-nlp-part2-b2103dbf0c65?source=rss-da6911ab637b------2</link>
            <guid isPermaLink="false">https://medium.com/p/b2103dbf0c65</guid>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[nlp]]></category>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[ai]]></category>
            <dc:creator><![CDATA[Avinash Lepakshi]]></dc:creator>
            <pubDate>Mon, 19 Nov 2018 11:15:55 GMT</pubDate>
            <atom:updated>2024-12-18T10:33:16.916Z</atom:updated>
            <content:encoded><![CDATA[<h4>Here we’ll get to know about NLP terms :</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/700/1*IfHc0tX71qLedcwJREnf1g.jpeg" /><figcaption><a href="https://www.pinterest.com/pin/173177548156131039/"><strong>https://www.pinterest.com/pin/173177548156131039/</strong></a></figcaption></figure><p>In <a href="https://medium.com/@lepakshi.avinash/getting-started-with-nlp-677ddd4b396b">previous post we had a brief introduction of NLP</a> and use-cases, now we’ll learn about<strong> NLP TERMS and BASICS </strong>which will be helpful before going into in depth Analysis..</p><p>Let’s understand all of these by a simple example</p><p>Ex:-</p><pre>import pandas as pd</pre><pre>df[‘notes’]= pd.DataFrame(‘This is my first Text on NLP to learn for , welcome to the World of Machine Learning!!.’)</pre><h4><strong>1.Tokenisation &amp; Punctuation Removal:</strong></h4><p>With the text data ready our next step towards NLP will be tokenisation i.e. splitting sentences →words and excluding punctuation&#39;s sample code is shown below!</p><p>NLTK- An inbuilt python module which is very much helpful for NLP preprocessing ,along with NLTK there are other modules like spacy etc… Try out!!</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/677/1*NVQ2cA06H6VcDhHJQMguBw.png" /><figcaption><a href="https://tahirnaeem.net/performance-comparison-of-nlp-libraries-in-text-tokenization">https://tahirnaeem.net/performance-comparison-of-nlp-libraries-in-text-tokenization</a></figcaption></figure><blockquote>#Sample code</blockquote><pre>import string</pre><pre>from nltk.tokenize import word_tokenize</pre><pre>list(string.punctuation) </pre><p>#gives list of punctuation&#39;s add to this for any extra punctuation&#39;s</p><p>#considering df[‘notes’]-data frame with text ,df[‘tokens]-new dataframe with tokenized and punctuation removed</p><pre>df[‘tokens’] = df[‘notes’].apply(lambda doc: [i.lower() for i in word_tokenize(doc) if i.lower() not in punctuations])</pre><h4><strong>2.Vocabulary:</strong></h4><p>Vocabulary is the set of unique words from our text documents</p><p>Ex:- statement:- ‘Im good and he is also good’</p><p>vocabulary:-[Im good and he is also]</p><h4><strong>3.Stop Words:</strong></h4><p>As the name tells this is the process of removing unnecessary words from out text which might not be helping for our goal, NLTK package has an inbuilt set of stop-words we can add some more to the existing if needed</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/546/1*DFjozbJlaSEyGix83SrlpA.png" /><figcaption>(<a href="http://opening.download/view.php?pic=http://www.geeksforgeeks.org/wp-content/uploads/Stop-word-removal-using-NLTK.png">http://opening.download/view.php?pic=http://www.geeksforgeeks.org/wp-content/uploads/Stop-word-removal-using-NLTK.png</a>)</figcaption></figure><blockquote>#<strong>Sample code</strong></blockquote><pre>from nltk.corpus import stopwords</pre><pre>#gives list of stopwords from NLTK corpus</pre><pre>Stopwords=set(stopwords.words(&#39;english&#39;))<br>df[&#39;stop_words&#39;]=datadf&#39;stop_words&#39;].apply(lambda doc: [i.lower() for i in doc if i.lower() not in Stopwords])</pre><h4><strong>4. Parts of speech tagging:</strong></h4><p>We know parts of speech in English, the same one but with many variants below are few examples, This is very much helpful for the machine to understand what does a specific word mean to based on the context without tagging’s machine takes the words as Noun’s by default!</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/550/1*_eTmCUcPbsi5Szxs4Jk3TA.jpeg" /></figure><p><strong>Parts of Speech(POS),example</strong></p><ul><li>NN noun, singular ‘desk’</li><li>NNS noun plural ‘desks’</li><li>NNP proper noun, singular ‘Harrison’</li><li>NNPS proper noun, plural ‘Americans’</li><li>PDT predeterminer ‘all the kids’</li><li>POS possessive ending parent’s</li><li>PRP personal pronoun I, he, she</li></ul><p>etc…</p><blockquote>#<strong>Sample code :</strong></blockquote><pre>df[‘pos_tag’]=df[‘stop_words’].apply(lambda doc:[i for i,j in   nltk.pos_tag(doc)])</pre><h4><strong>4.Lemmatization:</strong></h4><p>This is the process of converting words to its base form like went-&gt;go.</p><p>we have an prebuilt models which has a dictionary of words and its base form,</p><p>For example if we are using “WordNetLemmatizer” an inbuilt dictionary of words with in nltk , every token of our corps would lookup for a word in WordNetLemmatizer dictionary and changes the word to its base form subsequently</p><p>Advantage: By this similar meaning words would be clubbed instead of having different words in vocabulary</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/252/1*kKcpmDEubhvjMOGtJkzULw.png" /><figcaption>Word →Lemma word</figcaption></figure><blockquote>#<strong>Sample code:</strong></blockquote><pre>from nltk.stem import WordNetLemmatizer</pre><pre>lemmatiser=WordNetLemmatizer() </pre><pre>data[‘lemmatise’]=data[‘pos_tag’].apply(lambda doc: [lemmatiser.lemmatize(i) for i in doc])</pre><h4><strong>5.Stemming:</strong></h4><p>This is a process of bringing word to its root form by removing suffixes+Prefixes</p><blockquote>#<strong>Sample code</strong></blockquote><pre>from nltk.stem.porter import PorterStemmer</pre><pre>stemmer = PorterStemmer()</pre><pre>df[‘stem’]=df[‘lemmatise’].apply(lambda doc: stemmer.stem(i) for i in doc)</pre><h4><strong>6.Named Entity Recognition:</strong></h4><p>This is very much useful for extracting things out of text like name of person, place, object etc…</p><p>Suppose we have a large documents which would be hard for a human to skim through all the documents and list out people , organisations,places involved in it, but with this Named entity recognition it will be easy!</p><blockquote>Ex:-</blockquote><figure><img alt="" src="https://cdn-images-1.medium.com/max/480/1*4AELiC2rYdgfkH9pS5XYKw.jpeg" /></figure><p>Person-&gt;Tim Cook</p><p>Place-&gt;Cuperting , Flint Center</p><p>Organisation -&gt;Apple</p><blockquote>#<strong>Sample code</strong></blockquote><pre>df[‘NER’] = nltk.ne_chunk(‘tagged_sentences’)</pre><h4>8.Unigram/Bigram/Trigram…:</h4><p>Let’s suppose we have a sentence like ‘I don’t like’ the sentiment here is pulled towards positive (like) but it’s opposite in these cases we can keep bigram which takes combo (don’t like) as a single word which solves our problem.</p><p>Unigram-as name says it’s word independent</p><p>Trigram-three words combo</p><p>Ex:-Bigram (‘dont’ , ‘like’) ; trigram(‘dont’,’like’,’ice-cream’)</p><h4><strong>9.Corpus:</strong></h4><p>A set of whole text combined is known as corpus, this is like an language learner to our machine.</p><p>There are lot of corpus like wiki text, wordnet etc..</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/210/1*3tS609Gdxv5me2cm7dpafg.jpeg" /><figcaption>Group of documents(sentences,words..) [<a href="https://www.thoughtco.com/what-is-a-grammatical-function-1690821">https://www.thoughtco.com/what-is-a-grammatical-function-1690821</a>]</figcaption></figure><h4><strong>10.Vectorisation:</strong></h4><p>Now that the cleaning part of our text (words) are done let’s get into world of machine by converting words to numbers through which machine can interpret .</p><p>The process of converting words – &gt; numerical vectors is vectorisation</p><p>Vector?- set of numerical values assigned to a word by its semantics i.e. Understanding it’s relation with neighbouring words check here for more info</p><p>There are many variations to do this below are some of the popular methods;</p><p><strong>10.1 Bag of words:</strong></p><p>As name says collect similar words into a bag and finally count the words in each bag</p><p><strong>10.2 TF-IDF :</strong></p><p>a.k.a Term frequency -Inverse document frequency, which compares words found in one doc to all other docs</p><p><strong>10.3 word 2 vec:</strong></p><p>This is an google built word-2-vec model which runs on neural networks including semantics.</p><p><strong>10.4 glove etc…..</strong></p><h4><strong>Total flow::</strong></h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/889/1*8lcI5gq7hWgZD4Eonrv-dA.png" /><figcaption>NLP-Process</figcaption></figure><h4>Will be writing more on “vectorisation” in a new blog!! :-)</h4><p>Thanks for reading!!</p><p><a href="https://medium.com/@lepakshi.avinash/getting-started-with-nlp-part3-2e00446517e">click here for next part</a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=b2103dbf0c65" width="1" height="1" alt="">]]></content:encoded>
        </item>
    </channel>
</rss>