<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Happiest Neurons - Medium]]></title>
        <description><![CDATA[We are innovative, transparent, reliable and human centric - Medium]]></description>
        <link>https://medium.com/happiestneurons?source=rss----e1b9c06853df---4</link>
        <image>
            <url>https://cdn-images-1.medium.com/proxy/1*TGH72Nnw24QL3iV9IOm4VA.png</url>
            <title>Happiest Neurons - Medium</title>
            <link>https://medium.com/happiestneurons?source=rss----e1b9c06853df---4</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Wed, 18 Nov 2026 16:20:39 GMT</lastBuildDate>
        <atom:link href="https://medium.com/feed/happiestneurons" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[How to add new SVG Illustrations to Pose-Animator ??]]></title>
            <link>https://medium.com/happiestneurons/how-to-add-new-svg-illustrations-to-pose-animator-aeef50ad0cd?source=rss----e1b9c06853df---4</link>
            <guid isPermaLink="false">https://medium.com/p/aeef50ad0cd</guid>
            <category><![CDATA[posenet]]></category>
            <category><![CDATA[tensorflowjs]]></category>
            <category><![CDATA[poae-animator]]></category>
            <category><![CDATA[svg-animation]]></category>
            <category><![CDATA[javascript]]></category>
            <dc:creator><![CDATA[Anjana Pradeep]]></dc:creator>
            <pubDate>Tue, 16 Jun 2020 03:16:29 GMT</pubDate>
            <atom:updated>2020-06-12T18:19:32.125Z</atom:updated>
            <content:encoded><![CDATA[<h3>How to add new SVG Illustrations in Pose-Animator ???</h3><p>There are so many Developers (who are not familiar with computer graphics, Illustrations) struggling to know the logic behind how to create own illustrations in Pose-animator tool.</p><p>Who should continue reading this full blog?</p><ol><li>Any Developer who is working with Pose-Animator.</li><li>Any Developer who has an interest in knowing what is Pose-Animator and how it works.</li><li>Any developer who wants to make your favourite cartoon illustrations to mimic you, in the browser.</li></ol><figure><img alt="" src="https://cdn-images-1.medium.com/max/800/1*m1pQS4WP-_wvl2D7dpQL_w.gif" /></figure><p>Starting with, there is a common explanation given in the <a href="https://github.com/yemount/pose-animator">GitHub repo</a> on how to create new illustrations. But, for people who still find it difficult, there is an alternative.</p><p><strong>STEPS TO FOLLOW:</strong></p><blockquote>Have a look at the Github repo of Pose-Animator</blockquote><blockquote>(link: <a href="https://github.com/yemount/pose-animator">https://github.com/yemount/pose-animator</a>)</blockquote><ol><li>In the folder, go to Resources=&gt;Illustration =&gt;and download any of your favourite illustrations(Let’s say I am downloading oy.svg file)</li><li>Go to Illustration=&gt;samples=&gt; download the skeleton Svg file.</li><li>Open any of t=your favourite graphic editor tools(Inkscape, CoralDraw, Illustrator etc…)</li><li>Open the Boy .svg file (you should be able to see a boy illustration with the skeleton file embedded to it.)</li><li>Go to Objects option and notice the layers in it. You can see that for any svg file that you download from the repo, there is a similarity in grouping and layering the objects of the illustration and the Skeleton. You can see that the layer 1 has a group of objects named illustration on top of which our skeleton group lies. So, our Object panel should show a hierarchy of something like this,</li><li>[Layer 1]<br> |---- skeleton<br> |---- illustration<br> |---- path 1<br> |---- path 2<br> |---- path 3</li><li>But, Pose-Animator works on well, only when it finds two layers as the file structure and nothing more than that. So, when we add any new object, there is a problem.</li><li>So, we should not disturb the hierarchy but still make changes in the file. We can accomplish this task by adding all the object paths to the Illustration layer. If you are dealing with an already vectorized svg file, then make sure you ungroup all the objects of the vectorized file and then add it to the existing Illustration layer.</li><li>Once done, delete all the other objects that belong to boy.svg file. If you are editing the Boy .svg file without including any additional objects, then do not disturb the hierarchy, let ti be the same.</li><li>As the last step, move the skeleton.svg on top of the present illustration in the same way how it was when downloaded.</li></ol><p>Now, save the illustration in the same filename or into a new file in the illustration folder, then open the demo site of Pose-Animator and drag&amp;drop the svg file. Your svg file should appear in Browser now!!</p><p><strong>TIPS THAT CAN HELP YOU:</strong></p><ul><li>Start with editing existing objects in the file.</li><li>While creating new svg illustrations, make it very simple. The simpler the layers are, the more effective Pose-animator will work.</li><li>When working with more number of layers, make sure that the particular number of layers forming a hand or an eye in your illustration is grouped so that the layers do not go out of scope while getting animated.</li><li>Always try to save it in another file(this minimises the number of downloads from the Github :P</li></ul><p>/////////////////////////</p><p>HAPPY ANIMATION</p><p>////////////////////////</p><p><strong>INSPIRATION FROM:</strong></p><p><strong>Pose Animator — An open source tool to bring SVG characters to life in the browser via motion capture</strong></p><p><a href="https://blog.tensorflow.org/2020/05/pose-animator-open-source-tool-to-bring-svg-characters-to-life.html">Pose Animator - An open source tool to bring SVG characters to life in the browser via motion capture</a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=aeef50ad0cd" width="1" height="1" alt=""><hr><p><a href="https://medium.com/happiestneurons/how-to-add-new-svg-illustrations-to-pose-animator-aeef50ad0cd">How to add new SVG Illustrations to Pose-Animator ??</a> was originally published in <a href="https://medium.com/happiestneurons">Happiest Neurons</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Extensive thoughts on Machine Learning in JavaScript]]></title>
            <link>https://medium.com/happiestneurons/extensive-thoughts-on-machine-learning-in-javascript-878802398459?source=rss----e1b9c06853df---4</link>
            <guid isPermaLink="false">https://medium.com/p/878802398459</guid>
            <category><![CDATA[tensorflowjs]]></category>
            <category><![CDATA[posenet]]></category>
            <category><![CDATA[javascript]]></category>
            <dc:creator><![CDATA[Anjana Pradeep]]></dc:creator>
            <pubDate>Tue, 16 Jun 2020 03:15:16 GMT</pubDate>
            <atom:updated>2020-06-07T07:47:20.821Z</atom:updated>
            <content:encoded><![CDATA[<p>Machine Learning models can now be deployed with so much of ease. Do you</p><ol><li><em>Want to track Jackie Chan’s Karate moves with at most accuracy?</em></li><li><em>Want to learn to master dance moves, online?</em></li><li><em>Want to create and play great VR games with your body gestures and movements?</em></li><li><em>Want to knock out the AR world with spectacular animations?</em></li></ol><p>All you have to know about is “<em>Pose Estimation</em>”. But how do we estimate poses?</p><p>Here’s the answer,</p><p><strong><em>PoseNet </em></strong>is a fantastic Machine Learning model, i.e.) a pre-trained model trained to detect and estimate poses from human images, videos and gifs. <em>PoseNet</em> helps in sketching a 2D skeletal animation of the human poses. <em>PoseNet</em> helps in determining single pose as well as multiple poses, which makes things a lot easier.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/222/1*iEF3xpEM1aDMgwgui_-ZTA.png" /><figcaption>2D skeleton image</figcaption></figure><p><strong>PART 1: A LITTLE UNDERSTANDING ABOUT THE MODEL IS WORTH KNOWING</strong></p><p>Before we get into the algorithm, there are few <strong><em>basic</em></strong> <strong><em>terminologies </em></strong>to get familiar with!</p><p><em>“Keypoints”</em>: These are the significant body parts/joints, which gives life to the skeleton being sketched. E.g. Right-shoulder, Left-leg, Eyes etc…). PoseNet works with detecting 17 key points to provide more accuracy from each pose.</p><p><em>“Keypoint-Position”</em>: These coordinates[x, y], provide the position corresponding to each key point.</p><p><em>“Pose Confidence-Score”</em>: These scores range from 0 to 1 which provide us with information about how confident or correct the model is about its prediction.</p><p><em>“Output-Stride</em>:<em> </em>This helps in increasing the speed of estimation and as well as the accuracy of the poses. It is usually 8 or 16(Ideal) or 32. Providing higher output stride makes the estimation fast, thereby reducing accuracy and vice versa.</p><h4>Working of the model:</h4><p>Firstly, an input of RGB image or Video is fed into a convolutional neural network, which returns nothing but (Terminology alert!) Heat maps and offset vectors.</p><p>“<em>Heat Maps</em>”: A 3D tensor of size (Resolution²) * key point count.</p><p>“<em>Offset Vectors</em>”: A 3D tensor of size (Resolution²) * twice the key point count.</p><p>Both help in locating the coordinates with a high confidence score, thereby resulting in returning the object properties.</p><h4>Now, what would be the output of the model?</h4><p>The model would return a bunch of objects which holds the information about the coordinates, confidence score, and many more. With these objects, we try to map and sketch a 2D illustration of the skeleton, which is further animated to provide mind-blowing results.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/399/1*2m34lpGnNqAVuiasY2jC1w.png" /><figcaption>Object properties</figcaption></figure><p><strong>PART-2: BUT TRAINING SUCH HEAVY MODELS IN LOCAL MACHINE DOES NOT SAVE TIME! :</strong></p><p>Google Developers have saved us and we are out of this trouble too!</p><p>Google has released “TensorFlow.js” which is a JavaScript library used to develop and deploy Machine Learning models in Web Browsers using Node.js or python server directly without any installation trouble. All you’ll have is, to play with the terminal for a few necessary installations. For developers, who are interested in Javascript much more than Machine Learning through Python, this is a great way to excel.</p><p><strong>INSTALLATION GUIDE:</strong></p><p>1.) Install npm.</p><p>DOWNLOAD LINK: <a href="https://www.npmjs.com/get-npm">https://www.npmjs.com/get-npm</a></p><p>2.) Install Node.js(latest version, preferably).</p><p>DOWNLOAD LINK: <a href="https://nodejs.org/en/">https://nodejs.org/en/</a></p><p>3.) Install React.js(latest version).</p><p>DOWNLOAD LINK: <a href="https://react-cn.github.io/react/downloads.html">https://react-cn.github.io/react/downloads.html</a></p><p>4.) Install TensorFlow.js via npm or yarn or script tag.</p><p>In command prompt type<em>, npm install @tensorflow/tfjs</em> or <em>yarn add @tensorflow/tfjs</em></p><p>Or, add the Script tag in the Index.html file of the project</p><p>&lt;script src=”https://cdn.jsdelivr.net/npm/@tensorflow/tfjs@2.0.0/dist/tf.min.js&quot;&gt;&lt;/script&gt;</p><p>5.) Install PoseNet Model via script tag or npm.</p><p>In command prompt type<em>, npm install @tensorflow-models/posenet</em></p><p>Now that you’ve installed all the libraries and packages create a new React application and load all the modules required. “React-PoseNet” helps as a wrapper component to help deploy models to React. With TensorFlow.js installed in the, you can develop and deploy PoseNet in web browsers directly with so much of ease.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=878802398459" width="1" height="1" alt=""><hr><p><a href="https://medium.com/happiestneurons/extensive-thoughts-on-machine-learning-in-javascript-878802398459">Extensive thoughts on Machine Learning in JavaScript</a> was originally published in <a href="https://medium.com/happiestneurons">Happiest Neurons</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Docker]]></title>
            <link>https://medium.com/happiestneurons/docker-203d41750580?source=rss----e1b9c06853df---4</link>
            <guid isPermaLink="false">https://medium.com/p/203d41750580</guid>
            <category><![CDATA[intro-to-docker]]></category>
            <category><![CDATA[docker]]></category>
            <dc:creator><![CDATA[kamalnath L P]]></dc:creator>
            <pubDate>Fri, 15 May 2020 14:43:04 GMT</pubDate>
            <atom:updated>2019-08-24T04:38:29.437Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*b45WH7HHExZq88GsFxtNDA.png" /></figure><h3><strong>Flashback of Docker:</strong></h3><p>DotCloud, Inc. in 2010 by <a href="https://fr.wikipedia.org/wiki/Solomon_Hykes">Solomon Hykes</a>. Hykes started the Docker project in France as an internal project within <a href="https://en.wikipedia.org/wiki/DotCloud">dotCloud</a>, a <a href="https://en.wikipedia.org/wiki/Platform_as_a_service">platform-as-a-service</a> company. On September 19, 2013, dotCloud and Red Hat announced an alliance to integrate <strong>Docker</strong> with OpenShift Red Hat’s Platform-as-a-Service (PaaS) offering.</p><p>In the early days dotCloud hosted a contest on 99Designs, many proposed different ideas, but the whale got the top rating and became the winner of the contest. The whale in Docker’s logo is named Moby Dock.</p><p>Docker logo is created by Ricky Asamanis, a graphic designer that lives in Indonesia, and is recognized as one of the top designers in the 99Designs community.</p><p>Docker and its libraries are written in Go programming language.</p><h3><strong>What is Docker?</strong></h3><p>Docker is a software container platform to deploy and run applications by using <strong>Containers</strong> →allow developers to package up an application with all libraries, dependencies, and development requirements.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/312/1*qL2cOmUYgr9_M0eEp1H6hA.jpeg" /></figure><h3>Docker in SDLC:</h3><p>Software Development Life Cycle(SDLC) aims to produce High-Quality software in the pre-defined frame and format.</p><p>SDLC consists of different phases, which are listed as Requirement Analysis, Design, Development, Testing, Deployment and Maintenance. <strong>Docker</strong> is used in the <strong>deployment stage</strong> of the software development life cycle.</p><h3><strong>Real-Time Example of Docker:</strong></h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/291/1*j961IRDYpFq4UVnYXZz1fQ.jpeg" /></figure><p>In the world, there are different varieties of industries are developing varieties of products ranging from Small/Medium / High. But all the products cannot be transported or shipped through the same medium.</p><p>Here the container comes into existence where any kind of product can be packed into a container and shipped with any transport. Here “container is the common medium”.</p><p>Likewise, In the software application development, all the software dependencies are put together in a single package and deployed. whereas it can be used in all environments and it is portable.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=203d41750580" width="1" height="1" alt=""><hr><p><a href="https://medium.com/happiestneurons/docker-203d41750580">Docker</a> was originally published in <a href="https://medium.com/happiestneurons">Happiest Neurons</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[MultiThreading in Python]]></title>
            <link>https://medium.com/happiestneurons/multithreading-in-python-c03e5a5396b7?source=rss----e1b9c06853df---4</link>
            <guid isPermaLink="false">https://medium.com/p/c03e5a5396b7</guid>
            <category><![CDATA[coding]]></category>
            <category><![CDATA[programming]]></category>
            <category><![CDATA[multithreading]]></category>
            <category><![CDATA[python]]></category>
            <category><![CDATA[learning]]></category>
            <dc:creator><![CDATA[Hari]]></dc:creator>
            <pubDate>Fri, 25 Oct 2019 19:20:08 GMT</pubDate>
            <atom:updated>2019-03-15T15:52:47.330Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*kzPuo0jzwg4knhNf" /></figure><p><strong>Thread to Process to Multithreading</strong></p><p>A <strong>thread</strong> of execution is the smallest sequence of programmed instructions that can be managed independently by a scheduler, which is typically a part of the operating system.We can call a thread an element of a process.</p><p>In computing, a <strong>process</strong> is the instance of a computer program that is being executed. It contains the program code and its activity. Depending on the operating system, a process may be made up of multiple <strong>threads</strong> that execute instructions concurrently.</p><p>The key difference between the two concepts is that a thread is typically a component of a process. Therefore, one process can include multiple threads, which can be executing simultaneously also called <strong>Multithreading</strong></p><p><strong>Threading module in Python</strong></p><p>The threading module in python was originally inspired by Java. The threading module is the most common ways of working with threads in python. Let’s get into an example straight away.</p><p>In the example below we wanted to run Two threads (thread_A &amp; thread_B ) concurrently. To see how these threads run we have created an object of the MyThread class which takes in the task and wait time as arguments. We have used the start and join method to start the calling thread object calling the run() method and waits for the calling thread object to complete before executing the rest of the statements in the program respectively.</p><pre><strong>from </strong>ThreadingClass <strong>import </strong>MyThread<br><br><br>thread_A = MyThread(<strong>&#39;Task_A&#39;</strong>, 1)<br>thread_B = MyThread(<strong>&#39;Task_B&#39;</strong>, 1)<br><br>thread_A.start()<br>thread_B.start()<br><br>thread_A.join()<br>thread_B.join()<br><br><br><strong>print</strong>(<strong>&#39;Finished.&#39;</strong>)</pre><p>To customize a new thread using the imported threading module there are few steps to be followed. Firstly define the subclass of the the threading.Thread class and override the default __init__ and run() methods inside the subclass.</p><pre><strong>import </strong>threading<br><strong>import </strong>time<br><br><br><strong>class </strong>MyThread(threading.Thread):<br>    <strong>def </strong>__init__(self, task, wait):<br>        threading.Thread.__init__(self)<br>        self.task = task<br>        self.wait = wait<br><br>    <strong>def </strong>run(self):<br>        <strong>print</strong>(<strong>&#39;Starting thread %s.&#39; </strong>% self.task)<br>        thread_count_down(self.task, self.wait)<br>        <strong>print</strong>(<strong>&#39;Finished thread %s.&#39; </strong>% self.task)<br><br><strong>def </strong>thread_count_down(task, wait):<br>    counter = 5<br><br>    <strong>while </strong>counter:<br>        time.sleep(wait)<br>        <strong>print</strong>(<strong>&#39;Thread %s counting down: %i...&#39; </strong>% (task, counter))<br>        counter -= 1</pre><p>From the below output we can see that both the threads ran concurrently, both tasks started at the same time independent to each other for drastically increasing the performance.</p><pre>Starting thread Task_A.<br>Starting thread Task_B.<br>Thread Task_A counting down: 5...<br>Thread Task_B counting down: 5...<br>Thread Task_A counting down: 4...<br>Thread Task_B counting down: 4...<br>Thread Task_A counting down: 3...<br>Thread Task_B counting down: 3...<br>Thread Task_A counting down: 2...<br>Thread Task_B counting down: 2...<br>Thread Task_A counting down: 1...<br>Finished thread Task_A.<br>Thread Task_B counting down: 1...<br>Finished thread Task_B.<br>Finished.</pre><pre>Process finished with exit code 0</pre><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=c03e5a5396b7" width="1" height="1" alt=""><hr><p><a href="https://medium.com/happiestneurons/multithreading-in-python-c03e5a5396b7">MultiThreading in Python</a> was originally published in <a href="https://medium.com/happiestneurons">Happiest Neurons</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Slowly Changing Dimension Type 2 (SCD2) in Big query]]></title>
            <link>https://medium.com/happiestneurons/slowly-changing-dimension-type-2-scd2-in-big-query-95330ce31625?source=rss----e1b9c06853df---4</link>
            <guid isPermaLink="false">https://medium.com/p/95330ce31625</guid>
            <category><![CDATA[bigquery]]></category>
            <category><![CDATA[sql]]></category>
            <category><![CDATA[dimensions]]></category>
            <category><![CDATA[data-warehouse]]></category>
            <category><![CDATA[google-cloud-platform]]></category>
            <dc:creator><![CDATA[Hari]]></dc:creator>
            <pubDate>Fri, 25 Oct 2019 19:19:29 GMT</pubDate>
            <atom:updated>2019-03-07T11:04:22.874Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*YAEq_a7haDVPeY4A" /></figure><p>Slowly Changing Dimension is the technique for implementing dimension history in a dimensional data warehouse. There are two predominantly used SCD techniques for most of the usecases, SCD1 and SCD2.</p><p><strong>SCD1: </strong>Updates dimension records by overwriting the existing data — no history of the records is maintained. SCD1 is normally used to directly rectify incorrect data. Basically when the source data changes you directly apply an update on that record in dimension table.</p><p>Considering we have a staging table which holds the records from the source and these records need to be pushed to the target dimension table in the data warehouse (Bigquery), then the below SQL will do a direct update on the existing records and performs an insert for the additional new records. This is acheived using the powerful merge command in bigquery.</p><p><strong>Note : </strong>The examples below just focuses on the generic way or an idea of performing SCD2 on a specific column(qunatity), in a real world scenario when we are dealing with ’n’ number of columns we can induce implementing hash or md5 before updating any record.</p><p><strong>Query:</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*G6KQ_vvdS-t0QIKIuOaEsg.png" /></figure><p><strong>Staging Table:</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/772/1*bovEAkHZ1krEdyIzihNKJg.png" /></figure><p><strong>Target Table before merge:</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1014/1*fqMvqiUXsKAqx7zsqd6Wew.png" /></figure><p><strong>Target Table after merge:</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1012/1*NLXU4otX7CPnrO8eH4Xseg.png" /></figure><p><strong>SCD2: </strong>Maintains dimension history by creating newer ‘versions’ of a dimension record whenever its source changes. SCD2 does not delete or modify existing data. Basically when the source data changes you expire the existing record in the dimension table via an indicator (boolean or date or any standard technique of marking a record as expired by setting a flag).</p><p>Considering we have a staging table which holds the records from the source and these records need to be pushed to the target dimension table in the data warehouse (Bigquery), then the below SQL will expire the old data and insert the new record and performs a normal insert for the additional new records , acheived using the merge command.</p><p><strong>Query:</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*XvUTPUoO9t5vni7h_ECINQ.png" /></figure><p><strong>Staging Table:</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/772/1*bovEAkHZ1krEdyIzihNKJg.png" /></figure><p><strong>Target Table before merge:</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*__Gy4RhWH8l1EKYUwKzWpg.png" /></figure><p><strong>Target Table after merge:</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*kCKPrQNLHrh4sCKDk6FlWw.png" /></figure><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=95330ce31625" width="1" height="1" alt=""><hr><p><a href="https://medium.com/happiestneurons/slowly-changing-dimension-type-2-scd2-in-big-query-95330ce31625">Slowly Changing Dimension Type 2 (SCD2) in Big query</a> was originally published in <a href="https://medium.com/happiestneurons">Happiest Neurons</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Archiving your HDFS data in AWS Cloud for disaster recovery]]></title>
            <link>https://medium.com/happiestneurons/archiving-your-hdfs-data-in-aws-cloud-for-disaster-recovery-2fc3d75ee15f?source=rss----e1b9c06853df---4</link>
            <guid isPermaLink="false">https://medium.com/p/2fc3d75ee15f</guid>
            <category><![CDATA[hadoop]]></category>
            <category><![CDATA[cloud]]></category>
            <category><![CDATA[aws]]></category>
            <dc:creator><![CDATA[Hari]]></dc:creator>
            <pubDate>Fri, 25 Oct 2019 19:18:19 GMT</pubDate>
            <atom:updated>2019-10-25T19:18:18.974Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*rPm49wT135zoXHQv" /></figure><p><strong><em>Requirement</em></strong></p><p>To take backup of our Cluster data for disaster recovery</p><p><strong><em>Approach</em></strong></p><p>We are going to use the Glacier Storage provided by AWS.</p><p><strong><em>About Glacier Storage</em></strong></p><p>Glacier is designed to address the shortcomings of a number of traditional archive solutions, like TAPE and DISK archiving none of which is completely satisfactory.</p><p>Glacier leverages the AWS infrastructure to provide archival storage that addresses the shortcomings of both tape and disk solutions:</p><ul><li><strong>It’s inexpensive:</strong>Glacier costs start at less than $.02 per gigabyte of archival storage. That’s significantly less expensive than disk archive, and even less expensive than tape archive, the previous low-cost archive solution.</li><li><strong>It’s durable:</strong>Glacier uses the S3 infrastructure, which means it can offer the same 99.999999999-percent durability as the S3 service. That’s a lot more reliable than the previous archive solutions.</li><li><strong>It’s convenient:</strong>You just send and retrieve archive files over the Internet, making it simple to extend your current backup solution to Glacier. Many of today’s newer, commercial backup solutions provide deduplication functionality, so if you use one of those, you can be sure that it will soon have an Archive to Glacier option.</li><li><strong>It’s highly scalable:</strong>An archive file can be as large as 40TB, which should be big — enough for anyone.</li><li><strong>It’s secure:</strong>Data is transmitted to and from Glacier over SSL encryption, and the archives themselves are encrypted as well while in storage.</li><li><strong>It’s fast:</strong>Data can be pulled from Glacier in as little as five hours, making it significantly faster than tape archive solutions, which require schlepping out to the archive storage facility.</li></ul><p><strong><em>So how do you send huge data over the Internet, AWS has a couple solutions to this issue</em></strong></p><ul><li><strong>AWS <em>Import/Export</em></strong> is a service that allows lets you to send Amazon physical disk drives with your data on them.</li><li><strong>AWS <em>Direct Connect</em> </strong>is a service offered by Amazon in partnership with network service providers that place a high bandwidth connection between their facilities (or, indeed, your own data center) and AWS.</li></ul><p><strong><em>Implementation architecture</em></strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/885/0*eGDc_6zbebYJ690v.jpg" /></figure><p>Amazon S3 allows users to manage objects’ lifecycles in order to lower the cost of S3 objects that should be removed in a certain period, archive log files that should be stored for backup, or auditing in the future. For example, you can configure a lifecycle policy to automatically delete objects in a week because the objects are a collection of data to create reports and are not needed anymore after that. Otherwise you can archive objects into Amazon Glacier in a month because the objects are system log files, which need not be examined immediately for auditing but need to be examined in a couple of days when it’s needed.</p><p><strong><em>Bottleneck</em></strong></p><p>It takes around 3 to 5 hours to restore the data back to S3</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=2fc3d75ee15f" width="1" height="1" alt=""><hr><p><a href="https://medium.com/happiestneurons/archiving-your-hdfs-data-in-aws-cloud-for-disaster-recovery-2fc3d75ee15f">Archiving your HDFS data in AWS Cloud for disaster recovery</a> was originally published in <a href="https://medium.com/happiestneurons">Happiest Neurons</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
    </channel>
</rss>