<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Swaroop on Medium]]></title>
        <description><![CDATA[Stories by Swaroop on Medium]]></description>
        <link>https://medium.com/@durgaswaroop?source=rss-35e37d0bb61d------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/2*R_dFLkdVzqPn0fBj8qcu8w.jpeg</url>
            <title>Stories by Swaroop on Medium</title>
            <link>https://medium.com/@durgaswaroop?source=rss-35e37d0bb61d------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Tue, 17 Nov 2026 13:31:48 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@durgaswaroop/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[List Tables & Databases in Apache Spark]]></title>
            <link>https://medium.com/@durgaswaroop/list-tables-and-databases-in-spark-2d03594d2883?source=rss-35e37d0bb61d------2</link>
            <guid isPermaLink="false">https://medium.com/p/2d03594d2883</guid>
            <category><![CDATA[apache-spark]]></category>
            <category><![CDATA[python]]></category>
            <category><![CDATA[databricks]]></category>
            <category><![CDATA[spark]]></category>
            <category><![CDATA[pyspark]]></category>
            <dc:creator><![CDATA[Swaroop]]></dc:creator>
            <pubDate>Sun, 26 Jul 2020 09:40:43 GMT</pubDate>
            <atom:updated>2020-08-01T14:09:58.422Z</atom:updated>
            <content:encoded><![CDATA[<p>Tables (or views) are needed if you want to perform SQL like queries on data in Spark.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/500/1*oDQP9Q3Ycz3du1hjhxHaEw.png" /></figure><h4>Databases in Spark</h4><p>Tables exist in Spark inside a database. So, We need to first talk about Databases before going to Tables. If we don’t specify any database, Spark uses the <em>default </em>database.</p><p>We can see the list of available databases with listDatabases:</p><pre><em>&gt;&gt;&gt; </em>spark.catalog.listDatabases()<br>[Database(name=&#39;default&#39;, description=&#39;default database&#39;, locationUri=&#39;file:/~/projects/pyspark-youtube/spark-warehouse&#39;)]</pre><p>As we have not created any new databases, we only see the default database in the list.</p><p>We can also use Spark SQL to get the same information.</p><pre>&gt;&gt;&gt; spark.sql(&#39;show databases&#39;).show()<br>+------------+<br>|databaseName|<br>+------------+<br>|     default|<br>+------------+</pre><p>Apart from the ‘default’ database, there is also a global database global_temp which is not listed here. There can be tables created in the global database too.</p><p>When you start a Spark application, default is the database Spark uses. We can see this with currentDatabase</p><pre><em>&gt;&gt;&gt; spark.catalog.currentDatabase()<br>&#39;default&#39;</em></pre><p>We can create new databases as well. This can be done only with SQL. There is no equivalent catalog API for this.</p><pre><em>&gt;&gt;&gt; spark.sql(&#39;create database freblogg&#39;)</em></pre><p>And now, listing databases will show the new database as well.</p><pre>&gt;&gt;&gt; spark.sql(&#39;show databases&#39;).show()<br>+------------+<br>|databaseName|<br>+------------+<br>|     default|<br>|    freblogg|<br>+------------+</pre><h4>Create Tables in Spark</h4><p>I have a file, shows.csv with some of the TV Shows that I love.</p><pre># shows.csv<br>Name,Release Year,Number of Seasons<br>The Big Bang Theory,2007,12<br>The West Wing,1999,7<br>The Secret Circle,2011,1<br>The Good Place,2016,4<br>Sherlock,2010,4,<br>Death Note,2006,1<br>How I Met Your Mother,2005,9<br>Friends,1994,10<br>Castle,2009,8<br>Elementary,2012,7</pre><p>To create a view called shows from that, we will do</p><pre>&gt;&gt;&gt; tv_shows = spark.read.option(<strong>&#39;</strong>header<strong>&#39;</strong>, <strong>&#39;</strong>true<strong>&#39;</strong>).csv(<strong>&#39;shows.csv&#39;</strong>)<br>&gt;&gt;&gt;<br>&gt;&gt;&gt; tv_shows.show(2)<br>+-------------------+------------+-----------------+<br>|               Name|Release Year|Number of Seasons|<br>+-------------------+------------+-----------------+<br>|The Big Bang Theory|        2007|               12|<br>|      The West Wing|        1999|                7|<br>+-------------------+------------+-----------------+<br>only showing top 2 rows<br>&gt;&gt;&gt;<br>&gt;&gt;&gt; tv_shows.<strong>createOrReplaceTempView</strong>(<strong>&#39;shows&#39;</strong>)</pre><p>That creates a table/view called shows in the default database.</p><p>We can create a view in the global_temp table as well as follows:</p><pre>&gt;&gt;&gt; tv_shows.<strong>createOrReplaceGlobalTempView</strong>(&#39;shows&#39;)</pre><h4>Tables in Spark</h4><p>Tables in Spark can be of two types. Temporary or Permanent. Both of these tables are present in a database. To list them we need to specify the database as well.</p><pre>&gt;&gt;&gt; spark.catalog.listTables(&#39;default&#39;)<br>[Table(name=&#39;shows&#39;, database=None, description=None, tableType=&#39;TEMPORARY&#39;, isTemporary=True)]</pre><p>Note that in the above list database is shown as None . This is because the default database is more metaphorical. The database by itself actually does not exist.</p><p>We can do this listing from Spark SQL as well.</p><pre>&gt;&gt;&gt; spark.sql(&#39;<strong>show tables from default</strong>&#39;).show()<br>+--------+---------+-----------+<br>|database|tableName|isTemporary|<br>+--------+---------+-----------+<br>|        |    shows|       true|<br>+--------+---------+-----------+</pre><p>To list tables from global_temp , we can do it in the same way.</p><pre>&gt;&gt;&gt; spark.sql(&#39;<strong>show tables from global_temp</strong>&#39;).show()<br>+-----------+---------+-----------+<br>|   database|tableName|isTemporary|<br>+-----------+---------+-----------+<br>|global_temp|    shows|       true|<br>|           |    shows|       true|<br>+-----------+---------+-----------+</pre><p>Note that, it also shows the tables in default database when we do this.</p><p>That is how we can see the tables present in various various databases of Spark.</p><p>I have made a youtube video about this topic. Do check it out for some more details and examples:</p><iframe src="https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Fwww.youtube.com%2Fembed%2FobVDb_S0X28%3Ffeature%3Doembed&amp;display_name=YouTube&amp;url=https%3A%2F%2Fwww.youtube.com%2Fwatch%3Fv%3DobVDb_S0X28&amp;image=https%3A%2F%2Fi.ytimg.com%2Fvi%2FobVDb_S0X28%2Fhqdefault.jpg&amp;key=a19fcc184b9711e1b4764040d3dc5c07&amp;type=text%2Fhtml&amp;schema=youtube" width="854" height="480" frameborder="0" scrolling="no"><a href="https://medium.com/media/271df864c84abf3ae424b052fcd1232b/href">https://medium.com/media/271df864c84abf3ae424b052fcd1232b/href</a></iframe><p>Thank you for reading this article. Here are some other articles you might like:</p><ul><li><a href="https://medium.com/swlh/walrus-of-the-sea-python-3-8-b064e6808784">Walrus of the sea | Python 3.8</a></li><li><a href="https://medium.com/@durgaswaroop/a-practical-introduction-to-kafka-storage-internals-d5b544f6925f">A Practical Introduction to Kafka Storage Internals</a></li><li><a href="https://medium.com/@durgaswaroop/understanding-git-octopus-merge-af8b9c688e84">Understanding Git Octopus Merge</a></li></ul><p><strong>References</strong></p><ul><li><a href="https://stackoverflow.com/questions/45929237/how-to-specify-the-database-when-registering-a-data-frame-to-a-table-in-spark1-6">How to specify the database when registering a data frame to a table in Spark1.6</a></li><li><a href="https://stackoverflow.com/questions/49108386/what-is-the-purpose-of-global-temporary-views">What is the purpose of global temporary views?</a></li><li><a href="https://stackoverflow.com/questions/38187084/how-to-create-a-database-in-spark-sql">How to Create a Database in Spark SQL</a></li><li><a href="https://stackoverflow.com/questions/42489130/showing-tables-from-specific-database-with-pyspark-and-hive">Showing tables from specific database with Pyspark and Hive</a></li><li><a href="https://spark.apache.org/docs/2.4.0/api/python/_modules/pyspark/sql/catalog.html">pyspark.sql.catalog - PySpark 2.4.0 documentation</a></li></ul><p><strong>Attribution</strong></p><p>Spark Logo from <a href="https://commons.wikimedia.org/wiki/File:Apache_Spark_logo.svg">https://commons.wikimedia.org/wiki/File:Apache_Spark_logo.svg</a></p><p>Used under License: Apache software foundation / Apache License 2.0 (<a href="http://www.apache.org/licenses/LICENSE-2.0">http://www.apache.org/licenses/LICENSE-2.0</a>)</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=2d03594d2883" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Walrus of the sea | Python 3.8]]></title>
            <link>https://medium.com/swlh/walrus-of-the-sea-python-3-8-b064e6808784?source=rss-35e37d0bb61d------2</link>
            <guid isPermaLink="false">https://medium.com/p/b064e6808784</guid>
            <category><![CDATA[operators]]></category>
            <category><![CDATA[python]]></category>
            <category><![CDATA[assignment]]></category>
            <category><![CDATA[walrus]]></category>
            <category><![CDATA[python3]]></category>
            <dc:creator><![CDATA[Swaroop]]></dc:creator>
            <pubDate>Sun, 12 Jul 2020 13:01:01 GMT</pubDate>
            <atom:updated>2020-07-13T18:40:01.192Z</atom:updated>
            <content:encoded><![CDATA[<h3>Walrus and the Python | Introduction to Walrus Operator</h3><p>Python 3.8 brought in several new features to the language like the positional only arguments, debug f-strings, improved type checking, etc, but none was as talked about as the Walrus operator.</p><h3>Why the name?</h3><p>The operator is <strong>:=</strong>. It looks like the eyes and tusks of a walrus. Hence the <em>affectionate </em>name, “walrus operator”.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*7WzjDUOmL87hV9SorW5u8A.jpeg" /></figure><h3>What is the Walrus Operator?</h3><p>Walrus operator is a new way to perform assignment in <a href="https://docs.python.org/3/whatsnew/3.8.html">Python 3.8</a>, introduced to prevent code duplication. (<a href="https://www.python.org/dev/peps/pep-0572/">PEP-572</a>)</p><h3>How to use it?</h3><p>Let’s see how we can use Walrus operator with five examples.</p><h4>If Conditions</h4><p>Take an example code as follows where we are printing if the length of the list is more than an expected value.</p><pre>numbers = [1, 2, 3, 4, 5]<br><strong>if </strong>len(numbers) &gt;= 4:<br>    print(<strong>f&#39;List is too long. Has {</strong>len(numbers)<strong>} elements&#39;</strong>)</pre><p>Here we are calculating the length twice with len(numbers). This can slow the code if the list is very long. This is where the Walrus operator comes in. We can be simplify this with walrus operator as follows:</p><pre>numbers = [1, 2, 3, 4, 5]<br><strong>if </strong>(length := len(numbers)) &gt;= 4:<br>    print(<strong>f&#39;List is too long. Has {</strong>length<strong>} elements&#39;</strong>)</pre><p>The len(numbers) is calculated once and stored in the length variable with the assignment expression length := len(numbers) . This eliminates the double calculation and still keeps the code length the same.</p><p>Note the brackets around the walrus expression in (length := len(numbers)) &gt;= 4 . This is needed because the Walrus operator has lower precedence than the &gt;= operator. Without the brackets, the expression would have been evaluated as length := (len(numbers) &gt;=4) , which would put a True in length instead of 4 .</p><h4>List Comprehension</h4><p>We can use Walrus operator in List comprehension as well to eliminate computational redundancy.</p><p>Say, we have a piece as follows where we are printing the even squares of numbers.</p><pre>numbers = [1, 2, 3, 4, 5]<br>even_squares = [n * n <strong>for </strong>n <strong>in </strong>numbers <strong>if </strong>n * n % 2 == 0]<br>print(even_squares) # [4, 16]</pre><p>Again here, we are doing the computation of n*n twice. Walrus to the rescue. That above code can be rewritten with := as follows:</p><pre>numbers = [1, 2, 3, 4, 5]<br>even_squares = [s <strong>for </strong>n <strong>in </strong>numbers <strong>if </strong>(s := n * n) % 2 == 0]<br>print(even_squares) # [4, 16]</pre><p>The n*n is computed once and stored in s with s := n*n and that s is returned from the list comprehension without having to recompute it.</p><h4>Calling Functions</h4><p>The next example is for the case where you are using the value from a function in multiple places.</p><p>Say, you have a piece as follows:</p><pre><strong>def </strong>get_value_from_db():<br>    <strong>return </strong>10<br><br><br>x = get_value_from_db()<br>y = [x // 2, x, x * 2] <br>print(y)# [5, 10, 20]</pre><p>The list y gets populated using x . The above code is completely fine on its own but we can choose to rewrite it with walrus operator as follows:</p><pre>y = [(x := get_value_from_db()) // 2, x, x * 2]<br>print(y) # [5, 10, 20]</pre><p>Instead of calculating the x value outside, we can compute it in the list itself and use it, making the code shorter by a line!</p><h4>While Loops</h4><p>Walrus operator can be used with while loops as well, especially with infinite loops.</p><p>Here, we have a password guessing game, where the code loops until the user enters the correct password.</p><pre><strong>while True</strong>:<br>    user_input := input(<strong>&#39;What is the secret word? &#39;</strong>)<br>    <strong>if </strong>user_input == <strong>&#39;swordfish&#39;</strong>:<br>        <strong>break<br><br>    </strong>print(<strong>f&#39;You have entered {</strong>user_input<strong>}. Try again..&#39;</strong>)</pre><p>That code can be made much shorter by using Walrus operator as follows:</p><pre><strong>while </strong>(user_input := input(<strong>&#39;What is the secret word? &#39;</strong>)) != <strong>&#39;swordfish&#39;</strong>:<br>    print(<strong>f&#39;You have entered {</strong>user_input<strong>}. Try again..&#39;</strong>)</pre><p>In just two lines, we were able to capture the same functionality as before.</p><h4>For Loops</h4><p>For the final example, we look at how we can use Walrus operator with in a for-loop.</p><p>Say, we have a list of people as follows:</p><pre>people = [<br>    {<strong>&#39;name&#39;</strong>: <strong>&#39;Adam&#39;</strong>, <strong>&#39;age&#39;</strong>: 25},<br>    {<strong>&#39;name&#39;</strong>: <strong>&#39;Adriana&#39;</strong>, <strong>&#39;age&#39;</strong>: 30},<br>    {<strong>&#39;name&#39;</strong>: <strong>&#39;&#39;</strong>, <strong>&#39;age&#39;</strong>: 28},<br>]</pre><p>Each element in the list is a dictionary with the keys name and age .</p><p>We would iterate over that list as follows, printing each name if it exists.</p><pre><strong>for </strong>person <strong>in </strong>people:<br>    <strong>if </strong>person[<strong>&#39;name&#39;</strong>]:<br>        print(<strong>f&#39;We have {</strong>person[<strong>&quot;name&quot;</strong>]<strong>}&#39;</strong>)</pre><p>This can also ‘simplified’ with the := operator.</p><pre><strong>for </strong>person <strong>in </strong>people:<br>    <strong>if </strong>name := person[<strong>&#39;name&#39;</strong>]:<br>        print(<strong>f&#39;We have {</strong>name<strong>}&#39;</strong>)</pre><p>Of Course, this is not really doing much as the lookup on dictionary is already fast enough but you could do something like that if you choose.</p><p>Those are five different ways in which you can use the Walrus operator.</p><h4>Word Of Caution</h4><p>The introduction of Walrus operator in Python was quite controversial and is infact one of the <a href="https://www.infoworld.com/article/3292936/guido-van-rossum-resigns-whats-next-for-python.html">main reasons for Guido Von Rossum (creator of Python) to step down</a> from his role as <a href="https://en.wikipedia.org/wiki/Benevolent_dictator_for_life">BDFL</a>. A lot of people thought it would obscure the readability of the Python language. This is possible, if the programmer is not careful about where they use Walrus operator.</p><p>As long as you use it responsibly and sparingly, Walrus operator has its place in the language providing a new syntactical outlet for your thoughts.</p><p>I have made a youtube video about this topic. Do check it out for some more details about the Walrus operator along with the above examples:</p><iframe src="https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Fwww.youtube.com%2Fembed%2Fihv0bzTAgZ0%3Ffeature%3Doembed&amp;display_name=YouTube&amp;url=https%3A%2F%2Fwww.youtube.com%2Fwatch%3Fv%3Dihv0bzTAgZ0&amp;image=https%3A%2F%2Fi.ytimg.com%2Fvi%2Fihv0bzTAgZ0%2Fhqdefault.jpg&amp;key=a19fcc184b9711e1b4764040d3dc5c07&amp;type=text%2Fhtml&amp;schema=youtube" width="854" height="480" frameborder="0" scrolling="no"><a href="https://medium.com/media/eac674acefd979e9ac93eb3d854ca01d/href">https://medium.com/media/eac674acefd979e9ac93eb3d854ca01d/href</a></iframe><p>Thank you for reading this article. Here are some other articles you might like:</p><ul><li><a href="https://medium.com/analytics-vidhya/mocking-in-python-with-pytest-mock-part-i-6203c8ad3606">Mocking in Python with pytest-mock | Part I</a></li><li><a href="https://medium.com/@durgaswaroop/a-practical-introduction-to-kafka-storage-internals-d5b544f6925f">A Practical Introduction to Kafka Storage Internals</a></li><li><a href="https://medium.com/@durgaswaroop/understanding-git-octopus-merge-af8b9c688e84">Understanding Git Octopus Merge</a></li><li><a href="https://medium.com/@durgaswaroop/introduction-to-statistics-for-aspiring-data-scientists-70dddfc8bbd4">Introduction To Statistics For Aspiring Data scientists</a></li></ul><h3>Attribution</h3><p>Documentation references from <a href="https://docs.python.org/3/whatsnew/3.8.html">https://docs.python.org/3/whatsnew/3.8.html</a></p><p>Walrus Image — Public Domain — <a href="https://commons.wikimedia.org/wiki/File:Pacific_Walrus_-_Bull_(8247646168).jpg">https://commons.wikimedia.org/wiki/File:Pacific_Walrus_-_Bull_(8247646168).jpg</a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=b064e6808784" width="1" height="1" alt=""><hr><p><a href="https://medium.com/swlh/walrus-of-the-sea-python-3-8-b064e6808784">Walrus of the sea | Python 3.8</a> was originally published in <a href="https://medium.com/swlh">The Startup</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Writing Better Tests in Python with pytest-mock | Part 2]]></title>
            <link>https://medium.com/@durgaswaroop/writing-better-tests-in-python-with-pytest-mock-part-2-92b828e1453c?source=rss-35e37d0bb61d------2</link>
            <guid isPermaLink="false">https://medium.com/p/92b828e1453c</guid>
            <category><![CDATA[python]]></category>
            <category><![CDATA[stub]]></category>
            <category><![CDATA[mock]]></category>
            <category><![CDATA[testing]]></category>
            <category><![CDATA[pytest]]></category>
            <dc:creator><![CDATA[Swaroop]]></dc:creator>
            <pubDate>Sat, 25 Apr 2020 12:51:50 GMT</pubDate>
            <atom:updated>2020-04-28T14:57:58.971Z</atom:updated>
            <content:encoded><![CDATA[<h4>Mocking Functions Part II | Writing Better Tests in Python with pytest-mock</h4><p>In the <a href="https://medium.com/analytics-vidhya/mocking-in-python-with-pytest-mock-part-i-6203c8ad3606?source=friends_link&amp;sk=9c6f3b8a67a9f5107343fb8f413df33b">previous article</a> on mocking, we have seen why mocking is useful and how to install pytest-mock. We have also seen how we can mock a slow function to make our test suite faster.</p><figure><img alt="Mocking with pytest" src="https://cdn-images-1.medium.com/max/1024/0*ll5pRauRXCcbxeFO.png" /></figure><p>In this article, we will learn a few more ways of mocking functions. To recap, we had a module named application1.py with two functions.</p><pre># application1.py<br><br>from time import sleep<br><br>def is_windows():<br>    # This sleep could be some complex operation instead<br>    sleep(5)<br>    return True<br><br><br>def get_operating_system():<br>    return &#39;Windows&#39; if is_windows() else &#39;Linux&#39;</pre><p>The get_operating_system function depends on is_windows function, which was slow (simulated with sleep). To be able to test get_operating_system, we decided to mock the is_windows function.</p><p>The test function was as follows:</p><pre># test_application1.py<br><br>from application1 import get_operating_system<br><br># mocker is a fixture  provided by &quot;pytest-mock&quot;<br>def test_get_operating_system(mocker):<br>    mocker.patch(&#39;application1.is_windows&#39;, return_value=True)<br>    assert get_operating_system() == &#39;Windows&#39;</pre><p>With the patch, the is_windows function will not take 5 seconds anymore and will return a value immediately.</p><p>This can be visualized as shown below. The default flow where get_operating_system calls is_windows would be as follows:</p><figure><img alt="mock visualization" src="https://cdn-images-1.medium.com/max/600/0*UcBOS3ZsfRPV7LDB.png" /></figure><p>When we do the patch, we create a new mocked function that gets called, bypassing the original function.</p><figure><img alt="mock visualization" src="https://cdn-images-1.medium.com/max/725/0*ksWatf1zOvBtQmFo.png" /></figure><p>That is how mocks work. They replace the original objects with a dummy.</p><p>Now, Let’s switch things up a bit. Say, the is_windows function, instead of being part of application, is imported from another module called windows_utils.</p><pre># windows_utils.py<br><br>from time import sleep<br><br>def is_windows():<br>    # This sleep could be some complex operation instead<br>    sleep(5)<br>    return True</pre><p>We import this function into our application module.</p><pre># application2.py<br><br>from windows_utils import is_windows<br><br>def get_operating_system():<br>    return &#39;Windows&#39; if is_windows() else &#39;Linux&#39;</pre><p>And it is the same story again here. We need to mock the slow function. Now, the is_windows function is coming from windows_utils module. So, you might be tempted to do as follows:</p><pre>mocker.patch(&#39;windows_utils.is_windows&#39;, return_value=True)</pre><p>But that will not work the way you might expect. When it comes to mocking, <strong>we always mock where the function is used</strong>. Even though the is_windows is from the windows_utils module, since it is being used in application2.py, we have to mock application2.is_windows instead. This is because of how imports work in python. <strong><em>In Python, whenever you import something into a module, it becomes part of the module</em></strong>. So, you refer to it always as &lt;module&gt;.&lt;imported_thing&gt;.</p><p>So, the test function would remain same as before.</p><pre># test_application2.py<br><br>from application2 import get_operating_system<br><br>def test_get_operation_system(mocker):<br>    mocker.patch(&#39;application2.is_windows&#39;, return_value=True)<br>    assert get_operating_system() == &#39;Windows&#39;</pre><p>As discussed above, your mocking target should reflect how something is imported. In application2.py we have imported is_windows with from windows_utils import is_windows. Hence, our mock target was application2.is_windows.</p><p>Instead, if we just do import windows_utils, our mocking target would also need to change. Let us see how this works with an example.</p><pre># application3.py<br><br>import windows_utils <br><br>def get_operating_system():<br>    return &#39;Windows&#39; if windows_utils.is_windows() else &#39;Linux&#39;</pre><p>Now the mock target for ‘is_windows’ is no longer application3.is_windows as there is no direct reference to &#39;is_windows&#39; in &#39;application3&#39;. Instead we use application3.windows_utils.is_windows. That is the fully qualified reference. The new test is as follows:</p><pre># test_application3.py<br><br>from application3 import get_operating_system<br><br>def test_get_operation_system(mocker):<br>    mocker.patch(&#39;application3.windows_utils.is_windows&#39;, return_value=True)<br>    assert get_operating_system() == &#39;Windows&#39;</pre><p>Instead of using application3.windows_utils.is_windows, we can simplify it to windows_utils.is_windows as well. But referring to everything as application3.&lt;something&gt; keeps it clean and easy to understand.</p><p>That is all for this article. We will see how we can mock objects in the next article. Stay tuned.</p><p>The full list of articles in this series:</p><p><a href="https://medium.com/analytics-vidhya/mocking-in-python-with-pytest-mock-part-i-6203c8ad3606">Mocking Functions Part I</a> <br><a href="https://medium.com/@durgaswaroop/writing-better-tests-in-python-with-pytest-mock-part-2-92b828e1453c">Mocking Functions Part II</a>🢠 Current Article</p><p>If you like this article, you can clap it to encourage me to put out the next article soon. If you think someone you know can benefit from this article, do share it with them.</p><p>If you want to thank me, you can say hi on twitter @durgaswaroop. And, if you want to support me here is my paypal link: paypal.me/durgaswaroop</p><p>Attribution: Python Logo â€” <a href="https://www.python.org/community/logos/">https://www.python.org/community/logos/</a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=92b828e1453c" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Mocking in Python with pytest-mock | Part I]]></title>
            <link>https://medium.com/analytics-vidhya/mocking-in-python-with-pytest-mock-part-i-6203c8ad3606?source=rss-35e37d0bb61d------2</link>
            <guid isPermaLink="false">https://medium.com/p/6203c8ad3606</guid>
            <category><![CDATA[python]]></category>
            <category><![CDATA[mocking]]></category>
            <category><![CDATA[patching]]></category>
            <category><![CDATA[pytest]]></category>
            <category><![CDATA[stubbing]]></category>
            <dc:creator><![CDATA[Swaroop]]></dc:creator>
            <pubDate>Sat, 11 Apr 2020 17:23:26 GMT</pubDate>
            <atom:updated>2020-04-26T13:27:32.108Z</atom:updated>
            <content:encoded><![CDATA[<h3>Mocking functions Part I | Better Unit Testing in Python with pytest-mock</h3><p>Mocking resources in unit tests is just as important and common as writing unit tests. However, a lot of people are not familiar with how to properly mock classes, objects or functions for tests, because the available documentation online is either too short or unnecessarily complicated. One of the main reasons for this confusion — several ways to do the same thing. Every other article out there seems to mock things in a different way. With this series of articles on mocking, I hope to bring some clarity on the topic.</p><figure><img alt="Mocking with pytest" src="https://cdn-images-1.medium.com/max/1024/0*3bM970bQ7UYqsDyy.png" /></figure><h4>Pre-requisite</h4><p>This is a tutorial on Mocking with pytest. I am operating with the assumption that you can write unit tests in Python using pytest.</p><h4>Why Mock?</h4><p>As you are here, reading this article, I will assume that you are familiar with mocking. In case you are not, let us do a quick overview of what it is and why we need it.</p><p>Say, you have a service that collects stock market data and gives information about the top gainers in a particular sector. You get the stock market information from a third party API, and process it to give out the results. Now, to test your code, you would not want to hit the API every time, as it will make the tests slower, and also the API provider would charge you for the extra hits. What you want here is a mock! A mock replaces a function with a dummy you can program to do whatever you choose. This is also called ‘Patching’. For the rest of this series, I am going to use ‘mock’ and ‘patch’ interchangeably.</p><h4>Packages needed for Mocking</h4><p>Unlike the majority of programming languages, Python comes with a built-in library for unit testing and mocking. They are powerful, self-sufficient and provide the functionality you need. The Pytest-mock plugin we will use, is a convenient wrapper around it which makes it easier to use it in combination with pytest.</p><p>If you look up articles on mocking, or if you read through the endless questions on Stackoverflow, you will frequently come across the words Mock, MagicMock, patch, etc. I&#39;m going to demystify them here.</p><p>In Python, to mock, be it functions, objects or classes, you will mostly use Mock class. Mock class comes from the built-in unittest.mock module. From now on, anytime you come across Mock, know that it is from the unittest library. MagicMock is a subclass of Mock with some of the magic methods implemented. Magic methods are your usual dunder methods like__str__, __len__, etc.</p><p>For the most part, it does not matter which one you use, Mock or MagicMock. Unless you need magic methods like the above implemented, you can stick to Mock. Pytest-mock gives you access to both of these classes with an easy to use interface.</p><p>patch is another function that comes from the &#39;unittest&#39; module that helps replace functions with mocks. Pytest mock has a wrapper for this too.</p><h4>Installing Pytest Mock</h4><p>Before you get started with using pytest-mock, you have to install it. You can install it with pip as follows:</p><pre>pip install pytest-mock</pre><p>This is a pytest plugin. So, it will also install pytest, if you have not installed it already.</p><h4>Mocking a simple function</h4><p>As this is the first article, we will keep it simple. We will start by mocking a simple function.</p><p>Say, we have a function get_operating_system that tells us whether we are using Windows or Linux.</p><pre># application.py </pre><pre>from time import sleep  </pre><pre>def is_windows():    <br>    # This sleep could be some complex operation instead<br>    sleep(5)    <br>    return True  </pre><pre>def get_operating_system():    <br>    return &#39;Windows&#39; if is_windows() else &#39;Linux&#39;</pre><p>This function uses another function is_windows to check if the current system is Windows or not. Assume that this is_windows function is quite complex taking several seconds to run. We can simulate this slow function by making the program sleep for 5 seconds every time it is called.</p><p>A pytest for get_operating_system() would be as follows:</p><pre># test_application.py<br><br>from application import get_operating_system<br><br>def test_get_operating_system():<br>    assert get_operating_system() == &#39;Windows&#39;</pre><p>Since, get_operating_system() calls a slower function is_windows, the test is going to be slow. This can be seen below in the output of running pytest which took 5.05 seconds.</p><pre>$ pytest<br>================ test session starts ========================<br>Python 3.7.3, pytest-5.4.1, py-1.8.1, pluggy-0.13.1<br>rootdir: /usr/Personal/Projects/pytest-and-mocking<br>plugins: mock-2.0.0<br>collected 1 item<br><br>test_application.py .                                    [100%]<br><br>================ 1 passed in 5.05s ==========================</pre><p>Unit tests should be fast. We should be able to run hundreds of tests in seconds. A single test that takes five seconds slows down the test suite. Enter mocking, to makes our lives easier. If we patch the slow function, we can verify get_operating_system&#39;s behavior without waiting for five seconds.</p><p>Let’s mock this function with pytest-mock.</p><p>Pytest-mock provides a fixture called mocker. It provides a nice interface on top of python&#39;s built-in mocking constructs. You use mocker by passing it as an argument to your test function, and calling the mock and patch functions from it.</p><p>Say, you want the is_windows function to return True without taking those five precious seconds. We can patch it as follows:</p><pre>mocker.patch(&#39;application.is_windows&#39;, return_value=True)</pre><p>You have to refer to is_windows here as application.is_windows, given that it is the function in the <em>application </em>module. If we only patch is_windows, it will try to patch a function called is_windows in the &#39;test_application&#39; file, which obviously does not exist. The format is always &lt;module_name&gt;.&lt;function_name&gt;. Knowing how to mock correctly is important and we will continue working on it in this series.</p><p>The updated test function with the patch is as follows:</p><pre># &#39;mocker&#39; fixture provided by pytest-mock<br>def test_get_operating_system(mocker):  <br>    # Mock the slow function and return True always<br>    mocker.patch(&#39;application.is_windows&#39;, return_value=True) <br>    assert get_operating_system() == &#39;Windows&#39;</pre><p>Now when you run the test, it will finish much faster.</p><pre>$ pytest<br>============ test session starts ==================<br>Python 3.7.3, pytest-5.4.1, py-1.8.1, pluggy-0.13.1<br>rootdir: /mnt/c/Personal/Projects/pytest-and-mocking<br>plugins: mock-2.0.0<br>collected 1 item<br><br>test_application.py .                          [100%]<br><br>=========== 1 passed in 0.11s ======================</pre><p>As you can see, the test only 0.11 seconds. We have successfully patched the slow function and made the test suite faster.</p><p>Another advantage of mocking - you can make the mock function return anything. You can even make it raise errors to test how your code behaves in in those scenarios. We will see how all of this works and more, in the future articles.</p><p>For now, if you want to test the case where is_windows returnsFalse, write the following test:</p><pre>def test_operation_system_is_linux(mocker):<br>    mocker.patch(&#39;application.is_windows&#39;, return_value=False) # set the return value to be False<br>    assert get_operating_system() == &#39;Linux&#39;</pre><p>Note that all of the mocks &amp; patches set with mocker are function scoped i.e., they will only be available for that specific function. Therefore, you can patch the same function in multiple tests and they will not conflict with each other.</p><p>That is your first introduction to the world of mocking with pytest. We will cover more scenarios in the upcoming articles. Stay tuned, stay safe and stay awesome till then.</p><p>List of articles in this series:</p><p><a href="https://medium.com/analytics-vidhya/mocking-in-python-with-pytest-mock-part-i-6203c8ad3606">Mocking Functions Part I</a> 🢠 Current Article<br><a href="https://medium.com/@durgaswaroop/writing-better-tests-in-python-with-pytest-mock-part-2-92b828e1453c">Mocking Functions Part II</a></p><p>If you like this article, you can like this article to encourage me to put out the next article soon. If you think someone you know can benefit from this article, do share it with them.</p><p>If you want to thank me, you can say hi on twitter <a href="http://twitter.com/durgaswaroop">@durgaswaroop</a>. And, if you want to support me here’s my paypal link: paypal.me/durgaswaroop</p><p>Attribution: Python Logo — <a href="https://www.python.org/community/logos/">https://www.python.org/community/logos/</a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=6203c8ad3606" width="1" height="1" alt=""><hr><p><a href="https://medium.com/analytics-vidhya/mocking-in-python-with-pytest-mock-part-i-6203c8ad3606">Mocking in Python with pytest-mock | Part I</a> was originally published in <a href="https://medium.com/analytics-vidhya">Analytics Vidhya</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Understanding Git Octopus Merge]]></title>
            <link>https://medium.com/@durgaswaroop/understanding-git-octopus-merge-af8b9c688e84?source=rss-35e37d0bb61d------2</link>
            <guid isPermaLink="false">https://medium.com/p/af8b9c688e84</guid>
            <category><![CDATA[git]]></category>
            <category><![CDATA[rebase]]></category>
            <category><![CDATA[git-merge]]></category>
            <category><![CDATA[octopus]]></category>
            <category><![CDATA[git-octopus]]></category>
            <dc:creator><![CDATA[Swaroop]]></dc:creator>
            <pubDate>Mon, 08 Jul 2019 15:54:49 GMT</pubDate>
            <atom:updated>2019-07-09T06:27:13.526Z</atom:updated>
            <content:encoded><![CDATA[<p>The Code for Git merge is one of the most sophisticated piece of software ever written. There is so much stuff that goes inside during a merge that its just mind boggling. Just for that alone, Linus Torvalds could be considered a programming genius. Too bad for other geniuses, he has Linux kernel on his resume.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/383/0*l5AdgHZT8a-GK-3t.png" /></figure><p>As the title suggests this article is about <em>Octopus Merge</em> in Git. For this, I hope you know what a basic Git merge is and what it means to merge. If you’re completely unfamiliar with Git, then I’ve no idea what you’re doing here. You better read up on some Git 101 before jumping to this article.</p><p>Anyway, Just to brush up, this is how a simple/familiar Git merge goes…</p><p>We have a branch called <em>feature</em> that diverged from <em>master</em> at the second commit and went to have four commits of its own.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/600/0*MpK5FXiE6m7-Gejl.png" /></figure><blockquote>Note<em>: For the branch pictures in this article I am using a Git GUI tool called </em><a href="https://www.gitkraken.com/"><em>Git Kraken</em></a><em>. I have been trying it for a few days now, and it looks quite promising. I am a fan of its clean and minimalist UI and have been using it extensively for its beautiful visualization of branches. And above all, it is free for personal, non-commercial use. You can try it out for free.</em></blockquote><p>Now, you want to add those cool new changes on the feature branch to master. The way you do it is by merging (Let’s not talk about ‘Rebasing’ for now. We will look at it another time). So, when you merge this is how it looks like.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/600/0*dBsIOBsOdwM0Ph_J.png" /></figure><p>This is all the usual stuff that we are all familiar with.</p><p>Now there is another type of merge called the <strong>The Octopus Merge</strong>. At least some of you must have heard about it, either from an online video or from a colleague in your office that seems to know everything. Either way the Octopus merge is a really fun way of Merging. You probably won’t get to do this at your work as a lot of companies think this complicates things and we all know how much companies hate complexity. Anyway, Let’s see what it looks like. I have a local git repository with three branches <em>branch1</em>, <em>branch2</em>, <em>branch3</em> along with <em>master</em>. All four of these branches have two extra commits from the point they diverged.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/600/0*CcJhbj7KpuHiBAJe.png" /></figure><p>Now, If you want to merge them, the usual way would be to merge two branches at a time, to finally get to the final combination after three merges as follows.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/600/0*OjqVHhPLdF3cxaz_.png" /></figure><p>This may seem fine and might actually be the only way you would think if not for the Octopus merge. You have three merge commits here and as we know merge commits are noise. They pollute the history of your repository and interrupt the story told by your Git history. So, how about keeping the noise low by just having one Merge commit instead of three. How you ask? Octopus, My friend. All hail the great and mighty Octopus. The way you perform an Octopus is by merging all the branches at once on to the master. To do that you give a command like this</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/600/0*p5r6RavgxCRskZNE.png" /></figure><p>This will merge all the three branches to master. The branches will look something like this. Do you see the reference of Octopus now?</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/800/0*Qdud1_Shp4Rz0vlL.png" /></figure><p>Now, if you know anything about octopuses, you might be wondering that we only have four legs here while an Octopus has 8. Well you are right. Octopuses do have 8 legs (technically 6, as two of them are used as hands) but 4 is good enough. Actually any merge can be called Octopus if you’re merging three or more branches.</p><p>If you are using Git for sometime, you might be wondering why more people haven’t heard about it If ‘Octopus merge’ is so freaking cool and Why more people are not using it. You are right. Octopus sure is awesome, but as I said before, It certainly does complicate things a lot. Especially when dealing with merge conflicts. Merge is hard enough as it is when dealing with just two branches. But if you are merging 5 or 10 branches together it would feel like you’re performing a surgery. You have to be really careful in that case and I am not even sure if any modern GUI tools support diffing 10-way! Also a lot of people tend to go overboard with Octopus.</p><p>Look at this <a href="http://marc.info/?l=linux-kernel&amp;m=139033182525831">message</a> where Linus Torvalds yells (pleasantly) at a guy for creating an Octopus with 66 branches! Imagine that for a second. 66 branches! I wouldn’t want to be the guy that handles merge conflicts on that one! Linux aptly says “<em>that’s not an octopus, that’s a </em><strong><em>Cthulhu</em></strong><em> merge!”</em></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/404/0*Ag1KqxNJ442ysKAG.jpg" /></figure><p>So, a lot of companies don’t really use this. A lot of people won’t even consider this for their Merge strategies.</p><p>A rule of thumb to follow with <strong>Octopus</strong> is to never overdo it. An 8-way octopus merge though borders on crazy hard and insane, is fine but more than that is an overkill. The situations where you have to merge more than 5 or 6 branches tend to be very rare and in those cases may be you can go for an Octopus on a subset of branches at a time and do a Octopus for those. Or may be rethink your merging strategy.</p><p>Either way, I hope this article helped you in understanding something new and gives you some ideas for dealing with complex merges. I hope you will educate your peers and colleagues about this new merge and share this article with them.</p><p>Well, That is all for this article folks. See you again in the next one. Until then, Good Bye.</p><p>Special Thanks to <a href="https://www.gitkraken.com/">Git Kraken</a> team at Axosoft for developing a great tool like Kraken.</p><p>I am <a href="https://twitter.com/durgaswaroop">@durgaswaroop</a> on Twitter</p><h4>Attributions:</h4><p>Cthulhu Image : <a href="https://commons.wikimedia.org/wiki/File:Cthulhu_and_R&#39;lyeh.jpg">https://commons.wikimedia.org/wiki/File:Cthulhu_and_R&#39;lyeh.jpg</a></p><p><em>Originally published at </em><a href="https://www.freblogg.com/2016/12/git-octopus-merge.html"><em>https://www.freblogg.com</em></a><em>.</em></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=af8b9c688e84" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Json Parsing With Python]]></title>
            <link>https://medium.com/@durgaswaroop/json-parsing-with-python-15a41c6fe03a?source=rss-35e37d0bb61d------2</link>
            <guid isPermaLink="false">https://medium.com/p/15a41c6fe03a</guid>
            <category><![CDATA[parsing]]></category>
            <category><![CDATA[json]]></category>
            <category><![CDATA[python]]></category>
            <category><![CDATA[python3]]></category>
            <category><![CDATA[libraries]]></category>
            <dc:creator><![CDATA[Swaroop]]></dc:creator>
            <pubDate>Fri, 26 Apr 2019 12:40:18 GMT</pubDate>
            <atom:updated>2019-04-26T12:40:18.428Z</atom:updated>
            <content:encoded><![CDATA[<p>JSON has become an ubiquitous data exchange format everywhere. Pretty much every service has a JSON API. And since it is so popular, most of the programming languages have built-in JSON parsers. And Of course, Python is no exception. In this article, I’ll show you how you can parse JSON with Python’s json library.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/601/0*rwlPtiwASawh6FGc.png" /></figure><p>JSON parsing in Python is quite straight forward and easy unlike in some languages, where it is unnecessarily cumbersome. Like everything else in Python, You start by importing the library you want.</p><pre>import json</pre><p>In this article, I am going to use the following JSON I got from <a href="http://json.org/example.html">json.org</a></p><pre>{<br>  &quot;menu&quot;: {<br>    &quot;id&quot;: &quot;file&quot;,<br>    &quot;value&quot;: &quot;File&quot;,<br>    &quot;popup&quot;: {<br>      &quot;menuitem&quot;: [<br>        {&quot;value&quot;: &quot;New&quot;, &quot;onclick&quot;: &quot;CreateNewDoc()&quot;},<br>        {&quot;value&quot;: &quot;Open&quot;, &quot;onclick&quot;: &quot;OpenDoc()&quot;},<br>        {&quot;value&quot;: &quot;Close&quot;, &quot;onclick&quot;: &quot;CloseDoc()&quot;}<br>      ]<br>    }<br>  }<br>}</pre><p>We have got a good set of dictionaries and arrays to work with in this data. If you want to follow along, you can use the same JSON or you can use anything else as well.</p><p>The first thing to do is to get this json string into a variable.</p><pre>json_string = &quot;&quot;&quot;{&quot;menu&quot;: {<br>  &quot;id&quot;: &quot;file&quot;,<br>  &quot;value&quot;: &quot;File&quot;,<br>  &quot;popup&quot;: {<br>    &quot;menuitem&quot;: [<br>      {&quot;value&quot;: &quot;New&quot;, &quot;onclick&quot;: &quot;CreateNewDoc()&quot;},<br>      {&quot;value&quot;: &quot;Open&quot;, &quot;onclick&quot;: &quot;OpenDoc()&quot;},<br>      {&quot;value&quot;: &quot;Close&quot;, &quot;onclick&quot;: &quot;CloseDoc()&quot;}<br>    ]<br>  }<br>}}&quot;&quot;&quot;</pre><p>And now we parse this string into a dictionary object with the help of the json library&#39;s loads() method.</p><pre>json_dict = json.loads(json_string)</pre><p>And you’re done. The JSON is parsed and is stored in the json_dict object. The json_dict here is a python dictionary object. If you want to verify, you can do that by calling the type() on it with</p><pre>print(type(json_dict))</pre><p>And it will show that it is &lt;class &#39;dict&#39;&gt;.</p><p>Getting back, We have the entire json object as a dictionary in json_dict object and you can just drill down into the dictionary with the keys. On the top level, We just have one key in the dictionary which is menu. We get can get that by indexing the dictionary with that key.</p><pre>menu = json_dict[&#39;menu&#39;]</pre><p>And of course menu is a dictionary too with the keys id, value, and popup. We can access them and print them as well.</p><pre>print(menu[&#39;id&#39;])            ## =&gt; &#39;file&#39;<br>print(menu[&#39;value&#39;])         ## =&gt; &#39;File&#39;</pre><p>And then finally we’ve got popup which is another dictionary as well with the key menuitem which is a list. We can verify this by checking the types of these objects.</p><pre>popup = menu[&#39;popup&#39;]<br>print(type(popup))           ## =&gt; &lt;class &#39;dict&#39;&gt;</pre><pre>menuitem = popup[&#39;menuitem&#39;]<br>print(type(menuitem))        ## =&gt; &lt;class &#39;list&#39;&gt;</pre><p>And Since menuitem is a list, we can iterate on it and print the values.</p><pre>for item in menuitem:<br>    print(item)</pre><p>And the output is</p><pre>{&#39;value&#39;: &#39;New&#39;, &#39;onclick&#39;: &#39;CreateNewDoc()&#39;}<br>{&#39;value&#39;: &#39;Open&#39;, &#39;onclick&#39;: &#39;OpenDoc()&#39;}<br>{&#39;value&#39;: &#39;Close&#39;, &#39;onclick&#39;: &#39;CloseDoc()&#39;}</pre><p>And of course each of these elements are dictionaries and so you can go further inside and access those keys and values.</p><p>For example, If you want to access New from the above output, you can do this:</p><pre>print(menuitem[0][&#39;value&#39;])  ## =&gt; New</pre><p>And so on and so forth to get any value in the JSON.</p><p>And not only that, json library can also accept JSON responses from web services. One cool thing here is that, web server responses are byte strings which means that if you want to use them in your program you&#39;d have convert them to regular strings by using the decode() method. But for json you don&#39;t have to do that. You can directly feed in the byte string and it will give you a parsed object. That&#39;s pretty cool!</p><p>That is all for this article.</p><p>For more programming articles, checkout <a href="http://freblogg.com/">Freblogg</a> <a href="http://www.freblogg.com/search/label/Python">Freblogg/Python</a></p><p>Some articles on automation:</p><p><a href="https://medium.com/@durgaswaroop/web-scraping-with-python-introduction-7b3c0bbb6053">Web Scraping For Beginners with Python</a></p><p><a href="https://medium.com/@durgaswaroop/my-semi-automated-blogging-workflow-62cba2827986">My semi automated workflow for blogging</a></p><p><a href="https://medium.com/@durgaswaroop/publish-articles-on-blogger-in-just-one-second-2ef45586901">Publish articles to Blogger automatically</a></p><p><a href="http://www.freblogg.com/2017/12/publish-articles-to-your-medium-blog-in_83.html">Publish articles to Medium automatically</a></p><p>If you are interested in contributing to any open source projects and haven’t found the right project or if you were unsure on how to begin, I would like to suggest my own project, <a href="https://github.com/durgaswaroop/delorean">Delorean</a> which is a Distributed Version control system, built from scratch in scala. You can contribute not only in the form of code, but also with usage documentation and also by identifying any bugs in its functionality.</p><p>Thanks for reading. See you again in the next article.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=15a41c6fe03a" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[A Practical Introduction to Kafka Storage Internals]]></title>
            <link>https://medium.com/@durgaswaroop/a-practical-introduction-to-kafka-storage-internals-d5b544f6925f?source=rss-35e37d0bb61d------2</link>
            <guid isPermaLink="false">https://medium.com/p/d5b544f6925f</guid>
            <category><![CDATA[topics]]></category>
            <category><![CDATA[partition]]></category>
            <category><![CDATA[apache]]></category>
            <category><![CDATA[zookeeper]]></category>
            <category><![CDATA[kafka]]></category>
            <dc:creator><![CDATA[Swaroop]]></dc:creator>
            <pubDate>Sun, 05 Aug 2018 19:49:33 GMT</pubDate>
            <atom:updated>2021-09-02T07:43:24.650Z</atom:updated>
            <content:encoded><![CDATA[<p>Kafka is everywhere these days. With the advent of Microservices and distributed computing, Kafka has become a regular occurrence in the architecture of every product. In this article, I’ll try to explain how Kafka’s internal storage mechanism works.</p><p>Since this is going to be a deep dive into Kafka’s internals, I would expect you to have some understanding about Kafka. Although I’ve tried to keep the entry-level for this article pretty low, you might not be able to understand everything if you’re not familiar with the general workings of it. Proceed further with that in mind.</p><figure><img alt="Kafka" src="https://cdn-images-1.medium.com/max/1024/0*5cC2ijcO2fQq10bX.png" /></figure><p>Kafka is typically referred to as a <em>Distributed, Replicated Messaging Queue</em>, which although technically true, usually leads to some confusion depending on your definition of a <em>messaging queue</em>. Instead, I prefer to call it a <strong>Distributed, Replicated Commit Log</strong>. This, I think, clearly represents what Kafka does, as all of us understand how logs are written to disk. And in this case, it is the messages pushed into Kafka that are stored to disk.</p><p>With reference to storage in Kafka, you’ll always hear two terms, Partition and Topic. <strong><em>Partitions</em></strong> are the units of storage in Kafka for messages. And <strong><em>Topic</em></strong> can be thought of as being a container in which these partitions lie.</p><p>You can find all of my new articles on <a href="https://freblogg.com/">https://freblogg.com/</a> including the current article.</p><p>With the basic stuff out of our way, let’s understand these concepts better by working with Kafka.</p><p>I am going to start by creating a topic in Kafka with three partitions. If you want to follow along, the command looks like this for a local Kafka setup on windows.</p><pre>kafka-topics.bat --create --topic freblogg --partitions 3 --replication-factor 1 --zookeeper localhost:2181</pre><p>If I go into Kafka’s log directory, I see three directories created as follows.</p><pre>$ tree freblogg*<br>freblogg-0<br>|-- 00000000000000000000.index<br>|-- 00000000000000000000.log<br>|-- 00000000000000000000.timeindex<br>`-- leader-epoch-checkpoint<br>freblogg-1<br>|-- 00000000000000000000.index<br>|-- 00000000000000000000.log<br>|-- 00000000000000000000.timeindex<br>`-- leader-epoch-checkpoint<br>freblogg-2<br>|-- 00000000000000000000.index<br>|-- 00000000000000000000.log<br>|-- 00000000000000000000.timeindex<br>`-- leader-epoch-checkpoint</pre><p>We have three directories created because we’ve given three partitions for our topic, which means that each partition gets a directory on the file system. You also see some files like <em>index, log etc</em>. We’ll get to them shortly.</p><p>One more thing that you should be able to see from here is that in Kafka, the <strong>topic</strong> is more of a logical grouping than anything else and that the <strong>Partition is the actual unit of storage in Kafka</strong>. That is what is physically stored on the disk. Let’s understand partitions in some more detail.</p><h4>Partitions</h4><p>A partition, in theory, can be described as an immutable collection (or sequence) of messages. We can only append messages to a partition but cannot delete from it. And by “We”, I am talking about the Kafka producer. A producer can’t delete the messages in the topic.</p><p>Now we’ll send some messages into the topic. But before that, I want you to see the sizes of files in our partition folders.</p><pre>$ ls -lh freblogg-0<br>total 20M<br>- freblogg 197121 10M Aug  5 08:26 00000000000000000000.index<br>- freblogg 197121   0 Aug  5 08:26 00000000000000000000.log<br>- freblogg 197121 10M Aug  5 08:26 00000000000000000000.timeindex<br>- freblogg 197121   0 Aug  5 08:26 leader-epoch-checkpoint</pre><p>You see the index files combined are about 20M in size while the log file is completely empty. This is the same case with freblogg-1 and freblogg-2 folders.</p><p>Now let us send a couple of messages and see what happens. To send the messages I’m using the console producer as follows:</p><pre>kafka-console-producer.bat --topic freblogg --broker-list localhost:9092</pre><p>I have sent two messages, first a customary “hello world” and then I pressed the Enter key, which becomes the second message. Now if I print the sizes again:</p><pre>$ ls -lh freblogg*<br>freblogg-0:<br>total 20M<br>- freblogg 197121 10M Aug  5 08:26 00000000000000000000.index<br>- freblogg 197121   0 Aug  5 08:26 00000000000000000000.log<br>- freblogg 197121 10M Aug  5 08:26 00000000000000000000.timeindex<br>- freblogg 197121   0 Aug  5 08:26 leader-epoch-checkpoint<br><br>freblogg-1:<br>total 21M<br>- freblogg 197121 10M Aug  5 08:26 00000000000000000000.index<br>- freblogg 197121  68 Aug  5 10:15 00000000000000000000.log<br>- freblogg 197121 10M Aug  5 08:26 00000000000000000000.timeindex<br>- freblogg 197121  11 Aug  5 10:15 leader-epoch-checkpoint<br><br>freblogg-2:<br>total 21M<br>- freblogg 197121 10M Aug  5 08:26 00000000000000000000.index<br>- freblogg 197121  79 Aug  5 09:59 00000000000000000000.log<br>- freblogg 197121 10M Aug  5 08:26 00000000000000000000.timeindex<br>- freblogg 197121  11 Aug  5 09:59 leader-epoch-checkpoint</pre><p>Our two messages went into two of the partitions where you can see that the log files have a non zero size. This is because <strong>the messages in the partition are stored in the ‘xxxx.log’ file</strong>. To confirm that the messages are indeed stored in the log file, we can just see what’s inside that log file.</p><pre>$ cat freblogg-2/*.log<br>@^@^BÂ°Â£Ã¦Ãƒ^@^K^XÃ¿Ã¿Ã¿Ã¿Ã¿Ã¿^@^@^@^A&quot;^@^@^A^VHello World^@</pre><p>The file format of the ‘log’ file is not one that is conducive for textual representation but nevertheless, you should see the ‘Hello World’ at the end indicating that this file got updated when we have sent the message into the topic. The second message we have sent went into the other partition.</p><p>Notice that the first message we sent, went into the third partition (freblogg-2) and the second message went into the second partition (freblogg-1). This is because Kafka arbitrarily picks the partition for the first message and then distributes the messages to partitions in a round robin fashion. If a third message comes now, it would go into freblogg-0 and this order of partition continues for any new message that comes in. We can also make Kafka choose the same partition for our messages by adding a key to the message. Kafka stores all the messages with the same key into a single partition.</p><p>Each new message in the partition gets an Id which is one more than the previous Id number. This Id number is also called as the <em>Offset</em>. So, the first message is at ‘offset’ 0, the second message is at offset 1 and so on. These offset Id’s are always incremented from the previous value.</p><figure><img alt="Kafka Partitions" src="https://cdn-images-1.medium.com/max/1024/0*vJkjcHVs9ZAN42Gt.png" /></figure><h4>&lt;Quick detour&gt;</h4><p>We can understand those random characters in the log file, using a Kafka tool. Those extra characters might not seem useful to us, but they are useful for Kafka as they are the metadata for each message in the queue. If I run,</p><pre>kafka-run-class.bat kafka.tools.DumpLogSegments --deep-iteration --print-data-log --files logs\freblogg-2\00000000000000000000.log</pre><p>This gives the output</p><pre>Dumping logs\freblogg-2\00000000000000000000.log<br>Starting offset: 0<br><br>offset: 0 position: 0 CreateTime: 1533443377944 isvalid: true keysize: -1 valuesize: 11 producerId: -1 headerKeys: [] payload: Hello World<br><br>offset: 1 position: 79 CreateTime: 1533462689974 isvalid: true keysize: -1 valuesize: 6 producerId: -1 headerKeys: [] payload: amazon</pre><p>(I’ve removed a couple of things from this output which are not necessary for this discussion.)</p><p>You can see that it stores information of the <strong>offset</strong>, <strong>time of creation</strong>, <strong>key and value sizes</strong> etc along with the actual message payload in the log file.</p><h4>&lt;/Quick detour&gt;</h4><p>It is also important to note that <strong>a partition is tied to a broker</strong>. In other words, If we have three brokers and if the folder freblogg-0 exists on broker-1, you can be sure that it will not appear in any of the other brokers. Partitions of a topic can be spread out to multiple brokers but a partition is always present on one single Kafka broker (When the replication factor has its default value, which is 1. Replication is mentioned further below).</p><figure><img alt="Parititions across brokers" src="https://cdn-images-1.medium.com/max/1024/0*gdxNBjcfbC1WUI5-.png" /></figure><h4>Segments</h4><p>We’ll finally talk about those index and log files we’ve seen in the partition directory. Partition might be the standard unit of storage in Kafka, but it is not the lowest level of abstraction provided. Each partition is sub-divided into <strong>segments</strong>.</p><p>A segment is simply a collection of messages of a partition. Instead of storing all the messages of a partition in a single file (think of the log file analogy again), Kafka splits them into chunks called segments. Doing this provides several advantages. Divide and Conquer FTW!</p><p>Most importantly, it makes purging data easy. As previously introduced partition is immutable from a consumer perspective. But Kafka can still remove the messages based on the “Retention policy” of the topic. Deleting segments is much simpler than deleting things from a single file, especially when a producer might be pushing data into it.</p><pre>$ ls -lh freblogg-0<br>total 20M<br>- freblogg 197121 10M Aug  5 08:26 00000000000000000000.index<br>- freblogg 197121   0 Aug  5 08:26 00000000000000000000.log<br>- freblogg 197121 10M Aug  5 08:26 00000000000000000000.timeindex<br>- freblogg 197121   0 Aug  5 08:26 leader-epoch-checkpoint</pre><p>The 00000000000000000000 in front of the log and the index files in each partition folder, is the name of our segment. Each segment file has segment.log, segment.index and segment.timeindex files.</p><p>Kafka always writes the messages into these segment files under a partition. There is always an <em>active</em> segment to which Kafka writes to. Once the segment’s size limit is reached, a new segment file is created and that becomes the newly active segment.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/910/1*OryjuBEx2V_qQ1NRuWzi2Q.png" /></figure><p>Each segment file is created with the offset of the first message as its file name. So, In the above picture, segment 0 has messages from offset 0 to offset 2, segment 3 has messages from offset 3 to 5 and so on. Segment 6 which is the last segment is the active segment.</p><pre>$ ls -lh freblogg*<br>freblogg-0:<br>total 20M<br>- freblogg 197121 10M Aug  5 08:26 00000000000000000000.index<br>- freblogg 197121   0 Aug  5 08:26 00000000000000000000.log<br>- freblogg 197121 10M Aug  5 08:26 00000000000000000000.timeindex<br>- freblogg 197121   0 Aug  5 08:26 leader-epoch-checkpoint<br><br>freblogg-1:<br>total 21M<br>- freblogg 197121 10M Aug  5 08:26 00000000000000000000.index<br>- freblogg 197121  68 Aug  5 10:15 00000000000000000000.log<br>- freblogg 197121 10M Aug  5 08:26 00000000000000000000.timeindex<br>- freblogg 197121  11 Aug  5 10:15 leader-epoch-checkpoint<br><br>freblogg-2:<br>total 21M<br>- freblogg 197121 10M Aug  5 08:26 00000000000000000000.index<br>- freblogg 197121  79 Aug  5 09:59 00000000000000000000.log<br>- freblogg 197121 10M Aug  5 08:26 00000000000000000000.timeindex<br>- freblogg 197121  11 Aug  5 09:59 leader-epoch-checkpoint</pre><p>In our case, we only had one segment in each of our partitions which is 00000000000000000000. Since we don&#39;t see another segment file present, it means that 00000000000000000000 is the active segment in each of those partitions.</p><p>The default value for segment size is a high value (1 GB) but let’s say we’ve tweaked Kafka configuration so that each segment can hold only three messages. Let’s see how that would play out.</p><p>Say this is the current state of the freblogg-2 partition. We&#39;ve three messages pushed into it.</p><figure><img alt="Kafka segment with message" src="https://cdn-images-1.medium.com/max/1024/0*9nEybSlGeO13UESL.png" /></figure><p>Since ‘three messages’ is the limit we’ve set, If a new message comes into this partition, Kafka will automatically close the current segment, create a new segment, make that the active segment and store that new message in the new segment’s log file.</p><pre>(I&#39;m not showing the preceding zeroes to make it easy on the eyes)<br><br>freblogg-2<br>|-- 00.index<br>|-- 00.log<br>|-- 00.timeindex<br>|-- 03.index<br>|-- 03.log<br>|-- 03.timeindex<br>`--</pre><p>You should’ve noted that the name of the newer segment is not 01. Instead, you see 03.index, 03.log. So, what is going on?</p><figure><img alt="Kafka segment with new message" src="https://cdn-images-1.medium.com/max/1024/0*xt_h9Nl1dI87v77Q.png" /></figure><p>This is because Kafka makes the lowest offset in the segment as its name. Since the new message that came into the partition has the offset 3, that is the name Kafka gives for the new segment. It also means that since we have 00 and 03 as our segments, we can be sure that the messages with offsets 0,1 and 2 are indeed present in the 00 segment. New messages coming into freblogg-2 partition with offsets 3,4 and 5 will be stored in the segment 03.</p><p>One of the common operations in Kafka is to read the message at a particular offset. For this, if it has to go to the log file to find the offset, it becomes an expensive task especially because the log file can grow to huge sizes (Default — 1G). This is where the .index file becomes useful. <strong>Index file stores the offsets and physical position of the message in the log file</strong>.</p><p>An index file for the log file I’ve showed in the ‘Quick detour’ above would look something like this:</p><figure><img alt="Index and log file" src="https://cdn-images-1.medium.com/max/1024/0*13sZ658YqtxS_XP0.png" /></figure><p>If you need to read the message at offset 1, you first search for it in the index file and figure out that the message is in position 79. Then you directly go to position 79 in the log file and start reading. This makes it quite effective as you can use binary search to quickly get to the correct offset in the already sorted index file.</p><h4>Parallelism with Partitions</h4><p>To guarantee the order of reading messages from a partition, Kafka restricts to having only one consumer (from a consumer group) per partition. So, if a partition gets messages a,f and k, the consumer will also read them in the order a,f and k. This is an important thing to make a note of as the <strong>order of message consumption is not guaranteed at a topic level</strong> when you have multiple partitions.</p><p>Just increasing the number of consumers won’t increase the parallelism. You need to scale your partitions accordingly. To read data from a topic in parallel with two consumers, you create two partitions so that each consumer can read from its own partition. Also since partitions of a topic can be on different brokers, two consumers of a topic can read the data from two different brokers.</p><h4>Topics</h4><p>We’ve finally come to what a topic is. We’ve covered a lot of things about topics already. The most important thing to know is that <strong>a Topic is merely a logical grouping of several partitions</strong>.</p><p>A topic can be distributed across multiple brokers. This is done using the partitions. But a partition still needs to be on a single broker. Each topic will have its unique name and the partitions will be named from that.</p><h4>Replication</h4><p>Let’s talk about replication. Whenever we’re creating a topic in Kafka, we need to specify what the replication factor we need for that topic. Let&#39;s say we&#39;ve two brokers and so we&#39;ve given the replication-factor as 2. What this means is that Kafka will try to always ensure that each partition of this topic has a backup/replica. The way Kafka distributes the partitions is quite similar to how HDFS distributes its data blocks across nodes.</p><p>Say for the freblogg topic that we&#39;ve been using so far, we&#39;ve given the replication factor as 2. The resulting distribution of its three partitions will look something like this.</p><figure><img alt="Kafka Partition distribution" src="https://cdn-images-1.medium.com/max/769/0*OYXOtpzP24Aq41TB.png" /></figure><p>Even when you have a replicated partition on a different broker, Kafka wouldn’t let you read from it because in each replicated set of partitions, there is a LEADER and the rest of them are just mere FOLLOWERS serving as backup. The followers keep on syncing the data from the leader partition periodically, waiting for their chance to shine. When the leader goes down, one of the in-sync follower partitions is chosen as the new leader and now you can consume data from this partition.</p><p>A Leader and a Follower of a single partition are never in a single broker. It should be quite obvious why that is so.</p><p>Finally, this long article ends. Congratulations on making this far. You now know most of what there is to know about Kafka’s data storage. To ensure that you retain this information let’s do a quick recap.</p><h4>Recap</h4><ul><li>Data in Kafka is stored in topics</li><li>Topics are partitioned</li><li>Each partition is further divided into segments</li><li>Each segment has a log file to store the actual message and an index file to store the position of the messages in the log file</li><li>Various partitions of a topic can be on different brokers but a partition is always tied to a single broker</li><li>Replicated partitions are passive. You can consume messages from them only when the leader is down</li></ul><p>That ought to cover everything we’ve talked about. Thanks for reading. See you again in the next one.</p><p>Attribution:</p><p>Kafka image — <a href="https://kafka.apache.org/images/kafka_diagram.png">https://kafka.apache.org/images/kafka_diagram.png</a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=d5b544f6925f" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Reshaping Pandas Dataframes Melt And Unmelt]]></title>
            <link>https://medium.com/@durgaswaroop/reshaping-pandas-dataframes-melt-and-unmelt-9f57518c7738?source=rss-35e37d0bb61d------2</link>
            <guid isPermaLink="false">https://medium.com/p/9f57518c7738</guid>
            <category><![CDATA[melt]]></category>
            <category><![CDATA[dataframes]]></category>
            <category><![CDATA[pandas]]></category>
            <category><![CDATA[pivot]]></category>
            <category><![CDATA[python]]></category>
            <dc:creator><![CDATA[Swaroop]]></dc:creator>
            <pubDate>Sun, 17 Jun 2018 13:13:26 GMT</pubDate>
            <atom:updated>2021-11-25T19:51:48.629Z</atom:updated>
            <content:encoded><![CDATA[<h4>Reshaping Pandas Data frames with Melt &amp; Pivot</h4><p>Pandas is a wonderful data manipulation library in python. Working in the field of Data science and Machine learning, I find myself using Pandas pretty much everyday. It’s an invaluable tool for data analysis and manipulation.</p><p>In this short article, I will show you what Melt and Reverse melt (Unmelt) are in Pandas, and how you can use them for reshaping data frames.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/384/1*GujwvK5tx3nJAM8TFEhUcg.png" /><figcaption>Happy Panda</figcaption></figure><p>Say, I have the data of the closing prices of stock market data of stock market closing prices of two major companies for last week as follows:</p><pre>+-----+--------+-------+<br>| Day | Google | Apple |<br>+:===:+:======:+:=====:+<br>| MON | 1129   | 191   |<br>+-----+--------+-------+<br>| TUE | 1132   | 192   |<br>+-----+--------+-------+<br>| WED | 1134   | 190   |<br>+-----+--------+-------+<br>| THU | 1152   | 190   |<br>+-----+--------+-------+<br>| FRI | 1152   | 188   |<br>+-----+--------+-------+</pre><p>For an analysis I want to do I need the names of the companies Google &amp; Apple to appear in a single column with the stock price as another column, as shown below.</p><pre>+-----+---------+---------------+<br>| Day | Company | Closing Price |<br>+:===:+:=======:+:=============:+<br>| MON | Google  | 1129          |<br>+-----+---------+---------------+<br>| TUE | Google  | 1132          |<br>+-----+---------+---------------+<br>| WED | Google  | 1134          |<br>+-----+---------+---------------+<br>| THU | Google  | 1152          |<br>+-----+---------+---------------+<br>| FRI | Google  | 1152          |<br>+-----+---------+---------------+<br>| MON | Apple   | 191           |<br>+-----+---------+---------------+<br>| TUE | Apple   | 192           |<br>+-----+---------+---------------+<br>| WED | Apple   | 190           |<br>+-----+---------+---------------+<br>| THU | Apple   | 190           |<br>+-----+---------+---------------+<br>| FRI | Apple   | 188           |<br>+-----+---------+---------------+</pre><p>This is exactly where melt comes into picture. Melt is used for converting a bunch of columns into a single row, which is exactly what I need here.</p><p>Let’s see how we can do this:</p><h4>Melt:</h4><p>First we need to import pandas</p><p>import pandas as pd</p><p>Now, we’ll create the Dataframe with the data I need:</p><pre>df = pd.DataFrame(data = {<br>    &#39;Day&#39; : [&#39;MON&#39;, &#39;TUE&#39;, &#39;WED&#39;, &#39;THU&#39;, &#39;FRI&#39;], <br>    &#39;Google&#39; : [1129,1132,1134,1152,1152], <br>    &#39;Apple&#39; : [191,192,190,190,188] <br>})</pre><p>And this will get us the dataframe we need as follows:</p><pre>+---+-----+--------+-------+<br>|   | Day | Google | Apple |<br>+:=:+:===:+:======:+:=====:+<br>| 0 | MON | 1129   | 191   |<br>+---+-----+--------+-------+<br>| 1 | TUE | 1132   | 192   |<br>+---+-----+--------+-------+<br>| 2 | WED | 1134   | 190   |<br>+---+-----+--------+-------+<br>| 3 | THU | 1152   | 190   |<br>+---+-----+--------+-------+<br>| 4 | FRI | 1152   | 188   |<br>+---+-----+--------+-------+</pre><p>Let’s melt this now. To melt this dataframe, you call the melt() on the dataframe with the id_vars parameter set.</p><pre>reshaped_df = df.melt(id_vars=[&#39;Day&#39;])</pre><p>And you’re done. Your reshaped_df would like this now:</p><pre>+---+-----+----------+-------+<br>|   | Day | variable | value |<br>+:=:+:===:+:========:+:=====:+<br>| 0 | MON | Apple    | 191   |<br>+---+-----+----------+-------+<br>| 1 | TUE | Apple    | 192   |<br>+---+-----+----------+-------+<br>| 2 | WED | Apple    | 190   |<br>+---+-----+----------+-------+<br>| 3 | THU | Apple    | 190   |<br>+---+-----+----------+-------+<br>| 4 | FRI | Apple    | 188   |<br>+---+-----+----------+-------+<br>| 5 | MON | Google   | 1129  |<br>+---+-----+----------+-------+<br>| 6 | TUE | Google   | 1132  |<br>+---+-----+----------+-------+<br>| 7 | WED | Google   | 1134  |<br>+---+-----+----------+-------+<br>| 8 | THU | Google   | 1152  |<br>+---+-----+----------+-------+<br>| 9 | FRI | Google   | 1152  |<br>+---+-----+----------+-------+</pre><p>The id_vars you&#39;ve passed into the melt() method is to specify which column you want to leave untouched. Since we want the Day column to stay even after the melt, we set id_vars=[&#39;Day&#39;].</p><p>Also, you would have noticed that the output dataframe of melt has the columns variable and value. These are the default names given by pandas for the columns. We can change this either manually with something like</p><pre>reshaped_df.columns = [[&#39;Day&#39;, &#39;Company&#39;, &#39;Closing Price&#39;]]</pre><p>Or we can specify the values for these columns in the melt() itself. Melt takes arguments var_name and value_name apart from id_vars. These options specify the names for the `variable` column and the `value` column respectively.</p><pre>reshaped_df = df.melt(id_vars=[&#39;Day&#39;], var_name=&#39;Company&#39;, value_name=&#39;Closing Price&#39;)</pre><p>That will give us:</p><pre>+---+-----+---------+---------------+<br>|   | Day | Company | Closing Price |<br>+:=:+:===:+:=======:+:=============:+<br>| 0 | MON | Apple   | 191           |<br>+---+-----+---------+---------------+<br>| 1 | TUE | Apple   | 192           |<br>+---+-----+---------+---------------+<br>| 2 | WED | Apple   | 190           |<br>+---+-----+---------+---------------+<br>| 3 | THU | Apple   | 190           |<br>+---+-----+---------+---------------+<br>| 4 | FRI | Apple   | 188           |<br>+---+-----+---------+---------------+<br>| 5 | MON | Google  | 1129          |<br>+---+-----+---------+---------------+<br>| 6 | TUE | Google  | 1132          |<br>+---+-----+---------+---------------+<br>| 7 | WED | Google  | 1134          |<br>+---+-----+---------+---------------+<br>| 8 | THU | Google  | 1152          |<br>+---+-----+---------+---------------+<br>| 9 | FRI | Google  | 1152          |<br>+---+-----+---------+---------------+</pre><h4>Unmelt/Reverse Melt/Pivot</h4><p>We can also do the reverse of the melt operation which is also called as pivoting. In Pivoting or Reverse Melting, we convert a column with multiple values into several columns of their own.</p><p>The pivot() method on the dataframe takes two main arguments index and columns. The index parameter is similar to id_vars we have seen before i.e., It is used to specify the column you don&#39;t want to touch. The columns parameter is to specify which column should be used to create the new columns.</p><pre>reshaped_df.pivot(index=&#39;Day&#39;, columns=&#39;Company&#39;)</pre><p>Running the above command gives you the following:</p><pre>+---------+---------------+-------+<br>|         |     Closing Price     |<br>+=========+:=============:+:=====:+<br>| Company | Google        | Apple |<br>+---------+---------------+-------+<br>| Day     |               |       |<br>+---------+---------------+-------+<br>| MON     | 1129          | 191   |<br>+---------+---------------+-------+<br>| TUE     | 1132          | 192   |<br>+---------+---------------+-------+<br>| WED     | 1134          | 190   |<br>+---------+---------------+-------+<br>| THU     | 1152          | 190   |<br>+---------+---------------+-------+<br>| FRI     | 1152          | 188   |<br>+---------+---------------+-------+</pre><p>This is close, but probably not exactly what you wanted. The Closing Price is an extra stacked column (index) on top of Google &amp; Apple. So to get exactly the reverse of melt and get the original df dataframe we started with, we do the following:</p><pre>original_df = reshaped_df.pivot(index=&#39;Day&#39;, columns=&#39;Company&#39;)[&#39;Closing Price&#39;].reset_index()<br>original_df.columns.name = None</pre><p>And that gets us back to what we have started with.</p><pre>+-----+--------+-------+<br>| Day | Google | Apple |<br>+:===:+:======:+:=====:+<br>| MON | 1129   | 191   |<br>+-----+--------+-------+<br>| TUE | 1132   | 192   |<br>+-----+--------+-------+<br>| WED | 1134   | 190   |<br>+-----+--------+-------+<br>| THU | 1152   | 190   |<br>+-----+--------+-------+<br>| FRI | 1152   | 188   |<br>+-----+--------+-------+</pre><p>That is all for this article. I hope this was useful for you and that you’ll try to use this in your data processing workflow.</p><p>For more programming articles, checkout <a href="http://freblogg.com/">Freblogg</a></p><p><a href="http://www.freblogg.com/search/label/Python">Freblogg/Python</a> <a href="http://www.freblogg.com/search/label/Apache%20Spark">Freblogg/Spark</a></p><p>Some python articles that you might find useful:</p><p><a href="https://medium.com/@durgaswaroop/web-scraping-with-python-introduction-7b3c0bbb6053">Web Scraping For Beginners with Python</a></p><p><a href="https://medium.com/@durgaswaroop/my-semi-automated-blogging-workflow-62cba2827986">My semi automated workflow for blogging</a></p><p><a href="http://www.freblogg.com/2017/12/publish-articles-to-your-medium-blog-in_83.html">Publish articles to Medium automatically</a></p><p>Thanks for reading. See you again in the next article.</p><p>Happy Panda Image: <a href="https://pixabay.com/en/animal-asian-bamboo-cartoon-china-2027894/">https://pixabay.com/en/animal-asian-bamboo-cartoon-china-2027894/</a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=9f57518c7738" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[How to stash uncommitted changes with Git Stash]]></title>
            <link>https://medium.com/@durgaswaroop/how-to-stash-uncommitted-changes-with-git-stash-f1c38a1c4fc3?source=rss-35e37d0bb61d------2</link>
            <guid isPermaLink="false">https://medium.com/p/f1c38a1c4fc3</guid>
            <category><![CDATA[git]]></category>
            <category><![CDATA[save]]></category>
            <category><![CDATA[stash]]></category>
            <category><![CDATA[vc]]></category>
            <category><![CDATA[git-stash]]></category>
            <dc:creator><![CDATA[Swaroop]]></dc:creator>
            <pubDate>Sat, 13 Jan 2018 18:24:35 GMT</pubDate>
            <atom:updated>2021-11-28T12:56:42.907Z</atom:updated>
            <content:encoded><![CDATA[<h4>Stash Your Worries Away With Git Stash</h4><p>You are in the middle of developing an important feature and suddenly your manager tells you to work on an urgent fix for a bug in the production code. You haven’t committed your changes, as they are not complete yet. How can you switch to a new branch without losing your local uncommitted changes? Git Stash to your rescue.</p><figure><img alt="Git Logo" src="https://cdn-images-1.medium.com/max/383/0*mbhMaDOeTQn_MKVL.png" /></figure><p>Say, I’ve two commits in my git repository as follows:</p><pre>$ git log --oneline --decorate --graph<br>* 10c532b (HEAD -&gt; master) Add File2<br>* d19fe8d Add File1</pre><p>And I’ve those two files file1 and file2 in my directory.</p><pre>$ ls<br>file1  file2</pre><p>After I have added file2, I&#39;ve made some changes to it that have not been committed yet, as you see from the output of git status below:</p><pre>$ git status<br>On branch master<br>Changes not staged for commit:<br>  (use &quot;git add &lt;file&gt;...&quot; to update what will be committed)<br>  (use &quot;git checkout -- &lt;file&gt;...&quot; to discard changes in working directo<br><br>        modified:   file2<br><br>no changes added to commit (use &quot;git add&quot; and/or &quot;git commit -a&quot;)</pre><p>At this stage, say I have to switch to a different branch or a different commit. I have two usual options. Before switching, I either have to commit these changes or lose them by switching to the other commit. As I don’t want to pick either of those options, I will go with the third option available, which is Stashing.</p><p>You can stash your changes with the following command:</p><pre>git stash save &quot;Changes in file2&quot;</pre><p>To git stash, I pass in the command save along with a message. This message is similar to commit messages, by which you can identify a particular stash.</p><p>You can do the same with just git stash as well. But in this case, you will not be able to give a message for your stash, and so it is good to stick with the previous version of the command.</p><p>You can see the existing list of stashes as follows:</p><pre>$ git stash list<br>stash@{0}: On master: Changes in file2</pre><p>That is the stash I’ve just created, and you can see the branch name master and the stash message I have given as well. The stash@{0} is sort of like the name of your stash.</p><p>Now we have created the stash, how can we use the stash later. For that, you have two options.</p><ol><li>apply — Adds the changes in stash to working directory but keeps the stash</li><li>pop — Adds the changes in stash to working directory and deletes the stash</li></ol><p>To apply a stash:</p><pre>git stash apply stash@{0}</pre><p>This will still keep the stash, and you can see it with list command.</p><p>To pop a stash:</p><pre>git stash pop stash@{0}</pre><p>If you want to delete the stash, you can do:</p><pre>git stash drop stash@{0}</pre><p>And that will delete the stash.</p><p>Git stashes are a great way to quickly stow away your unsaved changes for some later use. Try this out and this can be a really useful tool in your development workflow.</p><p>That is all for this article.</p><p>This article is originally published on <a href="https://freblogg.com/">Freblogg</a>. You can find the original article here — <a href="https://www.freblogg.com/git-stash">https://www.freblogg.com/git-stash</a></p><p>For more programming articles, checkout <a href="http://freblogg.com/">Freblogg</a> , <a href="http://www.freblogg.com/search/label/Git">Freblogg/Git</a></p><p>If you are interested in contributing to any open source projects and haven’t found the right project or if you were unsure on how to begin, I would like to suggest my own project, <a href="https://github.com/durgaswaroop/delorean">Delorean</a> which is a Distributed Version control system, built from scratch in scala. You can contribute not only in the form of code, but also with usage documentation and also by identifying any bugs in its functionality.</p><p>Thanks for reading. See you again in the next article.</p><p>Image attribution:</p><p>Git Logo — Git Logo by Jason Long is licensed under the Creative Commons Attribution 3.0 Unported License.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=f1c38a1c4fc3" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Functions as First class citizens in Python]]></title>
            <link>https://medium.com/@durgaswaroop/functions-as-first-class-citizens-in-python-2e70cdc6e357?source=rss-35e37d0bb61d------2</link>
            <guid isPermaLink="false">https://medium.com/p/2e70cdc6e357</guid>
            <category><![CDATA[functional-programming]]></category>
            <category><![CDATA[objects]]></category>
            <category><![CDATA[function]]></category>
            <category><![CDATA[programming]]></category>
            <category><![CDATA[python]]></category>
            <dc:creator><![CDATA[Swaroop]]></dc:creator>
            <pubDate>Thu, 11 Jan 2018 18:23:41 GMT</pubDate>
            <atom:updated>2018-01-11T18:23:41.641Z</atom:updated>
            <content:encoded><![CDATA[<h4>First Class Functions In Python</h4><p>If you have been using Python for some time, you certainly would have come across the phrase “Functions are First class citizens in Python”. If you are wondering what that means and wondering what about other citizens, then this article is for you.</p><figure><img alt="Python Logo" src="https://cdn-images-1.medium.com/max/601/0*ZYR--yWFeggZ6H87.png" /></figure><p>Functions as first class citizens means that, you can pass functions around just like other objects. That means, you can assign functions to variables, you can pass them along as arguments, you can store them in bigger data structures, define them inside another function and also return them from another function, just like any other objects.</p><p>This is achieved in Python because functions are nothing but objects. It has so many parameters/methods in it which you can inspect using the dir() method.</p><pre>def hello_func():<br>  print(&quot;hello&quot;)<br><br>print(dir(hello_func))</pre><p>And the output is:</p><pre>[&#39;__annotations__&#39;, &#39;__call__&#39;, &#39;__class__&#39;, &#39;__closure__&#39;, &#39;__code__&#39;, &#39;__defaults__&#39;, &#39;__delattr__&#39;, &#39;__dict__&#39;, &#39;__dir__&#39;, <br>&#39;__doc__&#39;, &#39;__eq__&#39;, &#39;__format__&#39;, &#39;__ge__&#39;, &#39;__get__&#39;, &#39;__getattribute__&#39;, &#39;__globals__&#39;, &#39;__gt__&#39;, &#39;__hash__&#39;, &#39;__init__&#39;, <br>&#39;__init_subclass__&#39;, &#39;__kwdefaults__&#39;, &#39;__le__&#39;, &#39;__lt__&#39;, &#39;__module__&#39;, &#39;__name__&#39;, &#39;__ne__&#39;, &#39;__new__&#39;, &#39;__qualname__&#39;, <br>&#39;__reduce__&#39;, &#39;__reduce_ex__&#39;, &#39;__repr__&#39;, &#39;__setattr__&#39;, &#39;__sizeof__&#39;, &#39;__str__&#39;, &#39;__subclasshook__&#39;]</pre><p>So, Functions are objects of type function.</p><p>Let’s see the advantages of functions being objects.</p><h4>Assigning Functions To Variables</h4><p>Let’s say I define a method greet like this:</p><pre>def greet(name):<br>    print(&quot;Hello &quot; + name)<br><br>greet(&quot;World&quot;)                ## =&gt; &quot;Hello World&quot;<br>greet(&quot;Universe&quot;)                ## =&gt; &quot;Hello Universe&quot;</pre><p>But Since greet is a function object, I can assign it to a variable which I call say_hello</p><pre>say_hello = greet</pre><p>Now say_hello is also a functions object which means, I can call it just like greet.</p><pre>say_hello(&quot;Earth&quot;)                         ## =&gt; &quot;Hello Earth&quot;<br>say_hello(&quot;Mars&quot;)                         ## =&gt; &quot;Hello Mars&quot;</pre><p>You can do this because functions are first class citizens.</p><h4>Passing Functions As Arguments</h4><p>You can pass functions as arguments to other functions as well.</p><p>Let’s say you have a call_func which is defined like this:</p><pre>def call_func(x, func):<br>    print(func(x))</pre><p>The func is a function and you&#39;re just printing the value of the func(x).</p><p>Now, Let’ say I’ve two functions double and triple that are defined like this:</p><pre>def double(n): return 2*n<br><br>def triple(n): return 3*n</pre><p>Now you can pass these functions to call_func that we have defined earlier.</p><pre>call_func(4, double)     ## =&gt; 8<br>call_func(5, triple)     ## =&gt; 15</pre><blockquote>Note: You are passing double and triple to the function and not double() and triple()</blockquote><h4>Storing Functions In Other Data Structures</h4><p>You can also store functions in other data structures like Lists, Dictionaries and more. Depending on how you implement them these can be extremely useful.</p><p>I can define an list called operations that stores the previously defined double and triple methods.</p><pre>operations = [double, triple]</pre><p>And after that I can call these functions with their indices.</p><pre>operations[0](2)     ## =&gt; 4 <br>operations[1](10)    ## =&gt; 30</pre><p>So this way, functions are much more useful in python than in some other languages where this is not the case. These also have some good uses like generators. We’ll cover those in future articles.</p><p>That is all for this article.</p><p>For more programming articles, checkout <a href="http://freblogg.com/">Freblogg</a> <a href="http://www.freblogg.com/search/label/Python">Freblogg/Python</a></p><p>Some articles on python automation:</p><p><a href="https://medium.com/@durgaswaroop/web-scraping-with-python-introduction-7b3c0bbb6053">Web Scraping For Beginners with Python</a></p><p><a href="https://medium.com/@durgaswaroop/my-semi-automated-blogging-workflow-62cba2827986">My semi automated workflow for blogging</a></p><p><a href="https://medium.com/@durgaswaroop/publish-articles-on-blogger-in-just-one-second-2ef45586901">Publish articles to Blogger automatically</a></p><p><a href="http://www.freblogg.com/2017/12/publish-articles-to-your-medium-blog-in_83.html">Publish articles to Medium automatically</a></p><p>This is the 20th article as part of my twitter challenge <a href="https://twitter.com/durgaswaroop/status/944503750340702208">#30DaysOfBlogging</a>. Ten more articles on various topics, including but not limited to, <a href="http://www.freblogg.com/search/label/Java">Java</a>, <a href="http://www.freblogg.com/search/label/Git">Git</a>, <a href="http://www.freblogg.com/search/label/Vim">Vim</a>, <a href="http://www.freblogg.com/search/?q=software">Software Development</a>, <a href="http://www.freblogg.com/search/?q=python">Python</a>, to come.</p><p>If you are interested in this, make sure to follow me on Twitter <a href="https://twitter.com/durgaswaroop">@durgaswaroop</a>. While you’re at it, Go ahead and subscribe <a href="https://medium.com/@durgaswaroop/">here on medium</a> and my <a href="http://freblogg.com/">other blog</a> as well.</p><p>If you are interested in contributing to any open source projects and haven’t found the right project or if you were unsure on how to begin, I would like to suggest my own project, <a href="https://github.com/durgaswaroop/delorean">Delorean</a> which is a Distributed Version control system, built from scratch in scala. You can contribute not only in the form of code, but also with usage documentation and also by identifying any bugs in its functionality.</p><p>Thanks for reading. See you again in the next article.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=2e70cdc6e357" width="1" height="1" alt="">]]></content:encoded>
        </item>
    </channel>
</rss>