<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Kunal Dhariwal on Medium]]></title>
        <description><![CDATA[Stories by Kunal Dhariwal on Medium]]></description>
        <link>https://medium.com/@Mr.dhariwal?source=rss-9cba4434c54e------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/1*5_XDtJa7-veA54oigeco1A.png</url>
            <title>Stories by Kunal Dhariwal on Medium</title>
            <link>https://medium.com/@Mr.dhariwal?source=rss-9cba4434c54e------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Wed, 18 Nov 2026 16:08:55 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@Mr.dhariwal/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[Learn Pandas Pivot Tables The Excel Way]]></title>
            <link>https://medium.datadriveninvestor.com/learn-pandas-pivot-tables-the-excel-way-66b221dbf42e?source=rss-9cba4434c54e------2</link>
            <guid isPermaLink="false">https://medium.com/p/66b221dbf42e</guid>
            <category><![CDATA[artificial-intelligence]]></category>
            <category><![CDATA[data-analysis]]></category>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[programming]]></category>
            <dc:creator><![CDATA[Kunal Dhariwal]]></dc:creator>
            <pubDate>Wed, 03 Nov 2021 14:09:28 GMT</pubDate>
            <atom:updated>2021-11-03T14:09:28.791Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*AW-K2APaapiF4Sqm8_eUxA.jpeg" /><figcaption>Credits : <a href="https://pixabay.com/illustrations/spreadsheet-graph-chart-report-3324169/">Pixabay</a></figcaption></figure><p>We all are aware of the fact that both the Pandas are amazing. The one in the forest as well as the one that we use for Data Science Ops and Data Analytics. Let’s say we want to perform some of the operations like summarizing, reorganizing, grouping, transforming data, etc. Some of these can be very well done just by using <strong><em>pandas groupby </em></strong>and has some advantages over pivot tables<strong><em>, </em></strong>but it tends to be super confusing for beginners as well as for pro’s because it is not as intuitive as the pivot tables in MS Excel where you can just drag and drop columns and get your things done. But do not worry, Pandas is here to offer us similar pivot tables to those that excel or Google sheets offer. Using pandas pivot tables can make your life super easier and more organized.</p><blockquote><strong>Did you know that people find the Pandas Pivot Table easier to learn than the Pandas Groupby function — after reading this tutorial :P</strong></blockquote><p><strong>Let’s get to the basics of pivot tables:</strong></p><p>— What is a pivot table?</p><p>A pivot table is a dynamic tool for data analysis that lets us summarize and arrange data according to its dimensions.</p><p>— What can pivot tables do and how do they work?</p><p>A pivot table can arrange data dimensions on columns, rows, and as filters. When we place a data dimension on columns, the pivot table will display one column per dimension item. If we place multiple data dimensions on columns, the pivot table displays one column for all combinations of the items in the selected dimensions. When we place a data dimension on rows, the pivot table displays one row per dimension item similarly. The dimensions we select as filters will not be included in the pivot table, but will aggregate and filter the table data based on the selected filter items.</p><p><strong>Components of a pivot table (MS Excel ==Pandas):</strong></p><p>A pivot table in Google sheets/MS Excel and Pandas has five major components:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/488/1*lqhQiL9DrkfI6EN6R-9NFQ.png" /><figcaption><strong>Very important table for future reference, with examples below</strong></figcaption></figure><p><strong>data - </strong>Dataframe that we’ll use for the pivot table.</p><p><strong>values - </strong>columns to aggregate, This is optional.</p><p><strong>index - </strong>column, Grouper, or an array. Our table will be grouped on this.</p><p><strong>columns - </strong>column, Grouper or an array.</p><p><strong>aggfunc - </strong>functions to aggregate or transform like sum, mean, max, nunique, count, etc</p><p>If a list of functions is passed, the resulting pivot table will have hierarchical columns whose top level are the function names (inferred from the function objects themselves) If a dictionary is passed, the key is the column to aggregate, and the value is either a function or a list of functions.</p><p><strong>syntax</strong>:</p><p>pd.pivot_table(dataframe, values=’colA’, index=’colC’, columns=’ColB’, aggfunc=’count’)</p><blockquote><strong>Note</strong>:</blockquote><blockquote>The URL for the code used and the pivot tables created on Google Sheets for this demonstration can be found at the bottom of this article.</blockquote><p><strong>Let’s learn this practically:</strong></p><ol><li>Import the required libraries</li></ol><pre>import pandas as pd<br>import numpy as np<br>pd.set_option(‘display.max_rows’, None)</pre><p>2. Load and preview the dataset</p><pre>movies = pd.read_csv(&#39;<a href="https://raw.githubusercontent.com/kunaldhariwal/Learn-Pandas-Pivot-The-Excel-Way/main/pivot_excel_way_dataset.csv&#39;,index_col=0">https://raw.githubusercontent.com/kunaldhariwal/Learn-Pandas-Pivot-The-Excel-Way/main/pivot_excel_way_dataset.csv&#39;,index_col=0</a>)</pre><pre>movies.head()</pre><p>3. Basic Pivot 1:</p><p>To get the total count of movies released each year, we’ll just assign Year to the rows and Movie to the values and select COUNTA Function.</p><p>MS Excel/Sheets:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*aclvPo-AEEh_NQqypHajWA.png" /></figure><p>Pandas:</p><p>For pandas, this can be done in a single line of code in a similar format.</p><pre>pd.pivot_table(movies, index=&#39;Year&#39;,values=&#39;Movie&#39;,aggfunc=&#39;count&#39;)</pre><p>4. Basic Pivot 2:</p><p>To get the count of movies released each year based on their production house, we’ll assign LeadStudio to the rows, Year to the columns, and Movie to the values and select the COUNTA function.</p><p>MS Excel/Sheets:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Qr7hDLyAtGIjiJiF2lhT1A.png" /></figure><p>Pandas:</p><p>Here, we’ll assign LeadStudio to the index, Year to the columns, Movie to the values, and count as aggfunc.</p><pre>pd.pivot_table(movies, index=&#39;LeadStudio&#39;,columns=&#39;Year&#39;,values=&#39;Movie&#39;,aggfunc=&#39;count&#39;)</pre><p>5. Fill in the NaN values in the above results:</p><p>Just add fill_value=0 to the code given above.</p><pre>pd.pivot_table(movies, index=&#39;LeadStudio&#39;,columns=&#39;Year&#39;,values=&#39;Movie&#39;,aggfunc=&#39;count&#39;, fill_value=0)</pre><p>6. Basic Pivot 3:</p><p>Overall Profitability for each Genre.</p><p>MS Excel/Sheets:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Z9Tm_WDLRw_ScqfS699iqw.png" /></figure><p>Pandas:</p><pre>pd.pivot_table(movies, index=’Genre’,values=’Profitability’,aggfunc=’sum’)</pre><p>7. Advance Pivot 1 (Operations):</p><p>To get the count of movies released by each production house and the average audience score of their movies released, we’ll follow the same procedure as mentioned in the above steps.</p><p>MS Excel/Sheets:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*ILb9mg0T8zU_HQFmck8RxA.png" /></figure><p>Pandas:</p><p>Here, we’ll add both the column names and the function will be applied to them as key: value pairs in aggfunc and the rest of the steps are the same as above.</p><pre>pd.pivot_table(movies, index=&#39;LeadStudio&#39;,values=[&#39;Movie&#39;,&#39;AudienceScore&#39;], aggfunc={&#39;Movie&#39;:&#39;count&#39;, &#39;AudienceScore&#39;:&#39;mean&#39;}, fill_value=0)</pre><p>8. Advance Pivot 2 (Multiple Operations-Multiple Fields):</p><p>Get the maximum domestic gross, the minimum world gross, the unique number of genres filmed, and the min, max, and average budget of each production house.</p><p>MS Excel/Sheets:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*hLcsXRet3-AVxA_t6ZZvFQ.png" /></figure><p>Pandas:</p><pre>pd.pivot_table(movies, index=&#39;LeadStudio&#39;,values=[&#39;DomesticGross&#39;,&#39;ForeignGross&#39;,&#39;WorldGross&#39;,&#39;Budget&#39;,&#39;Genre&#39;], aggfunc={&#39;DomesticGross&#39;:&#39;max&#39;, &#39;WorldGross&#39;:&#39;min&#39;,&#39;Budget&#39;:[&#39;min&#39;,&#39;max&#39;,&#39;mean&#39;],&#39;Genre&#39;:&quot;nunique&quot;}, fill_value=0)</pre><p>9. Advance Pivot 3 (DataFrame Filtering):</p><p>We’ll apply the filter to the above data only for Sony, Dreamworks, and Disney.</p><p>MS Excel/Sheets:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*YOYumD0tHYCiyodEZRnRkQ.png" /></figure><p>Pandas:</p><p>Instead of passing the usual dataframe, we’ll pass the dataframe with filters to the pivot table function.</p><pre># DataFrame with Filters<br>movies[movies.LeadStudio.isin([&#39;Sony&#39;,&#39;DreamWorks&#39;,&#39;Disney&#39;])]<br></pre><pre>pd.pivot_table(movies[movies.LeadStudio.isin([&#39;Sony&#39;,&#39;DreamWorks&#39;,&#39;Disney&#39;])], index=&#39;LeadStudio&#39;,values=[&#39;DomesticGross&#39;,&#39;ForeignGross&#39;,&#39;WorldGross&#39;,&#39;Budget&#39;,&#39;Genre&#39;], aggfunc={&#39;DomesticGross&#39;:&#39;max&#39;, &#39;WorldGross&#39;:&#39;min&#39;,&#39;Budget&#39;:[&#39;min&#39;,&#39;max&#39;,&#39;mean&#39;],&#39;Genre&#39;:&quot;nunique&quot;}, fill_value=0)</pre><p>With this we are all set to use pandas pivot tables for the greater good!</p><p>Do you now agree with me that <em>Pandas pivot tables are as easy as the ones we use on Google Sheets/MS Excel?</em></p><p>Please Highlight the text above if you agree with me!</p><blockquote><strong>Jupyter Notebook (Code used): </strong><a href="https://github.com/kunaldhariwal/Learn-Pandas-Pivot-The-Excel-Way">https://github.com/kunaldhariwal/Learn-Pandas-Pivot-The-Excel-Way</a></blockquote><blockquote><strong><em>Google Sheets (Pivot Tables): </em></strong><a href="https://docs.google.com/spreadsheets/d/1x5hLGlBB_lzF9WhP8pfe8tZCVdBiVLtuW2flz0elMTo/edit#gid=2023555283">https://docs.google.com/spreadsheets/d/1x5hLGlBB_lzF9WhP8pfe8tZCVdBiVLtuW2flz0elMTo</a></blockquote><blockquote><strong><em>LinkedIn</em></strong><em>: </em><a href="https://bit.ly/2u4YPoF"><em>https://bit.ly/2u4YPoF</em></a></blockquote><p><strong>I hope that this has helped you to enhance your knowledge base :)</strong></p><p><strong>Follow me for more!</strong></p><p><strong><em>Cheers!</em></strong></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=66b221dbf42e" width="1" height="1" alt=""><hr><p><a href="https://medium.datadriveninvestor.com/learn-pandas-pivot-tables-the-excel-way-66b221dbf42e">Learn Pandas Pivot Tables The Excel Way</a> was originally published in <a href="https://medium.datadriveninvestor.com/">DataDrivenInvestor</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Pandas with Dask, For an Ultra-Fast Notebook]]></title>
            <link>https://medium.com/data-science/pandas-with-dask-for-an-ultra-fast-notebook-e2621c3769f?source=rss-9cba4434c54e------2</link>
            <guid isPermaLink="false">https://medium.com/p/e2621c3769f</guid>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[data-analysis]]></category>
            <category><![CDATA[programming]]></category>
            <category><![CDATA[artificial-intelligence]]></category>
            <category><![CDATA[data-science]]></category>
            <dc:creator><![CDATA[Kunal Dhariwal]]></dc:creator>
            <pubDate>Sun, 17 May 2020 21:12:58 GMT</pubDate>
            <atom:updated>2021-10-31T11:43:17.636Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*iKZzj7wohpQUXqVAYQBV6w.jpeg" /><figcaption>Courtesy: <a href="https://pixabay.com/photos/highway-drive-driving-road-travel-1209547/">Pixabay</a></figcaption></figure><p>If you are into the data world then the need of using pandas might occur very frequently. Well, we all are also aware of one thing that Pandas is amazing and is more like a boon to the data world. Most of us spend at least a few minutes or even hours using pandas for data manipulation and day to day analysis. Pandas does not need much introduction but here are a few things that you should be aware of:</p><p><strong>pandas</strong> is well suited for many different kinds of data:</p><ul><li>Tabular data with heterogeneously-typed columns, as in an SQL table or Excel spreadsheet</li><li>Ordered and unordered (not necessarily fixed-frequency) time-series data.</li><li>Arbitrary matrix data (homogeneously typed or heterogeneous) with row and column labels</li><li>Any other form of observational/statistical data sets. The data need not be labeled at all to be placed into a pandas data structure.</li></ul><p>Using pandas with gigabytes or terabytes of data is more like a pain in the notebook :D. I frequently use files that are into GBs and the notebook goes like <em>“wait a minute, let me freeze now!”</em>. By the way, my system has some amazing configuration. Dealing with a humongous amount of data cannot be done using pandas, Hence that’s where we have to start using <strong>dask</strong>. In this article I’ll demonstrate how you can use pandas with dask and speed up your notebook. In dask, reading GBs of files takes only a few seconds. Before jumping onto the demo, let me give you a brief introduction about dask.</p><p><strong>Dask:</strong></p><p>Dask has 3 parallel collections namely Dataframes, Bags, and Arrays. Which enables it to store data that is larger than RAM. Each of these can use data partitioned between RAM and a hard disk as well distributed across multiple nodes in a cluster. A Dask DataFrame is partitioned row-wise, grouping rows by index value for efficiency. These Pandas objects may live on disk or other machines. Dask DataFrames coordinate many Pandas DataFrames or Series arranged along the index</p><p>Dask can enable efficient parallel computations on single machines by leveraging their multi-core CPUs and streaming data efficiently from disk. It can run on a distributed cluster. Dask also allows the user to replace clusters with a single-machine scheduler which would bring down the overhead. These schedulers require no setup and can run entirely within the same process as the user’s session. Few other things about dask:</p><ul><li>The ability to work in parallel with NumPy array and Pandas DataFrame objects</li><li>integration with other projects.</li><li>Distributed computing</li><li>Faster operation because of its low overhead and minimum serialization</li><li>Runs resiliently on clusters with thousands of cores</li><li>Real-time feedback and diagnostics</li></ul><p><strong>Demonstration:</strong></p><p>The entire code used for this demonstration can be found in my Github repo and you can find the link at the end of this article. Now, let’s jump on to the demo:</p><p><strong>Importing the libraries:</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/758/1*dDum9Ac9fXWIhl2qT-viIw.png" /></figure><p><strong>Code to calculate elapsed time and file size:</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*6KLcB_D5CpCF_lFdSwXt9A.png" /></figure><p>The first snippet is to calculate the elapsed time for an operation that we perform in here, for example: reading a file. In the second snippet we are displaying the file size that we’ll be using for this demo. The file size is around 4GB.</p><blockquote>First, we’ll see a few things Dask</blockquote><blockquote>is better at and then we’ll skip to the things that pandas does better. This will help you to combine these two libraries and perform your analysis.</blockquote><h4>Dask over Pandas:</h4><p><strong>Reading a file — Pandas &amp; Dask:</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*_IbcYk5IIyb2izUTRqbY2Q.png" /></figure><p>Pandas took around 5 minutes to read a file of size 4gb. Wait, the size is not everything, the number of columns and rows present in a data set plays a major role in the time consumption. Let’s see how much time Dask takes for the same file.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*XUMBy7cVnrpbuXmLxq2_Lw.png" /></figure><p>Holy moly, It just took around 2 milliseconds to read the same file whereas pandas took around 5 minutes. Isn’t it just amazing? Let’s perform some more operations on both the pandas data frame and the dask data frame.</p><p><strong>Appending two files — Pandas &amp; Dask:</strong></p><p>To perform this operation, we’ll be reading another file and then append it to the previous one.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*3dIM7_ny0MMz293OlkTpzA.png" /></figure><p>It took around 9 minutes to perform the above-mentioned operations. Now let’s see how we can optimize it by using Dask.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*XJaJLJZxn9qoi3OvRj_e9A.png" /></figure><p>Well, Well, you just saved around another 9 minutes there. Let us see now other frequently used this we do with pandas.</p><p><strong>Grouping the data — Pandas &amp; Dask:</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*YqHiNDTH_kGQ-hvOCWdyAw.png" /></figure><p>Pandas took over 6 minutes to do a simple grouping. Let’s see now how much more time we can save with dask.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*DlV3B453l1hTPchi0mI0Qw.png" /></figure><p>Woah, you just saved whooping 6 minutes again. There numerous other things where you can save much more time if you are using dask.</p><p><strong>Merging the datasets — Pandas &amp; Dask:</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*S-0Fmi1aQbkYifLM6xrVkQ.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*6HBtPUF4Bxio6bAqG6vNKg.png" /></figure><p>So this is what happened. It tried for like 30 minutes and still couldn’t merge those two files with pandas. Let’s see if we can do it by using dask.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*QJmMBGJRrVn9Vvhyv-3D5A.png" /></figure><p>Well, It barely took me a second to do it using dask. What just happened is, Unlike pandas.read_csv which reads in the entire file before inferring data types, dask.dataframe.read_csv only reads in a sample from the beginning of the file (or first file if using a glob). These inferred data types are then enforced when reading all partitions.</p><h4>Pandas over Dask:</h4><p><strong>Sorting — Pandas &amp; Dask:</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*1EaQeDpb5K0YxmBUNLYUfw.png" /></figure><p>I tried to sort the data frame based on the values in a column and it took me around a minute and a half, that’s pretty decent. Let’s see how dask can help us with this.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*_JM9DbPT0tAxVX1DUhNkPA.png" /></figure><p>Unfortunately, dask cannot even start this task because dask has no functionality of sorting although it uses pandas API. All hail pandas!</p><p><strong>Unique &amp; notNA — Pandas &amp; Dask:</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*4pJrcAHhcbuI1J0NPoHnJA.png" /></figure><p>It took around only 1 minute to get these tasks done. Pandas does most of the things pretty well but screws in quite a few.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*-ecioZLH-OFeiJsNsLhWJQ.png" /></figure><p>Dask does not support these two things as well. There are numerous other things for which you’ll have to use pandas.</p><p><strong>Saving a Dataframe to a file — Pandas &amp; Dask:</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*J0oVmeAGrDMov1uWrl51Kw.png" /></figure><p>Pandas does a good job of saving the file. It took around 3 minutes for me to save the filtered file.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*diY14fUjuUZ7ZVvX87xnTA.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*w53BK-bvjt1tIF60i7a-3w.png" /></figure><p>Dask does not save the file properly. It breaks the file into multiple chunks and saves these files in a folder with that mentioned name. Another problem is that you can’t read this saved file ever again. It is just a waste of time.</p><p>The solution for saving a dask data frame to a file is to convert it into a pandas data frame like this and then save the pandas data frame to a file.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*vBNlok99yzO6k-FR26TpHQ.png" /></figure><blockquote><strong>Conclusion:</strong></blockquote><blockquote><strong>It is always the best option to use pandas and dask together because one can fill other’s limitations very well. When used individually, I suppose that you might run into different issues. Hence, we conclude that Pandas with Dask can save you a lot of time and resources.</strong></blockquote><p>Note:</p><p><em>Dask is faster because it doesn’t really execute anything until we use .compute(). Although, This can save a lot, a lot of time and give us more speed!</em></p><p>Tip:</p><pre># Use this to make pandas run faster<br>pd.read_csv(filepath, engine = &#39;c&#39;)</pre><p>Do let me know if you are aware of any better alternatives than dask.</p><blockquote><strong><em>Jupyter Notebook (Code used) : </em></strong><a href="https://github.com/kunaldhariwal/Pandas-with-Dask-For-an-Ultra-Fast-Notebook"><em>https://github.com/kunaldhariwal/Medium-12-Amazing-Pandas-NumPy-Functions</em></a></blockquote><blockquote><strong>LinkedIn</strong>: <a href="https://bit.ly/2u4YPoF">https://bit.ly/2u4YPoF</a></blockquote><p><strong>I hope that this has helped you to enhance your knowledge base :)</strong></p><p><strong>Follow me for more!</strong></p><blockquote>BTW, Did you know you can also use Dask with pandas for free on Saturn Cloud <a href="https://saturncloud.io/?utm_source=Kunal+Medium+&amp;utm_medium=Pandas+with+Dask+article&amp;utm_campaign=Pandas+with+Dask+article"><strong>Click Here</strong></a>, a scalable data science environment in the cloud. Teams can collaborate with powerful resources, jobs, deployments, and more.</blockquote><p><strong>Thanks for your read and valuable time!</strong></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=e2621c3769f" width="1" height="1" alt=""><hr><p><a href="https://medium.com/data-science/pandas-with-dask-for-an-ultra-fast-notebook-e2621c3769f">Pandas with Dask, For an Ultra-Fast Notebook</a> was originally published in <a href="https://medium.com/data-science">TDS Archive</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Learn SQL & MongoDb Simultaneously — The Easy Way (Part 1)]]></title>
            <link>https://medium.com/data-science/learn-sql-mongodb-simultaneously-the-easy-way-part-1-2d4ee20aa083?source=rss-9cba4434c54e------2</link>
            <guid isPermaLink="false">https://medium.com/p/2d4ee20aa083</guid>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[data-analysis]]></category>
            <category><![CDATA[programming]]></category>
            <category><![CDATA[mongodb]]></category>
            <category><![CDATA[sql]]></category>
            <dc:creator><![CDATA[Kunal Dhariwal]]></dc:creator>
            <pubDate>Sat, 16 May 2020 15:15:55 GMT</pubDate>
            <atom:updated>2020-05-17T07:24:55.476Z</atom:updated>
            <content:encoded><![CDATA[<h4>Learn Everything, The Easy Way.</h4><h3>Learn SQL &amp; MongoDB Simultaneously — The Easy Way (Part 1)</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*pnDOXfVP7q6-t-HgXx09GQ.jpeg" /><figcaption>Courtesy: <a href="https://pixabay.com/photos/server-space-the-server-room-dark-2160321/">Pixabay</a></figcaption></figure><p>By reading this title you may be wondering that who on earth would learn MongoDB along with SQL? Let me give you a small cupcake, It is super easy to learn a relational and non-relational database management language simultaneously. You can also call them as scripting languages.</p><p>Oh, are you telling us that they are super easy to learn when learned together? Yes, It is. Just follow me during this whole series and you’ll realize that I’m not wrong.</p><p>But, why would you want to learn even either one of these? Well, It is always great to know some languages that can help you interact and perform operations on a database. Another big hit is, If you’re into data science, data analytics, business analytics, web development, app development, and dev-ops then you got to learn these. Once a while you’ll have to interact with and use such DB’s and without prior knowledge of this I’m afraid that you might feel guilty of not learning it.</p><h4>Basic Overview:</h4><p><strong><em>SQL</em></strong>: SQL is a database computer language designed for the retrieval and management of data in a relational database. SQL stands <strong>for Structured Query Language</strong>. This tutorial will give you a quick start to SQL. It covers most of the topics required for a basic understanding of SQL. A SQL database contains :</p><ul><li>The data is stored in database objects which are called <strong>tables.</strong></li><li>Every table is broken up into smaller entities called <strong>fields.</strong></li><li>A <strong>record</strong> is also called a <strong>row</strong> of data in each entry that exists in a table.</li><li>A <strong>column</strong> is a vertical entity in a table that contains all information associated with a specific field in a table.</li></ul><p><strong><em>MongoDB</em></strong>: MongoDB database is queried using <strong>MongoDB Query Language</strong> (MQL). As a NoSQL database, MongoDB indeed does not use SQL as its querying language. Instead, MongoDB relies on several drivers that allow its engine to interact with a wide variety of languages. A No-SQL database contains :</p><ul><li>The data is stored in database objects which are called <strong>Collections.</strong></li><li>A <strong>document</strong> is a set of key-value pairs. Documents have dynamic schema. This can be called a <strong>row.</strong></li><li><strong>_id</strong>: This is a field required in every MongoDB document.</li><li><strong>Field </strong>is a name-value pair in a document. A document has zero or more fields. This can be called as a <strong>column.</strong></li></ul><p><strong>Overview of Datatypes:</strong></p><p>SQL:</p><ul><li>Integer</li><li>Smallint</li><li>Numeric(p,s) − Where <strong><em>p</em></strong> is a precision value; <strong><em>s</em></strong> is a scale value. For example, numeric(6,2) is a number that has 4 digits before the decimal and 2 digits after the decimal.</li><li>Decimal(p,s) − Where <strong><em>p</em></strong> is a precision value; <strong><em>s</em></strong> is a scale value.</li><li>Real − Single-precision floating-point number.</li><li>Double precision − Double-precision floating-point number.</li><li>Float(p) − Where <strong><em>p</em></strong> is a precision value.</li><li>Char(x) − Where <strong><em>x</em></strong> is the number of characters to store. It is space padded to fill the number of characters specified.</li><li>Varchar(x) − Where <strong><em>x</em></strong> is the number of characters to store. It does NOT space pad.</li><li>Bit(x) − Where <strong><em>x</em></strong> is the number of bits to store.</li><li>Bit varying(x) − Where <strong><em>x</em></strong> is the number of bits to store. The length can vary up to x.</li><li>Date − Stores year, month, and day values.</li><li>Time − Stores the hour, minute, and second values.</li><li>Timestamp − Stores year, month, day, hour, minute, and second values.</li><li>Time with time zone − The same as time, but also stores an offset from UTC of the time specified.</li><li>Timestamp with time zone − The same as timestamp, but also stores an offset from UTC of the time specified.</li><li>Year-month interval − Contains a year value, a month value, or both.</li><li>Day-time interval − Contains a day value, an hour value, a minute value, and/or a second value.</li></ul><p>MongoDB:</p><ul><li>String − This is the most commonly used datatype to store the data. String in MongoDB must be UTF-8 valid.</li><li>Integer − It is used to store a numerical value. Integer can be 32 bit or 64 bit depending upon your server.</li><li>Boolean − It is used to store a boolean (true/ false) value.</li><li>Double − It is used to store floating-point values.</li><li>Min/ Max keys − It is used to compare a value against the lowest and highest BSON elements.</li><li>Arrays − It is used to store arrays or lists or multiple values into one key.</li><li>Timestamp − ctimestamp. This can be handy for recording when a document has been modified or added.</li><li>Object − It is used for embedded documents.</li><li>Null − It is used to store a Null value.</li><li>Symbol − It is used identically to a string; however, it’s generally reserved for languages that use a specific symbol type.</li><li>Date<strong> </strong>− It is used to store the current date or time in UNIX time format. You can specify your own date time by creating object of Date and passing day, month, year into it.</li><li>Object ID − It is used to store the document’s ID.</li><li>Binary data − It is used to store binary data.</li><li>Code − It is used to store JavaScript code into the document.</li><li>Regular expression − It is used to store regular expression.</li></ul><p>I am not going to bombard you with all information in one part because even it is hard for me to handle so much information, I know how it feels. We will learn SQL &amp; MongoDB step by step and in a very smooth manner. I’ll cover around 4 staters in part 1.</p><p><strong>Create a Table / Collection and Insert a record:</strong></p><p>The creation of table/collection is pretty simple and you won’t need to do this more often if you focus more on extracting and analyzing the data. By the way, SQL is case sensitive.</p><p>SQL:</p><pre># Syntax to create a table<br>&gt;<strong> create</strong> <strong>table</strong> &quot;tablename&quot; (&quot;columnName&quot; &quot;datatype&quot;, &quot;columnName&quot; &quot;datatype&quot;);</pre><pre># Create a table:<br>&gt; <strong>create</strong> <strong>table</strong> winner ( Id INT, name VARCHAR(15))</pre><pre># Insert a row:<br>&gt; <strong>insert</strong> <strong>into</strong> winner (Id) <strong>values</strong> (1)<br>(or)<br>&gt; <strong>insert</strong> <strong>into</strong> winner <strong>values</strong> (1)</pre><p>MongoDB:</p><pre># Syntax to create a collection<br>&gt; db.createCollection(name, options)</pre><pre># Create a collection:<br>&gt; db.createCollection(&quot;winner&quot;)</pre><pre># Syntax to insert a record&gt; db.collectionName.insert({&quot;key&quot;:&quot;value&quot;})</pre><pre># Inserting a record:<br>&gt; db.winner.insert({&quot;name&quot; : &quot;Kunal&quot;})</pre><p><strong>Query the Database:</strong></p><p>The general definition of querying the database is getting or fetching the data from collections or tables. The basic process of querying the database is pretty simple. In <strong>SQL</strong>, you do it by using <strong>select </strong>and in <strong>MongoDB</strong> you do it by using either <strong>find </strong>or <strong>aggregate</strong>. Here is how you can fetch the data from a database:</p><p>SQL:</p><pre># Syntax to query a table<br>&gt;<strong> select</strong> column <strong>from </strong>tables</pre><pre># Query all the records<br>&gt; <strong>select</strong> * <strong>from</strong> winner</pre><pre># Query the DB, select all rows from the Id column<br>&gt; <strong>select</strong> Id <strong>from</strong> winner</pre><p>MongoDB:</p><pre># Syntax to query a collection, to select all the documents<br>&gt; db.COLLECTION_NAME.find({})</pre><pre># Get all the records using <strong>find</strong><br>&gt; db.winner.find({})</pre><pre># Fetch data using <strong>aggregate<br></strong>&gt; db.winner.aggregate(<br>[{<br>        $group: { <br>            &#39;_id&#39;:null,<br>            &#39;max&#39;:{&#39;$first&#39;:&quot;$$ROOT&quot;}<br>        }<br>    }])</pre><p>For the starters, I would suggest you get familiar with <strong>find</strong> function because switching directly to <strong>aggregate</strong> function would be a little tough. The reason is that at a later stage you’ve to learn about aggregate, <strong>find</strong> function has various limitations, and to fulfill that you have to use an <strong>aggregate </strong>function.</p><p><strong>Sorting the Queried Data:</strong></p><p>Sorting is very important when you want the data to be in a particular order. The query that you might have run earlier would give you results in a random order. Sorting is termed as <strong>Order</strong> in <strong>SQL </strong>and <strong>Sort</strong> in <strong>MongoDB</strong>. We can sort the data based on a column, ascending order, descending order, and by limit. Let me show you how it’s done.</p><p>SQL:</p><pre># Syntax for ordering the data<br>&gt; select * from tablename order by asc/desc/column/limit</pre><pre># Fetch all records and sort by ascending<br>&gt; <strong>select</strong> * <strong>from</strong> winners <strong>order</strong> <strong>by</strong> asc</pre><pre># Fetch all records and sort by descending<br>&gt; <strong>select</strong> * <strong>from</strong> winners <strong>order</strong> <strong>by</strong> desc</pre><pre># Fetch all records and sort by column<br>&gt; <strong>select</strong> * <strong>from</strong> winners <strong>order</strong> <strong>by</strong> Id</pre><pre># Fetch all records and sort by multiple columns<br>&gt; <strong>select</strong> * <strong>from</strong> winners <strong>order</strong> <strong>by</strong> Id, name</pre><p>MongoDB:</p><pre># Syntax to sort the data - find()<br>&gt; db.collectionName.find({}).sort({<strong>KEY</strong>:1})</pre><pre># Sort the data in descending order using find()<br>&gt; db.winners.find({}).sort({&quot;_id&quot;:-1})</pre><pre># Sort the data in ascending order using find()<br>&gt; db.winners.find({}).sort({&quot;_id&quot;:1})</pre><pre># Sort the data in ascending order using aggregate()<br>&gt; db.users.aggregate(<br>   [<br>     { <strong>$sort</strong> : { name: 1 } }<br>   ]<br>)</pre><pre># Sort the data in descending order using aggregate()<br>&gt; db.users.aggregate(<br>   [<br>     { <strong>$sort </strong>: { name: -1 } }<br>   ]<br>)</pre><p>The <strong>_id</strong> is the objectID that is captured for every record being inserted in the collection and it is compulsory. If you decode that ObjectID, you’ll get a timestamp out of it. For functions in aggregate pipeline you have to prefix a <strong>$.</strong></p><p><strong>Querying Database Based on Conditions:</strong></p><p>It is always simple to fetch all the records at a time but it might sometimes consume a lot of your time and sometimes, you don’t even need to see whole data. You can also filter the data based on certain conditions. What filtering does is, it eliminates the rest of them from your display and you get to see only the required data. In <strong>SQL</strong> it is done using <strong>where</strong> and in <strong>MongoDB</strong> it is done using $<strong>match</strong> in <strong>aggregate</strong> and just some simple <strong>assignment</strong> in <strong>find</strong>. Let me show you how it’s done.</p><p>SQL:</p><pre># Syntax to query data based on condition<br>&gt; <strong>select</strong> * <strong>from</strong> table <strong>where</strong> <em>condition</em></pre><pre># Fetch data based on a condition<br>&gt; <strong>select</strong> Id <strong>from</strong> winners <strong>where</strong> name=&quot;Kunal&quot; <strong>and</strong> Id&gt;4</pre><pre># Another condition<br>&gt; <strong>select</strong> name <strong>from</strong> winners <strong>where</strong> Id <strong>between</strong> 10 <strong>and</strong> 12</pre><p>Between condition will filter out the results that are falling in the mentioned condition.</p><p>MongoDB:</p><pre># Syntax to query based on a condition<br>&gt; db.winners.find({condition})</pre><pre># Fetch data based on a condition, Id greater than 2<br>&gt; db.winners.find({&quot;Id&quot;:{&quot;$gt&quot;:2}})</pre><pre># Using aggregate, match is just like where in SQL<br>&gt; db.winners.aggregate(<br>[{<br>&quot;<strong>$match</strong>&quot;:{&quot;Id&quot;:{&quot;$gt&quot;:2}}<br>}])</pre><p>We have used the same condition both in <strong>find </strong>and <strong>aggregate</strong>, but don’t you think <strong>find </strong>one is easier to implement? <strong>Find </strong>comes with its limitations, that’s why you have to switch to <strong>aggregate </strong>at a later stage. I suggest you get used to <strong>aggregate</strong>, It will save you a couple of hours of fuss of using find.</p><p><strong>Keynotes:</strong></p><ol><li>SQL is case sensitive.</li><li>SQL has tables, MongoDB has collections.</li><li><strong>Semicolon</strong> is the standard way to separate each <strong>SQL</strong> statement.</li><li>MongoDB pipeline functions require <strong>$</strong> as a prefix.</li><li>Use “” in MongoDB to improve code readability.</li><li>MongoDB has <strong>find </strong>and <strong>aggregate </strong>to query the DB.</li><li>In MongoDB,<strong> _id</strong> is an ObjectID with a timestamp in it.</li></ol><p>You’re just amazing if you have come this far. With this, we end the part 1 of our <strong>The Easy Way </strong>series.</p><blockquote><em>If you encounter any error or need any help, you can always make a comment or ping me on LinkedIn.</em></blockquote><blockquote><strong><em>LinkedIn:</em></strong><em> </em><a href="https://bit.ly/2u4YPoF"><em>https://bit.ly/2u4YPoF</em></a></blockquote><blockquote><em>Follow me on Github for more amazing projects and tutorials.</em></blockquote><blockquote><strong><em>Github: </em></strong><a href="https://bit.ly/2SQV7ss"><em>https://bit.ly/2SQV7ss</em></a></blockquote><p><strong>I hope that this has helped you to enhance your knowledge base :)</strong></p><p><strong>Follow me for more!</strong></p><p><strong>Thanks for your read and valuable time!</strong></p><blockquote>P.S — Part 2 of this series will be out soon. Always stay hungry for knowledge.</blockquote><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=2d4ee20aa083" width="1" height="1" alt=""><hr><p><a href="https://medium.com/data-science/learn-sql-mongodb-simultaneously-the-easy-way-part-1-2d4ee20aa083">Learn SQL &amp; MongoDb Simultaneously — The Easy Way (Part 1)</a> was originally published in <a href="https://medium.com/data-science">TDS Archive</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[12 Amazing Pandas & NumPy Functions]]></title>
            <link>https://medium.com/data-science/12-amazing-pandas-numpy-functions-22e5671a45b8?source=rss-9cba4434c54e------2</link>
            <guid isPermaLink="false">https://medium.com/p/22e5671a45b8</guid>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[programming]]></category>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[data-analysis]]></category>
            <category><![CDATA[artificial-intelligence]]></category>
            <dc:creator><![CDATA[Kunal Dhariwal]]></dc:creator>
            <pubDate>Sat, 07 Mar 2020 21:56:29 GMT</pubDate>
            <atom:updated>2020-03-08T10:39:00.700Z</atom:updated>
            <content:encoded><![CDATA[<h4>Make your day to day life easier by using these functions in your analysis</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*vuQ4fs_zYLoqVoIw_YS3sA.png" /><figcaption>courtesy: <a href="https://pixabay.com/photos/code-programming-python-1084923/">https://pixabay.com/photos/code-programming-python-1084923/</a></figcaption></figure><p>We all know that Pandas and NumPy are amazing, and they play a crucial role in our day to day analysis. Without Pandas and NumPy, we would be left deserted in this huge world of data analytics and science. Today, I am going to share 12 amazing Pandas and NumPy functions that will make your life and analysis much easier than before. <em>In the end, you can find a Jupyter Notebook for the code used in this article.</em></p><p><strong>Let’s start with NumPy:</strong></p><p>NumPy is the fundamental package for scientific computing with Python. It contains among other things:</p><ul><li>a powerful N-dimensional array object</li><li>sophisticated (broadcasting) functions</li><li>tools for integrating C/C++ and Fortran code</li><li>useful linear algebra, Fourier transform, and random number capabilities</li></ul><p>Besides its obvious scientific uses, NumPy can also be used as an efficient multi-dimensional container of generic data. Arbitrary data-types can be defined. This allows NumPy to seamlessly and speedily integrate with a wide variety of databases.</p><ol><li><strong>argpartition()</strong></li></ol><p>NumPy has this amazing function which can find N largest values index. The output will be the N largest values index, and then we can sort the values if needed.</p><pre>x = np.array([12, 10, 12, 0, 6, 8, 9, 1, 16, 4, 6, 0])</pre><pre>index_val = np.argpartition(x, -4)[-4:]<br>index_val<br>array([1, 8, 2, 0], dtype=int64)</pre><pre>np.sort(x[index_val])<br>array([10, 12, 12, 16])</pre><p>2.<strong> allclose()</strong></p><p>Allclose() is used for matching two arrays and getting the output in terms of a boolean value. It will return False if items in two arrays are not equal within a tolerance. It is a great way to check if two arrays are similar, which can actually be difficult to implement manually.</p><pre>array1 = np.array([0.12,0.17,0.24,0.29])<br>array2 = np.array([0.13,0.19,0.26,0.31])</pre><pre># with a tolerance of 0.1, it should return False:<br>np.allclose(array1,array2,0.1)<br>False</pre><pre># with a tolerance of 0.2, it should return True:<br>np.allclose(array1,array2,0.2)<br>True</pre><p>3. <strong>clip()</strong></p><p>Clip() is used to keep values in an array within an interval. Sometimes, we need to keep the values within an upper and lower limit. For the mentioned purpose, we can make use of NumPy’s clip(). Given an interval, values outside the interval are clipped to the interval edges.</p><pre>x = np.array([3, 17, 14, 23, 2, 2, 6, 8, 1, 2, 16, 0])</pre><pre>np.clip(x,2,5)<br>array([3, 5, 5, 5, 2, 2, 5, 5, 2, 2, 5, 2])</pre><p>4. <strong>extract()</strong></p><p>Extract() as the name goes, is used to extract specific elements from an array based on a certain condition. With extract(), we can also use conditions like <strong>and </strong>and <strong>or</strong>.</p><pre># Random integers<br>array = np.random.randint(20, size=12)<br>array<br>array([ 0,  1,  8, 19, 16, 18, 10, 11,  2, 13, 14,  3])</pre><pre>#  Divide by 2 and check if remainder is 1<br>cond = np.mod(array, 2)==1<br>cond<br>array([False,  True, False,  True, False, False, False,  True, False, True, False,  True])</pre><pre># Use extract to get the values<br>np.extract(cond, array)<br>array([ 1, 19, 11, 13,  3])</pre><pre># Apply condition on extract directly<br>np.extract(((array &lt; 3) | (array &gt; 15)), array)<br>array([ 0,  1, 19, 16, 18,  2])</pre><p>5. <strong>where()</strong></p><p>Where() is used to return elements from an array that satisfy a certain condition. It returns the index position of values that fall in a certain condition. This is almost similar to the where condition that we use in SQL, I’ll demonstrate that in the examples below.</p><pre>y = np.array([1,5,6,8,1,7,3,6,9])</pre><pre># Where y is greater than 5, returns index position<br>np.where(y&gt;5)<br>array([2, 3, 5, 7, 8], dtype=int64),)</pre><pre># First will replace the values that match the condition, <br># second will replace the values that does not<br>np.where(y&gt;5, &quot;Hit&quot;, &quot;Miss&quot;)<br>array([&#39;Miss&#39;, &#39;Miss&#39;, &#39;Hit&#39;, &#39;Hit&#39;, &#39;Miss&#39;, &#39;Hit&#39;, &#39;Miss&#39;, &#39;Hit&#39;, &#39;Hit&#39;],dtype=&#39;&lt;U4&#39;)</pre><p>6.<strong> percentile()</strong></p><p>Percentile() is used to compute the nth percentile of the array elements along the specified axis.</p><pre>a = np.array([1,5,6,8,1,7,3,6,9])</pre><pre>print(&quot;50th Percentile of a, axis = 0 : &quot;,  <br>      np.percentile(a, 50, axis =0))<br>50th Percentile of a, axis = 0 :  6.0</pre><pre>b = np.array([[10, 7, 4], [3, 2, 1]])</pre><pre>print(&quot;30th Percentile of b, axis = 0 : &quot;,  <br>      np.percentile(b, 30, axis =0))<br>30th Percentile of b, axis = 0 :  [5.1 3.5 1.9]</pre><p>Let me know if you’ve used them earlier and how far did it help you. Let’s move on to the amazing Pandas.</p><p><strong>Pandas:</strong></p><p>pandas is a Python package providing fast, flexible, and expressive data structures designed to make working with structured (tabular, multidimensional, potentially heterogeneous) and time-series data both easy and intuitive.</p><p>pandas is well suited for many different kinds of data:</p><ul><li>Tabular data with heterogeneously-typed columns, as in an SQL table or Excel spreadsheet</li><li>Ordered and unordered (not necessarily fixed-frequency) time-series data.</li><li>Arbitrary matrix data (homogeneously typed or heterogeneous) with row and column labels</li><li>Any other form of observational/statistical data sets. The data actually need not be labeled at all to be placed into a pandas data structure.</li></ul><p>Here are just a few of the things that pandas does well:</p><ul><li>Easy handling of missing data (represented as NaN) in floating point as well as non-floating point data</li><li>Size mutability: columns can be inserted and deleted from DataFrame and higher dimensional objects</li><li>Automatic and explicit data alignment: objects can be explicitly aligned to a set of labels, or the user can simply ignore the labels and let Series, DataFrame, etc. automatically align the data for you in computations</li><li>Powerful, flexible group by functionality to perform split-apply-combine operations on data sets, for both aggregating and transforming data</li><li>Make it easy to convert ragged, differently-indexed data in other Python and NumPy data structures into DataFrame objects</li><li>Intelligent label-based slicing, fancy indexing, and subsetting of large data sets</li><li>Intuitive merging and joining data sets</li><li>Flexible reshaping and pivoting of data sets</li><li>Hierarchical labeling of axes (possible to have multiple labels per tick)</li><li>Robust IO tools for loading data from flat files (CSV and delimited), Excel files, databases, and saving/loading data from the ultrafast HDF5 format</li><li>Time series-specific functionality: date range generation and frequency conversion, moving window statistics, date shifting and lagging.</li></ul><ol><li><strong>read_csv(nrows=n)</strong></li></ol><p>You might already be aware of the use of read_csv function. But, most of us still make a mistake of reading the entire .csv file even when it is not required. Let’s consider a situation where we are unaware of the columns and the data present in a .csv file of 10gb, reading whole .csv file here would not be a smart decision because it would be the unnecessary use of our memory and would take a lot of time. We can just import a few rows from the .csv file and then proceed further as per our need.</p><pre>import io<br>import requests</pre><pre># I am using this online data set just to make things easier for you guys<br>url = &quot;<a href="https://raw.github.com/vincentarelbundock/Rdatasets/master/csv/datasets/AirPassengers.csv">https://raw.github.com/vincentarelbundock/Rdatasets/master/csv/datasets/AirPassengers.csv</a>&quot;<br>s = requests.get(url).content</pre><pre># read only first 10 rows<br>df = pd.read_csv(io.StringIO(s.decode(&#39;utf-8&#39;)),nrows=10 , index_col=0)</pre><p>2. <strong>map()</strong></p><p>The map() function is used to map values of Series according to input correspondence. Used for substituting each value in a Series with another value, that may be derived from a function, a dict or a Series.</p><pre># create a dataframe<br>dframe = pd.DataFrame(np.random.randn(4, 3), columns=list(&#39;bde&#39;), index=[&#39;India&#39;, &#39;USA&#39;, &#39;China&#39;, &#39;Russia&#39;])</pre><pre>#compute a formatted string from each floating point value in frame<br>changefn = lambda x: &#39;%.2f&#39; % x</pre><pre># Make changes element-wise<br>dframe[&#39;d&#39;].map(changefn)</pre><p>3. <strong>apply()</strong></p><p>The apply() allows the users to pass a function and apply it on every single value of the Pandas series.</p><pre># max minus mix lambda fn<br>fn = lambda x: x.max() - x.min()</pre><pre># Apply this on dframe that we&#39;ve just created above<br>dframe.apply(fn)</pre><p>4. <strong>isin()</strong></p><p>The isin() is used to filter data frames. isin() helps in selecting rows with having a particular(or Multiple) value in a particular column. It is the most useful function I’ve come across.</p><pre># Using the dataframe we created for read_csv<br>filter1 = df[&quot;value&quot;].isin([112]) <br>filter2 = df[&quot;time&quot;].isin([1949.000000])</pre><pre>df [filter1 &amp; filter2]</pre><p>5. <strong>copy()</strong></p><p><em>The copy()</em> is used to create a copy of a Pandas object. When you assign a data frame to another data frame, its value changes when you make changes in the other one. To prevent the mentioned issue, we can make use of copy().</p><pre># creating sample series <br>data = pd.Series([&#39;India&#39;, &#39;Pakistan&#39;, &#39;China&#39;, &#39;Mongolia&#39;])</pre><pre># Assigning issue that we face<br>data1= data<br># Change a value<br>data1[0]=&#39;USA&#39;<br># Also changes value in old dataframe<br>data</pre><pre># To prevent that, we use<br># creating copy of series <br>new = data.copy()</pre><pre># assigning new values <br>new[1]=&#39;Changed value&#39;</pre><pre># printing data <br>print(new) <br>print(data)</pre><p>6.<strong> select_dtypes()</strong></p><p>The select_dtypes() function returns a subset of the data frame&#39;s columns based on the column dtypes. The parameters of this function can be set to include all the columns having some specific data type or it could be set to exclude all those columns which has some specific data types.</p><pre># We&#39;ll use the same dataframe that we used for read_csv<br>framex =  df.select_dtypes(include=&quot;float64&quot;)</pre><pre># Returns only time column</pre><p><strong>Bonus:</strong></p><p><strong>pivot_table()</strong></p><p>The most amazing and useful function of pandas is pivot_table. If you hesitate to use groupby and want to extend its functionalities then you can very well use the pivot_table. If you’re aware of how pivot table works in excel, then it’s might be a piece of cake for you. Levels in the pivot table will be stored in MultiIndex objects (hierarchical indexes) on the index and columns of the result DataFrame.</p><pre># Create a sample dataframe<br>school = pd.DataFrame({&#39;A&#39;: [&#39;Jay&#39;, &#39;Usher&#39;, &#39;Nicky&#39;, &#39;Romero&#39;, &#39;Will&#39;], <br>      &#39;B&#39;: [&#39;Masters&#39;, &#39;Graduate&#39;, &#39;Graduate&#39;, &#39;Masters&#39;, &#39;Graduate&#39;], <br>      &#39;C&#39;: [26, 22, 20, 23, 24]})<br></pre><pre># Lets create a pivot table to segregate students based on age and course<br>table = pd.pivot_table(school, values =&#39;A&#39;, index =[&#39;B&#39;, &#39;C&#39;], <br>                         columns =[&#39;B&#39;], aggfunc = np.sum, fill_value=&quot;Not Available&quot;) <br>  <br>table</pre><p>Do let me know down below in the comments if you guys have come across or used any other amazing functions. I would love to know more about them.</p><blockquote><strong>Jupyter Notebook (Code used) : </strong><a href="https://github.com/kunaldhariwal/Medium-12-Amazing-Pandas-NumPy-Functions">https://github.com/kunaldhariwal/Medium-12-Amazing-Pandas-NumPy-Functions</a></blockquote><blockquote><strong><em>LinkedIn</em></strong><em>: </em><a href="https://bit.ly/2u4YPoF"><em>https://bit.ly/2u4YPoF</em></a></blockquote><p><strong>I hope that this has helped you to enhance your knowledge base :)</strong></p><p><strong>Follow me for more!</strong></p><p><strong>Thanks for your read and valuable time!</strong></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=22e5671a45b8" width="1" height="1" alt=""><hr><p><a href="https://medium.com/data-science/12-amazing-pandas-numpy-functions-22e5671a45b8">12 Amazing Pandas &amp; NumPy Functions</a> was originally published in <a href="https://medium.com/data-science">TDS Archive</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Dataset unavailable? No problem!]]></title>
            <link>https://medium.com/data-science/dataset-unavailable-no-problem-b5031311d59d?source=rss-9cba4434c54e------2</link>
            <guid isPermaLink="false">https://medium.com/p/b5031311d59d</guid>
            <category><![CDATA[data-analysis]]></category>
            <category><![CDATA[python]]></category>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[programming]]></category>
            <category><![CDATA[data]]></category>
            <dc:creator><![CDATA[Kunal Dhariwal]]></dc:creator>
            <pubDate>Tue, 26 Nov 2019 16:15:55 GMT</pubDate>
            <atom:updated>2019-11-27T06:10:46.639Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*iFN_PWPWs6TQ9JzDp2v9Wg.jpeg" /><figcaption>source: <a href="https://pixabay.com/photos/code-code-editor-coding-computer-1839406/">https://pixabay.com/photos/code-code-editor-coding-computer-1839406/</a></figcaption></figure><p>There’s a plethora of datasets available on the internet, but let me still mention a few amazing resources. If you are already connected with me on LinkedIn then you might have seen this list, If you’re not connected please hit this <a href="https://www.linkedin.com/in/kunaldhariwal/"><strong>link</strong></a><strong>. </strong>Following sites might help you in getting the dataset that you need:</p><p><strong>Popular open data repositories:</strong></p><ol><li>UC Irvine Machine Learning Repository: <a href="https://lnkd.in/f5CUDj7">https://lnkd.in/f5CUDj7</a></li><li>Kaggle datasets: <a href="https://lnkd.in/fKWqVbw">https://lnkd.in/fKWqVbw</a></li><li>Amazon’s AWS datasets: <a href="https://lnkd.in/fnVYnm5">https://lnkd.in/fnVYnm5</a></li></ol><p><strong>Meta portals (they list open data repositories):</strong></p><ol><li><a href="http://dataportals.org/">http://dataportals.org/</a></li><li><a href="http://opendatamonitor.eu/">http://opendatamonitor.eu/</a></li><li><a href="http://quandl.com/">http://quandl.com/</a></li></ol><p><strong>Other pages listing many popular open data repositories:</strong></p><ol><li>Wikipedia’s list of Machine Learning datasets: <a href="https://lnkd.in/f6wujcB">https://lnkd.in/f6wujcB</a></li><li><a href="http://quora.com/">Quora.com</a> question: <a href="https://lnkd.in/fuHPfFF">https://lnkd.in/fuHPfFF</a></li><li>Datasets subreddit: <a href="https://lnkd.in/fmqkisy">https://lnkd.in/fmqkisy</a></li></ol><h4>What if you couldn’t find a dataset even on those sites? what would you do?</h4><p>Would you blame the internet for not giving you the dataset of your need or you would curse the whole universe?</p><p>Well, I would not do both the things mentioned above. I would create my own dataset and trust me, It wouldn’t take any longer than 5 minutes.</p><p>Now, Let me show you how to create your own dataset quickly. We’ll be using a python package called <strong>Faker.</strong></p><ol><li>Install the Faker library.</li></ol><pre>pip install faker</pre><p>2. Import the library.</p><pre>from faker import Faker</pre><p>3. Assign the Faker function to a variable and set some random seed.</p><pre>fake = Faker()<br>fake.seed(4321)</pre><p>We’ll be creating a hotel dataset. Just to keep this tutorial short and precise we’ll create a total of 8 columns, you can create as many as you want.</p><p>4. Assign each variable an empty array so that we can insert some data into it.</p><pre>ID=[]<br>name=[]<br>phone=[]<br>age=[]<br>gender=[]<br>address=[]<br>checkin=[]<br>checkout=[]</pre><p>5. Let’s insert 100 entries for phone, address, check-in, checkout, and name.</p><pre>for _ in range(100):<br> phone.append(fake.phone_number())<br> address.append(fake.country())<br> checkin.append(fake.date_time_ad())<br> checkout.append(fake.date_time_ad())<br> name.append(fake.name())</pre><p>6. We have to insert values for ID, age, and gender separately. ID will have data from range 1 to 100. In the age variable, we’ll set the minimum age to be 8 and the maximum age to be 80, also we’ll assign the size as 100. The gender part is quite complex because you cannot assign males and females randomly. Also, some ML Algorithms accepts on only numbers. We’ll assign 0 and 1 for male and female and the ratio would be 60:40 respectively.</p><pre>ID=pd.Series(range(1,101))<br>age=np.random.randint(8, high=80, size=100)<br>gender=np.random.choice([0, 1], size=100, p=[.6, .4])</pre><p>7. Now, we have successfully inserted data into those empty arrays. Let’s assign that data to dataframe columns.</p><pre>data= pd.DataFrame({‘Timestamp’:timestamp,’ID’:ID,’name’:name,’Phone number’:phone,’age’:age, ‘gender’:gender, ‘address’:address,’checkin’:checkin,’checkout’:checkout})</pre><h4>That is it! We have successfully created our own custom dataset.</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*dZ0ojp7MhQdgbTAeg-SbgQ.png" /><figcaption>First 10 rows of our custom dataset</figcaption></figure><p>You might need to perform <strong>exploratory data analysis</strong> here because the dataset isn’t clean enough and you gotta find a variable relationship, outliers, and stuff. But that’s where you get to learn the most important part of data science i.e EDA.</p><p><strong>Let’s do some basic EDA.</strong></p><p>8. Replace 0 and 1 with males and females respectively.</p><pre>data.gender.replace(1,’female’,inplace=True)<br>data.gender.replace(0,’male’,inplace=True)</pre><p>9. Convert check-in and checkout to DateTime format.</p><pre>data[‘checkin’]=pd.to_datetime(data.checkin, errors=’coerce’)<br>data[‘checkout’]=pd.to_datetime(data.checkout, errors=’coerce’)</pre><p>10. Let’s now set the checkout date as check-in + 3 days</p><pre>import datetime<br>enddate = pd.to_datetime(data.checkin) + pd.DateOffset(days=3)<br>data[‘checkout’]=enddate</pre><p>11. Let’s plot something.</p><pre>plt.figure(figsize=(20,20))<br>plt.scatter(list(data.checkin.values),list(data.address.values),color=’r’)</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*NZbMyGtMZLSDF8us5iPthQ.png" /><figcaption>Scatter plot</figcaption></figure><p>We can perform here hell lots of EDA here but to keep the article short and precise I’m leaving the next steps to you. Have fun with EDA and this tutorial.</p><p>Here is a link to the full code: <a href="https://github.com/kunaldhariwal/Building-a-custom-dataset">https://github.com/kunaldhariwal/Building-a-custom-dataset</a></p><p><strong>I hope that this has helped you to enhance your knowledge base :)</strong></p><p><strong>Follow me for more!</strong></p><p><strong>Thanks for your read and valuable time!</strong></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=b5031311d59d" width="1" height="1" alt=""><hr><p><a href="https://medium.com/data-science/dataset-unavailable-no-problem-b5031311d59d">Dataset unavailable? No problem!</a> was originally published in <a href="https://medium.com/data-science">TDS Archive</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Become a video analysis expert using python]]></title>
            <link>https://medium.com/data-science/generate-any-sport-highlights-using-python-3695c98baead?source=rss-9cba4434c54e------2</link>
            <guid isPermaLink="false">https://medium.com/p/3695c98baead</guid>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[data]]></category>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[programming]]></category>
            <category><![CDATA[python]]></category>
            <dc:creator><![CDATA[Kunal Dhariwal]]></dc:creator>
            <pubDate>Mon, 09 Sep 2019 21:40:48 GMT</pubDate>
            <atom:updated>2019-10-14T07:08:02.413Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*KiYI6AmGN--NTEuy.jpeg" /><figcaption>Courtesy<a href="https://pixabay.com/photos/children-splash-asia-sunset-1822688/">: pixabay</a></figcaption></figure><iframe src="https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Fplayer.vimeo.com%2Fvideo%2F358783111%3Fapp_id%3D122963&amp;dntp=1&amp;url=https%3A%2F%2Fvimeo.com%2F358783111&amp;image=https%3A%2F%2Fi.vimeocdn.com%2Fvideo%2F812502344_640.jpg&amp;key=a19fcc184b9711e1b4764040d3dc5c07&amp;type=text%2Fhtml&amp;schema=vimeo" width="640" height="360" frameborder="0" scrolling="no"><a href="https://medium.com/media/58b25cab6e74e43ab18134449474c5d1/href">https://medium.com/media/58b25cab6e74e43ab18134449474c5d1/href</a></iframe><h3>What if tell you that this is not generated using Machine Learning or Deep Learning!</h3><p>The task of generating highlights for any sport is quite tedious and difficult. It involves a lot of tech and other software, hardware needs. You also need to invest in a lot of time to do it. If you’re into such a video company or department then it is okay, but if you’re a normal user just like me, you wouldn’t want to do it that way. I’ll be using python to get this done in just a few steps. There won’t be any external software involved. Just pure python and Also, we don’t have to rely on machine learning or deep learning techniques to do it. We will be using <strong>speech analysis</strong> for this task.</p><p>When something exciting happens during a game, there is a rise in the commentator’s voice. Let’s take cricket for example. Whenever a batsman hits a boundary or a bowler takes a wicket, there is a rise in the commentator’s voice. Both the audience and the commentators have high pitch during that event. We can use these changes in audio to capture interesting moments from a video.</p><p>The process we’ll be following is added down below:</p><ul><li>Input the video file</li><li>Extract the audio</li><li>Break the audio into chunks</li><li>Compute short-time energy of every chunk</li><li>Classify every chunk as excitement or not (based on a threshold value)</li><li>Merge all the excitement-clips to form the video highlights</li><li>Generate the final video highlights</li></ul><p><strong>Remember, We won’t use any external software here.</strong></p><p><strong>1. Download the video input</strong>: <a href="https://bit.ly/2lKfVa6">https://bit.ly/2lKfVa6</a></p><p>You can choose any other video of your favourite sport too.</p><p><strong><em>2. Extract the audio:</em></strong></p><p>For extracting the audio and other operations on the video file, we’ll be using a package called <strong>moviepy.</strong></p><p>a. Install the package.</p><pre>pip install moviepy</pre><p>b. Input the file path in this code.</p><pre># AudioExtract.py</pre><pre>import moviepy.editor as mp</pre><pre>clip = mp.VideoFileClip(&quot;filepath//videoplayback.mp4&quot;).subclip(1, 1380)<br>clip.audio.write_audiofile(&quot;filepath//audio.wav&quot;)</pre><blockquote><em>In the above code,</em> mp.VideoFileClip<em> is where we feed in our video file.</em></blockquote><blockquote>subclip<em>, we specify the the time limit for which we need the audio. *Not required if you want full clip audio*</em></blockquote><blockquote>audio.write_audiofile<em>, we specify the output path and file name here.</em></blockquote><p>That’s it, We have extracted the audio in 2 simple steps.</p><p><strong>3. Break the audio into chunks:</strong></p><p>We’ll be using the audio that we just extracted in <strong>step 2.</strong></p><p>librosa is a python package for music and audio analysis.</p><pre>import librosa<br>import IPython.display as ipd</pre><pre>filename = &quot;filepath\\audio.wav&quot;</pre><pre># loading the file with a sampling rate<br>x, sr = librosa.load(filename, sr=16000)</pre><pre># To get duration of the audio clip in minutes<br>int(librosa.get_duration(x, sr) / 60)</pre><pre># Dividing into chunks of 5 seconds <br>max_slice = 10<br>window_length = max_slice * sr</pre><pre># Playing the audio chunk<br>a = x[21 * window_length:22 * window_length]<br>ipd.Audio(a, rate=sr)</pre><p>Here, I’ve set the sampling rate as 16000, you can set it as per your convenience and requirement. Now, I have divided the audio into chunks of 10 seconds each, since we want to find out whether a particular audio chunk contains a rise in the audio voice or not. You can set time limit as per your choice, Try different time limits for different outputs. Also, we’ll play the audio chunk just for a check.</p><p><strong>4. Compute short-time energy of every chunk:</strong></p><pre>import numpy as np<br>s_energy = np.array([sum(abs(x[i:i + window_length] ** 2)) for i in range(0, len(x), window_length)])</pre><p>The energy or power of an audio signal refers to the loudness of the sound. It is computed by the sum of the square of the amplitude of an audio signal in the time domain. When energy is computed for a chunk of an entire audio signal, then it is known as <strong>Short Time Energy</strong>.</p><p>Let’s plot it:</p><pre>import matplotlib.pyplot as plt<br>plt.hist(s_energy)<br>plt.show()</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/640/0*BuOKhTcCUnH6WG3A.png" /></figure><p><strong>5. Classify every chunk as excitement or not (based on a threshold value):</strong></p><pre>import pandas as pd<br>df = pd.DataFrame(columns=[&#39;energy&#39;, &#39;start&#39;, &#39;end&#39;])<br>thresh = 180<br>row_index = 0<br>for i in range(len(s_energy)):<br>    value = energy[i]<br>    if value &gt;= thresh:<br>        i = np.where(s_energy == value)[0]<br>        df.loc[row_index, &#39;energy&#39;] = value<br>        df.loc[row_index, &#39;start&#39;] = i[0] * 10<br>        df.loc[row_index, &#39;end&#39;] = (i[0] + 1) * 10<br>        row_index = row_index + 1</pre><p>I am creating a dataframe based on the energy data. I chose the threshold as 180, you can choose it as per your choice and by seeing the graph above. And then merge consecutive time intervals of audio clips into one:</p><pre>temp = []<br>i = 0<br>j = 0<br>n = len(df) - 2<br>m = len(df) - 1<br>while (i &lt;= n):<br>    j = i + 1<br>    while (j &lt;= m):<br>        if (df[&#39;end&#39;][i] == df[&#39;start&#39;][j]):<br>            df.loc[i, &#39;end&#39;] = df.loc[j, &#39;end&#39;]<br>            temp.append(j)<br>            j = j + 1<br>        else:<br>            i = j<br>            break<br>df.drop(temp, axis=0, inplace=True)<br>print(df)</pre><p><strong>6. Merge all the excitement-clips to form the video highlights</strong></p><pre>start = np.array(df[&#39;start&#39;])<br>end = np.array(df[&#39;end&#39;])<br>for i in range(len(df)):<br>    if i != 0:<br>        start_lim = start[i] - 10<br>    else:<br>        start_lim = start[i]<br>    end_lim = end[i]<br>    filename = &quot;highlight&quot; + str(i + 1) + &quot;.mp4&quot;<br>    ffmpeg_extract_subclip(&quot;filepath//videofile.mp4&quot;, start_lim, end_lim, targetname=filename)</pre><p>This code will generate the highlights from each of the chunk based on the amplitude and then it will save these files in your system. You are not done yet, You have to merge all <strong>14</strong> files into one to actually create the highlights.</p><p><strong>7. Generate the final video highlights</strong></p><pre># Final_joined.py</pre><pre>from moviepy.editor import VideoFileClip, concatenate_videoclips</pre><pre>clip1 = VideoFileClip(&quot;filename\\highlight1.mp4&quot;)<br>clip2 = VideoFileClip(&quot;filename\\highlight2.mp4&quot;)<br>clip3 = VideoFileClip(&quot;filename\\highlight3.mp4&quot;)<br>clip4 = VideoFileClip(&quot;filename\\highlight5.mp4&quot;)<br>clip5 = VideoFileClip(&quot;filename\\highlight6.mp4&quot;)<br>clip6 = VideoFileClip(&quot;filename\\highlight7.mp4&quot;)<br>clip7 = VideoFileClip(&quot;filename\\highlight8.mp4&quot;)<br>clip8 = VideoFileClip(&quot;filename\\highlight9.mp4&quot;)<br>clip9 = VideoFileClip(&quot;filename\\highlight10.mp4&quot;)<br>clip10 = VideoFileClip(&quot;filename\\highlight11.mp4&quot;)<br>clip11= VideoFileClip(&quot;filename\\highlight12.mp4&quot;)<br>clip12 = VideoFileClip(&quot;filename\\highlight13.mp4&quot;)<br>clip13= VideoFileClip(&quot;filename\\highlight14.mp4&quot;)<br>clip14= VideoFileClip(&quot;filename\\highlight4.mp4&quot;)</pre><pre>final_clip = concatenate_videoclips([clip1,clip2,clip3,clip4,clip5,clip6,clip7,clip8,clip9,clip9,clip10,clip11,clip12,clip13,clip14])<br>final_clip.write_videofile(&quot;filepath\\Final_Highlights.mp4&quot;)</pre><p>We are using concatenate to merge all the files into one. After this the final file will be generated.</p><blockquote><strong><em>Complete code: </em></strong><a href="https://bit.ly/2lE55mi"><em>https://bit.ly/2lE55mi</em></a></blockquote><blockquote><em>You can get the </em><strong><em>video demo </em></strong><em>here: </em><a href="https://bit.ly/2lDEd5V"><em>https://bit.ly/2lDEd5V</em></a></blockquote><blockquote><em>If you encounter any error or need any help, you can always comment or ping me on LinkedIn.</em></blockquote><blockquote><strong><em>LinkedIn</em></strong><em>: </em><a href="https://bit.ly/2u4YPoF"><em>https://bit.ly/2u4YPoF</em></a></blockquote><blockquote><em>Reference: </em><a href="https://bit.ly/2kry0tB"><em>https://bit.ly/2kry0tB</em></a></blockquote><blockquote><em>I hope that this has helped you to enhance your knowledge base :)</em></blockquote><blockquote><em>Follow me for more!</em></blockquote><blockquote><em>Thanks for your read and valuable time!</em></blockquote><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=3695c98baead" width="1" height="1" alt=""><hr><p><a href="https://medium.com/data-science/generate-any-sport-highlights-using-python-3695c98baead">Become a video analysis expert using python</a> was originally published in <a href="https://medium.com/data-science">TDS Archive</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Create your own Virtual Personal Assistant]]></title>
            <link>https://medium.com/data-science/create-your-own-virtual-personal-assistant-94be5df65ced?source=rss-9cba4434c54e------2</link>
            <guid isPermaLink="false">https://medium.com/p/94be5df65ced</guid>
            <category><![CDATA[voice-assistant]]></category>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[python]]></category>
            <category><![CDATA[ai]]></category>
            <dc:creator><![CDATA[Kunal Dhariwal]]></dc:creator>
            <pubDate>Mon, 20 May 2019 13:52:05 GMT</pubDate>
            <atom:updated>2019-05-20T13:52:05.749Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*bqekIYkD8ddhxNyfb3MPhw.jpeg" /><figcaption>Courtesy: <a href="https://pixabay.com/photos/robot-mech-machine-technology-2301646/#">Pixabay</a></figcaption></figure><p>You know about Cortana, Siri and Google Assistant, right? Have you ever imagined that you can make your own virtual personal assistant and customize it as you want? Today, we’ll be doing it here. We’ll be building a personal assistant from scratch in python. Oh, Before getting into it, let me tell you that by no means it’s an AI but just a powerful example of what AI can do, and how versatile and amazing python is. Also, you need to have some experience with python in order to get started with it. So, Let’s begin:</p><p>First, We need to install some important packages:</p><ul><li>SpeechRecognition : Library for performing speech recognition, with support for several engines and APIs, online and offline.</li><li>Pyttsx3 : Pyttsx is a good text to speech conversion library in python.</li><li>Wikipedia : Wikipedia is a Python library that makes it easy to access and parse data from Wikipedia.</li><li>Wolframalpha : Python Client built against the <a href="http://wolframalpha.com/">Wolfram|Alpha</a> v2.0 API.</li><li>PyAudio : Python Bindings for PortAudio.</li></ul><p>Make sure you have installed all these packages or else you might run into some errors and this is how you install it:</p><pre>pip install PackageName</pre><p><strong>PyAudio Installation:</strong></p><p>You might run into some errors while installing Pyaudio, I’ve been through the same. You can make use of these steps to avoid the installation errors:</p><ul><li>find your Python version by python --version mine is 3.7.3 for example</li><li>find the appropriate .whl file from <a href="https://www.lfd.uci.edu/~gohlke/pythonlibs/#pyaudio"><strong>here</strong></a>, for example mine is PyAudio‑0.2.11‑cp37‑cp37m‑win_amd64.whl, and download it.</li><li>go to the folder where it is downloaded for example cd C:\Users\foobar\Downloads</li><li>install the .whl file with pip for example in my case:</li></ul><pre>pip install PyAudio-0.2.11-cp37-cp37m-win_amd64.whl</pre><p>Also, install this to avoid unnecessary errors:</p><pre>pip install pypiwin32</pre><p>If you are done installing those packages then we can import them and get back to the code:</p><pre>import os<br>import sys<br>import datetime<br>import pyttsx3<br>import speech_recognition as sr<br>import wikipedia<br>import wolframalpha<br>import webbrowser<br>import smtplib<br>import random</pre><p>Now, We’’ll be using ‘SAPI5’ as the TTS engine for pyttsx3 and getting the key for wolframaplha, and defining the client.</p><pre>engine = pyttsx3.init(‘sapi5’) <br>client = wolframalpha.Client(‘Get your own key’)</pre><p><em>You can get your own key from wolframalpha.com -&gt; Apps -&gt; Key.</em></p><p>Now, We’ll be initializing a variable and getting the necessary voice parameter we need. For female voice, you can set the value as -1 in the second line and -2 for male voice. Next, We’ll be creating a function <strong>talk </strong>with audio as the input parameter.</p><pre>voices = engine.getProperty(‘voices’)<br>engine.setProperty(‘voice’, voices[len(voices) — 2].id)</pre><pre>def talk(audio): <br>    print(‘KryptoKnite: ‘ + audio) <br>    engine.say(audio) <br>    engine.runAndWait()</pre><p>Next, let’s create another function <strong>greetMe </strong>and this will be used to greet the user when he runs the program. <em>datetime.datetime.now().hour</em> is being used to get the current time in terms of hour and give the output as per the time and the conditions written below. <strong>Talk </strong>fn will be used to give the output in terms of voice.</p><pre>def greetMe():<br>    CurrentHour = int(datetime.datetime.now().hour)<br>    if CurrentHour &gt;= 0 and CurrentHour &lt; 12:<br>        talk(&#39;Good Morning!&#39;)</pre><pre>    elif CurrentHour &gt;= 12 and CurrentHour &lt; 18:<br>        talk(&#39;Good Afternoon!&#39;)</pre><pre>    elif CurrentHour &gt;= 18 and CurrentHour != 0:<br>        talk(&#39;Good Evening!&#39;)<br></pre><pre>greetMe()</pre><pre>talk(&#39;Hey Buddy, It\&#39;s  your assistant KryptoKnite!&#39;)<br>talk(&#39;tell me about today?&#39;)</pre><p>Next, we’ll be creating another function <strong>GivenCommand</strong> which is used to recognize the user input, it will define that microphone is used as the source of input and We’ll set the pause threshold as 1. Try except block is used and language to recognized will be set as English-India, and if the voice isn’t recognized or heard We’ll send the text input as a kind of error message.</p><pre>def GivenCommand():<br>    k = sr.Recognizer()<br>    with sr.Microphone() as source:<br>        print(&quot;Listening...&quot;)<br>        k.pause_threshold = 1<br>        audio = k.listen(source)<br>    try:<br>        Input = k.recognize_google(audio, language=&#39;en-in&#39;)<br>        print(&#39;Kunal Dhariwal: &#39; + Input + &#39;\n&#39;)</pre><pre>    except sr.UnknownValueError:<br>        talk(&#39;Sorry! I didn\&#39;t get that! Try typing it here!&#39;)<br>        Input = str(input(&#39;Command: &#39;))</pre><pre>    return Input</pre><p>Now, let’s start the main function:</p><p>Here, We’ll be declaring some important functions and conditions which will enhance the functionality of our personal assistant and help him to deliver the output and receive the inputs from the user.</p><pre>if __name__ == &#39;__main__&#39;:</pre><pre>    while True:</pre><pre>        Input = GivenCommand()<br>        Input = Input.lower()</pre><pre>        if &#39;open google&#39; in Input:<br>            talk(&#39;sure&#39;)<br>            webbrowser.open(&#39;www.google.co.in&#39;)</pre><pre>        elif &#39;open youtube&#39; in Input:<br>            talk(&#39;sure&#39;)<br>            webbrowser.open(&#39;www.youtube.com&#39;)</pre><pre>        elif &quot;what\&#39;s up&quot; in Input or &#39;how are you&#39; in Input:<br>            setReplies = [&#39;Just doing some stuff!&#39;, &#39;I am good!&#39;,                                     <br>                          &#39;Nice!&#39;, &#39;I am amazing and full of power&#39;]<br>            talk(random.choice(setReplies))</pre><p>Similarly you can add up more <strong><em>elif </em></strong>with other functionalities like I’ve added one for sending the email.</p><pre>elif &#39;email&#39; in Input:<br>    talk(&#39;Who is the recipient? &#39;)<br>    recipient = GivenCommand()</pre><pre>    if &#39;me&#39; in recipient:<br>        try:<br>            talk(&#39;What should I say? &#39;)<br>            content = GivenCommand()</pre><pre>            server = smtplib.SMTP(&#39;smtp.gmail.com&#39;, 587)<br>            server.ehlo()<br>            server.starttls()<br>            server.login(&quot;Your_Username&quot;, &#39;Your_Password&#39;)<br>            server.sendmail(&#39;Your_Username&#39;, &quot;Recipient_Username&quot;, content)<br>            server.close()<br>            talk(&#39;Email sent!&#39;)</pre><pre>        except:<br>            talk(&#39;Sorry ! I am unable to send your message at this moment!&#39;)</pre><p>Or may be play some music?</p><pre>elif &#39;play music&#39; in Input:<br>    music_folder = &#39;Path<br>    music = [&#39;song&#39;]<br>    random_music = music_folder + random.choice(music) + &#39;.mp3&#39;<br>    os.system(random_music)</pre><pre>    talk(&#39;Okay, here is your music! Enjoy!&#39;)</pre><p>Next, We’ll be adding functionalities to use the input to make some searches on wikipedia, google and use wolframalpha.</p><pre>else:<br>    Input = Input<br>    talk(&#39;Searching...&#39;)<br>    try:<br>        try:<br>            res = client.Input(Input)<br>            outputs = next(res.outputs).text<br>            talk(&#39;Alpha says&#39;)<br>            talk(&#39;Gotcha&#39;)<br>            talk(outputs)</pre><pre>        except:<br>            outputs = wikipedia.summary(Input, sentences=3)<br>            talk(&#39;Gotcha&#39;)<br>            talk(&#39;Wikipedia says&#39;)<br>            talk(outputs)<br></pre><pre>    except:<br>        talk(&quot;searching on google for &quot; + Input)<br>        say = Input.replace(&#39; &#39;, &#39;+&#39;)<br>        webbrowser.open(&#39;https://www.google.co.in/search?q=&#39; + Input)</pre><pre>talk(&#39;Next Command! Please!&#39;)</pre><p>When all this is done, it is very important for the program to exit. Let’s write a condition for it here:</p><pre>elif &#39;nothing&#39; in Input or &#39;abort&#39; in Input or &#39;stop&#39; in Input:<br>    talk(&#39;okay&#39;)<br>    talk(&#39;Bye, have a good day.&#39;)<br>    sys.exit()</pre><pre>elif &#39;bye&#39; in Input:<br>    talk(&#39;Bye, have a great day.&#39;)<br>    sys.exit()</pre><p>That’s it! You have created your own virtual personal assistant.</p><p>You can now customize it as you like and set any conditions for it, Also you can add N number of functionalities to it and make it even more amazing.</p><blockquote><strong>Complete code:</strong> <a href="https://bit.ly/2VaBsEU">https://bit.ly/2VaBsEU</a></blockquote><blockquote>You can get the <strong><em>video demo </em></strong>here on my LinkedIn Post: <a href="https://bit.ly/2DW8qU0">https://bit.ly/2DW8qU0</a></blockquote><blockquote>If you encounter any error or need any help, you can always make a comment or ping me on LinkedIn.</blockquote><blockquote><strong>LinkedIn</strong>: <a href="https://bit.ly/2u4YPoF">https://bit.ly/2u4YPoF</a></blockquote><blockquote><strong>Github</strong>: <a href="https://bit.ly/2SQV7ss">https://bit.ly/2SQV7ss</a></blockquote><blockquote>I hope that this has helped you to enhance your knowledge base :)</blockquote><blockquote>Follow me for more!</blockquote><blockquote>Thanks for your read and valuable time!</blockquote><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=94be5df65ced" width="1" height="1" alt=""><hr><p><a href="https://medium.com/data-science/create-your-own-virtual-personal-assistant-94be5df65ced">Create your own Virtual Personal Assistant</a> was originally published in <a href="https://medium.com/data-science">TDS Archive</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Taking Data Visualization to Another Level]]></title>
            <link>https://medium.com/hackernoon/taking-data-visualization-to-another-level-4d1c47bb01a2?source=rss-9cba4434c54e------2</link>
            <guid isPermaLink="false">https://medium.com/p/4d1c47bb01a2</guid>
            <category><![CDATA[data-visualization]]></category>
            <category><![CDATA[python]]></category>
            <category><![CDATA[data-analysis]]></category>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[hackernoon-top-story]]></category>
            <dc:creator><![CDATA[Kunal Dhariwal]]></dc:creator>
            <pubDate>Mon, 11 Mar 2019 13:51:00 GMT</pubDate>
            <atom:updated>2019-05-10T13:31:07.864Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*7SYBgrbNOfbD45I3NRxeKA.jpeg" /><figcaption>Courtesy:<a href="https://pixabay.com/illustrations/background-data-network-web-3228704/"> Pixabay</a></figcaption></figure><p>When you tend to use one library for a certain period of time, you get used to it. But, you need to evolve and learn something new every day. If you are still stuck up with Matplotlib(Which is amazing), Seaborn(This is amazing too), Pandas(Basic, yet easy Visualization) and Bokeh, You need to move on and try something new. Many amazing visualization libraries are available in python, which turns to be very versatile. Here, I’m going to discuss about these amazing libraries:</p><ul><li><strong><em>Plotly</em></strong></li><li><strong><em>Cufflinks</em></strong></li><li><strong><em>Folium</em></strong></li><li><strong><em>Altair + Vega</em></strong></li><li><strong>D3.js (My best pick)</strong></li></ul><p>If you are aware and use the libraries mentioned above then you are on the right track of evolution. They can help in generating some amazing visualizations and syntax ain’t difficult too. Generally, I prefer more of plotly+cufflinks and <strong>D3.js. </strong>Alright, lets get back to the basics:</p><h4><strong><em>Plotly</em></strong></h4><p>Plotly is an open-source, interactive and browser-based graphing library for python. Plotly is a library that allows you to create interactive plots that you can use in dashboards or websites (you can save them as html files or static images). Plotly is built on top of plotly.js which in turn is built on D3.js and it is a high-level charting library. plotly comes with over 30 chart types, including scientific charts, statistical charts, 3D graphs, financial charts and more. The best things about plotly is that you can use it in Jupyter Notebooks, as well as stand alone HTML pages. You can also use it online on their site but i prefer more to use it offline and you can also save the visualization as an image. It’s pretty simple to use it and get it to work.</p><p><strong><em>— Method to use it in Jupyter Notebook (Offline)</em></strong></p><p>First, install the plotly library.</p><pre>pip install plotly</pre><p>Then open jupyter notebook and type this:</p><pre>from plotly import __version__<br>from plotly.offline import download_plotlyjs, init_notebook_mode, plot, iplot<br>init_notebook_mode(connected=True)</pre><p>The syntax is quite simpler or simplest, i say? In P<em>andas</em>, you use <em>dataframe.plot() </em>and here, you use<em> dataframe.iplot(). </em>This <em>“</em><strong><em>i</em></strong><em>” </em>changes the whole definition of the visualization.</p><p>With just one line, i generated this scatter plot. You can customize it as you want. Remember to specify mode markers or you’ll just get some cluster of lines.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*s4O6EKAv1tTCtDbKLcS_Yw.gif" /><figcaption>Scatter plot generated using plotly</figcaption></figure><blockquote>Please note that as the data increases, plotly begins to choke. So, I would only use plotly when I have less than 500K data points.</blockquote><blockquote>Try it all in your Jupyter Notebook.</blockquote><h4><strong>Cufflinks</strong></h4><p>Cufflinks bind Plotly directly to pandas dataframes. The combination is just amazing, the power of plotly combined with flexibility of Pandas. It is more effective than plotly. The syntax is even simpler than plotly. With plotly’s Python library, you can describe figures with DataFrame’s series and index, but with cufflinks you can plot it directly. Here is an example:</p><pre>df = cf.datagen.lines()<br><br>py.iplot([{<br>    &#39;x&#39;: df.index,<br>    &#39;y&#39;: df[col],<br>    &#39;name&#39;: col<br>}  <strong>for</strong> col <strong>in</strong> df.columns])</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*F1KmpY26BfKmu9VjiTobQQ.gif" /><figcaption>With Plotly</figcaption></figure><pre>df.iplot(kind=&#39;scatter&#39;)</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*j_AyLUvIKNAZJLk-UgN2PA.gif" /><figcaption>With Cufflinks</figcaption></figure><p>Cufflinks makes it much easier to plot stuffs. You can also generated amazing 3D charts with cufflinks too. I have generated this 3D with just couple of lines of code.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*6aPmG867ymQSow3XxccUeg.gif" /><figcaption>3D chart with Cufflinks</figcaption></figure><p>You can always try it out in your Jupyter Notebook.</p><p><strong>— Quick Hack:</strong></p><p>Set in the config:</p><blockquote>c.NotebookApp.iopub_data_rate_limit = 1.0e10</blockquote><p>Import it the following way:</p><pre>import plotly.graph_objs as go<br>import plotly.plotly as py<br>import cufflinks as cf<br>from plotly.offline import iplot, init_notebook_mode<br>cf.go_offline()<br># Set global theme<br>cf.set_config_file(world_readable=True, theme=’pearl’, offline=True)<br>init_notebook_mode()</pre><p>And it works inline.</p><p>Next, I’m going to talk about yet another amazing Viz library.</p><h4><strong>Folium</strong></h4><p>Folium is built on the data wrangling strengths of the Python ecosystem and the mapping strengths of the Leaflet.js library. You can manipulate your data in python, then visualize it in a Leaflet map via folium. Folium is turning out be an amazing library for plotting spatial data. You can also generate heat maps and choropleth maps using folium. Let’s learn something about folium:</p><ol><li>Maps are defined as a <em>folium.Map </em>object, addition of other folium objects on top of the <em>folium.Map can be done </em>to improve the map rendered</li><li>You can use different map tiles for the map rendered by Folium, such as MapBox, OpenStreetMap , and several other tiles, for that you can visit <a href="https://github.com/python-visualization/folium/tree/master/folium/templates/tiles">this github repo folder</a> or <a href="http://python-visualization.github.io/folium/docs-v0.5.0/modules.html">this documentation page</a>.</li><li>You can also select different map projections. Many projections are available out there.</li></ol><p>Let’s generate a Choropleth map with Geojson of US unemployment. Here is the snippet:</p><pre>map = folium.Map([43, -100], zoom_start=4)<br><br>choropleth = folium.Choropleth(<br>    geo_data=us_states,<br>    data=state_data,<br>    columns=[&#39;State&#39;, &#39;Unemployment&#39;],<br>    key_on=&#39;feature.id&#39;,<br>    fill_color=&#39;YlGn&#39;,<br>    name=&#39;Unenployment&#39;,<br>    show=<strong>False</strong>,<br>).add_to(m)<br><br><em># The underlying GeoJson and StepColormap objects are reachable</em><br>print(type(choropleth.geojson))<br>print(type(choropleth.color_scale))<br><br>folium.LayerControl(collapsed=<strong>False</strong>).add_to(m)<br><br>map.save(os.path.join(&#39;results&#39;, &#39;GeoChoro.html&#39;))<br><br>map</pre><p>This is just a basic one, you can add markers, pop ups and a lot more to it. Here is how it would look like.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*ZgguNbAGJQg-drUMwDWSkQ.gif" /><figcaption>Map with leaflet and folium</figcaption></figure><h4><strong>Altair + Vega</strong></h4><p>Altair is a declarative statistical visualization library and it is based on Vega and Vega-Lite. Altair enables you to build a wide range of statistical visualizations quickly with a powerful and concise visualization grammar. You need to install it the following way, if you are using Jupyter Notebook. It also includes some example vega datasets.</p><pre>pip install -U altair vega_datasets notebook vega</pre><p>Altair’s main dependency is Vega, in order to make the plots to be visible on the screen, you need to install it and also, you need to run this command for every new session.</p><pre>alt.renderers.enable(‘notebook’)</pre><p>Data in Altair is built around the Pandas Dataframe. One of the defining characteristics of statistical visualization is that it begins with tidy Dataframes. You can also save the plot as an image or open it in vega editor for more options. It’s definitely not the best one out there, but definitely worth a try for the sake of creators hard work.</p><p>Here is an example, I’m using cars dataset for this;</p><pre>import altair as alt<br>from vega_datasets import data</pre><pre>source = data.cars()</pre><pre>brush = alt.selection(type=&#39;interval&#39;)</pre><pre>points = alt.Chart().mark_point().encode(<br>    x=&#39;Horsepower:Q&#39;,<br>    y=&#39;Miles_per_Gallon:Q&#39;,<br>    color=alt.condition(brush, &#39;Origin:N&#39;, alt.value(&#39;lightgray&#39;))<br>).add_selection(<br>    brush<br>)</pre><pre>bars = alt.Chart().mark_bar().encode(<br>    y=&#39;Origin:N&#39;,<br>    color=&#39;Origin:N&#39;,<br>    x=&#39;count(Origin):Q&#39;<br>).transform_filter(<br>    brush<br>)</pre><pre>alt.vconcat(points, bars, data=source)</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/815/1*fvdV6JcNRZEQyq9A8RHXuQ.gif" /><figcaption>Scatter plot and Histogram with Altair and Vega</figcaption></figure><blockquote>You can try it out in your own Notebook and let me know if you get stuck anywhere!</blockquote><h4><strong>D3.js (Data Driven Documents)</strong></h4><p><strong>D3.js</strong> is a JavaScript library for manipulation of documents based on the data. You can bring data to life using HTML, SVG, and CSS. D3 does not require you to tie tie yourself to any proprietary framework because modern browsers have it all that D3 needs and it is also used for combining powerful visualization components and for a data-driven approach to DOM manipulation.</p><p>D3.js is the best data visualization library out in the market. I prefer to use it almost every time. You can use it with python and as well as with R. Originally, it works with JavaScript and that becomes quite tough because JS has wide range of functions and requires much learning and experience but if you are a JS pro then you don’t need to make a second thought. Although, <em>Python</em> and <em>R</em> has made it a bit simpler, just a bit! But you get the best stuff out there with this library.</p><p>D3py has 3 three main dependencies:</p><ol><li>Numpy</li><li>Pandas</li><li>NetworkX</li></ol><p>I would suggest you to use it with JavaScript or R, not with python because the version is out of date and was last updated in 2016. Though, it was just a thin python wrapper for D3.js.</p><p>R has an interface for D3 visualizations. With <strong>r2d3</strong>, you can bind data from R to D3 visualizations. D3 visualizations created with <strong>r2d3</strong> work just like R plots within RStudio, R Markdown documents, and Shiny applications. You can install the <strong>r2d3</strong> package from CRAN as follows:</p><pre>install.packages(“r2d3”)</pre><p>You can make some amazing visualizations with this one, Let me show you a couple of it here.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*hVqpVQEOgHJpDEDHc-ae4w.gif" /><figcaption>Sequences Sunburst — <a href="https://bl.ocks.org/kerryrodden">Kerry Rodden</a>’s Block (Source)</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*a1VGwPlHodZwluVfPBtxnw.gif" /><figcaption>Activity Status of an Year — Kunal Dhariwal (Me, lol)</figcaption></figure><p>From basics to High end, you can build anything with D3.js, Don’t forget to try it out.</p><blockquote>If you encounter any error or need any help, you can always make a comment or ping me on LinkedIn.</blockquote><blockquote><strong>LinkedIn</strong>: <a href="https://bit.ly/2u4YPoF">https://bit.ly/2u4YPoF</a></blockquote><blockquote><strong>Github</strong>: <a href="https://bit.ly/2SQV7ss">https://bit.ly/2SQV7ss</a></blockquote><p><strong><em>P.S- Special Thanks to the creators and contributors of those amazing libraries.</em></strong></p><blockquote>I hope that this has helped you to enhance your knowledge base :)</blockquote><blockquote>Follow me for more!</blockquote><blockquote>Thanks for your read and valuable time!</blockquote><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=4d1c47bb01a2" width="1" height="1" alt=""><hr><p><a href="https://medium.com/hackernoon/taking-data-visualization-to-another-level-4d1c47bb01a2">Taking Data Visualization to Another Level</a> was originally published in <a href="https://medium.com/hackernoon">HackerNoon.com</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Bitcoin Raw Transaction Breakdown]]></title>
            <link>https://medium.datadriveninvestor.com/bitcoin-raw-transaction-breakdown-c0a5a3aa8688?source=rss-9cba4434c54e------2</link>
            <guid isPermaLink="false">https://medium.com/p/c0a5a3aa8688</guid>
            <category><![CDATA[cryptocurrency]]></category>
            <category><![CDATA[blockchain]]></category>
            <category><![CDATA[crypto]]></category>
            <category><![CDATA[blockchain-technology]]></category>
            <category><![CDATA[bitcoin]]></category>
            <dc:creator><![CDATA[Kunal Dhariwal]]></dc:creator>
            <pubDate>Mon, 04 Feb 2019 06:14:08 GMT</pubDate>
            <atom:updated>2019-02-04T08:08:15.386Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/700/0*1CeRgkUU1O1f_gkU" /></figure><p><strong><em>If you are new to the Blockchain world then reading my articles posted earlier would be a valuable deal for you.</em></strong> Here, you get some basic knowledge and also all sort of information you must know about the Blockchain world.</p><p><a href="https://bit.ly/2SeNJat"><strong><em>https://bit.ly/2SeNJat</em></strong></a><strong><em> &lt;- Blockchain Basics and important terminologies.</em></strong></p><p><a href="https://bit.ly/2FzolZT"><strong><em>https://bit.ly/2FzolZT</em></strong></a><strong><em> &lt;- How Blockchain works.</em></strong></p><p>In this article, we’ll be breaking down the Bitcoin raw transaction for our better understanding of how a transaction works and what are all its contents in the Bitcoin Blockchain. Firstly, you need to know that transactions are stored on the blockchain by their <em>Transaction Hash</em>. Transactions stored in the bitcoin blockchain are stored in a double-hashed form. This means the raw transaction was put through SHA256 twice to get the Transaction hash we see on the blockchain.</p><blockquote>For example, a transaction with this hash: 2621c2609d114f652dadf6fd95820c021da1cf2d0ac15e0361fd5d136e30a3c4</blockquote><blockquote>Transaction in the bitcoin blockchain are stored in a double-hashed form:<br>SHA256(SHA256(01000…) = 2621c2609d114f652dadf6fd95820c021da1cf2d0ac15e0361fd5d136e30a3c4</blockquote><p>To get the raw transaction, you can use this W<strong>eb API: </strong><a href="https://bit.ly/2VV6y56"><strong>https://bit.ly/2VV6y56</strong></a><strong> </strong>.You just need to enter the transaction id in order to get the raw transaction details.</p><p>Or</p><p>You can use <strong><em>getrawtransaction “txid”</em></strong><em> command in the Bitcoin Core Console.</em></p><blockquote><em>Given the raw transaction= </em>0100000001f3f6a909f8521adb57d898d2985834e632374e770fd9e2b98656f1bf1fdfd427010000006b48304502203a776322ebf8eb8b58cc6ced4f2574f4c73aa664edce0b0022690f2f6f47c521022100b82353305988cb0ebd443089a173ceec93fe4dbfe98d74419ecc84a6a698e31d012103c5c1bc61f60ce3d6223a63cedbece03b12ef9f0068f2f3c4a7e7f06c523c3664ffffffff0260e31600000000001976a914977ae6e32349b99b72196cb62b5ef37329ed81b488ac063d1000000000001976a914f76bc4190f3d8e2315e5c11c59cfc8be9df747e388ac00000000<em>,</em></blockquote><blockquote><em>We were able to see more details about the Transaction Data model.</em></blockquote><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*oFl5V471MunuwG2Bv_-Eyg.png" /><figcaption>Raw Transaction Breakdown</figcaption></figure><p><strong>Version</strong> — All transactions include information about the Bitcoin Version number so we know which rules this transaction follows.</p><p><strong>Input Count</strong> — Which is how many inputs were used for this transaction.</p><p><em>Data stored in Input information:</em></p><ul><li><strong>Previous output hash</strong> — All the inputs reference back to an output (UTXO). This points back to the transaction containing the UTXO (Unspent Transaction Output) that will be spent in this input. The hash value of this UTXO is saved in reverse order.</li><li><strong>Previous output index</strong> — The transaction may have more than one UTXO which are referenced by their index number. The first index is 0.</li><li><strong>Unlocking Script Size</strong> — This is the size of the unlocking script in bytes.</li><li><strong>Unlocking Script</strong> — This is the hash of the unlocking script that fulfils the conditions of the UTXO locking script.</li><li><strong>Sequence Number</strong> — This is a deprecated feature of bitcoin, currently set to ffffffff by default.</li></ul><p><strong>Output Count</strong> — which tells us how many outputs were produced from this transaction.</p><p><em>Data stored in Output Information:</em></p><ul><li><strong>Amount</strong> — The amount of Bitcoin outputted in Satoshis (the smallest bitcoin unit). ¹⁰⁸ Satoshis = 1 Bitcoin.</li><li><strong>Locking Script Size</strong> — This is the size of the locking script in bytes.</li><li><strong>Locking Script</strong> — This is the hash of the locking script that specifies the conditions that must be met in order to spend this output.</li></ul><p><strong>Locktime</strong> — The locktime field indicates the earliest time or the earliest block that a transaction can be added to the blockchain.<br>If the locktime is non-zero &amp; less than 500 million, it is interpreted that block height and miners have to wait until that block height is reached before attempting to add it to a block. If the locktime is above 500 million, it is read as a UNIX timestamp which means the number of seconds since the date January 1st 1970. It is usually 0 which means confirm as soon as possible.</p><h4>Thanks for your read!</h4><h4>Show some love, if you find it helpful :)</h4><h4>Related Stories from DDI:</h4><ul><li><a href="https://medium.com/datadriveninvestor/why-data-will-transform-investment-management-4a60966c1c81">Why Data will Transform Investment Management</a></li><li><a href="https://medium.com/datadriveninvestor/a-quick-guide-to-investment-algorithms-628450b9dc55">A Quick Guide to Investment Algorithms</a></li></ul><iframe src="https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Fupscri.be%2Fb2a0d6%3Fas_embed%3Dtrue&amp;dntp=1&amp;display_name=Upscribe&amp;url=https%3A%2F%2Fupscri.be%2Fb2a0d6%2F&amp;key=a19fcc184b9711e1b4764040d3dc5c07&amp;type=text%2Fhtml&amp;schema=upscri" width="800" height="400" frameborder="0" scrolling="no"><a href="https://medium.com/media/0707f5c806284d01a4a13c7b13a91ce3/href">https://medium.com/media/0707f5c806284d01a4a13c7b13a91ce3/href</a></iframe><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=c0a5a3aa8688" width="1" height="1" alt=""><hr><p><a href="https://medium.datadriveninvestor.com/bitcoin-raw-transaction-breakdown-c0a5a3aa8688">Bitcoin Raw Transaction Breakdown</a> was originally published in <a href="https://medium.datadriveninvestor.com/">DataDrivenInvestor</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Blockchain Consensus Mechanisms]]></title>
            <link>https://medium.datadriveninvestor.com/blockchain-consensus-mechanisms-7aa0176d488a?source=rss-9cba4434c54e------2</link>
            <guid isPermaLink="false">https://medium.com/p/7aa0176d488a</guid>
            <category><![CDATA[ethereum]]></category>
            <category><![CDATA[cryptocurrency]]></category>
            <category><![CDATA[blockchain]]></category>
            <category><![CDATA[bitcoin]]></category>
            <category><![CDATA[hyperledger]]></category>
            <dc:creator><![CDATA[Kunal Dhariwal]]></dc:creator>
            <pubDate>Tue, 27 Nov 2018 10:31:39 GMT</pubDate>
            <atom:updated>2018-11-29T05:52:56.124Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*ZT-C-MG5j1tclQtbOwSXeA.png" /><figcaption>Courtesy: <a href="https://pixabay.com/en/blockchain-block-chain-technology-3019120/">https://pixabay.com/en/blockchain-block-chain-technology-3019120/</a></figcaption></figure><p><strong><em>Newbie? Read this article to know about blockchain and then you’re welcome back here: </em></strong><a href="https://bit.ly/2SeNJat"><strong><em>https://bit.ly/2SeNJat</em></strong></a></p><p>The buzz around blockchain remains too high. A blockchain is a decentralized, distributed and public digital ledger that is used to record transactions across many computers so that the record cannot be altered retroactively without altering the subsequent blocks and the collusion of the network. The blockchain needs a functional, efficient and secure consensus algorithm in order to function on global scale with a shared public ledger.</p><p>Here, I’ll be discussing about the consensus mechanisms that a blockchain uses. It differs from blockchain to blockchain and the technology they are using, for example: <em>Ethereum</em> uses <strong><em>Proof of stake </em></strong>and<strong> </strong><em>Bitcoin</em> uses <strong><em>Proof of work.</em></strong></p><p><strong>Blockchain Consensus Mechanisms:</strong></p><ol><li><strong>Proof of Work**</strong></li></ol><p>Proof of work is a protocol that has the main goal of deterring cyber-attacks such as a distributed denial-of-service attack (DDoS) which has the purpose of exhausting the resources of a computer system by sending multiple fake requests. This is how it works:</p><ul><li>Proof of work is a requirement to define an expensive computer calculation popularly called as mining, that needs to be performed in order to create a new group of trustless transactions (Block) on the blockchain.</li><li>The first block miner who solves the problem gets a reward in return.</li><li>The network miners compete to be the first to find a solution for the mathematical problem.</li><li>The transactions that are verified are only stored in the public blockchain.</li></ul><p>The party with the<em> most computational power</em> usually mines the block and others just gets their energy wasted because multiple miners compete to create a block at one instance. So, here we introduce the mechanism of Proof of Stake.</p><p><strong>2. Proof of Stake**</strong></p><p>It is the most common alternative to Proof of Work. It’s an alternative way to create blocks in a blockchain. The block producers are called as a validator instead of miners. Validators must put down a deposit or stake in order to participate in the process of block creation.</p><p>Here, the validators are selected on the basis of some selection algorithm that takes their stake into account. Only the selected validator can create a block and others cannot participate, Hence saving the energy of the other validators. It is much better for the environment. If the validator does some wrong stuff then they lose their stake and hence you get reward for your honesty. The miners take their <em>transaction fees </em>because they get no reward unlike PoW.</p><p><strong><em>Ethereum </em></strong>has shifted from <strong><em>PoW to PoS.</em></strong></p><p><strong>3. Proof of Burn</strong></p><p>Proof of Burn is an alternative to Proof of Work and Proof of Stake and also a method for distributed consensus. It is also a consensus mechanism implemented by a blockchain network to ensure that all participating nodes come to an agreement about the true and valid state of the blockchain network thereby avoiding any possibility of cryptocoin double spending.</p><p>It works on the principle of allowing the miners to “burn” or “destroy” the virtual currency tokens, which grants them the right to write blocks in proportion to the coins burnt. The more coins you burn, the better chance you have of being selected to mine the next block. It tries to address the energy consumption issue of PoW. In order to burn the coins, miners need to send it to the address where it couldn’t be spent in order to verify the block. <strong><em>Slim coin </em></strong>is based on this consensus mechanism algorithm.</p><p><strong>4. Proof of Capacity</strong></p><p>Well, this is also a consensus mechanism algorithm. This is a bit different from other algorithms. Here, you’ve to pay with the hard drive space. The more hard drive space you have, the better chance you have for mining the next block and earning the block reward. Prior to mining in a PoC system, the algorithm generates large data sets known as ‘plots’ and they are stored on your hard drive.</p><p>Proof of capacity involves two parts. There is the plotting of the hard drive and the actual mining of the blocks. Depending on the size of your hard drive, it can take days or even weeks to make your unique plot files. <strong>Plotting</strong> uses a very slow hash known as Shabal. This also includes variations like Proof of Space and Proof of Storage. This is more like aplay to pay thing. The only cryptocurrency which is based on PoC is <strong><em>Burst coin.</em></strong></p><p><strong>5. Delegated Proof of Stake</strong></p><p>This is the extension to PoS. In DPoS blockchain consensus protocols, cryptocoin holders use their coin balances to elect delegates called as witnesses. These witnesses have the opportunity to stake blocks of new transactions and add them to the blockchain. Voting power is actually determined by the amount of coins they have, popularly known as cyberwealth.</p><p>Those who have more coins or tokens will have a greater impact on the network than those with few coins. Anybody can take part in the block generator election and the chance to create the blocks will be determined by the amount of votes they get with respect to every single other generator. In the event that they can really pull it off, at that point they will have the DPoS to thank. The cryptocurrency based on this mechanism is <strong><em>CaptainAlt coin.</em></strong></p><p><strong>6. Proof of Activity</strong></p><p>This algorithm is much more like Proof of Work with the complexity reduced to a great extent due to which the solution of the problem takes from fractions of a second to several minutes. Proof of Activity is a hybrid approach that combines proof of work and proof of stake. As in PoW, miners race to solve a cryptographic puzzle. Then the system switches to PoS. The difference is that the blocks being mined do not contain transactions. They are simply templates with header information and the mining reward address.</p><p>The correctness of the created blocks is verified by limiting the minimum possible time for the block creation. This allows you to limit the maximum speed of adding blocks to the chain and hence preventing the emergence of flood in the network, which we also call as spam. The coins that use this mechanism are <strong><em>Decred </em></strong>and <strong><em>Espers.</em></strong></p><p><strong>7. Practical Byzantine fault tolerant Mechanism **</strong></p><p>PBFT is the most popular permissioned blockchain platform protocol and its currently used by <strong><em>Hyperledger Fabric (IBM)</em></strong>. So have you heard about Byzantine general’s problem? No? Let me explain that problem first and then you’ll be able to understand PBFT better. Imagine a group of Byzantine generals and their armies have surrounded a castle and preparing to attack. To win, they must attack simultaneously. But they know that there is at least one traitor among them. So, how do they launch a successful attack with at least one, unknown, bad actor in their group?</p><p>The analogy is clear: In any distributed computing environment there is a risk that rogue actors could wreak havoc. So, its reliance on community consensus makes Byzantine faults an especially thorny problem for blockchain. To overcome that problem PBFT is used. A consensus decision is determined based on the total decisions submitted by all the generals. PBFT consensus mechanism does not require any hashing energy to approve exchanges in a blockchain, which implies there is no requirement for high energy utilization and the danger of centralization is lower. It also yields high rewards and also there’s transaction finality.</p><p><strong>8. Proof of Elapsed Time</strong></p><p><strong><em>Intel</em></strong> has come up with its own alternative consensus protocol called proof of elapsed time. PoET is now the consensus model of choice for <strong><em>Hyperledger Sawtooth’s</em></strong> modular framework. The PoET algorithm is often used to give permissions to determine the rights a miner has. The networks using this identify the participants before they can participate. Each node has exactly the same chance to be the winner of the block. The PoET mechanism is based on spreading and fairly distributing the odds for the largest possible number of participants.</p><p>Each participating node in the network must wait for a certain amount of time. In this, every member asks for a hold-up time from its local reliable enclave. The member with the briefest hold-up time is next to offer a block after it holds-up for the allotted waiting time. Each node generates its own waiting time each time, after which it goes into a kind of sleep mode. As soon as the node wakes up and a block is available, that node is the lucky winner. Then a node can spread the information over the entire network, keeping it decentralized and receiving the reward.</p><p>BTW, we have to rely on intel for this and trust me, this is just against the concept of decentralization. I mean, why do we have to rely on third party, yet again?</p><p><strong>9. Proof of Importance</strong></p><p>PoI was first introduced by <strong><em>NEM</em></strong> for its cryptocurrency named as <strong><em>XEM coin. </em></strong>With PoI, it isn’t just the coin balance that values. The reward system, as per POI approach, should be based on the contribution of a user to the network in all capacities. The staking of the block, therefore, is based on multiple factors including reputation, overall balance, and the number of transactions done through or from a particular address.</p><p>Here, participants who frequently send and receive transactions are more likely to find blocks. The Proof of Importance only counts the coins that have been on the account for a number of days. 10% of the XEM that has not been set apart for the network is fixed every day. It’s very expensive to attack the network and also it generously rewards the user for protecting the network.</p><p><strong>10. Leased Proof of Stake</strong></p><p>LPoS was developed by WAVES, a decentralized blockchain platform that allows for the creation of custom tokens. In PoS, You can only add a block if you have certain amount coins or else you can’t even participate in the process of creating a block and not everyone gets the chance to participate in the maintenance of the chain and also get the rewards. This problem has been solved by LPoS.</p><p>More the number of users or participants, better becomes the security of the network. Greater the number of coins on the account, increases the likelihood that it will add the next block to the chain and the reward.</p><p><strong>11. Proof of Identity</strong></p><p>The concept of PoI is to compare the private key to an authorized identity. Proof of Identity (PoI) is basically a cryptographic evidence (piece of data) which tells that any user knows a private key that compares to an authorized identity and cryptographically attached to a specific transaction. Every individual from an existing group can create a block of data and present it to anyone for instance to the processing node.</p><blockquote><strong><em>Congrats, Now you know about all the blockchain consensus mechanisms.</em></strong></blockquote><p><em>Apart from this, I also wanted to mention this important consensus mechanism that you need to know about.</em></p><p><strong>Directed Acyclic Graph</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/260/1*606YwvAtdzf1qvhgB7OJEg.png" /></figure><p>As a blockchain grows over time, nodes will need more and more storage capacity and a higher bandwidth (volume increases) to keep up-to-date with the transactions added to the ledger.</p><p>Within a DAG, the nodes itself become the ‘miners’. This process is also much more lightweight because you only have to verify transactions for the two closest nodes. As miners are eliminated and we validate transactions ourselves, transaction fees are reduced to zero. The word acyclic just means that no node in the graph can reference back to itself, it can’t be its own mother node. <strong><em>IOTA</em></strong> make use of <strong><em>DAG (Tangle)</em></strong>. Besides, It’s data structure allows for the network’s easy scalability. Everyone is participating in reaching a consensus and, therefore, the more people are using IOTA, the faster the network becomes. More people, faster network and secure network!</p><blockquote><strong>That’s it, you’re all set to tell others that you know how things get done in a blockchain!</strong></blockquote><p><strong>Thanks for your read!</strong></p><p><strong><em>Do support and show some love (Claps xD)</em></strong></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=7aa0176d488a" width="1" height="1" alt=""><hr><p><a href="https://medium.datadriveninvestor.com/blockchain-consensus-mechanisms-7aa0176d488a">Blockchain Consensus Mechanisms</a> was originally published in <a href="https://medium.datadriveninvestor.com/">DataDrivenInvestor</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
    </channel>
</rss>