<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Byte Tales - Medium]]></title>
        <description><![CDATA[Space for Emotional Code and Binary Stories. - Medium]]></description>
        <link>https://medium.com/byte-tales?source=rss----1c1d036af958---4</link>
        <image>
            <url>https://cdn-images-1.medium.com/proxy/1*TGH72Nnw24QL3iV9IOm4VA.png</url>
            <title>Byte Tales - Medium</title>
            <link>https://medium.com/byte-tales?source=rss----1c1d036af958---4</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Sun, 22 Nov 2026 01:58:30 GMT</lastBuildDate>
        <atom:link href="https://medium.com/feed/byte-tales" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[Learning Data Analysis With Python — Pandas DataFrame]]></title>
            <link>https://medium.com/byte-tales/learning-data-analysis-with-python-pandas-dataframe-2f2d40d6c11f?source=rss----1c1d036af958---4</link>
            <guid isPermaLink="false">https://medium.com/p/2f2d40d6c11f</guid>
            <category><![CDATA[data-analysis]]></category>
            <category><![CDATA[tutorial-python]]></category>
            <category><![CDATA[python]]></category>
            <category><![CDATA[pandas-dataframe]]></category>
            <category><![CDATA[data-science]]></category>
            <dc:creator><![CDATA[James Shah]]></dc:creator>
            <pubDate>Fri, 04 Sep 2020 13:43:58 GMT</pubDate>
            <atom:updated>2020-09-04T13:43:58.049Z</atom:updated>
            <content:encoded><![CDATA[<h3>Learning Data Analysis With Python — Pandas DataFrame</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/770/1*rLrBoEJU3CMe51PijdO80Q.png" /></figure><blockquote><em>⚠️ </em><strong><em>Note</em></strong><em> — </em>This post is a part of Learning data analysis with python series. If you haven’t read the first post, some of the content won’t make sense. Check it out <a href="https://bytetales.co/learning-data-analysis-with-python-introduction-to-pandas">here</a>.</blockquote><p>In the previous article, we talked about Pandas Series, working with real world data and handling missing values in data. Although series are very useful but most real world datasets contain multiple rows and columns and that’s why Dataframes are used much more than series. In this post, we’ll talk about dataframe and some operations that we can do on dataframe objects.</p><h3>What is a DataFrame?</h3><p>As we saw in the previous post, A Series is a container of scalars,A DataFrame is a container for Series. It’s a dictionary like data structure for Series. A DataFrame is similar to a two-dimensional hetrogeneous tabular data(SQL table). A DataFrame is created using many different types of data such as dictionary of Series, dictionary of ndarrays/lists, a list of dictionary, etc. We’ll look at some of these methods to create a DataFrame object and then we’ll see some operations that we can apply on a DataFrame object to manipulate the data.</p><h3>DataFrame using dictionary of Series</h3><pre>In[1]:<br>    d = {<br>    &#39;col1&#39; : pd.Series([1,2,3], index = [&quot;row1&quot;, &quot;row2&quot;, &quot;row3&quot;]),<br>    &#39;col2&#39; : pd.Series([4,5,6], index = [&quot;row1&quot;, &quot;row2&quot;, &quot;row3&quot;])<br>    }</pre><pre>    df = pd.DataFrame(d)</pre><pre>Out[1]:<br>        col1    col2<br>row1     1       4<br>row2     2       5<br>row3     3       6</pre><p>As shown in above code, the keys of dict of Series becomes column names of the DataFrame and the index of the Series becomes the row name and all the data gets mapped by the row name i.e.,order of the index in the Series doesn’t matter.</p><h3>DataFrame using ndarrays/lists</h3><pre>In[2]:<br>    d = {<br>        &#39;one&#39; : [1.,2.,3.],<br>        &#39;two&#39; : [4.,5.,6.]<br>    }</pre><pre>    df = pd.DataFrame(d)</pre><pre>Out[2]:<br>    one  two<br>0   1.0  4.0<br>1   2.0  5.0<br>2   3.0  6.0</pre><p>As shown in the above code, when we use ndarrays/lists, if we don’t pass the index then the range(n) becomes the index of the DataFrame.And while using the ndarray to create a DataFrame, the length of these arrays must be same and if we pass an explicit index then the length of this index must also be of same length as the length of the arrays.</p><h3>DataFrame using list of dictionaries</h3><pre>In[3]:<br>    d = [<br>        {&#39;one&#39;: 1, &#39;two&#39; : 2, &#39;three&#39;: 3},<br>        {&#39;one&#39;: 10, &#39;two&#39;: 20, &#39;three&#39;: 30, &#39;four&#39;: 40}<br>    ]</pre><pre>    df = pd.DataFrame(d)</pre><pre>Out[3]:<br>    one two three four<br>0    1   2   3     NaN<br>1    10  20  30    40.0<br></pre><pre>In[4]:<br>    df = pd.DataFrame(d, index=[&quot;first&quot;, &quot;second&quot;])</pre><pre>Out[4]:</pre><pre>        one two three four<br>first    1   2   3     NaN<br>second   10  20  30    40.0</pre><p>And finally, as described above we can create a DataFrame object using a list of dictionary and we can provide an explicit index in this method,too.</p><p>Although learning to create a DataFrame object using these methods is necessary but in real world, we won’t be using these methods to create a DataFrame but we’ll be using external data files to load data and manipulate that data. So, let’s take a look how to load a csv file and create a DataFrame.In the previous post, we worked with the Nifty50 data to demonstrate how Series works and similarly in this post, we’ll load Nifty50 2018 data, but in this dataset we have data of Open, Close, High and Low value of Nifty50. First let’s see what this dataset looks like and then we’ll load it into a DataFrame.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/756/0*Qfnj-jD4Fd401Z3B.png" /><figcaption>Nifty 50 Data (2018)</figcaption></figure><pre>In[5]:<br>    df = pd.read_csv(&#39;NIFTY50_2018.csv&#39;)</pre><pre>Out[5]:<br>        Date      Open        High        Low        Close<br>0   31 Dec 2018  10913.20   10923.55    10853.20    10862.55<br>1   28 Dec 2018  10820.95   10893.60    10817.15    10859.90<br>2   27 Dec 2018  10817.90   10834.20    10764.45    10779.80<br>3   26 Dec 2018  10635.45   10747.50    10534.55    10729.85<br>4   24 Dec 2018  10780.90   10782.30    10649.25    10663.50<br>...     ...         ...        ...         ...         ...<br>241 05 Jan 2018  10534.25   10566.10    10520.10    10558.85<br>242 04 Jan 2018  10469.40   10513.00    10441.45    10504.80<br>243 03 Jan 2018  10482.65   10503.60    10429.55    10443.20<br>244 02 Jan 2018  10477.55   10495.20    10404.65    10442.20<br>245 01 Jan 2018  10531.70   10537.85    10423.10    10435.55</pre><pre>In[6]:<br>    df = pd.read_csv(&#39;NIFTY50_2018.csv&#39;, index_col=0)</pre><pre>Out[6]:<br>                  Open        High        Low        Close<br>    Date<br>31 Dec 2018     10913.20    10923.55    10853.20    10862.55<br>28 Dec 2018     10820.95    10893.60    10817.15    10859.90<br>27 Dec 2018     10817.90    10834.20    10764.45    10779.80<br>26 Dec 2018     10635.45    10747.50    10534.55    10729.85<br>24 Dec 2018     10780.90    10782.30    10649.25    10663.50<br>     ...           ...        ...         ...          ...<br>05 Jan 2018     10534.25    10566.10    10520.10    10558.85<br>04 Jan 2018     10469.40    10513.00    10441.45    10504.80<br>03 Jan 2018     10482.65    10503.60    10429.55    10443.20<br>02 Jan 2018     10477.55    10495.20    10404.65    10442.20<br>01 Jan 2018     10531.70    10537.85    10423.10    10435.55</pre><p>As shown above, we have loaded the dataset and created a DataFrame called df and looking at the data, we can see that we can set the index of our DataFrame to the Date column and in the second cell we did that by providing the index_col parameter in the read_csv method.</p><p>There are many more parameters available in the read_csv method such as usecols using which we can deliberately ask the pandas to only load provided columns, na_values to provide explicit values that pandas should identify as null values and so on and so forth. Read more about all the parameters in pandas <a href="https://pandas.pydata.org/pandas-docs/stable/reference/api/pandas.read_csv.html">documentation</a>.</p><p>Now, let’s look at some of the basic operations that we can perform on the dataframe object in order to learn more about our data.</p><pre>In[7]:<br>    # Shape(Number of rows and columns) of the DataFrame<br>    df.shape</pre><pre>Out[7]:<br>    (246,4)</pre><pre>In[8]:<br>    # List of index<br>    df.index</pre><pre>Out[8]:<br>    Index([&#39;31 Dec 2018&#39;, &#39;28 Dec 2018&#39;, &#39;27 Dec 2018&#39;, &#39;26 Dec 2018&#39;,<br>       &#39;24 Dec 2018&#39;, &#39;21 Dec 2018&#39;, &#39;20 Dec 2018&#39;, &#39;19 Dec 2018&#39;,<br>       &#39;18 Dec 2018&#39;, &#39;17 Dec 2018&#39;,<br>       ...<br>       &#39;12 Jan 2018&#39;, &#39;11 Jan 2018&#39;, &#39;10 Jan 2018&#39;, &#39;09 Jan 2018&#39;,<br>       &#39;08 Jan 2018&#39;, &#39;05 Jan 2018&#39;, &#39;04 Jan 2018&#39;, &#39;03 Jan 2018&#39;,<br>       &#39;02 Jan 2018&#39;, &#39;01 Jan 2018&#39;],<br>      dtype=&#39;object&#39;, name=&#39;Date&#39;, length=246)</pre><pre>In[9]:<br>    # List of columns<br>    df.columns</pre><pre>Out[9]:<br>    Index([&#39;Open&#39;, &#39;High&#39;, &#39;Low&#39;, &#39;Close&#39;], dtype=&#39;object&#39;)</pre><pre>In[10]:<br>    # Check if a DataFrame is empty or not<br>    df.empty</pre><pre>Out[10]:<br>    False</pre><p>It’s very crucial to know data types of all the columns because sometimes due to corrupt data or missing data, pandas may identify numeric data as ‘object’ data-type which isn’t desired as numeric operations on the ‘object’ type of data is costlier in terms of time than on float64 or int64 i.e numeric datatypes.</p><pre>In[11]:<br>    # Datatypes of all the columns<br>    df.dtypes</pre><pre>Out[11]:<br>    Open     float64<br>    High     float64<br>    Low      float64<br>    Close    float64<br>    dtype: object</pre><p>We can use iloc and loc to index and get the particular data from our dataframe.</p><pre>In[12]:<br>    # Indexing using implicit index<br>    df.iloc[0]</pre><pre>Out[12]:<br>    Open     10913.20<br>    High     10923.55<br>    Low      10853.20<br>    Close    10862.55<br>    Name: 31 Dec 2018, dtype: float64</pre><pre>In[13]:<br>    # Indexing using explicit index<br>    df.loc[&quot;01 Jan 2018&quot;]</pre><pre>Out[13]:<br>    Open     10531.70<br>    High     10537.85<br>    Low      10423.10<br>    Close    10435.55<br>    Name: 01 Jan 2018, dtype: float64</pre><p>We can also use both row and column to index and get specific cell from our dataframe.</p><pre>In[14]:<br>    # Indexing using both the axes(rows and columns)<br>    df.loc[&quot;01 Jan 2018&quot;, &quot;High&quot;]</pre><pre>Out[14]:<br>    10537.85</pre><p>We can also perform all the math operations on a dataframe object same as we did on series.</p><pre>In[15]:<br>    # Basic math operations<br>    df.add(10)</pre><pre>Out[15]:<br>                      Open        High        Low        Close<br>    Date<br>31 Dec 2018     10923.20    10933.55    10863.20    10872.55<br>28 Dec 2018     10830.95    10903.60    10827.15    10869.90<br>27 Dec 2018     10827.90    10844.20    10774.45    10789.80<br>26 Dec 2018     10645.45    10757.50    10544.55    10739.85<br>24 Dec 2018     10790.90    10792.30    10659.25    10673.50<br>     ...           ...        ...         ...          ...<br>05 Jan 2018     10544.25    10576.10    10530.10    10568.85<br>04 Jan 2018     10479.40    10523.00    10451.45    10514.80<br>03 Jan 2018     10492.65    10513.60    10439.55    10453.20<br>02 Jan 2018     10487.55    10505.20    10414.65    10452.20<br>01 Jan 2018     10541.70    10547.85    10433.10    10445.55</pre><p>We can also aggregate the data using the agg method. For instance, we can get the mean and median values from all the columns in our data using this method as show below.</p><pre>In[16]:<br>    # Aggregate one or more operations<br>    df.agg([&quot;mean&quot;, &quot;median&quot;])</pre><pre>Out[16]:<br>            Open            High            Low            Close<br>mean    10758.260366    10801.753252    10695.351423    10749.392276<br>median  10704.100000    10749.850000    10638.100000    10693.000000</pre><p>However, pandas provide a more convenient method to get a lot more than just minimum and maximum values across all the columns in our data. And that method is describe. As the name suggests, it describes our dataframe by applying mathematical and statistical operations across all the columns.</p><pre>In[17]:<br>    df.describe()</pre><pre>Out[17]:<br>            Open           High            Low             Close<br>count   246.000000      246.000000      246.000000      246.000000<br>mean    10758.260366    10801.753252    10695.351423    10749.392276<br>std     388.216617      379.159873      387.680138      382.632569<br>min     9968.800000     10027.700000    9951.900000     9998.050000<br>25%     10515.125000    10558.650000    10442.687500    10498.912500<br>50%     10704.100000    10749.850000    10638.100000    10693.000000<br>75%     10943.100000    10988.075000    10878.262500    10950.850000<br>max     11751.800000    11760.200000    11710.500000    11738.500000</pre><p>And to get the name, data types and number of non-null values in each columns, pandas provide info method.</p><pre>In[18]:<br>    df.info()</pre><pre>Out[18]:<br>    &lt;class &#39;pandas.core.frame.DataFrame&#39;&gt;<br>    Index: 246 entries, 31 Dec 2018 to 01 Jan 2018<br>    Data columns (total 4 columns):<br>    #   Column  Non-Null Count  Dtype<br>    ---  ------  --------------  -----<br>    0   Open    246 non-null    float64<br>    1   High    246 non-null    float64<br>    2   Low     246 non-null    float64<br>    3   Close   246 non-null    float64<br>    dtypes: float64(4)<br>    memory usage: 19.6+ KB</pre><p>We are working with a small data with less than 300 rows and thus, we can work with all the rows but when we have tens or hundreds of thousand rows in our data, it’s very difficult to work with such huge number of data. In statistics, ‘sampling’ is a technique that solves this problem. Sampling means to choose a small amount of data from the whole dataset such that the sampling dataset contains somewhat similar features in terms of diversity as that of the whole dataset. Now, it’s almost impossible to manually select such peculiar rows but as always, pandas comes to our rescue with the sample method.</p><pre>In[19]:<br>    # Data Sampling -  Get random n examples from the data.<br>    df.sample(5)</pre><pre>Out[19]:<br>              Open        High        Low         Close<br>Date<br>04 Jul 2018 10715.00    10777.15    10677.75    10769.90<br>22 Jun 2018 10742.70    10837.00    10710.45    10821.85<br>14 Mar 2018 10393.05    10420.35    10336.30    10410.90<br>09 Jan 2018 10645.10    10659.15    10603.60    10637.00<br>27 Apr 2018 10651.65    10719.80    10647.55    10692.30</pre><p>But, executing this method produces different results everytime and that may be unacceptable in some cases. But that can be solved by providing random_state parameter in the sample method to reproduce same result everytime.</p><p>As shown above, we can perform many operations on the DataFrame object to get information of the DataFrame and from the DataFrame. These are just basic operations that we can perform on the DataFrame object, there are many more interesting methods and operations that we can perform on the DataFrame object such as pivot , merge , join and many more. Also, in this given dataset, we have time as the index of our DataFrame i.e this is the TimeSeries dataset and pandas also provide many methods to manipulate the TimeSeries data such as rolling_window.</p><p>That will be all for this post. In the next post we’ll look at some of these methods and we’ll perform 5 analysis tasks using these methods. Till then, you can take a look at the <a href="https://pandas.pydata.org/pandas-docs/stable/index.html">pandas documentation</a> and find more information about DataFrame objects and the methods that can be applied on the DataFrame object.</p><p><em>Originally Published At : </em><a href="https://www.bytetales.co/pandas-data-frames-learning-data-analysis-with-python/">https://www.bytetales.co/pandas-data-frames-learning-data-analysis-with-python/</a></p><p><strong>Thank you for reading!</strong></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=2f2d40d6c11f" width="1" height="1" alt=""><hr><p><a href="https://medium.com/byte-tales/learning-data-analysis-with-python-pandas-dataframe-2f2d40d6c11f">Learning Data Analysis With Python — Pandas DataFrame</a> was originally published in <a href="https://medium.com/byte-tales">Byte Tales</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Learning Data Analysis With Python— Pandas Series, Missing Values and Real World Data]]></title>
            <link>https://medium.com/byte-tales/learning-data-analysis-with-python-pandas-series-missing-values-and-real-world-data-41caf0795f68?source=rss----1c1d036af958---4</link>
            <guid isPermaLink="false">https://medium.com/p/41caf0795f68</guid>
            <category><![CDATA[data-analysis]]></category>
            <category><![CDATA[python]]></category>
            <category><![CDATA[pandas-series]]></category>
            <category><![CDATA[data-analytics]]></category>
            <category><![CDATA[data-science]]></category>
            <dc:creator><![CDATA[James Shah]]></dc:creator>
            <pubDate>Thu, 16 Jul 2020 10:55:33 GMT</pubDate>
            <atom:updated>2020-07-16T10:55:33.678Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/600/1*THLPLG5SwImyPkwVut7Wmg.png" /><figcaption>Learning Data Analysis With Python</figcaption></figure><blockquote><em>⚠️ </em>This post is a part of Learning data analysis with python series. If you haven’t read the first post, some of the content won’t make sense. Check it out.</blockquote><p><a href="https://medium.com/byte-tales/learning-data-analysis-with-python-introduction-to-pandas-2cea73dadca9">Learning Data Analysis with Python — Introduction to Pandas</a></p><p>In the previous article, we talked about <a href="https://pandas.pydata.org/">pandas</a> library in python, Series data structure, different ways to create a Series object using a list,a dictionary and a numpy array. and how we can apply some functions on it to get some statistical data from it. In this post, we’ll talk about how you can create a Series object from an external file(csv), what is loc and iloc and some tasks that we can achieve using Series. Also, we’ll see an introduction to the DataFrame object as in real world data, almost in every case we come across tabular data(data with multiple columns and rows) rather than a single column data and thus, we’ll be using DataFrame more frequently than Series.</p><h3>What is loc and iloc?</h3><p><strong>loc</strong> and <strong>iloc</strong> are probably the most asked topic in the data science interviews and many newcomers and data science aspirants get confused initially about loc and iloc. However, it’s not rocket science and if you understand loc and iloc in Series objects, it will be easier for you to understand how to use them for indexing the DataFrame object.</p><p>In the last post, we saw that we can give index of our choice for the convenience of accessing data using that index but sometimes we may want to use the default integer indexing that pandas provide (0,1,2,…) to access some data. For example, let’s say, we have data consisting 100 rows and you want to get the first 50 rows, we can do this using slicing but now that we have custom indexing, we must know the first and 50th index to get the data. This is just an oversimplified case of such scenarios, as in real world data analysis tasks you may be working on a dataset with thousands or millions of rows. In that case, it is not feasible to find custom index and then access data. And that’s when loc and iloc come-in handy. loc and iloc are builtin pandas method for indexing data from a pandas Series or DataFrame. In plain english loc is label based indexing which means that use name(custom index) of row and column whereas iloc is integer index-based which means we use integer index(default pandas index) of row and column to access the data. Let&#39;s look at the code and it will make things more clear.</p><pre>In[1]:<br>    s = pd.Series([0.0, 1, 1, 2, 3, 5, 8], index=[1, 2, 3, 4, 5, 6, 7])</pre><pre>Out[1]:<br>    1    0.0<br>    2    1.0<br>    3    1.0<br>    4    2.0<br>    5    3.0<br>    6    5.0<br>    7    8.0<br>    dtype: float64</pre><pre>In[2]:<br>    s.loc[3] </pre><pre>Out[2]:<br>    1.0</pre><pre>In[3]:<br>    s.iloc[3] </pre><pre>Out[3]:<br>    2.0</pre><p>As you can see, in our Series object s, we have given custom index starting from 1 to 7. And then, when we use the loc method on this Series and pass 3 as index, we get the data corresponding to the 3 in our index column. And when we use the iloc method on this Series and again pass 3 as index, we get the data corresponding to the 3 in the default pandas index and as the dafault pandas indexing starts from 0, we get the data from 4th row. We can use the reset_index method to reset the index and get the default interger indexing in our Series. Looking at both the indexing side by side will make things more clear.</p><pre>In[4]:<br>    s.reset_index()</pre><pre>Out[4]:</pre><pre>        index   0<br>    0     1    0.0<br>    1     2    1.0<br>    2     3    1.0<br>    3     4    2.0<br>    4     5    3.0<br>    5     6    5.0<br>    6     7    8.0</pre><p>As you can see, the first unnamed column represents the default pandas indexing, second column named as ‘index’ represents the custom index that we provided and the third column named as ‘0’ represents corresponding values. I hope now you get the idea of what is loc and iloc and what&#39;s the difference between them. We&#39;ll talk more about it in the DataFrame objects article.</p><h3>Missing values or NaNs</h3><p>Missing values or NaN(Not A Number) values are very common in real world datasets and before using such datasets we have to handle these missing values and NaNs in our dataset in the data cleaning phase. There are many different ways in which we can handle missing values such as dropping rows that contain missing value or replacing missing values with some other values such as mean of that column or 0 or any other value. It’s a subjective matter that how you handle the missing values but anyways, Pandas provide very easy way to find missing values using isnan(), isna(), notna() in our data and using it&#39;s advance indexing methods, changing those missing values become very easy. Now let&#39;s look at the very basic example of handling missing values by changing it to the mean of that column.</p><pre>In[5]:<br>    &#39;&#39;&#39;Series of Mass of Planets, where index is the name of the planet and the corresponding value is the mass of that planet.&#39;&#39;&#39;</pre><pre>    mass = pd.Series([0.33, 4.87, 5.97, 0.642, 1898, 568, 86.8, 102, 0.0146],index=[&#39;Mercury&#39;, &#39;Venus&#39;, &#39;Earth&#39;, &#39;Mars&#39;, &#39;Jupiter&#39;, &#39;Saturn&#39;, &#39;Uranus&#39;, &#39;Neptune&#39;, &#39;Pluto&#39;])<br>    print(mass)</pre><pre>Out[5]:<br>    Mercury       0.3300<br>    Venus         4.8700<br>    Earth         5.9700<br>    Mars          0.6420<br>    Jupiter    1898.0000<br>    Saturn      568.0000<br>    Uranus       86.8000<br>    Neptune     102.0000<br>    Pluto         0.0146<br>    dtype: float64</pre><pre>In[6]:<br>    &#39;&#39;&#39;Series of Diameter of Planets, where index is the name of the planet and the corresponding value is the diameter of that planet.&#39;&#39;&#39;</pre><pre>    diameter = pd.Series([4879, 12104, 12756, 3475, 6792, 142984, 120536, 51118, 49528, 2370],index=[&#39;Mercury&#39;, &#39;Venus&#39;, &#39;Earth&#39;, &#39;Moon&#39;, &#39;Mars&#39;, &#39;Jupiter&#39;, &#39;Saturn&#39;,  &#39;Uranus&#39;, &#39;Neptune&#39;, &#39;Pluto&#39;])</pre><pre>    print(diameter)</pre><pre>Out[6]:<br>    Mercury      4879<br>    Venus       12104<br>    Earth       12756<br>    Moon         3475<br>    Mars         6792<br>    Jupiter    142984<br>    Saturn     120536<br>    Uranus      51118<br>    Neptune     49528<br>    Pluto        2370<br>    dtype: int64</pre><pre>In[7]:<br>    # Finding density using the mass and diameter series.<br>    density = mass / (np.pi * np.power(diameter, 3) / 6)<br>    print(density)</pre><pre>Out[7]:<br>    Earth      5.493286e-12<br>    Jupiter    1.240039e-12<br>    Mars       3.913302e-12<br>    Mercury    5.426538e-12<br>    Moon                NaN<br>    Neptune    1.603427e-12<br>    Pluto      2.094639e-12<br>    Saturn     6.194402e-13<br>    Uranus     1.241079e-12<br>    Venus      5.244977e-12<br>    dtype: float64</pre><pre>In[8]:<br>    pd.isna(density)</pre><pre>Out[8]:<br>    Earth      False<br>    Jupiter    False<br>    Mars       False<br>    Mercury    False<br>    Moon        True<br>    Neptune    False<br>    Pluto      False<br>    Saturn     False<br>    Uranus     False<br>    Venus      False<br>    dtype: bool</pre><pre>In[9]:<br>    # Get the index of data in density where the value is NaN and change it to the mean of density.<br>    density[pd.isna(density)] = np.mean(density)<br>    print(density)</pre><pre>Out[9]:<br>    Earth      5.493286e-12<br>    Jupiter    1.240039e-12<br>    Mars       3.913302e-12<br>    Mercury    5.426538e-12<br>    Moon       2.986303e-12<br>    Neptune    1.603427e-12<br>    Pluto      2.094639e-12<br>    Saturn     6.194402e-13<br>    Uranus     1.241079e-12<br>    Venus      5.244977e-12<br>    dtype: float64</pre><p>If that was a lot to take in, let me walk you through it. First I’ve created two Series objects named mass and diameter containing mass and diameter of the planets respectively. The only change in these two is that diameter Series contains ‘Moon’ in it but mass doesn’t. And thus, when we find density using these two Series we get the NaN value for Moon as mass doesn’t contain it. I infer you thinking <em>“How did this happen?”</em>. Let me explain, when we merge two Series or create a new Series by applying some functions on two different Series, what pandas does is that it matches both the Series by their indices. Also, the order of the indices doesn’t matter. Thus, when we created a new Series ‘density’ using ‘mass’ and ‘diameter’, pandas did the calculations and merged these two Series by their indices but in the case ‘Moon’ it couldn’t find the ‘mass’ and thus, it added NaN.</p><p>After creating this new ‘density’ Series, we see that we have NaN value in our data and to change it we have to find it’s index. Now, in our example for the sake of simplicity, we had a single NaN value and also our Series had only 10 rows so we could have changed the NaN value manually using the index of ‘Moon’, but with large data and many more NaN values we can’t go through all the rows and find the indices of each NaN values, therefore pandas has a builtin function to find all the values in which has NaN values. And using the indexing techniques that pandas offer we changed the value of Moon’s diameter in one line, otherwise we would have to go through all the data in the Series using a for loop and change the NaN values with the mean value.</p><p>As you can see, pandas package provides builtin methods for all the major tasks that we have to do while working with data and it uses numpy library that makes these operations fast as numpy use vectorization methods for all the computer heavy tasks and that’s the reason why <a href="https://numpy.org/">numpy</a> and <a href="https://pandas.pydata.org/">pandas</a> are the most popular python packages for data science.</p><h3>Reading Data From External File &amp; Working with Real World Data</h3><p>Till now, we have created and worked with the data that we created using list and dictionaries but we don’t create data by ourself in real world tasks, we use the data from external files such as csv(comma seperated values) files, text files, excels files, etc. And pandas provides many builtin methods to load the data from external files and make Series and DataFrame objects from that data very easily. Let’s look at the example of loading ‘csv’ file and creating a Series object from it.</p><p>We have a real world data of closing value of Nifty index from the 1st January 2019 to 31st December 2019. <a href="https://en.wikipedia.org/wiki/NIFTY_50">Nifty</a> is a benchmark Indian stock market index that represents the weighted average of 50 of the largest Indian companies listed on NSE. Let’s see what this data looks like.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/387/0*tgyH_PyoDOS79YKT.png" /><figcaption>Nifty50 2019 Dataset</figcaption></figure><p>Now let’s see how to load this data in our code and create a Series object. Pandas has a method named read_csv() which takes file path and some other parameters such asseperator,skip_rows,header, index_col and many <a href="https://pandas.pydata.org/pandas-docs/stable/reference/api/pandas.read_csv.html">more</a> and creates a DataFrame object from it. But we want to create a Series object, thus we&#39;ll use iloc and get only the first column from the DataFrame that padas creates.</p><pre>In[10]:<br>    # Using first column as index column create a Series from &#39;nifty_data.csv&#39;<br>    nifty = pd.read_csv(&quot;nifty_data.csv&quot;, index_col = 0).iloc[:,0]<br>    print(nifty.head(5)) # Print first 5 rows</pre><pre>Out[10]:<br>    Date<br>    01-Jan-2019    10910.10<br>    02-Jan-2019    10792.50<br>    03-Jan-2019    10672.25<br>    04-Jan-2019    10727.35<br>    07-Jan-2019    10771.80</pre><p>Now, let’s look at some basic operations on this data such as mean, median, max and other statistical data. But instead of doing all these operations one by on the Series, we can use a builtin pandas method describe() to get all these data together using a single function.</p><pre>In[11]:<br>    nifty.describe()</pre><pre>Out[11]:<br>    count      245.000000<br>    mean     11432.632245<br>    std        454.214611<br>    min      10604.350000<br>    25%      11003.050000<br>    50%      11512.400000<br>    75%      11841.550000<br>    max      12271.800000<br>    Name: Close, dtype: float64</pre><p>And just like that, we can easily get all the statistics information from the data with a single line. Here 25%,50% and 75% represents percentile of the data meaning that 25% of values in our data is below 11003.050000 and so on. You may ask that here we dont’ have median of the data but that’s because the median is equal to 50th percentile ,thus 50% also represents median of the data. From a little statistics knowledge I can infer that there are not many significant outliers as there isn’t much difference between mean and median.</p><p>Now remember I told you, when we apply some operations on two Series and get a new Series, pandas match them using their indices. Let’s see that with another example and why keeping that mind is important. Let’s say you want to find the total days on which the closing value was higher than the previous day’s closing value. Well that’s pretty simple, we can just take two slices from this Series, first being the data without the first column and second being the data without the last column and we can just subtract and see if the result is greater than zero or not.Right? Let’s see what happens when we do that.</p><pre>In[12]:<br>    nifty[1:]</pre><pre>Out[12]:<br>    Date<br>    02-Jan-2019    10792.50<br>    03-Jan-2019    10672.25<br>    04-Jan-2019    10727.35<br>    07-Jan-2019    10771.80<br>    08-Jan-2019    10802.15<br>        ...          ...<br>    24-Dec-2019    12214.55<br>    26-Dec-2019    12126.55<br>    27-Dec-2019    12245.80<br>    30-Dec-2019    12255.85<br>    31-Dec-2019    12168.45<br>    Name: Close, Length: 244, dtype: float64</pre><pre>In[13]:<br>    nifty[:-1]</pre><pre>Out[13]:<br>    Date<br>    01-Jan-2019    10910.10<br>    02-Jan-2019    10792.50<br>    03-Jan-2019    10672.25<br>    04-Jan-2019    10727.35<br>    07-Jan-2019    10771.80<br>        ...          ...<br>    23-Dec-2019    12262.75<br>    24-Dec-2019    12214.55<br>    26-Dec-2019    12126.55<br>    27-Dec-2019    12245.80<br>    30-Dec-2019    12255.85<br>    Name: Close, Length: 244, dtype: float64</pre><pre>In[14]:<br>    nifty[1:] - nifty[:-1]</pre><pre>Out[14]:<br>    Date<br>    01-Apr-2019    0.0<br>    01-Aug-2019    0.0<br>    01-Feb-2019    0.0<br>    01-Jan-2019    NaN<br>    01-Jul-2019    0.0<br>                ...<br>    31-Dec-2019    NaN<br>    31-Jan-2019    0.0<br>    31-Jul-2019    0.0<br>    31-May-2019    0.0<br>    31-Oct-2019    0.0<br>    Name: Close, Length: 245, dtype: float64</pre><p>Ooops!! That’s not what we expected it to be right? Yes and that’s because pandas matched indices of both the slices and subtracted their corresponding values and for the values that it couldn’t find matching indices (01-Jan-2019 and 31-Dec-2019), it addded NaN values. Thus, when we want to subtract two series by element by element and don’t want the index to be matched, we can use values() method on these slices and get the numpy array of values for both the slices and then subtract them.</p><pre>In[15]:<br>    (nifty.values[1:] - nifty.values[:-1]) &gt; 0</pre><pre>Out[15]:<br>    array([False, False,  True,  True,  True,  True, False, False, False,........,False])</pre><pre>In[16]:<br>    np.sum((nifty.values[1:] - nifty.values[:-1]) &gt; 0)</pre><pre>Out[16]:<br>    129</pre><p>Well that’s pretty much it for this post and Series object. There are many more things that you can do with the pandas Series object but including each and everything in my posts is pretty much impossible. Look at the pandas <a href="https://pandas.pydata.org/pandas-docs/stable/index.html">documentation</a> and see what you can do with the Series object. In the next post, We’ll talk about DataFrame objects and all the good things we can do with it. <strong>Thank you for reading.</strong></p><p><em>Originally Published at : </em><a href="http://bytetales.co/learning-data-analysis-with-python-pandas-series-missing-values-and-real-world-data"><em>bytetales.co</em></a></p><p>Get the full code notebook here : <a href="https://github.com/jamesshah/pandas-notebooks/blob/master/pandas-2.ipynb">Github</a></p><p>Get the Nifty 50 data here : <a href="https://github.com/jamesshah/pandas-notebooks/blob/master/nifty_data.csv">Data</a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=41caf0795f68" width="1" height="1" alt=""><hr><p><a href="https://medium.com/byte-tales/learning-data-analysis-with-python-pandas-series-missing-values-and-real-world-data-41caf0795f68">Learning Data Analysis With Python— Pandas Series, Missing Values and Real World Data</a> was originally published in <a href="https://medium.com/byte-tales">Byte Tales</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Learning Data Analysis with Python — Introduction to Pandas]]></title>
            <link>https://medium.com/byte-tales/learning-data-analysis-with-python-introduction-to-pandas-2cea73dadca9?source=rss----1c1d036af958---4</link>
            <guid isPermaLink="false">https://medium.com/p/2cea73dadca9</guid>
            <category><![CDATA[data-analysis]]></category>
            <category><![CDATA[python-data-analysis]]></category>
            <category><![CDATA[pandas]]></category>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[python]]></category>
            <dc:creator><![CDATA[James Shah]]></dc:creator>
            <pubDate>Sun, 05 Jul 2020 14:02:36 GMT</pubDate>
            <atom:updated>2020-07-05T14:02:36.682Z</atom:updated>
            <content:encoded><![CDATA[<h3>Learning Data Analysis with Python — Introduction to Pandas</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1000/1*j5uNQE7O7bjR8pi7KtlaVA.png" /></figure><h3>What is Pandas and Why do we need it</h3><p><a href="https://pandas.pydata.org/">Pandas</a> is python library that provides rich data structures and functions that makes working with relational and structured data easy, fast and convenient. It is developed on top of the <a href="https://numpy.org/">Numpy</a> package for the high performance computing and it gives flexible data manipulation techniques of relational databases. Okay, now why do we need panadas at all? If we want to mainpulate high dimensional data, we have numpy, so why do we need it?<br>Well, indeed numpy is there for the high dimensional data manipulation but what it doesn’t provide is a way to index or label our data like this.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/626/1*GVQDwZIw1e6INt-HXZgVGg.png" /><figcaption>Pandas DataFrame</figcaption></figure><p>And, although we can do so many interesting things to our data using numpy functions such as finding statistics like mean,median,mode and standard deviation but we can’t fill missing values in our data. Numpy doesn’t provide any built-in method to handle missing values in our data and in real world problems, we don’t always come around the perfect datasets.</p><p>There are other limitations to numpy such as no way to group data and no way to pivot data, and thus, Pandas was created to help programmers and data scientists with the data analysis task in Python.</p><h3>Data Structures in Pandas</h3><p>Well as we’ve mentioned earlier, pandas provides rich data structures and functions, let’s start by looking at the data structures and then we’ll look at the functions and methods that we can apply on these data structures to crunch data.</p><p>Pandas provide two major data structures namely “Series” and “DataFrames”. In plain english, Series is a one-dimensional labeled array, whereas DataFrames is two-dimensional labeled array with columns of potentially different types. Although, Series are useful in many tasks, DataFrames are the most commonly used pandas object. You can think of DataFrame as an SQL table.</p><p>Let’s look at the example for both of these data structures.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1000/1*RZ1nbLkRCn-8Hu7DK3I4jA.png" /><figcaption>Pandas Data Structure — Series &amp; DataFrame</figcaption></figure><p>Let’s start by looking at the different ways of creating a Series object.But, Before we get into coding, we need to install pandas as python package. This can be done using pip install pandas or you can install it using conda install pandas if you are using Anaconda Distribution for Python. If you don&#39;t have python installed on your machine, the best way to get started without all the hustle of installing it is Google Colab. Learn more about it <a href="https://colab.research.google.com/notebooks/intro.ipynb">here</a>. I&#39;m going to use jupyter notebooks which you can find <a href="https://github.com/jamesshah/pandas-notebooks/blob/master/pandas-1.ipynb">here</a>.</p><h3>Series</h3><p>As mentioned above, Series is a one-dimensional array-like object containing an array of data and an array of labels associated with the data. The data labels are called ‘index’.<br>We can create a Series from a list or a dictionary or a numpy array.<br>Let’s start with a list.</p><h3>Series using a list</h3><pre><strong>In[1]:</strong><br>    import pandas as pd<br>    import numpy as np</pre><pre>    friends = [&quot;Monica&quot;, &quot;Chandler&quot;, &quot;Joey&quot;, &quot;Pheobe&quot;, &quot;Ross&quot;, &quot;Rachel&quot;]<br>    series = pd.Series(friends)<br>    series</pre><pre><strong>Out[1]:</strong><br>    0      Monica<br>    1    Chandler<br>    2        Joey<br>    3      Pheobe<br>    4        Ross<br>    5      Rachel<br>    dtype: object</pre><p>As we can see, we get the Series object with the data that we provided with the associated index of for each row. We can access our data from this series using the index just like we do with the list.</p><pre><strong>In[2]:</strong><br>    series[0]</pre><pre><strong>Out[2]:</strong><br>    &#39;Monica&#39;</pre><p>Or we can get all the values in a list using .value attribute on series object.</p><pre><strong>In[3]:</strong><br>    series.values</pre><pre><strong>Out[3]:</strong><br>    array([&#39;Monica&#39;, &#39;Chandler&#39;, &#39;Joey&#39;, &#39;Pheobe&#39;, &#39;Ross&#39;, &#39;Rachel&#39;],<br>      dtype=object)</pre><p>Similarly, we can get all the indices in a list using .index attribute.</p><pre><strong>In[4]:</strong><br>    series.index</pre><pre><strong>Out[4]:</strong><br>    RangeIndex(start=0, stop=6, step=1)</pre><p>Note something different? Right, it doesn’t return a list of indices as it did in values, that’s because the implicit index that pandas automatically creates for us is just a list of consecutive numbers starting from 0. A more intiuative approach would be to provide index identifying each data point while creating a Series.</p><pre><strong>In[5]:</strong><br>    nick_names = [&quot;Monana&quot;, &quot;Officer Bing&quot;, &quot;Ken Adams&quot;, &quot;Regina Filange&quot;, &quot;Dinosaur Guy&quot;, &quot;Fun Aunt Rachel&quot;]<br>    series = pd.Series(data = friends, index = nick_names)<br>    series</pre><pre><strong>Out[6]:</strong><br>    Monana               Monica<br>    Officer Bing       Chandler<br>    Ken Adams              Joey<br>    Regina Filange       Pheobe<br>    Dinosaur Guy           Ross<br>    Fun Aunt Rachel      Rachel<br>    dtype: object</pre><p>And now, we can access our data using the index that we provided like this.</p><pre><strong>In[7]:</strong><br>    series[&quot;Dinosaur Guy&quot;]</pre><pre><strong>Out[7]:</strong><br>    &#39;Ross&#39;</pre><p><strong><em>It’s clearly a bad example of indexing but you got the idea, right?</em></strong></p><h3>Series using a dictionary</h3><p>Now, let’s instantiate Series from a dictionary. When creating a Series from dictionaries, we don’t pass index, the Series gets ordered by dictionary’s insertion order and the keys of dictionary becomes indices of Series.</p><pre><strong>In[8]:</strong><br>    d = {&quot;a&quot; : 1, &quot;b&quot; : 2, &quot;c&quot; : 3}<br>    series = pd.Series(d)<br>    series</pre><pre><strong>Out[8]:</strong><br>    a    1<br>    b    2<br>    c    3<br>    dtype: int64</pre><p>Series object also works like dictionaries. You can get or set the values in the Series by index label just like dictionaries.</p><pre><strong>In[9]:</strong><br>    series[&quot;c&quot;]</pre><pre><strong>Out[9]:</strong><br>    3</pre><pre><strong>In[10]:</strong><br>    series[&quot;d&quot;] = 4<br>    series</pre><pre><strong>Out[10]:</strong><br>    a    1<br>    b    2<br>    c    3<br>    d    4<br>    dtype: int64</pre><h3>Series using a numpy array</h3><p>And finally, we can create a Series from a numpy array.</p><pre><strong>In[11]:</strong><br>    d = np.random.randn(5) # Create a numpy array of 5 random numbers<br>    series = pd.Series(d)<br>    series</pre><pre><strong>Out[11]:</strong><br>    0   -0.507243<br>    1   -0.115731<br>    2    0.526216<br>    3   -0.495840<br>    4    0.065357<br>    dtype: float64</pre><p>Series is also similar to numpy array in functionalities. We can index data from a Series object as we do from a numpy array.</p><pre><strong>In[12]:</strong><br>    series[:3] # Returns first 3 elements</pre><pre><strong>Out[12]:</strong><br>    0   -0.507243<br>    1   -0.115731<br>    2    0.526216<br>    dtype: float64</pre><pre><strong>In[13]:</strong><br>    series[series &gt; 0] # Returns elements whose value is greater than 0</pre><pre><strong>Out[13]:</strong><br>    2    0.526216<br>    4    0.065357<br>    dtype: float64</pre><p>Now, once we created a Series, we can apply many different methods on it to achieve different tasks. A few of them are as below. Read more about all the methods available on Series object at <a href="https://pandas.pydata.org/pandas-docs/stable/reference/series.html">pandas official documentation</a>.</p><pre><strong>In[14]:</strong><br>    series.size # Returns size of the series.<br><strong>Out[14]:</strong><br>    5</pre><pre><strong>In[15]:</strong><br>    series.shape # Returns shape of the series.<br><strong>Out[15]:</strong><br>    (5,)</pre><pre><strong>In[16]:</strong><br>    series.add(1) # Returns series with elements&#39; value increased by 1<br><strong>Out[16]:</strong><br>    0    0.492757<br>    1    0.884269<br>    2    1.526216<br>    3    0.504160<br>    4    1.065357<br>    dtype: float64</pre><pre><strong>In[17]:</strong><br>    series.sub(1) # Returns series with elements&#39; value decreased by 1<br><strong>Out[17]:</strong><br>    0   -1.507243<br>    1   -1.115731<br>    2   -0.473784<br>    3   -1.495840<br>    4   -0.934643<br>    dtype: float64</pre><pre><strong>In[18]:</strong><br>    series.mul(10) # Returns series with elements&#39; value multiplied by 10<br><strong>Out[18]:</strong><br>    0   -5.072431<br>    1   -1.157307<br>    2    5.262160<br>    3   -4.958401<br>    4    0.653565<br>    dtype: float64</pre><pre><strong>In[19]:<br></strong>    series.div(10) # Returns series with elements&#39; value divided by 10<br><strong>Out[19]:<br></strong>    0   -0.050724<br>    1   -0.011573<br>    2    0.052622<br>    3   -0.049584<br>    4    0.006536<br>    dtype: float64</pre><pre><strong>In[20]:<br></strong>    series.abs() # Returns series with absolute value of elements<br><strong>Out[20]:<br></strong>    0    0.507243<br>    1    0.115731<br>    2    0.526216<br>    3    0.495840<br>    4    0.065357<br>    dtype: float64</pre><pre><strong>In[21]:<br></strong>    series.max() # Returns maximum of all the elements.<br><strong>Out[21]:</strong><br>    0.5262159987471329</pre><pre><strong>In[22]:</strong><br>    series.min() # Returns minimum of all the elements.<br><strong>Out[22]:</strong><br>    -0.5072430551642492</pre><pre><strong>In[23]:</strong><br>    series.sum() # Returns sum of all the elements.<br><strong>Out[23]:</strong><br>    -0.5272413324692681</pre><pre><strong>In[24]:</strong><br>    series.mean() # Returns mean of the series.<br><strong>Out[24]:</strong><br>    -0.10544826649385361</pre><pre><strong>In[25]:</strong><br>    series.median() # Returns median of the series.<br><strong>Out[25]:</strong><br>    -0.1157306666767307</pre><pre><strong>In[26]:</strong><br>    series.std() # Returns standard deviation of the series.<br><strong>Out[26]:</strong><br>    0.43073327047221427</pre><pre><strong>In[27]:</strong><br>    series.drop(4) # Returns series with specified index removed.<br><strong>Out[27]:</strong><br>    0   -0.507243<br>    1   -0.115731<br>    2    0.526216<br>    3   -0.495840<br>    dtype: float64</pre><pre><strong>In[28]:</strong><br>    series<br><strong>Out[28]:</strong><br>    0   -0.507243<br>    1   -0.115731<br>    2    0.526216<br>    3   -0.495840<br>    4    0.065357<br>    dtype: float64</pre><p>Look at the last code when we print the original series object, we get the elements same as we created, because all the operations that we applied on the series object gives us the view of the series and doesn’t affect the original object.<br>Okay, so this is just a beginning, there are many more interesting things that we can do with pandas such as dropping null values or replacing them with some other values from data, visualize data using so many plots and charts, combine and merge two or more Series, create and manipulate Time Series, etc.</p><p>That’s it for this post.In the next couple of posts, I’ll be playing with some real world datasets and make some analysis on those data using Pandas.</p><p>Get full code notebook here : <a href="https://github.com/jamesshah/pandas-notebooks/blob/master/pandas-1.ipynb">Pandas-1 Notebook</a></p><h4>Thank you for reading. Happy coding.👨‍💻👩‍💻</h4><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=2cea73dadca9" width="1" height="1" alt=""><hr><p><a href="https://medium.com/byte-tales/learning-data-analysis-with-python-introduction-to-pandas-2cea73dadca9">Learning Data Analysis with Python — Introduction to Pandas</a> was originally published in <a href="https://medium.com/byte-tales">Byte Tales</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Six Elements Of Machine Learning — A Beginner’s Guide]]></title>
            <link>https://medium.com/byte-tales/six-elements-of-machine-learning-a-beginners-guide-a00fd8b532be?source=rss----1c1d036af958---4</link>
            <guid isPermaLink="false">https://medium.com/p/a00fd8b532be</guid>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[beginner]]></category>
            <category><![CDATA[deep-learning]]></category>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[learning]]></category>
            <dc:creator><![CDATA[James Shah]]></dc:creator>
            <pubDate>Wed, 03 Jun 2020 11:31:29 GMT</pubDate>
            <atom:updated>2020-06-03T11:31:29.107Z</atom:updated>
            <content:encoded><![CDATA[<h3>Six Elements Of Machine Learning — A Beginner’s Guide</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1000/1*EI8BPkHtAP9ZQFySq36ekQ.png" /></figure><p><em>Note : Content of this article including some images are taken from the Deep Learning Course by One Fourth Labs.</em></p><p>Before introducing the 6 elements of Machine Learning, let’s understand Why do we need Machine Learning? Why not the old school expert systems where we used to come up with a solution and then code it in one of our favourite programming languages for the computer to understand it and then feeding it some input to get the desired result? A simple example of such expert systems is “Whether a patient has dengue or not?” Now, how do doctors decide that? They ask the patient “Do they have high fever and cold? Do thay have headache or rashes?” i.e. Whether the patient has symptoms of Dengues or not? And from the past experience doctor decides “If the patient has Dengue or not?”</p><p>Now, How do we solve this problem using a computer? We can simply write some if-else conditions to get the output as shown below.</p><pre>if HIGH_FEVER and COLD:<br>    return no_dengue<br>elif HIGH_FEVER and VOMITING and not RASH:<br>    return no_dengue<br>elif HIGH_FEVER and VOMITING and RASH:<br>    return dengue</pre><p>And now we can simply run this program and feed it with the symptoms of a new patient and can decide “Whether he/she has dengue or not?” Cool, right? So <em>Why do we need Machine Learning?</em></p><p>The answer is the limitations of the expert systems that we talked about. Predicting dengue or not was somewhat easy task as we know about it’s symptoms but let’s say we want to predict whether a patient has Ebola or not? In this case, the symptoms(rules to check) are unknown and thus we can’t feed the computer the rules that we don’t know.</p><p>Another limitations is that sometimes we have lots of data. For example, “Should a bank give you a loan or not?” Now, giving loan depends on lots of factors including your salary, your credit score, size of your family and many more. Now in this case, we have so much data to make sense from that writing if-else blocks is not an easy and effective task.</p><p>And sometimes, there are rules that we can’t express to the computer to understand. For example, sometimes your salary may be a little less than the required salary to get loan but you’ve been an honest and a regular customer to the bank and the bank manager knows that though your salary is not enough to meet the minimum requirements but they believe you and your honesty and gives you loan anyway.How do you feed such rule to the computer? How can a computer understand honesty?</p><p>These are the limitations that needs to be conquered and Machine Learning Does exactly that. Using Machine Learning, we give some data as input and some matching output for that data and then find the relationship between the input data and output data. The 6 elements of the the Machine Learning are:</p><ol><li><a href="https://www.bytetales.co/6-elements-of-machine-learning-a-beginner-s-guide-to-machine-learning/#1-data">Data</a></li><li><a href="https://www.bytetales.co/6-elements-of-machine-learning-a-beginner-s-guide-to-machine-learning/#2-task">Task</a></li><li><a href="https://www.bytetales.co/6-elements-of-machine-learning-a-beginner-s-guide-to-machine-learning/#3-model">Model</a></li><li><a href="https://www.bytetales.co/6-elements-of-machine-learning-a-beginner-s-guide-to-machine-learning/#4-loss-function">Loss Function</a></li><li><a href="https://www.bytetales.co/6-elements-of-machine-learning-a-beginner-s-guide-to-machine-learning/#5-learning-algorithm">Learning Algorithm</a></li><li><a href="https://www.bytetales.co/6-elements-of-machine-learning-a-beginner-s-guide-to-machine-learning/#6-evaluation">Evaluation</a></li></ol><figure><img alt="" src="https://cdn-images-1.medium.com/max/800/1*rc3Rp0v5M0gyAXGAFxGBfg.png" /><figcaption>6 Elements Of Machine Learning</figcaption></figure><p>Now, let’s dive deep into these elements one by one and see how we can see Machine Learning through the lens of these elements.</p><h3>1. Data</h3><p>Data simply means information. All types and formats of information. Today,there is enormous amount of data produced every second and this data can be used to answer so many questions. There is text data as well as audio-video data, there is structured data as well as unstructured data.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/515/1*mR_KfuzHlds3c1snaLPTfw.png" /><figcaption>Abundant Data</figcaption></figure><blockquote><em>“Data is the fuel of Machine Learning.”</em></blockquote><p>When starting a Machine Learning project, we need some answers to the questions like, “Do we have the required data? If no, how do we get the data and where from? And if yes, is it in a structured form or unstructured and raw?”</p><p>These questions lead us to different paths in the project. If we don’t have the data needed, then Data Curation comes in picture. There are many ways to get the data.</p><p><em>If you are lucky enough</em>, there are many resources where you can get datasets that are publically available. Few examples of such resources are <a href="https://datasetsearch.research.google.com/">Google Dataset Search</a>, <a href="https://www.kaggle.com/datasets">Kaggle Datasets</a>, <a href="https://data.gov.in/">Indian Government Data</a>.</p><p><em>If you are rich enough</em>, there are many websites on which you can assign some people to curate and collect data for your needs. For example, let’s say you have images of animals and you want these images to be labelled with the name of animal, then you can assign such task to people and you can get the data easily and in a short span of time. Few examples of such crowdsourcing marketplaces are <a href="https://www.mturk.com/">Amazon mturk</a>, <a href="https://dataturks.com/">Dataturks</a>.</p><p><em>If you are smart enough</em>, you can conduct surveys and get data from people or using the data scrapping techniques you can collect data from different websites.</p><p>One important thing to remember is that it doesn’t matter in which format you get the data, at the end all the data needs to be encoded as numbers before feeding it to the computers. For example, images can be converted into matrix of pixel values.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/484/1*vrElsxQd21z_i7rVfNkjeg.jpeg" /><figcaption>Data encoded as numbers</figcaption></figure><h3>2. Task</h3><p>Now, you have data. The next question you should ask is “What task do I need to achieve? Or What task can I achieve using the data that I have?”</p><p>Generally, Machine Learning tasks are divided into two parts: <strong>Supervised Learning And Unsupervised Learning.</strong> Supervised Learning is when you have some input data as well as some output data corresponding to the input data. Then you can use Machine Learning to get the relationship between the input and output data and then use that relationship to predict output for the new input. And Unsupervised Learning is when you have only input data.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/924/1*LeKxA-aLsDekBfKNQK8VNg.png" /><figcaption>Machine Learning Tasks</figcaption></figure><p>Supervised learning is further divided into two sub sections as <strong>Regression and Classification</strong> where Regression means you want to predict some continous value for example House Price Prediction and Classification means you want a discrete answer such as yes or no. An example of Classification is, given an image you want to detect whether there is a dog or not in that image.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*pDIjVFI4yu1uTP3fE-Du7w.png" /><figcaption>Regression Using Supervised Learning</figcaption></figure><p>Unsupervised learning is often used for Generation and Clustering. For example, the image given below shows that given the tweets of US President Donald Trump as an input, machine generates more such tweets.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Sr8V4yCEf86mza0IJMbd7g.png" /><figcaption>Generation using Unsupervised Learning</figcaption></figure><p>Although Unsupervised learning is useful in some cases, most of the real world problems are solved using Supervised learning as it gives more accurate results than Unsupervised learning.</p><blockquote><em>“Supervised learning has created 99% of economic value in AI.” — Andrew NG, Co-founder of Coursera.</em></blockquote><h3>3. Model</h3><p>A model is nothing but <em>a mathematical function</em> which defines relationship between input data and output data.</p><p>A model can be a simple Linear Function such as y = mx + c or it can be a complex function such as y = sin(cos(ax)) + tan(sin(bx^2 + cx)) .</p><p>The example that we talked about before, given an image(<em>data</em>), predict if it has a dog or not(<em>task</em>), we have to build a system that answers this question and such system is called a <em>model</em>.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/793/1*CSHuAsKcdsuWPLwQP9Rn1A.png" /><figcaption>Models</figcaption></figure><p>There are many models available that are created over the years and each one has it’s own pros and cons and each one is suitable for a unique task. For a human, it is almost impossible to come up with a function that defines relationship between the input and output, given that the input data has over thousand different features.</p><h3>4. Loss Function</h3><p>Although, it’s not possible to come up with a <em>model</em> just by looking at the given data, but let’s assume, you have come up with a model that you think correctly defines the relationship between input and output. But, two of your friends are solving the same problem and they have come up with a slightly different <em>model</em>, then how do you decide, which model is better? Loss function gives the answer to that question.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/881/1*Z19cX76jrxzoeEyeJjMn3Q.png" /><figcaption>Which function is better?</figcaption></figure><p>As shown in the image above, three different values predicted by the models that you and your friends have written are f1(x), f2(x) and f3(x), respectively. And the true value is y. Now, to know which model is best suited for our data, we compute <em>the difference between the true value and predicted value</em> for all the data using a loss function. One such loss function is <strong>Square Error Loss</strong>.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/577/1*uRD50R9VGD8JMQMcCuiOQg.png" /><figcaption>Square Error Loss</figcaption></figure><p>And counting loss function value for all three models, we get to know that the best model for our data is the one with the minimum value of loss function which is 1.38.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/907/1*nXh765JnhjXQqz4hWQAWxA.png" /><figcaption>Minimized Loss</figcaption></figure><p>This way using a loss function, we can get the best model that defines the relationship between our input data and output data. More such examples of Loss Function are <strong>Cross Entropy Loss</strong> and <strong>KL Divergence</strong>.</p><h3>5. Learning Algorithm</h3><p>Now, we assumed a hypothetical situation where you and your friends came up with a model for our data and task but that won’t happen in reality as there will be hundred and thousands of features in the data and you can’t just look at it and propose a model that satisfies the relationship. Instead, you can propose a function such as y = ax^3 + bx^2 + cx and say that I think this should define the relationship between the input x and output y, but you don&#39;t know the paramteres a, b and c. So, to identify these parameters for a model, we need a learning algorithm.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/941/1*sWDba4Vrblu7kUOaKHvytg.png" /><figcaption>Learning Algorithms</figcaption></figure><p>As shown in the picture above, we have a model with three parameters a,b and c and we need values of a,b and c at which the loss function’s value is minimum. In calculus, there are many optimization strategies available that we can use to find the best values of parameters which minimizes the loss. Some examples of such optimization solvers are Gradient Descent, Adagrad, RMSProp, etc.</p><p><em>Learning algorithm is one task that computer does, other than that from data collection to deciding task and then proposing a suitable model is done by humans but then finding parameters of the proposed model is done by computer.</em></p><h3>6. Evaluation</h3><p>Okay,let’s say using the above 5 elements, you’ve created an ML model to detect the animal in the image. And you want to sell it to some company or just show off to your friends. How do you know that your model is detecting the right animal? You guessed it right! you have to test your model. That’s what evaluation is. Testing your model by feeding it some test data and then checking if it predicts the correct output in all the cases? If not, then how accurate your model is?</p><p><em>Note: You evaluate your model using test data and not the training data, but why so? that’s beyond the scope of this article!</em></p><p>There are many performance matrices available for evaluation, one such matrix is accuracy. We find the accuracy of a model diving &quot;Number of correct predictions&quot; by &quot;Total numbers of predictions&quot; , as shown in the image below.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/936/1*5I1eFm3EeUSe-KHytvWMZg.png" /><figcaption>Evaluation</figcaption></figure><h4>Thank you for reading! Hope you liked it. #HappyLearning</h4><p><em>Originally published at: </em><a href="https://www.bytetales.co/6-elements-of-machine-learning-a-beginner-s-guide-to-machine-learning/"><em>bytetales.co</em></a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=a00fd8b532be" width="1" height="1" alt=""><hr><p><a href="https://medium.com/byte-tales/six-elements-of-machine-learning-a-beginners-guide-a00fd8b532be">Six Elements Of Machine Learning — A Beginner’s Guide</a> was originally published in <a href="https://medium.com/byte-tales">Byte Tales</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Learning Data Structures And Algorithms]]></title>
            <link>https://medium.com/byte-tales/learning-data-structures-and-algorithms-e6028502ac06?source=rss----1c1d036af958---4</link>
            <guid isPermaLink="false">https://medium.com/p/e6028502ac06</guid>
            <category><![CDATA[coding]]></category>
            <category><![CDATA[learning]]></category>
            <category><![CDATA[productivity]]></category>
            <category><![CDATA[algorithms]]></category>
            <category><![CDATA[programming]]></category>
            <dc:creator><![CDATA[James Shah]]></dc:creator>
            <pubDate>Fri, 15 May 2020 15:03:54 GMT</pubDate>
            <atom:updated>2020-05-15T15:03:53.918Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1000/1*SC4SlxC2RsVJiustaWr1gw.png" /></figure><p>Data Structures and Algorithms are like backbone of a well designed and efficient code. For a software developer, Data Structures and Algorithms are the most important assets for programming, but Algorithms are not only limited to software engineering but in any field, Algorithms are a great way to define instructions to achieve some specific task. As a software development student, I’ve always tried to learn Data Structures and Algorithms, and I know basics and some intermediate level stuff, but I’ve always wanted to learn more about it and I tried so many times but I lacked consistency. But in this quarantine period, I had all the time and decided to start learning Data Structures And Algorithms.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*_4-uH9wikJ0P5UkuHI08hA.png" /></figure><h3>What is Data Structures and Algorithms?</h3><p><strong>Data Structure</strong> is a way to store and organize the data in order to facilitate access and modification to the data. There many different data structures such as Stacks , Linked List, Queues, Trees, Graphs, Hash Tables, etc.<br>And there are some programming language specific data structures,too. For example, Python provides Dictionaries, Tuples, etc.</p><p><strong>Algorithms</strong> have many different definitions all around the world. In general terms, am Algorithm is a collection of explicit, precise and unambiguous instructions, implemented in defined sequence to accomplish a certain task. And thus, Algorithms are not limited to software development but it has many other uses, too. There are so many problems that can be solved using specific algorithms such as simple Sorting, Searching and also some complex problems such as Identifying all the genes in a human DNA, Predictions and Recommendation Systems.</p><h3>Motivation To Learn</h3><p>Well as we all know, Data Structures and Algorithms are very important to crack coding interviews but that isn’t the only thing where learning Data Structures and Algorithms are useful. Data Structures and Algorithms Knowledge give us the ability to improve our solution to the problem and the ability to write much better and efficient code.</p><p>But most importantly, it helps to build problem solving mindset. And programming is almost all about problem solving. We can adapt any programming language syntax but the problem solving skill can’t be adapted overnight. Thus, learning Data Structures and Algorithms can be a major learning curve for any computer science student.</p><h3>What I Have To Learn?</h3><ol><li>Algorithm Design</li><li>Algorithm Analysis</li><li>Algorithm Techniques</li><li>How to improve the time and space complexity</li><li>Advanced Data Structures (Trees, Graphs,etc.)</li></ol><h3>My Resources &amp; Plan To Learn Data Structures and Algorithms</h3><p>There are so many books, courses, blogs and videos to learn Data Structures and Algorithms. And every Data Structures and Algorithms learner has this question that “Which course/book is the best to learn Data Structures and Algorithms?” And I am no exception. I had the same question as well, but over the past 3 years, I’ve learned many programming languages and frameworks using online courses and books. And what I’ve realized is that You don’t have to find the best resources.<em> Just take a book or a course and start learning.</em></p><p>While learning anything, I start with an online course and then refer to some book. So I started a <a href="https://www.coursera.org/specializations/data-structures-algorithms">Data Structures and Algorithms Specialization</a> by University Of California, San Diego on coursera. It consists total of 6 courses from Basic Data Structures and Algorithms to Advance Algorithms and Complexity. And in the last course they have a project “Genome Assembly Programming Challenge” . They also have a book to refer during the course which can be of great use if you prefer text based content over videos. I’m almost halfway through the first course of the specialization and What I like the most about this specialization is that all the courses have a lots of programming challenges as assignments to solve and so, you don’t just learn theory but also, implement it in code.</p><p>Along with this course, I’m using <a href="https://amzn.to/2zJbd3K">Introduction To Algorithms</a> book (aka CLRS). It’s a great book for someone who knows basics of Data Structures and Algorithms but for a total beginner, it can be overwhelming. In my opinion, if you are starting from scratch you should refer to <a href="http://geeksforgeeks.org/">GeeksForGeeks</a> for basic Data Structures and try to implement them in a programming language of your choice.</p><p>Also, Mathematics plays very important role in learning Data Structures and Algorithms. I have a pretty good background in Mathematics but still, I have to go through some topics of Algebra and Calculus to refresh the dead cells of my brain.😅</p><h3>Other Resources :</h3><h4>Courses :</h4><ol><li><a href="https://www.coursera.org/specializations/algorithms">Algorithms by Standford University</a></li><li><a href="https://practice.geeksforgeeks.org/courses/dsa-online">DSA by GeeksForGeeks</a></li><li><a href="https://www.youtube.com/watch?v=HtSuA80QTyo&amp;list=PLUl4u3cNGP61Oq3tWYp6V_F-5jb5L2iHb">MIT 6.006 On Youtube</a></li></ol><h4>Books :</h4><ol><li><a href="https://amzn.to/2T8858e">Algortithms</a> By Robert Sedgewick</li><li><a href="https://amzn.to/3bD1BVc">Data Structures and Algorithms Made Easy</a> by Narasimha Karumanchi</li></ol><h4>Youtube Channels</h4><ol><li><a href="https://www.youtube.com/user/codechefofficial">Codechef Official</a></li><li><a href="https://www.youtube.com/channel/UC1kBxkk2bcG78YBX7LMl9pQ/">code_report</a></li><li><a href="https://www.youtube.com/channel/UC0RhatS1pyxInC00YKjjBqQ">GeeksForGeeks Official channel</a></li></ol><h3>How to be consistent ?</h3><p>Learning Data Structures and Algorithms can be overwhelming sometimes but consistency is very important. The skills will fade away if you stop learning. I’ll tell you what worked for me to be consistent. If you are stuck at some problem for a long time and by long time I mean hours. Then stop trying to solve it at that time because your mind won’t be able to think about the solution at that time and eventually you will get frustrated and disappointed. But don’t skip the problem, if you are not able to solve it after many attempts and you feel like skipping it, go to google and search for the solution and analyze your mistake and your thinking process and then close that solution and implement it by yourself. But, don’t make it a habit. Be patient. Data Structures and Algorithms is a vast ocean and you can’t learn it all in a day. Keep learning, keep making mistakes, and keep solving them.<br>😉</p><h3>What’s Next ?</h3><p>Obviously, one cannot master Data Structures and Algorithms with just one book and a specialization course. It’s a huge field and this is just a beginning. After this course, I’ll be practicing my Data Structures And Algorithms skills on platforms like <a href="https://leetcode.com/">Leetcode</a> and <a href="https://hackerrank.com/">Hackerrank</a>. Also, some of my friends have recommended me this book called <a href="https://amzn.to/2ZipqiT">Crack Coding Interview</a> by Gayle Laakmann McDowell, so I’ll solve some problems from that book, too.</p><p>Also, if you have any suggestions, please let me know in the comments down below.</p><p><em>Originally published at:</em> <a href="https://www.bytetales.co/learning-data-structures-and-algorithms/">bytetales.co</a></p><p>Thank You For Reading. Happy Coding👩‍💻 👨‍💻</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=e6028502ac06" width="1" height="1" alt=""><hr><p><a href="https://medium.com/byte-tales/learning-data-structures-and-algorithms-e6028502ac06">Learning Data Structures And Algorithms</a> was originally published in <a href="https://medium.com/byte-tales">Byte Tales</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Building A Node.js — Express App (covid19-updates)|Part-4]]></title>
            <link>https://medium.com/byte-tales/building-a-node-js-express-app-covid19-updates-part-4-d4bdb9980d91?source=rss----1c1d036af958---4</link>
            <guid isPermaLink="false">https://medium.com/p/d4bdb9980d91</guid>
            <category><![CDATA[expressjs]]></category>
            <category><![CDATA[nodejs]]></category>
            <category><![CDATA[javascript]]></category>
            <category><![CDATA[deployment]]></category>
            <category><![CDATA[web-development]]></category>
            <dc:creator><![CDATA[James Shah]]></dc:creator>
            <pubDate>Tue, 21 Apr 2020 08:28:59 GMT</pubDate>
            <atom:updated>2020-04-21T08:28:59.153Z</atom:updated>
            <content:encoded><![CDATA[<h3>Building A Node.js — Express App (covid19-updates)|Part-4</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*OUjtNn5egreEGK6eO9NCUw.png" /></figure><p><em>If you haven’t read the last post, I recommend you to check it out </em><a href="https://medium.com/byte-tales/building-a-node-js-express-app-covid19-updates-part-3-b4948d53d80d"><em>here</em></a><em>, else many things in this post will not make sense.</em></p><p>In the last post we finally completed the database creation and user registration and also scheduled twilio sms api to send the case updates sms to the users daily and now in this post we will receive an sms from the user, in case they want to stop the service and we’ll deactivate their account to stop sending messages to them anymore and will send them a reply back. And after that, we’ll deploy our webapp on <a href="https://www.heroku.com/">Heroku</a>. Although, there are many cloud services available to serve Node.JS application online but I chose heroku as it’s makes deployment very easy using github integration. Let’s start with receiving an SMS from the user.</p><h3>Receiving and Replying to SMS using Twilio</h3><p>Twilio can send our web application an HTTP request when certain events happen, such as an incoming text message to our Twilio phone number.So, we’ll create a route that twilio will call whenever a user sends a message on our Twilio phone number to stop or restart the service. Now, We need to configure our Twilio phone number so that it can send an HTTP request to the route in our application, whenever our it receieves an incoming message. But, we’ll need our websites URL for that,which we don’t have for now. So, let’s skip this and create the route in our application and after we deploy our application on heroku, we’ll add the URL in the Twilio console.</p><p>Create a new file in routes directory called sms.js and add the following code to it. <em>Twilio provides code snippet for almost every programming language to achieve this task.</em></p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/b6f0254aee070542e38e356266be5ca4/href">https://medium.com/media/b6f0254aee070542e38e356266be5ca4/href</a></iframe><p>So, now when a user sends an sms to our Twilio phone number with ‘STOP’ or ‘START’ in the message body, Twilio will call the /sms route in our application and our application will handle the sms according to the message body and will reply to the user. We&#39;ll need to add the following code in app.js to add this route in our application.</p><p>And we are done with the coding part. Now, let’s deploy the application on Heroku and set it live for the users to use.I will be deploying the app on Heroku using Github repository, however we can also make use of heroku-cli to deploy. You can learn more about Heroku CLI at their website, <a href="https://devcenter.heroku.com/articles/heroku-cli">here</a>.</p><h3>Creating a Github Repository And Deploying using Heroku</h3><p>Before creating a Github repository, we’ll add a .gitignore file in our project directory to specify what parts of our application we want to ignore(i.e, node modules, .env file) while uploading it to the Github repository. We can manually write the .gitignore file but for the best practice we can get the specific file for specific programming language or framework that we use in our application. Github itself provides a repository that contains the collection of such files that we can use in our application and I used one for this application as well. You can find it <a href="https://github.com/github/gitignore/blob/master/Node.gitignore">here</a>.</p><p>Now, let’s create a new Github Repository and use our local project directory to upload.If you don’t know how to create a github repository, you can learn how to do it from their <a href="https://help.github.com/en/github/getting-started-with-github/create-a-repo">documentation</a>.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*mA_EX4qqyTlG4CbG" /></figure><p>Now, let’s go to Heroku and sign in to your account, create a new account,if you don’t have one by simplying filling out a form for free.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/800/0*p1orQ0ZnadtTWH9O" /></figure><p>Now, on the dashboard, let’s creat a new app and then connect your Github account on the next page and select the Project Repository that we’ve just created and connect.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*GVRK6UGQsDwWlCTE" /></figure><p>Now your app has been created but to tell Heroku that this is a Node.JS app, we’ll need to add a buildpack. Go to settings and select “Add Buildpack” tab and select <strong>nodejs</strong> option from the list.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/800/0*4an1DtZr6K747foS" /></figure><p>Remember, we used .env file to hide our secrets but we&#39;ve not uploaded it to our Github repository for the very obvious reason. So let&#39;s add those secrets in <strong>Heroku Config Variables</strong>. In settings tab select Reveal Config Var and then add all the keys and values one by one.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/686/0*sYSo3hUQjL62GffR" /></figure><p>And that’s it, Now we can visit the url by clicking <strong>Open App</strong> from dashboard and we have our app live on server!!</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*WU3etL7fQrjlGrKw" /></figure><p>Now remember, we skipped a step above, configuring the Twilio console to add the URL of our app. Let’s finish it by going into console page of our Twilio Account. Then click on your Twilio phone number. Find the Messaging section and the “A MESSAGE COMES IN” option and add the url to the application and add /sms as it is the route that handles incoming messages in our application.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/795/0*5yq-HF0ivBrVf3j4" /></figure><p>And Finally, We’ve completed all our project from the scratch to find an idea, learning tech stack, creating and then deploying the application.</p><p><strong><em>Full code can be found here:</em></strong></p><p><a href="https://github.com/jamesshah/Covid19-Updates">jamesshah/covid19-updates</a></p><h4>Thank You For Reading. #HappyCoding👩‍💻 👨‍💻</h4><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=d4bdb9980d91" width="1" height="1" alt=""><hr><p><a href="https://medium.com/byte-tales/building-a-node-js-express-app-covid19-updates-part-4-d4bdb9980d91">Building A Node.js — Express App (covid19-updates)|Part-4</a> was originally published in <a href="https://medium.com/byte-tales">Byte Tales</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Building A Node.js — Express App (covid19-updates)|Part-3]]></title>
            <link>https://medium.com/byte-tales/building-a-node-js-express-app-covid19-updates-part-3-b4948d53d80d?source=rss----1c1d036af958---4</link>
            <guid isPermaLink="false">https://medium.com/p/b4948d53d80d</guid>
            <category><![CDATA[tutorial]]></category>
            <category><![CDATA[programming]]></category>
            <category><![CDATA[javascript]]></category>
            <category><![CDATA[nodejs]]></category>
            <category><![CDATA[web-development]]></category>
            <dc:creator><![CDATA[James Shah]]></dc:creator>
            <pubDate>Wed, 15 Apr 2020 13:33:57 GMT</pubDate>
            <atom:updated>2020-04-15T13:33:57.098Z</atom:updated>
            <content:encoded><![CDATA[<h3>Building A Node.js — Express App (covid19-updates)|Part-3</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*OUjtNn5egreEGK6eO9NCUw.png" /></figure><p><em>If you haven’t read the last post, I recommend you to check it out here, else many things in this post will not make sense.</em></p><p><a href="https://medium.com/byte-tales/building-a-node-js-express-app-covid19-updates-part-2-954184599b9a">Building A Node.js — Express App (covid19-updates)|Part-2</a></p><p>In the previous post, we created database and user registration for our application and now in this post we’ll add Twilio API to send SMS to users whenever they register successfully and we’ll also add external API call to get the corona-virus data to send to our users. Lots of work to do!! So, let’s get started.</p><h3>Using Twilio API</h3><p><a href="https://www.twilio.com/">Twilio</a> provides many <a href="https://www.twilio.com/docs">services</a> ranging from SMS and Chat to AI Bots, Programmable SMS API being one of them. Using the Programmable Twilio SMS API we can add SMS service in our web application and that’s exactly what we’ll doing.</p><p>To use the Twilio SMS API,we need to create an account on Twilio website and buy a twilio number with SMS support. Twilio provides free number with some limitations. But luckily, As a part of #DevTwilioHackathon they are providing Free 20$ credits using the promo code DEVHACK20. So I went ahead and bought a number with this promo code. And now if we go to the console on the twilio website, we can get &#39;ACCOUNT SID&#39; and &#39;AUTH TOKEN&#39; which will be used to connect to the Twilio API in future. Copy that and paste it into the .env file in our project directory as we don&#39;t want anyone to be able to see it.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*t053RxcIv4vVKe-z" /></figure><p>To use the twilio API in our application, we’ll need Twilio Helper Library and we can download it using the following code in the terminal.</p><pre>$ npm install twilio</pre><p>Now, Twilio provides code snippets to connect to it’s various APIs in many different programming language, And we’ll use it to connect to the twilio api in our app.Let’s go to the users.js file in our project directory and add some code to connect to the API.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/7fa839209f02516d1d05fd23aef0b96e/href">https://medium.com/media/7fa839209f02516d1d05fd23aef0b96e/href</a></iframe><p>We’ll add this snippet in the /users/add_user route, so that whenever a new user gets added to the database, we&#39;ll send the SMS to the user. Here is the sample SMS that our application sends to the user.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*JplSYz8sY1A7Z7NS" /></figure><h3>Getting Data From The External API</h3><p>Now, we want to send corona virus case updates to our users but where do we get the data from? And there comes the External API. There are so many Corona Virus datasets available publically online. I’ve used the <a href="https://api.covid19india.org/">api.covid19india.org</a> to get the data as it is crowdsourced and it is based on the data declared by the State Governments so it’s a reliable source of data and also, it gets updated every 5 minutes, so we’ll get latest updates.And also, this api requires no authentication at all.</p><p>In node.js, we cannot use fetch as we do in the client-side javascript, but there is an npm package to our rescue called node-fetch. So let&#39;s install it.</p><pre>$ npm install node-fetch</pre><p>Now, let’s add a new file in our project directory called script.js and let’s add some code in it to get the data from the api and use it.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/b01fbaa391a1ce556e8f7373a2b9b623/href">https://medium.com/media/b01fbaa391a1ce556e8f7373a2b9b623/href</a></iframe><p>Okay, let me explain it a little bit. Here, we are fetching the state wise data from the api and it returns array of objects in which there are many properties like, total cases, new cases, total deaths, new deaths, total recovered,etc. We want to find the total cases and new cases in states. Here i’ve shown only 2 states (Gujarat &amp; Maharashtra) but we can do the same for any state and can get the same data for all the states of India. So i’ve created an object called ‘cases’ and in this object I’ve added 6 keys namely, ‘total_cases’, ‘total_new’, ‘mh_total’, ‘mh_new’, ‘gj_total’, ‘gj_new’ and their values gives the number of ‘Total Cases in India’, ‘New Cases in India’, ‘Total Cases in Maharashtra’,’New Cases in Maharashtra’, ‘Total Cases in Gujarat’, ‘New Cases in Gujarat’ respectively.</p><h3>Sending Message To The Users According To Their State</h3><p>We want to send a distinct message to the user according to his/her state.i.e, if the user registered with the state ‘Gujarat’ then we want to send updates of cases in Gujarat and in India and for users of Maharashtra, we want to send updates of cases in Maharashtra and in India. So to do that we’ll create two different messages to send to the user and for all the users in our database we’ll check the state and send the according message to them.Let’s create a new function get_users to get the users mobile numbers and divide them into different arrays according to their state.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/db32d4050ccb5f79e7f5cc37bb3632ab/href">https://medium.com/media/db32d4050ccb5f79e7f5cc37bb3632ab/href</a></iframe><p>So, as we can see we first get all the active users from our database and insert their mobile numbers in different arrays and add that arrays to the ‘numbers’ object with the corresponding key.</p><p>Now, let’s create a new function send_msg() to create and send message to our users using the Twilio API.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/8a7706316f1355a89d674e2eb0e48902/href">https://medium.com/media/8a7706316f1355a89d674e2eb0e48902/href</a></iframe><h3>Scheduling Automatic SMS</h3><p>We have our message ready with us to send to users but how do we schedule it to send automatically at a specific time? Cron Jobs! The node-cron package helps us schedule the job to perform at a specific time.So, let&#39;s install this npm package and use it to schedule our send_msg function to send SMS automatically everyday.</p><pre>$ npm install node-cron</pre><p>The schedule method of cron module takes two arguments, first the pattern of time that we want to schedule and second is the function that we want to schedule. I’ve set time to 9a.m. and 8p.m. Everyday. <a href="https://www.npmjs.com/package/node-cron">Read More About Cron Patterns Here</a>.</p><p>And now we’ll have to call the method tasks.start() in our app.js file when the server starts like this.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/c5df5c6c0d6185d4d191a96d135ed12b/href">https://medium.com/media/c5df5c6c0d6185d4d191a96d135ed12b/href</a></iframe><p>And that’s it.Now when we deploy our application on server, it will send sms to the users automatically everyday at 9 a.m. and 8 p.m. using the data retrieved by the external api. Here is what this message looks like.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*87fp7yd_kr8-1Au0" /></figure><p>And Bingo! We’ve created a fully functional application which registers users and sends them cases updates daily automatically. Now in the next post, we’ll add functionality to receive sms to users in case they want to stop the service and then we’ll deploy our application on heroku server.</p><h4>Thank You For Reading. #HappyCoding 👩‍💻 👨‍💻</h4><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=b4948d53d80d" width="1" height="1" alt=""><hr><p><a href="https://medium.com/byte-tales/building-a-node-js-express-app-covid19-updates-part-3-b4948d53d80d">Building A Node.js — Express App (covid19-updates)|Part-3</a> was originally published in <a href="https://medium.com/byte-tales">Byte Tales</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Building A Node.js — Express App (covid19-updates)|Part-2]]></title>
            <link>https://medium.com/byte-tales/building-a-node-js-express-app-covid19-updates-part-2-954184599b9a?source=rss----1c1d036af958---4</link>
            <guid isPermaLink="false">https://medium.com/p/954184599b9a</guid>
            <category><![CDATA[javascript]]></category>
            <category><![CDATA[covid19]]></category>
            <category><![CDATA[nodejs]]></category>
            <category><![CDATA[web-development]]></category>
            <category><![CDATA[programming]]></category>
            <dc:creator><![CDATA[James Shah]]></dc:creator>
            <pubDate>Sat, 11 Apr 2020 09:46:58 GMT</pubDate>
            <atom:updated>2020-04-11T09:46:58.484Z</atom:updated>
            <content:encoded><![CDATA[<h3>Building A Node.js — Express App (covid19-updates)|Part-2</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*OUjtNn5egreEGK6eO9NCUw.png" /></figure><p><em>If you haven’t read the last post, I recommend you to check it out else many things in this post will not make sense.</em></p><p><a href="https://medium.com/byte-tales/building-a-node-js-express-app-covid19-updates-part-1-1b7e297aa2dc">Building A Node.js — Express App (covid19-updates)|Part-1</a></p><p>In the last post we’ve created a simple index(‘/’) route with HTML file as response using EJS template engine.<br>Now, it’s time to look into middlewares and how can we use them to serve our routes with the same prefix easily and create user registration.</p><h3>Middle-wares</h3><p>Middleware functions are functions that have access to the request object (req), the response object (res), and the next function in the application’s request-response cycle.The next function is a function in the Express router which, when invoked, executes the middleware succeeding the current middleware.</p><p>For example, let’s say you have a route (‘user/login’) in your express app to login the user. Now you want to check the credentials given by users with the credentials stored in the database and authorize that user to the account.And when do you check this credentials? After the user sends request and before your application sends the response. And that’s when the middlewares comes in handy. You can write your logic to authenticate the user in the middleware function and response to the user according the return value of the middleware function.Other than authentication, middlewares have many uses as well. <a href="https://expressjs.com/en/guide/writing-middleware.html">Read More About Middleware Here</a>.<br> <br>To use the middleware function, we need to add app.use(middlewareFunctionName); in our code. And that will add the defined middleware function to all the routes that we define in our application. It is called the ‘application level middleware’. But we can also add middlewares to particular routes of our choice by providing that route as the first argument in the app.use function. i.e : app.use(&#39;/&#39;, middlewareFunctionName);</p><p>Now, let’s add middlewares in our code to serve the routes from routes directory.</p><pre>app.use(‘/’, require(‘./routes/index’));</pre><p>Now let’s create the user registration page in our app. But that will require MongoDB database. So, I created an account on <a href="https://www.mongodb.com/cloud/atlas">MongoDB Atlas</a>, they provide 500 MB of free database hosting. And I got the database URL from that which we’ll be using to connect to our database. the url seems something like this.</p><p>mongodb+src://&lt;username&gt;:&lt;password&gt;@cluster0-kmdpn.mongodb.net/test?retryWrites=true&amp;w=majority</p><h3>Creating Database Model and Connecting to The Database</h3><p>First of all, let’s use mongoose and the database URL and see if we can connect to our databse.We will need mongoose package and we’ll use it’s connect method to connect to our database. Let’s add the below code to our app.js file.</p><pre>const mongoose = require(“mongoose”);</pre><pre>const DATABASE_URL = “mongodb+srv://&lt;username&gt;:&lt;password&gt;<a href="http://twitter.com/cluster0">@cluster0</a>-kmdpn.mongodb.net/test?retryWrites=true&amp;w=majority”;</pre><pre>mongoose.connect(DATABASE_URL, (err) =&gt; {<br> if (err) console.log(err);<br> else console.log(“Connected to DB.”);<br>});</pre><p>Now we can start the server and see if we’ve connected to the database or not but we are currently working on a localhost server and this code is in our machine only but when we deploy our application online on cloud, we don’t want everyone to see the DATABASE_URL as it contains our username and password of the database. So, to protect it we can use .env file to store our secret credentials.And while uploading it online we won’t be adding this .env file. To use it let’s install the <a href="https://www.npmjs.com/package/dotenv">‘dotenv’ npm package</a>.</p><pre>$ npm install dotenv</pre><p>Then,create a .env file in the project directory and add DATABASE_URL to this file. And to use the value from this file in our application we can make following changes.</p><pre>require(“dotenv”).config();</pre><pre>// To connect to MongoDB Database<br>mongoose.connect(<br> process.env.DB_CONN, <br> (err) =&gt; {<br> if (err) console.log(err);<br> else console.log(“Connected to DB!”);<br> }<br>);</pre><p>Okay!! So now we have our application connected to our database.<br>Now,let’s create a database model which is basically a schema of our user’s information. So first let’s create a new directory ‘Models’ in our app directory and create a new file User.js. In this file, we’ll write the below code to create and export our model. <br>We’ll need 4 field in our User model. Name, Contact Number, State, and Active. In which, Name,Number and State fields will be String type and Active field will be Boolean type.</p><pre>const mongoose = require(“mongoose”);<br>const UserSchema = mongoose.Schema({<br> name: {<br> type: String,<br> required: true,<br> },<br> number: {<br> type: String,<br> required: true,<br> },<br> state: {<br> type: String,<br> required: true,<br> },<br> active: {<br> type: Boolean,<br> required: true,<br> default: true,<br> },<br>});</pre><pre>module.exports = mongoose.model(“User”, UserSchema);</pre><p>Now, we have our User model ready with us. Now, we we’ll create routes and create HTML page to register a user. Let’s start by creating (‘/users/register’) route. Create a new file users.js in routes directory and add the below code.</p><pre>const express = require(“express”);<br>const router = express.Router();</pre><pre>router.get(‘/register’, (req,res) =&gt; {<br> res.render(‘register’);<br>});</pre><pre>module.exports = router;</pre><h3>Creating User Registration Form and Registering Users</h3><p>Here we’ve used an in-built express middleware Router. Using router instance, we can easily serve routes with same prefix which is ‘user’. We’ll be creating more routes with this ‘user’ prefix and therefore we can take advantage of the Router middleware. Now, let’s create a form for users to register in the register view that we are rendering to the register route.</p><pre>&lt;div class=”jumbotron”&gt;<br> &lt;form action=”/users/add_user” method=”post”&gt;<br> &lt;fieldset&gt;<br> &lt;legend<br> style=”text-align: center; text-decoration: underline;”<br> class=”mt-3&quot;<br> &gt;<br> Register Now<br> &lt;/legend&gt;<br> &lt;div class=”form-group”&gt;<br> &lt;label for=”name”&gt;Name&lt;/label&gt;<br> &lt;input<br> type=”text”<br> name=”name”<br> class=”form-control”<br> id=”name”<br> required=”true”<br> placeholder=”Enter Your Name”<br> /&gt;<br> &lt;/div&gt;<br> &lt;div class=”form-group”&gt;<br> &lt;label for=”phone”&gt;Phone Number&lt;/label&gt;<br> &lt;input<br> name=”number”<br> type=”text”<br> class=”form-control”<br> id=”number”<br> required=”true”<br> aria-describedby=”phoneHelp”<br> placeholder=”Enter Your Number”<br> /&gt;<br> &lt;small id=”phoneHelp” class=”form-text text-muted”<br> &gt;We’ll never share your number with anyone else. Promise!&lt;/small<br> &gt;<br> &lt;/div&gt;<br> &lt;div class=”form-group”&gt;<br> &lt;label for=”state”&gt;State&lt;/label&gt;<br> &lt;select<br> class=”form-control”<br> id=”state”<br> name=”state”<br> style=”cursor: pointer;”<br> required=”true”<br> &gt;<br> &lt;option<br> selected=”selected”<br> id=”state_default”<br> value=””<br> style=”display: none;”<br> &gt;Select State&lt;/option<br> &gt;<br> &lt;option value=”Gujarat”&gt;Gujarat&lt;/option&gt;<br> &lt;option value=”Maharashtra”&gt;Maharashtra&lt;/option&gt;<br> &lt;/select&gt;<br> &lt;/div&gt;<br> &lt;button type=”submit” class=”btn btn-primary”&gt;Count Me In!&lt;/button&gt;<br> &lt;/fieldset&gt;<br> &lt;/form&gt;<br>&lt;/div&gt;</pre><p>It should look something like this,</p><figure><img alt="User Registration Form" src="https://cdn-images-1.medium.com/max/1024/1*y0_nbqDP5DdLwfsIu3T0cg.png" /><figcaption>User Registration Form</figcaption></figure><p>As we’ve added the path /users/add_user in the action attribute of the form tag, let’s create that route in our users.js file to register user to our database.</p><p>In this, first we’ll check if the phone number entered by the user already exist in our database, if it exists we’ll redirect the user to the already_registered page and if it doesn’t exist already, we’ll add it our database and redirect the user to the register/successful page. This is going to be a little bit complex, let’s start coding.</p><pre>const User = require(“../models/User”); // To access the User Model</pre><pre>router.post(“/add_user”, async (req, res) =&gt; {<br> try { <br> const { name, number, state } = req.body; // Getting Form Fields From The Request Body. <br> const new_user = await User.aggregate([{ $match: { number: number } }]); // Checking if the number already exists.<br> if (new_user.length != 0) {<br> res.render(“already_registered”,<br> {<br> message: “It Seems Like, This number is already registered with us.”, <br> }<br> );<br> } else {<br> // Create a new user object to add to the DB.<br> const user = new User({<br> name: name,<br> number: number,<br> state: state,<br> });<br> const addedUser = await user.save(); // Saving the user to the database.<br> res.redirect(“/users/register/successful”);<br> }<br> } catch (error) {<br> console.log(error);<br> res.json({ message: error });<br> }<br>};</pre><p>And Done!! Really?? No, This code will generate an error and that’s because we can’t directly use the request body in our app. We’ll have to convert it to JSON to be able to use it in our application. And to do that we’ll use body-parser package. Let’s add that in app.js and we are done with the registration!!</p><pre>$ npm install body-parser;</pre><pre>// app.js </pre><pre>const bodyParser = require(“body-parser”);<br>app.use(bodyParser.urlencoded({ extended: false }));<br>app.use(bodyParser.json());</pre><p>And now we are finally done!! Now in the next post we’ll add the most exciting part of this application — the Twilio API and we’ll send the users an SMS when they register to let them know they have registered successfully and also we’ll create a route to receive the message from the users, if they want to stop our service. Fun things are coming up!!</p><p>Full code can be found at this repository :</p><p><a href="https://github.com/jamesshah/covid19-updates">jamesshah/covid19-updates</a></p><h4>Thank you for reading!! #HappyCoding👩‍💻 👨‍💻</h4><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=954184599b9a" width="1" height="1" alt=""><hr><p><a href="https://medium.com/byte-tales/building-a-node-js-express-app-covid19-updates-part-2-954184599b9a">Building A Node.js — Express App (covid19-updates)|Part-2</a> was originally published in <a href="https://medium.com/byte-tales">Byte Tales</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Building A Node.js — Express App (covid19-updates)|Part-1]]></title>
            <link>https://medium.com/byte-tales/building-a-node-js-express-app-covid19-updates-part-1-1b7e297aa2dc?source=rss----1c1d036af958---4</link>
            <guid isPermaLink="false">https://medium.com/p/1b7e297aa2dc</guid>
            <category><![CDATA[javascript]]></category>
            <category><![CDATA[nodejs]]></category>
            <category><![CDATA[expressjs]]></category>
            <category><![CDATA[covid19]]></category>
            <category><![CDATA[web-development]]></category>
            <dc:creator><![CDATA[James Shah]]></dc:creator>
            <pubDate>Thu, 09 Apr 2020 14:16:58 GMT</pubDate>
            <atom:updated>2020-04-09T14:16:58.223Z</atom:updated>
            <content:encoded><![CDATA[<h3>Building A Node.js — Express App (covid19-updates)|Part-1</h3><figure><img alt="Photo Courtesy: undraw.co" src="https://cdn-images-1.medium.com/max/1024/1*OUjtNn5egreEGK6eO9NCUw.png" /><figcaption>Photo Courtesy: undraw.co</figcaption></figure><h3>Brainstorming</h3><p>Last week Dev.to announced a h<a href="https://dev.to/devteam/announcing-the-twilio-hackathon-on-dev-2lh8">ackathon</a> in collaboration with Twilio to make a web app using any of the Twilio APIs, So I decided to participate and build something in this quarantine time which will help me learn some new tech and build a project to showcase.</p><p>So I started brainstorming for an idea to create an app for and just then I heard my mother saying that we are privileged to have an Internet service which helps us getting latest news and updates in this Covid-19 lockdown period but for those who doesn’t have this privilege, It’s hard to get information about new cases in their state as well as in the country.And bingo!! I got the idea to create my web app.</p><p>I decided to create a web app in which user can register with their phone number once and they can get daily updates of new cases in their area as well as in the country with the help of <a href="https://www.twilio.com/docs/sms">Twilio Programmable SMS API</a>.</p><h3>Deciding Tech-Stack</h3><p>Now it was time to choose the Tech-Stack that i will be using to create this web app.I’ve created web apps using Django and Flask(Python Web Frameworks) in the past,and I am pretty much used to creating backend in Python but I wanted to learn new Stack and so I decided to take on Node.JS &amp; [Express Framework](<a href="https://expressjs.com/">https://expressjs.com/</a>) for this web app and for database, I’ve used SQL Databases (MySql, PostGreSql) in past and I wanted to learn NoSql Databases, so I decided to learn [MongoDB](<a href="https://www.mongodb.com/">https://www.mongodb.com</a>) by implementing it in this project. And That’s it. I was ready with the Idea and Technologies to use in my mind and I started searching and learning basics about these techs from here and there.</p><h4>Some Recommended Resources To Start Learning Node.Js, Express And MongoDB (Pre-requisite : Vanilla Javascript)</h4><p>1. <a href="https://www.youtube.com/watch?v=zQRrXTSkvfw">Node.Js Tutorial For Beginners | Dev Ed</a><br>2. <a href="https://www.youtube.com/watch?v=vjf774RKrLc&amp;t=515s">Build A Restful Api With Node.js Express &amp; MongoDB | Dev Ed</a><br>3. <a href="https://developer.mozilla.org/en-US/docs/Learn/Server-side/Express_Nodejs/Tutorial_local_library_website">Express Tutorial — MDN Docs</a><br>4. <a href="https://www.youtube.com/watch?v=-56x56UppqQ&amp;t=358s">MongoDB Crash Course | Traversy Media</a></p><h3>Creating Package.json &amp; Installing Dependencies</h3><p>After all the tutorials and documentation reading, it was time to get my hands dirty and writing some code.</p><p>Any node application is generally started by creating a `package.json` file. This file keeps record of all the dependencies used in your application and every information about your project. It is very essential as you cannot upload all your installed node modules online,let’s say Github or GitLab. So using this file you can install all the dependencies on the go.</p><p>You can create such file by writing following code in the terminal.</p><pre>$ npm init</pre><p>Now let’s install express and some other useful npm packages by writing the following line in terminal.</p><pre>$ npm install express nodemon mongoose</pre><p><a href="https://mongoosejs.com/">mongoose</a> is an object modeling package for node.js which helps us to connect to our MongoDB database and create model for our database.<br>And <a href="https://www.npmjs.com/package/nodemon">nodemon</a> is a tool that helps develop node.js based applications by automatically restarting the node application when file changes in the directory are detected.</p><p>As soon as these packages get installed, you can check in your ‘package.json’ and you will see that in dependencies these packages along with their installed versions are stored.</p><h3>Creating Express Server And Writing First Route</h3><p>After installing the express, I started writing my first route in express in the app.js file.</p><pre>// Importing express in our app<br>const express = require(‘express’); <br>// Creating express app instance<br>const app = express.app();</pre><pre>//First Get Route<br>app.get(‘/’, (req,res) =&gt; {<br> res.send(“Hello World. This is my first route in express!”)<br>});</pre><pre>// Start listening on port 3000.<br>app.listen(3000, () =&gt; console.log(‘Server Up And Running. Listening On Port 3000’) );</pre><p>Now, we can run this script with following code in terminal and it will run just fine.</p><pre>$ node app.js</pre><p>But as I’ve mentioned before, we’ll be using nodemon so that we don’t have to stop and rerun our server after every change and so now we’ll add an npm script in our `package.json` file as below.</p><pre>“start”: “nodemon app.js”</pre><p>And now, all we have to do is write npm start<br>and nodemon will run our server whenever we change any file in our project.</p><p>Then, By visiting <a href="http://localhost:3000/">http://localhost:3000</a>I could see my server up and running with my response to the index(‘/’) route as it is.</p><p>But we don’t want to just send single line responses in our website but we want to send styled html pages to our users.We can do so by Rendering HTML files using many different <a href="https://expressjs.com/en/guide/using-template-engines.html">template engines</a> such as Jade, EJS, Pug, Mustache, etc.</p><h3>Rendering HTML files using EJS template engine</h3><p>I decided to use EJS because in this application I won’t need much complex front-end. To use EJS, first we have to install EJS using npm.</p><pre>$ npm install ejs express-ejs-layouts</pre><p>This, express-ejs-layouts package helps us create HTML code blocks and use them in in different HTML files to reduce the code redundancy.For example, we can create a basic layout in which we can create the basic HTML structure and then we don’t have to repeat the `head, title, meta, body` tags again in every file.</p><p>To use EJS and express-ejs-layouts, we’ll have to add following lines in our app.js file</p><pre>const expressLayouts = require(‘express-ejs-layouts’);<br>app.set(‘view engine’, ‘ejs’);<br>app.use(expressLayouts);</pre><p>Now we can render HTML files into our application and using EJS we can also use variables in our HTML code. Now we can write HTML code in our route response but wait, we shouldn’t mix our front-end with our back-end in the same file right? Because, that will make debugging task backbreaking and our code will be unreadable.</p><p>Also, We won’t be writing routes in this app.js file as well for the same reason as above. So now is a good time to restructure our code before it gets unmanageable and tiresome.</p><h3>Restructuring Project Directory</h3><p>Directory structure is a very important point in any project and therefore, we will choose the best practice to create our project structure.</p><p>Below is the restructured project directory.</p><pre><strong>covid19-updates/</strong><br> <strong>views/<br>  index.ejs <br> routes/<br>  index.js<br> app.js<br> package.json</strong></pre><p>We’ll have to make some changes in our app.js file as well. We’ll have to set views directory so that our server will know where to search for our HTML files that we want to render and also we’ll have to import route in app.js file.Let’s see how our app.js file looks like after these changes.</p><pre>// Importing express in our app<br>const express = require(‘express’); <br>// Creating express app instance<br>const app = express.app();</pre><pre>const expressLayouts = require(‘express-ejs-layouts’);</pre><pre>app.set(‘views’,__dirname + ‘/views’);<br>app.set(‘view engine’, ‘ejs’);<br>app.use(‘expressLayouts’);</pre><pre>// To serve Route<br>require(‘./routes/index’)(app);</pre><pre>// Started listening on port 3000.<br>app.listen(3000, () =&gt; console.log(‘Server Up And Running. Listening On Port 3000’) );</pre><p>And now in routes/index.jsfile, we’ll have to export the module that we’ve created for our index(‘/’) route using module.exports.<br><em>Note: In node.js, every file is treated as a module.</em></p><p><a href="https://nodejs.org/api/modules.html">Read more about modules in node here.</a></p><pre>module.exports = {<br> app.get(‘/’(req,res) =&gt; {<br> res.render(‘index.ejs’);<br> });<br>}</pre><p>Though, using Router Middle-ware to serve routes of same prefix is the best practice in an express app. <br>And what in the name of god are Middle-wares,now??Well, that’s a whole another article. For now, we are ready with our index route with HTML file as response!!</p><p>Full code can be found at this repository :</p><p><a href="https://github.com/jamesshah/covid19-updates">jamesshah/covid19-updates</a></p><p>Thank you for reading!! <strong>#HappyCoding</strong>👩‍💻 👨‍💻</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=1b7e297aa2dc" width="1" height="1" alt=""><hr><p><a href="https://medium.com/byte-tales/building-a-node-js-express-app-covid19-updates-part-1-1b7e297aa2dc">Building A Node.js — Express App (covid19-updates)|Part-1</a> was originally published in <a href="https://medium.com/byte-tales">Byte Tales</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[How I Automated The Google Form Filling For My College Attendance Using Python]]></title>
            <link>https://medium.com/byte-tales/how-i-automated-the-google-form-filling-for-my-college-attendance-using-python-4a7907f60ea0?source=rss----1c1d036af958---4</link>
            <guid isPermaLink="false">https://medium.com/p/4a7907f60ea0</guid>
            <category><![CDATA[python]]></category>
            <category><![CDATA[automation]]></category>
            <category><![CDATA[coding]]></category>
            <category><![CDATA[programming]]></category>
            <category><![CDATA[productivity]]></category>
            <dc:creator><![CDATA[James Shah]]></dc:creator>
            <pubDate>Thu, 26 Mar 2020 10:53:06 GMT</pubDate>
            <atom:updated>2020-03-26T10:53:06.144Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*dht5kbd4Htb94Qk9ElGaVA.png" /></figure><p>To fight Corona Virus, our Prime Minister has declared lockdown for 21 days in all over the country. And so, our college decided to take online lectures and also, sent us a google form which is to be filled every day for each lecture to fill attendance. Filling form 4 times a day with the subject name and corresponding time and professor details is a tedious task. But also, Attendance is Engineer’s Only Motivation to attend lectures.</p><p>And So, I decided to use Python and to write a script to automate this humdrum task.</p><p>As usual, I started by searching on google to see how can I submit the google form using python script. And I came to know that Every Google Form Input element has a “name” attribute attached to it and we can use it to send the data for the corresponding field in the form.</p><p>So, I opened the google form in chrome and started inspecting input fields in which I will be filling the data, and I found a bunch of <em>ss-form-entry</em> objects with name attribute like</p><pre>&quot;name = entry.&lt;id&gt;&quot;</pre><p>I copied every name attribute value of the field that I wanted to submit data into.</p><p>And then, in python I created a dictionary(python data-type) with the value of name attributes that we copied from the browser as keys for the dictionary and the corresponding data that I want to fill in the field as the value of that key.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/01b2788071b10bae2cbbfb2ee3d794cd/href">https://medium.com/media/01b2788071b10bae2cbbfb2ee3d794cd/href</a></iframe><p>Then I created a dictionary with Days-name(Monday, Tuesday, etc.) as keys and the lectures’ details list as values to the corresponding key.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/3dd983e1d5b45ddeef9e6f94bd31d971/href">https://medium.com/media/3dd983e1d5b45ddeef9e6f94bd31d971/href</a></iframe><p>And then, put both the dictionaries in a function that returns the list of all the data dictionaries containing the information about the current day’s lectures.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/c4a8e2dacf7a23bf1eb3abbba605e006/href">https://medium.com/media/c4a8e2dacf7a23bf1eb3abbba605e006/href</a></iframe><p>And now, finally the function for sending the form response using the <strong>POST</strong> method of <strong>requests</strong> library. It takes two arguments — URL and data that you want to send with the URL, respectively.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/806d63ef242a8f55b287819cb1268a5d/href">https://medium.com/media/806d63ef242a8f55b287819cb1268a5d/href</a></iframe><p><strong><em>Note:</em></strong><em> Change the URL of google form by replacing ‘/viewform’ at the end of the URL with ‘formResponse’. Now, your URL should look like this</em></p><pre>https://docs.google.com/forms/d/&lt;form_id&gt;/formResponse</pre><p>And that’s it, now I just have to run this script once a day and it will fill my attendance for all the lectures of that day.</p><p>I can also deploy it on a server like <a href="https://www.pythonanywhere.com/">PythonAnyWhere</a> or <a href="https://www.digitalocean.com/">DigitalOcean</a> and it can run by itself every day, but as I’ve nothing much to do in this lockdown situation for the 21Days, I have more than enough time to run it manually every day.🤓🤓</p><p>You can get the full code from here: <a href="https://github.com/jamesshah/GoogleForm-AutoFill">Google Form Auto-Filling</a>.</p><p>Also check out my another python script to <a href="https://medium.com/byte-tales/automate-your-project-initialization-using-python-shell-script-6f6476cefb2e">Automate The Project Initialization</a>!!</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=4a7907f60ea0" width="1" height="1" alt=""><hr><p><a href="https://medium.com/byte-tales/how-i-automated-the-google-form-filling-for-my-college-attendance-using-python-4a7907f60ea0">How I Automated The Google Form Filling For My College Attendance Using Python</a> was originally published in <a href="https://medium.com/byte-tales">Byte Tales</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
    </channel>
</rss>