<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Ajaykumar Dev on Medium]]></title>
        <description><![CDATA[Stories by Ajaykumar Dev on Medium]]></description>
        <link>https://medium.com/@nikaljeajay36?source=rss-ea4e86d061b1------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/0*y46VCFkwtrK_3eDf.jpg</url>
            <title>Stories by Ajaykumar Dev on Medium</title>
            <link>https://medium.com/@nikaljeajay36?source=rss-ea4e86d061b1------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Sat, 21 Nov 2026 20:23:20 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@nikaljeajay36/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[Python: Understanding DataFrame data structure from Panda: Part 5.]]></title>
            <link>https://medium.com/@nikaljeajay36/python-understanding-dataframe-data-structure-from-panda-part-5-730d693ce4cc?source=rss-ea4e86d061b1------2</link>
            <guid isPermaLink="false">https://medium.com/p/730d693ce4cc</guid>
            <category><![CDATA[pandas]]></category>
            <category><![CDATA[join]]></category>
            <category><![CDATA[dataframes]]></category>
            <category><![CDATA[python-programming]]></category>
            <dc:creator><![CDATA[Ajaykumar Dev]]></dc:creator>
            <pubDate>Sat, 03 Aug 2024 16:18:09 GMT</pubDate>
            <atom:updated>2024-08-03T16:18:09.315Z</atom:updated>
            <content:encoded><![CDATA[<p>Pandas is an essential tool for data manipulation in Python, and understanding how to concatenate, merge, and join DataFrames is crucial for efficient data analysis. In this blog, we’ll explore:</p><ol><li><strong>Concatenating DataFrames Vertically and Horizontally</strong></li><li><strong>Merging DataFrames with Different Types of Joins</strong></li><li><strong>Finding Unique and Shared Values Between DataFrames</strong></li><li><strong>Joining DataFrames by Index Labels</strong></li></ol><p><strong>1. Concatenating DataFrames Vertically and Horizontally</strong></p><p>Concatenation in pandas allows you to combine multiple DataFrames along a particular axis. This is useful when you have data split across several DataFrames and want to aggregate them.</p><h4>Vertical Concatenation</h4><p>Vertical concatenation (stacking) is achieved by appending DataFrames on top of each other.</p><pre>import pandas as pd<br><br># Creating sample DataFrames<br>df1 = pd.DataFrame({<br>    &#39;Name&#39;: [&#39;Alice&#39;, &#39;Bob&#39;],<br>    &#39;Age&#39;: [25, 30]<br>})<br><br>print(&quot;df1: First Dataframe&quot;)<br>print(df1)<br><br>df2 = pd.DataFrame({<br>    &#39;Name&#39;: [&#39;Charlie&#39;, &#39;David&#39;],<br>    &#39;Age&#39;: [35, 40]<br>})<br><br>print(&quot;df2: Second Dataframe&quot;)<br>print(df2)<br><br># Concatenating DataFrames vertically<br>df_vertical = pd.concat([df1, df2], axis=0, ignore_index=True)<br>print(&quot;Vertical Concatenation:&quot;)<br>print(df_vertical)<br><br>O/P<br>df1: First Dataframe<br>    Name  Age<br>0  Alice   25<br>1    Bob   30<br>df2: Second Dataframe<br>      Name  Age<br>0  Charlie   35<br>1    David   40<br>Vertical Concatenation:<br>      Name  Age<br>0    Alice   25<br>1      Bob   30<br>2  Charlie   35<br>3    David   40</pre><h4>Horizontal Concatenation</h4><p>Horizontal concatenation (side-by-side) is achieved by aligning DataFrames next to each other.</p><pre>import pandas as pd<br><br># Creating sample DataFrames<br>df1 = pd.DataFrame({<br>    &#39;City&#39;: [&#39;New York&#39;, &#39;Los Angeles&#39;],<br>    &#39;Country&#39;: [&#39;USA&#39;, &#39;USA&#39;]<br>})<br><br>print(&quot;df1: First Dataframe&quot;)<br>print(df1)<br>print(&quot;\n&quot;)<br><br>df2 = pd.DataFrame({<br>    &#39;Occupation&#39;: [&#39;Engineer&#39;, &#39;Doctor&#39;],<br>    &#39;Salary&#39;: [100000, 120000]<br>})<br><br>print(&quot;df2: Second Dataframe&quot;)<br>print(df2)<br>print(&quot;\n&quot;)<br><br># Concatenating DataFrames horizontally<br>df_horizontal = pd.concat([df1, df2], axis=1)<br>print(&quot;\nHorizontal Concatenation:&quot;)<br>print(df_horizontal)<br><br>O/P<br>df1: First Dataframe<br>          City Country<br>0     New York     USA<br>1  Los Angeles     USA<br><br><br>df2: Second Dataframe<br>  Occupation  Salary<br>0   Engineer  100000<br>1     Doctor  120000<br><br><br><br>Horizontal Concatenation:<br>          City Country Occupation  Salary<br>0     New York     USA   Engineer  100000<br>1  Los Angeles     USA     Doctor  120000</pre><p><strong>2. Merging DataFrames with Different Types of Joins.</strong></p><p>Merging DataFrames is akin to SQL joins. You can perform inner, outer, left, and right joins based on common columns or indices.</p><p><strong>Inner Join</strong></p><p>An inner join returns only the rows with matching keys in both DataFrames.</p><pre>import pandas as pd<br><br># Creating sample DataFrames<br>df1 = pd.DataFrame({<br>    &#39;Employee&#39;: [&#39;John&#39;, &#39;Anna&#39;, &#39;Mike&#39;],<br>    &#39;Department&#39;: [&#39;HR&#39;, &#39;IT&#39;, &#39;IT&#39;]<br>})<br><br>print(&quot;df1: First Dataframe &quot;)<br>print(df1)<br>print(&quot;\n&quot;)<br><br>df2 = pd.DataFrame({<br>    &#39;Employee&#39;: [&#39;John&#39;, &#39;Mike&#39;, &#39;Sara&#39;],<br>    &#39;Salary&#39;: [70000, 85000, 60000]<br>})<br><br>print(&quot;df2: Second Dataframe &quot;)<br>print(df2)<br>print(&quot;\n&quot;)<br><br># Merging DataFrames with inner join<br>df_inner = pd.merge(df1, df2, on=&#39;Employee&#39;, how=&#39;inner&#39;)<br>print(&quot;\nInner Join:&quot;)<br>print(df_inner)<br><br>O/P<br>df1: First Dataframe <br>  Employee Department<br>0     John         HR<br>1     Anna         IT<br>2     Mike         IT<br><br>df2: First Dataframe <br>  Employee  Salary<br>0     John   70000<br>1     Mike   85000<br>2     Sara   60000<br><br><br>Inner Join:<br>  Employee Department  Salary<br>0     John         HR   70000<br>1     Mike         IT   85000</pre><p><strong>Outer Join</strong></p><p>An outer join returns all rows from both DataFrames, filling in NaNs where there are no matches.</p><pre>import pandas as pd<br><br># Creating sample DataFrames<br>df1 = pd.DataFrame({<br>    &#39;Employee&#39;: [&#39;John&#39;, &#39;Anna&#39;, &#39;Mike&#39;],<br>    &#39;Department&#39;: [&#39;HR&#39;, &#39;IT&#39;, &#39;IT&#39;]<br>})<br><br>print(&quot;df1: First Dataframe &quot;)<br>print(df1)<br>print(&quot;\n&quot;)<br><br>df2 = pd.DataFrame({<br>    &#39;Employee&#39;: [&#39;John&#39;, &#39;Mike&#39;, &#39;Sara&#39;],<br>    &#39;Salary&#39;: [70000, 85000, 60000]<br>})<br><br>print(&quot;df2: Second Dataframe &quot;)<br>print(df2)<br>print(&quot;\n&quot;)<br><br># Merging DataFrames with outer join<br>df_outer = pd.merge(df1, df2, on=&#39;Employee&#39;, how=&#39;outer&#39;)<br>print(&quot;\nOuter Join:&quot;)<br>print(df_outer)<br><br>O/P<br>df1: First Dataframe <br>  Employee Department<br>0     John         HR<br>1     Anna         IT<br>2     Mike         IT<br><br><br>df2: First Dataframe <br>  Employee  Salary<br>0     John   70000<br>1     Mike   85000<br>2     Sara   60000<br><br><br><br>Outer Join:<br>  Employee Department   Salary<br>0     Anna         IT      NaN<br>1     John         HR  70000.0<br>2     Mike         IT  85000.0<br>3     Sara        NaN  60000.0</pre><p><strong>Left Join</strong></p><p>A left join returns all rows from the left DataFrame and the matching rows from the right DataFrame. If there’s no match, NaN is used.</p><pre>import pandas as pd<br><br># Creating sample DataFrames<br>df1 = pd.DataFrame({<br>    &#39;Employee&#39;: [&#39;John&#39;, &#39;Anna&#39;, &#39;Mike&#39;],<br>    &#39;Department&#39;: [&#39;HR&#39;, &#39;IT&#39;, &#39;IT&#39;]<br>})<br><br>print(&quot;df1: First Dataframe &quot;)<br>print(df1)<br>print(&quot;\n&quot;)<br><br>df2 = pd.DataFrame({<br>    &#39;Employee&#39;: [&#39;John&#39;, &#39;Mike&#39;, &#39;Sara&#39;],<br>    &#39;Salary&#39;: [70000, 85000, 60000]<br>})<br><br>print(&quot;df2: Second Dataframe &quot;)<br>print(df2)<br>print(&quot;\n&quot;)<br><br># Merging DataFrames with left join<br>df_left = pd.merge(df1, df2, on=&#39;Employee&#39;, how=&#39;left&#39;)<br>print(&quot;\nLeft Join:&quot;)<br>print(df_left)<br><br>O/P<br>df1: First Dataframe <br>  Employee Department<br>0     John         HR<br>1     Anna         IT<br>2     Mike         IT<br><br><br>df2: First Dataframe <br>  Employee  Salary<br>0     John   70000<br>1     Mike   85000<br>2     Sara   60000<br><br><br><br>Left Join:<br>  Employee Department   Salary<br>0     John         HR  70000.0<br>1     Anna         IT      NaN<br>2     Mike         IT  85000.0<br></pre><p><strong>3. Finding Unique and Shared Values Between DataFrames</strong></p><p>You might need to find values that are unique to each DataFrame or shared between them.</p><pre>import pandas as pd<br><br># Creating sample DataFrames<br>df7 = pd.DataFrame({<br>    &#39;ID&#39;: [1, 2, 3, 4],<br>    &#39;Value&#39;: [&#39;A&#39;, &#39;B&#39;, &#39;C&#39;, &#39;D&#39;]<br>})<br><br>df8 = pd.DataFrame({<br>    &#39;ID&#39;: [3, 4, 5, 6],<br>    &#39;Value&#39;: [&#39;C&#39;, &#39;D&#39;, &#39;E&#39;, &#39;F&#39;]<br>})<br><br># Finding unique values in each DataFrame<br>unique_df7 = df7[~df7[&#39;ID&#39;].isin(df8[&#39;ID&#39;])]<br>unique_df8 = df8[~df8[&#39;ID&#39;].isin(df7[&#39;ID&#39;])]<br><br>print(&quot;\nUnique values in df7:&quot;)<br>print(unique_df7)<br><br>print(&quot;\nUnique values in df8:&quot;)<br>print(unique_df8)<br><br># Finding shared values between DataFrames<br>shared_values = df7[df7[&#39;ID&#39;].isin(df8[&#39;ID&#39;])]<br>print(&quot;\nShared values between df7 and df8:&quot;)<br>print(shared_values)<br><br>O/P<br>Unique values in df7:<br>   ID Value<br>0   1     A<br>1   2     B<br><br>Unique values in df8:<br>   ID Value<br>2   5     E<br>3   6     F<br><br>Shared values between df7 and df8:<br>   ID Value<br>2   3     C<br>3   4     D</pre><p><strong>4. Joining DataFrames by Index Labels.</strong></p><p>You can join DataFrames based on their index labels using the join method.</p><pre>import pandas as pd<br><br># Creating sample DataFrames with indices<br>df1 = pd.DataFrame({<br>    &#39;A&#39;: [1, 2, 3],<br>    &#39;B&#39;: [4, 5, 6]<br>}, index=[&#39;a&#39;, &#39;b&#39;, &#39;c&#39;])<br><br>df2 = pd.DataFrame({<br>    &#39;C&#39;: [7, 8],<br>    &#39;D&#39;: [9, 10]<br>}, index=[&#39;b&#39;, &#39;c&#39;])<br><br># Joining DataFrames by index labels<br>df_joined = df1.join(df2)<br>print(&quot;\nJoined DataFrames by Index:&quot;)<br>print(df_joined)<br><br>O/P<br>Joined DataFrames by Index:<br>   A  B    C     D<br>a  1  4  NaN   NaN<br>b  2  5  7.0   9.0<br>c  3  6  8.0  10.0</pre><h3>Conclusion</h3><p>Mastering DataFrame operations in pandas, such as concatenation, merging, finding unique values, and joining by index, is essential for effective data manipulation. These techniques allow you to combine datasets, perform complex queries, and organize your data for insightful analysis. Whether you’re aligning data side-by-side or merging datasets on common attributes, pandas provides a robust framework to handle a variety of data tasks efficiently.</p><p><strong>Thank you for Reading</strong></p><p>If you like this post:</p><ul><li>Please show your support by following and with a clap 👏 or several claps!</li><li>Feel free to share this guide with your friends.</li><li>Your feedback is invaluable — it inspires and guides my future posts.</li><li>Or drop me a message: <a href="https://www.linkedin.com/in/ajaykumar-dev/">https://www.linkedin.com/in/ajaykumar-dev/</a></li></ul><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=730d693ce4cc" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Python: Understanding DataFrame data structure from Panda: Part 4.]]></title>
            <link>https://medium.com/@nikaljeajay36/python-understanding-dataframe-data-structure-from-panda-part-4-01a41d11370b?source=rss-ea4e86d061b1------2</link>
            <guid isPermaLink="false">https://medium.com/p/01a41d11370b</guid>
            <category><![CDATA[python-programming]]></category>
            <category><![CDATA[pandas]]></category>
            <category><![CDATA[dataframes]]></category>
            <category><![CDATA[group-by]]></category>
            <dc:creator><![CDATA[Ajaykumar Dev]]></dc:creator>
            <pubDate>Sat, 03 Aug 2024 15:54:28 GMT</pubDate>
            <atom:updated>2024-08-03T15:54:28.404Z</atom:updated>
            <content:encoded><![CDATA[<p>Pandas provides powerful tools for data analysis, and one of the most versatile features is the groupby method. This method allows you to split your DataFrame into groups based on some criteria, apply functions to these groups, and perform various operations. In this blog, we’ll cover how to:</p><ol><li><strong>Split a DataFrame into Groups.</strong></li><li><strong>Extract First and Last Rows from Groups.</strong></li><li><strong>Perform Aggregate Operations.</strong></li><li><strong>Iterate Over DataFrames in a GroupBy Object.</strong></li></ol><p>We will go through each concepts,</p><p><strong>1. Splitting a DataFrame into Groups</strong></p><p>The groupby method is used to split a DataFrame into groups based on one or more columns. This is particularly useful for performing operations on subsets of your data.</p><pre>import pandas as pd<br><br># Creating a DataFrame<br>df = pd.DataFrame({<br>    &#39;Department&#39;: [&#39;HR&#39;, &#39;IT&#39;, &#39;IT&#39;, &#39;HR&#39;, &#39;Finance&#39;, &#39;Finance&#39;],<br>    &#39;Employee&#39;: [&#39;John&#39;, &#39;Anna&#39;, &#39;Mike&#39;, &#39;Sarah&#39;, &#39;Tom&#39;, &#39;Jerry&#39;],<br>    &#39;Salary&#39;: [70000, 80000, 85000, 72000, 90000, 92000]<br>})<br><br># Printing the current dataframe data<br>print(&quot;Printing the current dataframe data&quot;)<br>print(df)<br><br># Grouping by the &#39;Department&#39; column<br>grouped = df.groupby(&#39;Department&#39;)<br><br># Displaying the groups<br>print(&quot;Groups:&quot;)<br>print(grouped.groups)<br><br>O/P<br>Printing the current dataframe data<br>  Department Employee  Salary<br>0         HR     John   70000<br>1         IT     Anna   80000<br>2         IT     Mike   85000<br>3         HR    Sarah   72000<br>4    Finance      Tom   90000<br>5    Finance    Jerry   92000<br>Groups:<br>{&#39;Finance&#39;: [4, 5], &#39;HR&#39;: [0, 3], &#39;IT&#39;: [1, 2]}</pre><p><strong>2. Extracting First and Last Rows from Groups</strong></p><p>Once you have grouped your data, you might want to extract specific rows from each group. You can use first() and last() to get the first and last rows in each group, respectively.</p><pre>import pandas as pd<br><br># Creating a DataFrame<br>df = pd.DataFrame({<br>    &#39;Department&#39;: [&#39;HR&#39;, &#39;IT&#39;, &#39;IT&#39;, &#39;HR&#39;, &#39;Finance&#39;, &#39;Finance&#39;],<br>    &#39;Employee&#39;: [&#39;John&#39;, &#39;Anna&#39;, &#39;Mike&#39;, &#39;Sarah&#39;, &#39;Tom&#39;, &#39;Jerry&#39;],<br>    &#39;Salary&#39;: [70000, 80000, 85000, 72000, 90000, 92000]<br>})<br><br># Printing the current dataframe data<br>print(&quot;Printing the current dataframe data&quot;)<br>print(df)<br><br># Grouping by the &#39;Department&#39; column<br>grouped = df.groupby(&#39;Department&#39;)<br><br># Extracting the first row of each group<br>first_rows = grouped.first()<br>print(&quot;\nFirst row in each group:&quot;)<br>print(first_rows)<br><br># Extracting the last row of each group<br>last_rows = grouped.last()<br>print(&quot;\nLast row in each group:&quot;)<br>print(last_rows)<br><br>O/P<br>Printing the current dataframe data<br>  Department Employee  Salary<br>0         HR     John   70000<br>1         IT     Anna   80000<br>2         IT     Mike   85000<br>3         HR    Sarah   72000<br>4    Finance      Tom   90000<br>5    Finance    Jerry   92000<br><br>First row in each group:<br>           Employee  Salary<br>Department                 <br>Finance         Tom   90000<br>HR             John   70000<br>IT             Anna   80000<br><br>Last row in each group:<br>           Employee  Salary<br>Department                 <br>Finance       Jerry   92000<br>HR            Sarah   72000<br>IT             Mike   85000</pre><p><strong>3. Performing Aggregate Operations on GroupBy Groups.</strong></p><p>The groupby object allows you to perform aggregate operations such as sum, mean, and count on each group. This is useful for summarizing data.</p><pre>import pandas as pd<br><br># Creating a DataFrame<br>df = pd.DataFrame({<br>    &#39;Department&#39;: [&#39;HR&#39;, &#39;IT&#39;, &#39;IT&#39;, &#39;HR&#39;, &#39;Finance&#39;, &#39;Finance&#39;],<br>    &#39;Employee&#39;: [&#39;John&#39;, &#39;Anna&#39;, &#39;Mike&#39;, &#39;Sarah&#39;, &#39;Tom&#39;, &#39;Jerry&#39;],<br>    &#39;Salary&#39;: [70000, 80000, 85000, 72000, 90000, 92000]<br>})<br><br># Printing the current dataframe data<br>print(&quot;Printing the current dataframe data&quot;)<br>print(df)<br><br># Grouping by the &#39;Department&#39; column<br>grouped = df.groupby(&#39;Department&#39;)<br><br>aggregated = grouped.agg({<br>    &#39;Salary&#39;: [&#39;mean&#39;, &#39;sum&#39;, &#39;max&#39;]  # Compute mean, sum, and max of &#39;Salary&#39;<br>})<br><br>print(&quot;\nAggregated data:&quot;)<br>print(aggregated)<br><br>O/P<br>Printing the current dataframe data<br>  Department Employee  Salary<br>0         HR     John   70000<br>1         IT     Anna   80000<br>2         IT     Mike   85000<br>3         HR    Sarah   72000<br>4    Finance      Tom   90000<br>5    Finance    Jerry   92000<br><br>Aggregated data:<br>             Salary               <br>               mean     sum    max<br>Department                        <br>Finance     91000.0  182000  92000<br>HR          71000.0  142000  72000<br>IT          82500.0  165000  85000</pre><p><strong>4. Iterating Over DataFrames in a GroupBy Object.</strong></p><p>You can iterate over the groups in a GroupBy object to perform more customized operations. Each iteration yields a tuple where the first element is the group name and the second element is the DataFrame containing the group’s data.</p><pre>import pandas as pd<br><br># Creating a DataFrame<br>df = pd.DataFrame({<br>    &#39;Department&#39;: [&#39;HR&#39;, &#39;IT&#39;, &#39;IT&#39;, &#39;HR&#39;, &#39;Finance&#39;, &#39;Finance&#39;],<br>    &#39;Employee&#39;: [&#39;John&#39;, &#39;Anna&#39;, &#39;Mike&#39;, &#39;Sarah&#39;, &#39;Tom&#39;, &#39;Jerry&#39;],<br>    &#39;Salary&#39;: [70000, 80000, 85000, 72000, 90000, 92000]<br>})<br><br># Printing the current dataframe data<br>print(&quot;Printing the current dataframe data&quot;)<br>print(df)<br><br># Grouping by the &#39;Department&#39; column<br>grouped = df.groupby(&#39;Department&#39;)<br><br>print(&quot;\nIterating over groups:&quot;)<br>for name, group in grouped:<br>    print(f&quot;\nDepartment: {name}&quot;)<br>    print(group)<br><br>O/P<br>Printing the current dataframe data<br>  Department Employee  Salary<br>0         HR     John   70000<br>1         IT     Anna   80000<br>2         IT     Mike   85000<br>3         HR    Sarah   72000<br>4    Finance      Tom   90000<br>5    Finance    Jerry   92000<br><br>Iterating over groups:<br><br>Department: Finance<br>  Department Employee  Salary<br>4    Finance      Tom   90000<br>5    Finance    Jerry   92000<br><br>Department: HR<br>  Department Employee  Salary<br>0         HR     John   70000<br>3         HR    Sarah   72000<br><br>Department: IT<br>  Department Employee  Salary<br>1         IT     Anna   80000<br>2         IT     Mike   85000</pre><h3>Conclusion</h3><p>The groupby method in pandas is a powerful tool for data analysis, enabling you to split, aggregate, and iterate over DataFrame groups. By mastering these techniques, you can efficiently perform complex data manipulations and gain deeper insights from your datasets. Whether you’re summarizing data, extracting specific rows, or applying custom functions, the groupby method is an invaluable part of your data analysis toolkit.</p><p><strong>Thank you for Reading</strong></p><p>If you like this post:</p><ul><li>Please show your support by following and with a clap 👏 or several claps!</li><li>Feel free to share this guide with your friends.</li><li>Your feedback is invaluable — it inspires and guides my future posts.</li><li>Or drop me a message: <a href="https://www.linkedin.com/in/ajaykumar-dev/">https://www.linkedin.com/in/ajaykumar-dev/</a></li></ul><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=01a41d11370b" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Python: Understanding DataFrame data structure from Panda: Part 3.]]></title>
            <link>https://medium.com/@nikaljeajay36/python-understanding-dataframe-data-structure-from-panda-part-3-f614c0e15f92?source=rss-ea4e86d061b1------2</link>
            <guid isPermaLink="false">https://medium.com/p/f614c0e15f92</guid>
            <category><![CDATA[dataframes]]></category>
            <category><![CDATA[python-programming]]></category>
            <category><![CDATA[pandas]]></category>
            <dc:creator><![CDATA[Ajaykumar Dev]]></dc:creator>
            <pubDate>Sat, 03 Aug 2024 15:19:41 GMT</pubDate>
            <atom:updated>2024-08-03T15:19:41.656Z</atom:updated>
            <content:encoded><![CDATA[<p>Pandas is an essential library for data analysis in Python, offering powerful tools to manipulate and analyze data.</p><p>We will learn below concepts.</p><ol><li>Reducing a DataFrame’s memory use.</li><li>Extracting DataFrame rows by one or more conditions.</li><li>Filtering a DataFrame for rows that include or exclude null values.</li><li>Selecting column values that fall between a range.</li><li>Removing duplicate and null values from a DataFrame.</li></ol><p>Let’s go through each concepts,</p><p><strong>1. Reducing a DataFrame’s Memory Use</strong></p><p>DataFrames can consume a significant amount of memory, especially with large datasets. Optimizing memory usage is important for performance and scalability. You can reduce memory usage by changing data types to more efficient ones.</p><pre>import pandas as pd<br>import numpy as np<br><br># Creating a DataFrame<br>data = {<br>    &#39;A&#39;: np.random.randint(0, 100, size=1000000),<br>    &#39;B&#39;: np.random.random(size=1000000),<br>    &#39;C&#39;: np.random.choice([&#39;foo&#39;, &#39;bar&#39;, &#39;baz&#39;], size=1000000)<br>}<br>df = pd.DataFrame(data)<br><br># Checking initial memory usage<br>print(&quot;Initial memory usage:\n&quot;, df.memory_usage(deep=True))<br><br># Optimize data types<br>df[&#39;A&#39;] = df[&#39;A&#39;].astype(np.int8)  # Convert integer column to a smaller type<br>df[&#39;B&#39;] = df[&#39;B&#39;].astype(np.float32)  # Convert float column to a smaller type<br>df[&#39;C&#39;] = df[&#39;C&#39;].astype(&#39;category&#39;)  # Convert string column to category type<br><br># Checking reduced memory usage<br>print(&quot;\nReduced memory usage:\n&quot;, df.memory_usage(deep=True))<br><br>O/P<br>Initial memory usage:<br> Index         128<br>A         8000000<br>B         8000000<br>C        60000000<br>dtype: int64<br><br>Reduced memory usage:<br> Index        128<br>A        1000000<br>B        4000000<br>C        1000288<br>dtype: int64</pre><p>2. Extracting DataFrame Rows by One or More Conditions</p><p>Filtering rows based on specific conditions is a common operation in data analysis. You can extract rows that meet one or more criteria using boolean indexing.</p><pre>import pandas as pd<br><br># Creating a DataFrame<br>df = pd.DataFrame({<br>    &#39;Name&#39;: [&#39;Alice&#39;, &#39;Bob&#39;, &#39;Charlie&#39;, &#39;David&#39;, &#39;Eva&#39;],<br>    &#39;Age&#39;: [25, 30, 35, 40, 45],<br>    &#39;Salary&#39;: [70000, 80000, 75000, 90000, 95000]<br>})<br><br># Extract rows where Age is greater than 30<br>filtered_df = df[df[&#39;Age&#39;] &gt; 30]<br>print(&quot;Rows where Age &gt; 30:\n&quot;, filtered_df)<br><br># Extract rows where Age is between 30 and 40 and Salary is greater than 75000<br>filtered_df = df[(df[&#39;Age&#39;] &gt; 30) &amp; (df[&#39;Salary&#39;] &gt; 75000)]<br>print(&quot;\nRows where Age is between 30 and 40 and Salary &gt; 75000:\n&quot;, filtered_df)<br><br>O/P<br>Rows where Age &gt; 30:<br>       Name  Age  Salary<br>2  Charlie   35   75000<br>3    David   40   90000<br>4      Eva   45   95000<br><br>Rows where Age is between 30 and 40 and Salary &gt; 75000:<br>     Name  Age  Salary<br>3  David   40   90000<br>4    Eva   45   95000</pre><p>3, 4 . Filtering a DataFrame for Rows that Include or Exclude Null Values.</p><p>Handling missing values is crucial for accurate analysis. You can filter rows that contain or exclude null values using isna() and dropna() methods.</p><pre>import pandas as pd<br>import numpy as np<br><br># Creating a DataFrame with missing values<br>df = pd.DataFrame({<br>    &#39;Name&#39;: [&#39;Alice&#39;, &#39;Bob&#39;, np.nan, &#39;David&#39;, &#39;Eva&#39;],<br>    &#39;Age&#39;: [25, np.nan, 35, 40, 45],<br>    &#39;Salary&#39;: [70000, 80000, 75000, np.nan, 95000]<br>})<br><br><br>#Original Dataframe<br>print(df)<br>print(&quot;\n&quot;)<br><br># Filter rows where any value is NaN<br>rows_with_na = df[df.isna().any(axis=1)]<br>print(&quot;Rows with any NaN values:\n&quot;, rows_with_na)<br><br># Filter rows where no value is NaN<br>rows_without_na = df.dropna()<br>print(&quot;\nRows without NaN values:\n&quot;, rows_without_na)<br><br># Selecting rows where Salary is between 75000 and 90000<br>range_filtered_df = df[(df[&#39;Salary&#39;] &gt;= 75000) &amp; (df[&#39;Salary&#39;] &lt;= 90000)]<br>print(&quot;\n Rows where Salary is between 75000 and 90000:\n&quot;, range_filtered_df)<br><br>O/P<br>Name   Age   Salary<br>0  Alice  25.0  70000.0<br>1    Bob   NaN  80000.0<br>2    NaN  35.0  75000.0<br>3  David  40.0      NaN<br>4    Eva  45.0  95000.0<br><br><br>Rows with any NaN values:<br>     Name   Age   Salary<br>1    Bob   NaN  80000.0<br>2    NaN  35.0  75000.0<br>3  David  40.0      NaN<br><br>Rows without NaN values:<br>     Name   Age   Salary<br>0  Alice  25.0  70000.0<br>4    Eva  45.0  95000.0<br><br>Rows where Salary is between 75000 and 90000:<br>   Name   Age   Salary<br>1  Bob   NaN  80000.0<br>2  NaN  35.0  75000.0</pre><p>5. Removing Duplicate and Null Values from a DataFrame</p><p>Cleaning data by removing duplicates and null values is essential for accurate analysis. You can use drop_duplicates() and dropna() to clean your DataFrame.</p><pre>import pandas as pd<br>import numpy as np<br><br># Creating a DataFrame with duplicates and missing values<br>df = pd.DataFrame({<br>    &#39;Name&#39;: [&#39;Alice&#39;, &#39;Bob&#39;, &#39;Charlie&#39;, &#39;Alice&#39;, &#39;David&#39;, &#39;Bob&#39;],<br>    &#39;Age&#39;: [25, 30, 35, 25, 40, np.nan],<br>    &#39;Salary&#39;: [70000, 80000, 75000, 70000, 90000, np.nan]<br>})<br><br># Removing duplicate rows<br>df_no_duplicates = df.drop_duplicates()<br>print(&quot;DataFrame after removing duplicates:\n&quot;, df_no_duplicates)<br><br># Removing rows with any NaN values<br>df_cleaned = df_no_duplicates.dropna()<br>print(&quot;\nDataFrame after removing rows with NaN values:\n&quot;, df_cleaned)<br><br>O/P<br>DataFrame after removing duplicates:<br>       Name   Age   Salary<br>0    Alice  25.0  70000.0<br>1      Bob  30.0  80000.0<br>2  Charlie  35.0  75000.0<br>4    David  40.0  90000.0<br>5      Bob   NaN      NaN<br><br>DataFrame after removing rows with NaN values:<br>       Name   Age   Salary<br>0    Alice  25.0  70000.0<br>1      Bob  30.0  80000.0<br>2  Charlie  35.0  75000.0<br>4    David  40.0  90000.0</pre><h3>Conclusion</h3><p>Pandas provides a range of functionalities for efficiently managing and analyzing data. By reducing memory usage, filtering rows based on conditions, handling missing values, and removing duplicates, you can optimize your DataFrames for better performance and more accurate results. Mastering these operations is essential for effective data analysis and ensures that your datasets are clean and manageable.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=f614c0e15f92" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Python: Understanding DataFrame data structure from Panda: Part 2.]]></title>
            <link>https://medium.com/@nikaljeajay36/python-understanding-dataframe-data-structure-from-panda-part-2-6a12ad289a50?source=rss-ea4e86d061b1------2</link>
            <guid isPermaLink="false">https://medium.com/p/6a12ad289a50</guid>
            <category><![CDATA[dataframes]]></category>
            <category><![CDATA[pandas]]></category>
            <category><![CDATA[python-programming]]></category>
            <category><![CDATA[python]]></category>
            <dc:creator><![CDATA[Ajaykumar Dev]]></dc:creator>
            <pubDate>Sat, 03 Aug 2024 14:59:45 GMT</pubDate>
            <atom:updated>2024-08-03T15:00:19.873Z</atom:updated>
            <content:encoded><![CDATA[<p>Pandas is a powerful library in Python for data manipulation and analysis. At the core of pandas is the DataFrame object, a two-dimensional, size-mutable, and potentially heterogeneous tabular data structure. Understanding how to create, manipulate, and organize DataFrames is crucial for effective data analysis. In this blog, we will cover essential concepts including instantiating DataFrames, sorting data, accessing data, and manipulating DataFrame indices and labels.</p><p>In this blog, we would learn below concepts,</p><ol><li>Instantiating DataFrame objects from dictionaries and NumPy ndarrays.</li><li>Sorting DataFrame columns Accessing rows and columns in a DataFrame</li><li>Setting and resetting a DataFrame index Renaming columns and index labels in a DataFrame.</li></ol><p>Let’s go through each concept one by one,</p><ol><li>Instantiating DataFrame Objects.</li></ol><pre>import pandas as pd<br><br># Creating a DataFrame from a dictionary<br>data_dict = {<br>    &#39;Name&#39;: [&#39;Alice&#39;, &#39;Bob&#39;, &#39;Charlie&#39;],<br>    &#39;Age&#39;: [25, 30, 35],<br>    &#39;Salary&#39;: [70000, 80000, 75000]<br>}<br>df = pd.DataFrame(data_dict)<br>print(&quot;DataFrame from dictionary:\n&quot;, df)<br><br>o/p<br>DataFrame from dictionary:<br>       Name  Age  Salary<br>0    Alice   25   70000<br>1      Bob   30   80000<br>2  Charlie   35   75000</pre><p>2. From NumPy ndarrays: You can also create a DataFrame from NumPy arrays. Each array will represent a column, and you can provide column names if needed.</p><pre>import pandas as pd<br>import numpy as np<br><br># Creating a DataFrame from NumPy arrays<br>data_array = np.array([<br>    [25, 70000],<br>    [30, 80000],<br>    [35, 75000]<br>])<br>columns = [&#39;Age&#39;, &#39;Salary&#39;]<br>df_from_array = pd.DataFrame(data_array, columns=columns)<br>print(&quot;\nDataFrame from NumPy array:\n&quot;, df_from_array)<br><br>O/P<br>DataFrame from NumPy array:<br>    Age  Salary<br>0   25   70000<br>1   30   80000<br>2   35   75000</pre><p>3. Sorting DataFrame Columns.</p><p>Sorting DataFrames can be useful for organizing data and identifying trends. While sort_values is used to sort rows, you can also sort columns using sort_index.</p><pre>import pandas as pd<br><br># Creating a DataFrame<br>df = pd.DataFrame({<br>    &#39;Name&#39;: [&#39;Bob&#39;, &#39;Alice&#39;,&#39;Charlie&#39;],<br>    &#39;Age&#39;: [25, 30, 35],<br>    &#39;Salary&#39;: [70000, 80000, 75000]<br>})<br><br># Before Sorting<br>print(&quot;Before Sorting:&quot;)<br>print(df)<br><br># Sorting columns labels order alphabetically<br>df_sorted_columns = df.sort_index(axis=1)<br>print(&quot;DataFrame with sorted columns:\n&quot;, df_sorted_columns)<br><br># Sorting columns alphabetically<br>df_np_new = df_sorted_columns.sort_values(by=[&#39;Name&#39;, &#39;Age&#39;])<br><br># After sorting<br>print(&quot;\nAfter sorting Name and Age Columns data :&quot;)<br>print(df_np_new)<br><br>O/P<br>Before Sorting:<br>      Name  Age  Salary<br>0      Bob   25   70000<br>1    Alice   30   80000<br>2  Charlie   35   75000<br><br>DataFrame with sorted columns:<br>    Age     Name  Salary<br>0   25      Bob   70000<br>1   30    Alice   80000<br>2   35  Charlie   75000<br><br>After sorting Name and Age Columns data :<br>   Age     Name  Salary<br>1   30    Alice   80000<br>0   25      Bob   70000<br>2   35  Charlie   75000</pre><p>4. Accessing Rows and Columns</p><p>Accessing specific rows and columns is a fundamental operation. You can use .loc[] for label-based indexing and .iloc[] for integer-based indexing.</p><pre>import pandas as pd<br><br># Creating a DataFrame<br>df = pd.DataFrame({<br>    &#39;Name&#39;: [&#39;Alice&#39;, &#39;Bob&#39;, &#39;Charlie&#39;],<br>    &#39;Age&#39;: [25, 30, 35],<br>    &#39;Salary&#39;: [70000, 80000, 75000]<br>})<br><br># Accessing a single column<br>print(&quot;\nAccessing &#39;Age&#39; column:\n&quot;, df[&#39;Age&#39;])<br><br># Accessing multiple columns<br>print(&quot;\nAccessing &#39;Name&#39; and &#39;Salary&#39; columns:\n&quot;, df[[&#39;Name&#39;, &#39;Salary&#39;]])<br><br># Accessing rows by index<br>print(&quot;\nAccessing row with index 1:\n&quot;, df.iloc[1])<br><br>O/P<br>Accessing &#39;Age&#39; column:<br> 0    25<br>1    30<br>2    35<br>Name: Age, dtype: int64<br><br>Accessing &#39;Name&#39; and &#39;Salary&#39; columns:<br>       Name  Salary<br>0    Alice   70000<br>1      Bob   80000<br>2  Charlie   75000<br><br>Accessing row with index 1:<br> Name        Bob<br>Age          30<br>Salary    80000<br>Name: 1, dtype: object</pre><p>5. Setting and Resetting DataFrame Index</p><p>The DataFrame index is crucial for data alignment and selection. You can set a new index or reset it to the default integer index.</p><pre>import pandas as pd<br><br># Creating a DataFrame<br>df = pd.DataFrame({<br>    &#39;Name&#39;: [&#39;Alice&#39;, &#39;Bob&#39;, &#39;Charlie&#39;],<br>    &#39;Age&#39;: [25, 30, 35],<br>    &#39;Salary&#39;: [70000, 80000, 75000]<br>}, index=[&#39;a&#39;, &#39;b&#39;, &#39;c&#39;])<br><br># Setting a new index<br>df_set_index = df.set_index(&#39;Name&#39;)<br>print(&quot;\nDataFrame with &#39;Name&#39; as index:\n&quot;, df_set_index)<br><br># Resetting the index<br>df_reset_index = df_set_index.reset_index()<br>print(&quot;\nDataFrame with reset index:\n&quot;, df_reset_index)<br><br>O/P<br>DataFrame with &#39;Name&#39; as index:<br>          Age  Salary<br>Name                <br>Alice     25   70000<br>Bob       30   80000<br>Charlie   35   75000<br><br>DataFrame with reset index:<br>       Name  Age  Salary<br>0    Alice   25   70000<br>1      Bob   30   80000<br>2  Charlie   35   75000</pre><p>6. Renaming Columns and Index Labels</p><p>Renaming columns and index labels helps in making the DataFrame more readable and relevant to your analysis.</p><pre>import pandas as pd<br><br># Creating a DataFrame<br>df = pd.DataFrame({<br>    &#39;Name&#39;: [&#39;Alice&#39;, &#39;Bob&#39;, &#39;Charlie&#39;],<br>    &#39;Age&#39;: [25, 30, 35],<br>    &#39;Salary&#39;: [70000, 80000, 75000]<br>})<br><br># Renaming columns<br>df_renamed_columns = df.rename(columns={&#39;Name&#39;: &#39;Full Name&#39;, &#39;Salary&#39;: &#39;Annual Salary&#39;})<br>print(&quot;\nDataFrame with renamed columns:\n&quot;, df_renamed_columns)<br><br># Renaming index labels<br>df_renamed_index = df.rename(index={0: &#39;First&#39;, 1: &#39;Second&#39;, 2: &#39;Third&#39;})<br>print(&quot;\nDataFrame with renamed index labels:\n&quot;, df_renamed_index)<br><br>O/P<br>DataFrame with renamed columns:<br>   Full Name  Age  Annual Salary<br>0     Alice   25          70000<br>1       Bob   30          80000<br>2   Charlie   35          75000<br><br>DataFrame with renamed index labels:<br>            Name  Age  Salary<br>First     Alice   25   70000<br>Second      Bob   30   80000<br>Third   Charlie   35   75000</pre><h3>Conclusion</h3><p>Pandas DataFrames are versatile and powerful for data manipulation. By understanding how to instantiate DataFrames from various data sources, sort columns and rows, access data, and manipulate indices and labels, you can efficiently manage and analyze your data. Mastering these operations will enhance your data analysis capabilities and enable you to derive meaningful insights from your data.</p><p><strong>Thank you for Reading</strong></p><p>If you like this post:</p><ul><li>Please show your support by following and with a clap 👏 or several claps!</li><li>Feel free to share this guide with your friends.</li><li>Your feedback is invaluable — it inspires and guides my future posts.</li><li>Or drop me a message: <a href="https://www.linkedin.com/in/ajaykumar-dev/">https://www.linkedin.com/in/ajaykumar-dev/</a></li></ul><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=6a12ad289a50" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Python: Understanding series data structure from Panda: Part 1.]]></title>
            <link>https://medium.com/@nikaljeajay36/python-understanding-series-data-structure-from-panda-part-1-12c19710d875?source=rss-ea4e86d061b1------2</link>
            <guid isPermaLink="false">https://medium.com/p/12c19710d875</guid>
            <category><![CDATA[python-programming]]></category>
            <category><![CDATA[python3]]></category>
            <category><![CDATA[pandas]]></category>
            <dc:creator><![CDATA[Ajaykumar Dev]]></dc:creator>
            <pubDate>Sat, 03 Aug 2024 13:58:48 GMT</pubDate>
            <atom:updated>2024-08-03T13:58:48.431Z</atom:updated>
            <content:encoded><![CDATA[<p>In the world of pandas, we often work with two main data structures: Series and DataFrame.</p><ol><li>Series: This is a one-dimensional labeled array that can hold data of any type (integers, floats, strings, etc.). You can think of it as a single column of data with an index.</li><li>DataFrame: This is a two-dimensional labeled data structure with columns of potentially different types. It’s essentially a table, similar to a spreadsheet or SQL table.</li></ol><p>The key relationship between these two structures is that a DataFrame is composed of multiple Series objects. Each column in a DataFrame is actually a Series. So, you can visualize a DataFrame as a collection of Series objects aligned side by side, sharing a common index.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*NCkTCZWBdvvF3gzuT-kLxw.png" /><figcaption>fig 1 Each column in dataframe is series.</figcaption></figure><p>Understanding that a DataFrame is essentially a collection of Series is particularly beneficial when you become familiar with the methods available for Series objects. Most of these methods are also applicable to DataFrames. However, when you apply a method to a DataFrame, it operates on each column individually, producing a result for each column. For instance, the mean method, when applied to a Series, calculates and returns the mean (average) of the values within that Series. When you call mean on a DataFrame, pandas automatically applies the mean method to each column, resulting in a collection of mean values for all columns. Interestingly, this collection of mean values is itself returned as a Series, which means you can further manipulate or analyze it using other Series methods. This hierarchical application of methods allows for powerful and flexible data analysis.</p><p>The Series is a one-dimensional labeled array for homogeneous data. An <em>array</em> is an ordered collection of values comparable to a Python list. The term <em>homogeneous</em> means that the values are of the same data type (all integers or all Booleans, for example).</p><p>Pandas assigns each Series value a <em>label</em>—an identifier we can use to locate the value. The library also assigns each Series value an <em>order</em>—a position in line. The order starts counting from 0; the first Series value occupies position 0, the second value occupies position 1, and so on. The Series is a one-dimensional data structure because we need one reference point to access a value: either a label or a position.</p><p>A Series combines and expands the best features of Python’s native data structures. Like a list, it holds its values in a sequenced order. Like a dictionary, it assigns a key/label to each value. We gain the benefits of both of those objects plus more than 180 methods for data manipulation.</p><p>the index of a Series acts as a label for each element, similar to row numbers or dictionary keys, and can be customized or default to integers. It enables efficient data access, modification, and alignment during operations. Though immutable by default, you can reassign the entire index with new labels.</p><p>We will look into a lot of examples to understand series data structure.</p><ol><li>Creation of series using pandas library.</li></ol><pre>import pandas as pd<br><br># Creating a Series from a list<br>s1 = pd.Series([1, 2, 3, 4, 5])<br>print(s1)<br><br># Creating a Series from a dictionary<br>s2 = pd.Series({&#39;a&#39;: 1, &#39;b&#39;: 2, &#39;c&#39;: 3})<br>print(s2) <br><br># Creating a Series with a custom index<br>s3 = pd.Series([10, 20, 30], index=[&#39;x&#39;, &#39;y&#39;, &#39;z&#39;])<br>print(s3)<br><br>o/p <br><br>0    1<br>1    2<br>2    3<br>3    4<br>4    5<br>dtype: int64<br>a    1<br>b    2<br>c    3<br>dtype: int64<br>x    10<br>y    20<br>z    30<br>dtype: int64</pre><p>Basic attributes for series object.</p><ul><li>s.index: Returns the index (labels) of the Series.</li><li>s.values: Returns the underlying data of the Series as an array.</li><li>s.dtype: Returns the data type of the elements in the Series.</li><li>s.size: Returns the number of elements in the Series.</li><li>s.shape: Returns the shape of the Series (number of elements).</li></ul><h3>Key Methods</h3><ol><li><strong>Accessing Elements.</strong></li></ol><p>Below is code to demonstrate how to access the series data.</p><pre>import pandas as pd<br><br>s = pd.Series([10, 20, 30], index=[&#39;a&#39;, &#39;b&#39;, &#39;c&#39;])<br><br># Access by label<br>print(s[&#39;a&#39;])  # Output: 10<br><br># Access by position<br>print(s.iloc[1])  # Output: 20<br><br># Access by position<br>print(s.loc[&#39;a&#39;])  # Output: 10<br><br>o/p <br>10<br>20<br>10<br></pre><p>2. Basic Operations:</p><p>We will study how to perform basic operation like arithmetic operations and aggregate operations.</p><pre>import pandas as pd<br><br># Creating a Series<br>data = pd.Series([100, 200, 300, None, 500])<br><br>print(&quot;data:&quot;)<br>print(data)<br><br>print(&quot;Adds 2 to each element &quot;)  # Adds 2 to each element<br>print((data + 2))<br><br># Accessing elements<br>print(&quot;Accessing elements second element:&quot;)<br>print(data[1])       # Output: 200<br><br># Filtering<br>print(&quot;Filtering data greater than 150&quot;)<br>print(data[data &gt; 150])  # Output: 200, 300, 500<br><br># Filling missing values<br>print(&quot;Filling missing values&quot;)<br>print(data.fillna(0))  # Output: 100, 200, 300, 0, 500<br><br># Basic statistics<br>print(&quot;Basic statistics&quot;)<br>print(data.mean())    # Output: 275.0<br>print(data.sum())<br><br>o/p<br>data:<br>0    100.0<br>1    200.0<br>2    300.0<br>3      NaN<br>4    500.0<br>dtype: float64<br>Adds 2 to each element <br>0    102.0<br>1    202.0<br>2    302.0<br>3      NaN<br>4    502.0<br>dtype: float64<br>Accessing elements second element:<br>200.0<br>Filtering data greater than 150<br>1    200.0<br>2    300.0<br>4    500.0<br>dtype: float64<br>Filling missing values<br>0    100.0<br>1    200.0<br>2    300.0<br>3      0.0<br>4    500.0<br>dtype: float64<br>Basic statistics<br>275.0<br>1100.0</pre><p>3. Indexing and Slicing:</p><p>Here, we will provide the code for performing index and slice operations.</p><pre>import pandas as pd<br><br># Create a Pandas Series with custom indices<br>data = pd.Series([10, 20, 30, 40, 50], index=[&#39;a&#39;, &#39;b&#39;, &#39;c&#39;, &#39;d&#39;, &#39;e&#39;])<br><br># Display the original Series<br>print(&quot;Original Series:&quot;)<br>print(data)<br>print(&quot;\n&quot;)<br><br># Access a single element using a label<br>print(&quot;Access element at label &#39;b&#39;:&quot;)<br>print(data[&#39;b&#39;])<br>print(&quot;\n&quot;)<br><br># Access a single element using an integer position<br>print(&quot;Access element at position 2:&quot;)<br>print(data[2])<br>print(&quot;\n&quot;)<br><br># Access multiple elements using a list of labels<br>print(&quot;Access elements at labels &#39;a&#39;, &#39;c&#39;, and &#39;e&#39;:&quot;)<br>print(data[[&#39;a&#39;, &#39;c&#39;, &#39;e&#39;]])<br>print(&quot;\n&quot;)<br><br># Access multiple elements using a list of integer positions<br>print(&quot;Access elements at positions 1, 3, and 4:&quot;)<br>print(data[[1, 3, 4]])<br>print(&quot;\n&quot;)<br><br># Slice the Series using labels (end label is inclusive)<br>print(&quot;Slice from label &#39;b&#39; to &#39;d&#39;:&quot;)<br>print(data[&#39;b&#39;:&#39;d&#39;])<br>print(&quot;\n&quot;)<br><br># Slice the Series using integer positions (end position is exclusive)<br>print(&quot;Slice from position 1 to 3:&quot;)<br>print(data[1:3])<br>print(&quot;\n&quot;)<br><br># Boolean indexing to filter elements<br>print(&quot;Filter elements greater than 25:&quot;)<br>print(data[data &gt; 25])<br>print(&quot;\n&quot;)<br><br># Access using iloc for position-based indexing<br>print(&quot;Access element using iloc at position 3:&quot;)<br>print(data.iloc[3])<br>print(&quot;\n&quot;)<br><br># Slice using iloc for position-based indexing<br>print(&quot;Slice using iloc from position 1 to 4:&quot;)<br>print(data.iloc[1:4])<br>print(&quot;\n&quot;)<br><br># Access using loc for label-based indexing<br>print(&quot;Access element using loc at label &#39;c&#39;:&quot;)<br>print(data.loc[&#39;c&#39;])<br>print(&quot;\n&quot;)<br><br># Slice using loc for label-based indexing<br>print(&quot;Slice using loc from label &#39;b&#39; to &#39;e&#39;:&quot;)<br>print(data.loc[&#39;b&#39;:&#39;e&#39;])<br>print(&quot;\n&quot;)<br><br>o/p<br>riginal Series:<br>a    10<br>b    20<br>c    30<br>d    40<br>e    50<br>dtype: int64<br><br><br>Access element at label &#39;b&#39;:<br>20<br><br><br>Access element at position 2:<br>/Users/AjayLab/PythonPrograms/series_index.py:18: FutureWarning: Series.__getitem__ treating keys as positions is deprecated. In a future version, integer keys will always be treated as labels (consistent with DataFrame behavior). To access a value by position, use `ser.iloc[pos]`<br>  print(data[2])<br>30<br><br><br>Access elements at labels &#39;a&#39;, &#39;c&#39;, and &#39;e&#39;:<br>a    10<br>c    30<br>e    50<br>dtype: int64<br><br><br>Access elements at positions 1, 3, and 4:<br>/Users/AjayLab/PythonPrograms/series_index.py:28: FutureWarning: Series.__getitem__ treating keys as positions is deprecated. In a future version, integer keys will always be treated as labels (consistent with DataFrame behavior). To access a value by position, use `ser.iloc[pos]`<br>  print(data[[1, 3, 4]])<br>b    20<br>d    40<br>e    50<br>dtype: int64<br><br><br>Slice from label &#39;b&#39; to &#39;d&#39;:<br>b    20<br>c    30<br>d    40<br>dtype: int64<br><br><br>Slice from position 1 to 3:<br>b    20<br>c    30<br>dtype: int64<br><br><br>Filter elements greater than 25:<br>c    30<br>d    40<br>e    50<br>dtype: int64<br><br><br>Access element using iloc at position 3:<br>40<br><br><br>Slice using iloc from position 1 to 4:<br>b    20<br>c    30<br>d    40<br>dtype: int64<br><br><br>Access element using loc at label &#39;c&#39;:<br>30<br><br><br>Slice using loc from label &#39;b&#39; to &#39;e&#39;:<br>b    20<br>c    30<br>d    40<br>e    50<br>dtype: int64</pre><p>4. String Operations:</p><p>We will check example for string operations on series in Panda.</p><pre>import pandas as pd<br><br># Create a Pandas Series of strings<br>data = pd.Series([&#39;apple&#39;, &#39;banana&#39;, &#39;cherry&#39;, &#39;date&#39;, &#39;elderberry&#39;])<br><br># Convert all strings in the Series to uppercase<br>uppercase_data = data.str.upper()<br>print(&quot;Uppercase Strings:&quot;)<br>print(uppercase_data)<br>print(&quot;\n&quot;)<br><br># Convert all strings in the Series to lowercase<br>lowercase_data = data.str.lower()<br>print(&quot;Lowercase Strings:&quot;)<br>print(lowercase_data)<br>print(&quot;\n&quot;)<br><br># Check if each string contains the substring &#39;an&#39;<br>contains_an = data.str.contains(&#39;an&#39;)<br>print(&quot;Contains &#39;an&#39;:&quot;)<br>print(contains_an)<br>print(&quot;\n&quot;)<br><br># Replace the substring &#39;e&#39; with &#39;E&#39;<br>replace_e = data.str.replace(&#39;e&#39;, &#39;E&#39;)<br>print(&quot;Replace &#39;e&#39; with &#39;E&#39;:&quot;)<br>print(replace_e)<br>print(&quot;\n&quot;)<br><br># Get the length of each string<br>lengths = data.str.len()<br>print(&quot;Lengths of Strings:&quot;)<br>print(lengths)<br>print(&quot;\n&quot;)<br><br># Extract the first three characters of each string<br>first_three_chars = data.str[:3]<br>print(&quot;First Three Characters:&quot;)<br>print(first_three_chars)<br>print(&quot;\n&quot;)<br><br># Find the position of the substring &#39;r&#39; in each string<br>find_r = data.str.find(&#39;r&#39;)<br>print(&quot;Position of &#39;r&#39;:&quot;)<br>print(find_r)<br>print(&quot;\n&quot;)<br><br># Check if each string starts with the letter &#39;b&#39;<br>starts_with_b = data.str.startswith(&#39;b&#39;)<br>print(&quot;Starts with &#39;b&#39;:&quot;)<br>print(starts_with_b)<br>print(&quot;\n&quot;)<br><br># Check if each string ends with the letter &#39;y&#39;<br>ends_with_y = data.str.endswith(&#39;y&#39;)<br>print(&quot;Ends with &#39;y&#39;:&quot;)<br>print(ends_with_y)<br>print(&quot;\n&quot;)<br><br># Split each string by the letter &#39;e&#39; and expand into a DataFrame<br>split_by_e = data.str.split(&#39;e&#39;, expand=True)<br>print(&quot;Split by &#39;e&#39;:&quot;)<br>print(split_by_e)<br>print(&quot;\n&quot;)<br><br># Capitalize the first letter of each string<br>capitalize_data = data.str.capitalize()<br>print(&quot;Capitalized Strings:&quot;)<br>print(capitalize_data)<br>print(&quot;\n&quot;)<br><br>o/p<br>Uppercase Strings:<br>0         APPLE<br>1        BANANA<br>2        CHERRY<br>3          DATE<br>4    ELDERBERRY<br>dtype: object<br><br><br>Lowercase Strings:<br>0         apple<br>1        banana<br>2        cherry<br>3          date<br>4    elderberry<br>dtype: object<br><br><br>Contains &#39;an&#39;:<br>0    False<br>1     True<br>2    False<br>3    False<br>4    False<br>dtype: bool<br><br><br>Replace &#39;e&#39; with &#39;E&#39;:<br>0         applE<br>1        banana<br>2        chErry<br>3          datE<br>4    EldErbErry<br>dtype: object<br><br><br>Lengths of Strings:<br>0     5<br>1     6<br>2     6<br>3     4<br>4    10<br>dtype: int64<br><br><br>First Three Characters:<br>0    app<br>1    ban<br>2    che<br>3    dat<br>4    eld<br>dtype: object<br><br><br>Position of &#39;r&#39;:<br>0   -1<br>1   -1<br>2    3<br>3   -1<br>4    4<br>dtype: int64<br><br><br>Starts with &#39;b&#39;:<br>0    False<br>1     True<br>2    False<br>3    False<br>4    False<br>dtype: bool<br><br><br>Ends with &#39;y&#39;:<br>0    False<br>1    False<br>2     True<br>3    False<br>4     True<br>dtype: bool<br><br><br>Split by &#39;e&#39;:<br>        0     1     2     3<br>0    appl        None  None<br>1  banana  None  None  None<br>2      ch   rry  None  None<br>3     dat        None  None<br>4            ld    rb   rry<br><br><br>Capitalized Strings:<br>0         Apple<br>1        Banana<br>2        Cherry<br>3          Date<br>4    Elderberry<br>dtype: object</pre><p>To conclude the discussion on the Pandas Series data structure, here are some key points:</p><ol><li><strong>Definition</strong>: A Pandas Series is a one-dimensional array-like object that can hold data of any type (integers, floats, strings, etc.). It is similar to a list or an array in Python but comes with additional features, such as indexing.</li><li><strong>Indexing</strong>: Each element in a Series is associated with an index label, which can be either the default integer index or a custom index provided by the user. This allows for easy access and manipulation of data using index labels.</li><li><strong>Operations</strong>: Series supports a wide range of operations, including mathematical operations, string operations, and data alignment. You can perform operations on Series elements just like with NumPy arrays, and the operations are often vectorized for efficiency.</li><li><strong>Flexibility</strong>: Series are flexible in terms of data storage and can handle missing data with NaN (Not a Number) values. They also provide powerful capabilities for handling time series data, making them ideal for data analysis tasks.</li><li><strong>Usage</strong>: Series are typically used for storing and manipulating single-column data, such as time series, sensor readings, or any one-dimensional data. They are also a building block for DataFrames, which are two-dimensional and composed of multiple Series.</li></ol><p>Overall, the Pandas Series is a fundamental and versatile data structure in the Pandas library, widely used for data manipulation and analysis in Python.</p><p><strong>Thank you for Reading</strong></p><p>If you like this post:</p><ul><li>Please show your support by following and with a clap 👏 or several claps!</li><li>Feel free to share this guide with your friends.</li><li>Your feedback is invaluable — it inspires and guides my future posts.</li><li>Or drop me a message: <a href="https://www.linkedin.com/in/ajaykumar-dev/">https://www.linkedin.com/in/ajaykumar-dev/</a></li></ul><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=12c19710d875" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Data Science: Predicting Bajaj Finance Stock Prices Using Linear Regression in Python.]]></title>
            <link>https://medium.com/@nikaljeajay36/data-science-predicting-bajaj-finance-stock-prices-using-linear-regression-in-python-d6a7b9336db9?source=rss-ea4e86d061b1------2</link>
            <guid isPermaLink="false">https://medium.com/p/d6a7b9336db9</guid>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[scatter-plots]]></category>
            <category><![CDATA[linear-regression]]></category>
            <dc:creator><![CDATA[Ajaykumar Dev]]></dc:creator>
            <pubDate>Sun, 23 Jun 2024 04:01:47 GMT</pubDate>
            <atom:updated>2024-06-23T04:01:47.790Z</atom:updated>
            <content:encoded><![CDATA[<p>In the world of finance, predicting stock prices is a challenging yet fascinating task. With the advent of machine learning, we can leverage historical data to make informed predictions about future stock prices. In this blog post, we’ll walk through the process of using Linear Regression to predict the stock prices of Bajaj Finance, a prominent financial services company in India, using Python and the yfinance library.</p><p><strong>Disclaimer:</strong> This blog is intended for educational purposes only. The provided code should not be used for actual stock market price prediction. We do not assume any responsibility for any financial losses incurred as a result of using this code for trading or investment purposes.</p><h3>Step-by-Step Guide</h3><h3>1. Importing Necessary Libraries</h3><p>First, we need to import the necessary libraries for data manipulation, model creation, evaluation, and visualization.</p><pre>import yfinance as yf<br>import pandas as pd<br>import numpy as np<br>from sklearn.linear_model import LinearRegression<br>from sklearn.model_selection import train_test_split<br>from sklearn.metrics import mean_absolute_error, mean_squared_error, r2_score<br>import matplotlib.pyplot as plt<br>from datetime import datetime, timedelta</pre><h3>2. Fetching Historical Stock Data</h3><p>We use the yfinance library to fetch historical stock data for Bajaj Finance. We&#39;ll retrieve data for the past 60 days.</p><pre># Fetch data for Bajaj Finance<br>ticker = &#39;BAJFINANCE.NS&#39;<br>end_date = datetime.now()<br>start_date = end_date - timedelta(days=60)<br>data = yf.download(ticker, start=start_date, end=end_date)<br><br>data[&#39;Date&#39;] = data.index<br>data = data[[&#39;Date&#39;, &#39;Open&#39;, &#39;High&#39;, &#39;Low&#39;, &#39;Close&#39;, &#39;Volume&#39;]]<br>data.reset_index(drop=True, inplace=True)</pre><h3>3. Preparing the Data</h3><p>Next, we prepare the data for modeling. We extract the day of the month from the date and use it as one of the features. The features we use are ‘Open’, ‘High’, ‘Low’, ‘Volume’, and ‘Day’. The target variable is the ‘Close’ price.</p><pre># Prepare the data<br>data[&#39;Day&#39;] = data[&#39;Date&#39;].dt.day<br>features = [&#39;Open&#39;, &#39;High&#39;, &#39;Low&#39;, &#39;Volume&#39;, &#39;Day&#39;]<br>X = data[features]<br>y = data[&#39;Close&#39;]</pre><h3>4. Splitting the Data</h3><p>We split the data into training and testing sets using an 80–20 split. This allows us to train the model on a portion of the data and evaluate its performance on unseen data.</p><pre># Split data into training and testing sets<br>X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2, random_state=42)</pre><h3>5. Training the Linear Regression Model</h3><p>We create and train a Linear Regression model using the training data.</p><pre># Train Linear Regression model<br>model = LinearRegression()<br>model.fit(X_train, y_train)</pre><h3>6. Making Predictions and Evaluating the Model</h3><p>We use the trained model to make predictions on the test set and evaluate its performance using Mean Absolute Error (MAE), Mean Squared Error (MSE), and R-squared (R2) score.</p><pre># Predict and evaluate<br>y_pred = model.predict(X_test)<br>mae = mean_absolute_error(y_test, y_pred)<br>mse = mean_squared_error(y_test, y_pred)<br>r2 = r2_score(y_test, y_pred)<br><br>print(f&quot;Mean Absolute Error: {mae}&quot;)<br>print(f&quot;Mean Squared Error: {mse}&quot;)<br>print(f&quot;R2 Score: {r2}&quot;)</pre><h3>7. Visualizing the Predictions</h3><p>We plot the actual and predicted stock prices to visualize the model’s performance.</p><pre># Plot the predictions<br>plt.figure(figsize=(12, 6))<br>plt.plot(data[&#39;Date&#39;], data[&#39;Close&#39;], label=&#39;Actual Prices&#39;)<br>plt.plot(data[&#39;Date&#39;][X_test.index], y_pred, &#39;ro&#39;, label=&#39;Predicted Prices&#39;)<br>plt.xlabel(&#39;Date&#39;)<br>plt.ylabel(&#39;Stock Price&#39;)<br>plt.title(&#39;Linear Regression Prediction of Bajaj Finance Stock Prices&#39;)<br>plt.legend()<br>plt.show()</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*AGnqUjPvScp5cOJ-9AsXyg.png" /><figcaption>Linear Regression</figcaption></figure><h3>8. Understanding Feature Importance</h3><p>Finally, we print the coefficients of the features to understand their impact on the predictions.</p><pre># Print feature coefficients<br>print(&quot;# Print feature coefficients&quot;)<br>for feature, coef in zip(features, model.coef_):<br>    print(f&quot;{feature}: {coef}&quot;)</pre><ol><li>Model Performance Metrics:</li></ol><ul><li>Mean Absolute Error (MAE): 26.65<br>This indicates that, on average, the model’s predictions are off by about 26.65 rupees.</li><li>Mean Squared Error (MSE): 1101.96<br>This measures the average squared difference between predicted and actual values. It penalizes larger errors more heavily.</li><li>R2 Score: 0.9717<br>This suggests that the model explains about 97.17% of the variance in the stock price. This is a very high score, indicating that the model fits the data well.</li></ul><p>2. Feature Coefficients:</p><ul><li>Open: -0.6729<br>A 1 rupee increase in the opening price is associated with a 0.6729 rupee decrease in the closing price, holding other variables constant.</li><li>High: 0.9851<br>A 1 rupee increase in the day’s high price is associated with a 0.9851 rupee increase in the closing price.</li><li>Low: 0.6932<br>A 1 rupee increase in the day’s low price is associated with a 0.6932 rupee increase in the closing price.</li><li>Volume: -0.00001726<br>An increase of 1 in trading volume is associated with a very small decrease in the closing price.</li><li>Day: 0.6688<br>Each day of the month is associated with a 0.6688 rupee increase in the closing price.</li></ul><p><strong>Thank you for Reading</strong></p><p>If you like this post:</p><ul><li>Please show your support by following and with a clap 👏 or several claps!</li><li>Feel free to share this guide with your friends.</li><li>Your feedback is invaluable — it inspires and guides my future posts.</li><li>Or drop me a message: <a href="https://www.linkedin.com/in/ajaykumar-dev/">https://www.linkedin.com/in/ajaykumar-dev/</a></li></ul><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=d6a7b9336db9" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Data Science: Understanding Random Forest machine learning model for Stock Price Prediction with…]]></title>
            <link>https://medium.com/@nikaljeajay36/data-science-understanding-random-forest-machine-learning-model-for-stock-price-prediction-with-88d1b2f93047?source=rss-ea4e86d061b1------2</link>
            <guid isPermaLink="false">https://medium.com/p/88d1b2f93047</guid>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[random-forest-classifiers]]></category>
            <category><![CDATA[scatter-plots]]></category>
            <dc:creator><![CDATA[Ajaykumar Dev]]></dc:creator>
            <pubDate>Sat, 22 Jun 2024 09:47:29 GMT</pubDate>
            <atom:updated>2024-06-22T09:47:29.230Z</atom:updated>
            <content:encoded><![CDATA[<h3>Data Science: Understanding Random Forest machine learning model for Stock Price Prediction with Bajaj Finance Stock Data.</h3><p><strong>Disclaimer:</strong> This blog is intended for educational purposes only. The provided code should not be used for actual stock market price prediction. We do not assume any responsibility for any financial losses incurred as a result of using this code for trading or investment purposes.</p><p>Predicting stock prices is a complex task due to the volatile nature of the stock market. Machine learning models, like Random Forest, offer a way to analyze historical data and make predictions. In this blog, we’ll delve into the Random Forest algorithm, explain how it works, and demonstrate its application in predicting stock prices.</p><h3>What is Random Forest?</h3><p>Random Forest is an ensemble learning method that constructs multiple decision trees during training and merges their outputs to improve predictive accuracy and control overfitting. It can be used for both classification and regression tasks.</p><h3>How Does Random Forest Work?</h3><ol><li><strong>Bootstrapping</strong>: Random Forest uses a technique called bootstrapping to create multiple subsets of the original data by sampling with replacement. Each subset is used to train a different decision tree.</li><li><strong>Feature Randomness</strong>: When splitting a node during the training of each decision tree, only a random subset of the features is considered. This ensures that the trees are not highly correlated.</li><li><strong>Voting/Averaging</strong>: For classification tasks, the final prediction is based on the majority vote of the decision trees. For regression tasks, the average prediction of all trees is taken.</li></ol><h3>Advantages of Random Forest</h3><ul><li><strong>Improved Accuracy</strong>: Combining multiple trees helps improve the overall prediction accuracy.</li><li><strong>Robustness</strong>: It is less likely to overfit compared to a single decision tree.</li><li><strong>Feature Importance</strong>: Random Forest provides an estimate of feature importance, which helps in understanding the influence of different features.</li></ul><h3>1. Data Preparation</h3><p>Imagine we have historical stock data for Bajaj Finance for the last month, including the following columns: Open, High, Low, Close, and Volume. Our goal is to predict the Close price based on the other features.</p><h3>2. Bootstrap Sampling</h3><p><strong>Bootstrap Sampling</strong> involves creating multiple subsets of the original dataset by sampling with replacement. For example, if we have 10 days of data, we might generate several subsets, each with a different combination of days, some possibly repeated.</p><h3>3. Training Multiple Trees</h3><p>Each bootstrap sample is used to train a separate decision tree. Here’s how it works for a single tree:</p><h4>Example Decision Tree</h4><p>Consider we have a bootstrap sample of 5 days from the historical data:</p><pre>Day Open High Low Volume Close<br>1 1000 1020 990 10000 1010<br>2 1015 1035 1005 11000 1025<br>3 1020 1040 1010 10500 1030<br>4 1030 1050 1020 11500 1040<br>5 1040 1060 1030 12000 1050</pre><p><strong>Tree Construction</strong>:</p><ul><li><strong>Root Node</strong>: The decision tree algorithm starts at the root node and evaluates all possible splits. For example, it might consider splitting based on the Open price. The best split is chosen based on a criterion such as minimizing variance for regression.</li><li><strong>Split Decision</strong>: Suppose the best split is Open &lt; 1025. This creates two child nodes.</li><li>Left Node: Contains data where Open &lt; 1025 (Days 1 and 2).</li><li>Right Node: Contains data where Open &gt;= 1025 (Days 3, 4, and 5).</li><li><strong>Further Splits</strong>: The algorithm continues to split each node based on other features (High, Low, Volume, etc.) until stopping criteria are met (e.g., maximum depth or minimum samples per leaf).</li></ul><pre>Open &lt; 1025<br>├── Left Node (Day 1, Day 2)<br>│   ├── Close = 1010 (avg of Close prices)<br>├── Right Node (Day 3, Day 4, Day 5)<br>    ├── High &lt; 1050<br>    │   ├── Close = 1030 (avg of Close prices for Day 3)<br>    ├── High &gt;= 1050<br>        ├── Close = 1045 (avg of Close prices for Days 4 and 5)</pre><h3>4. Aggregation of Predictions</h3><p>Once all decision trees are trained, predictions are made using each tree. For regression tasks like stock price prediction, the final output is the average of all tree predictions.</p><h4>Example Prediction</h4><p>Let’s predict the Close price for a new day with features:</p><ul><li>Open: 1035</li><li>High: 1055</li><li>Low: 1025</li><li>Volume: 11000</li></ul><p>Each decision tree in the forest makes a prediction based on its structure. Suppose we have 3 trees with the following predictions:</p><ul><li>Tree 1: 1040</li><li>Tree 2: 1035</li><li>Tree 3: 1045</li></ul><p>The final prediction is the average of these predictions:</p><p>Final Prediction= ( 1040+1035+1045 ) / 3​=1040 .</p><h3><strong>Now, we can go actual code,</strong></h3><p>Step 1: Import Necessary Libraries</p><pre>import yfinance as yf<br>import pandas as pd<br>import numpy as np<br>from sklearn.ensemble import RandomForestRegressor<br>from sklearn.model_selection import train_test_split<br>from sklearn.metrics import mean_absolute_error, mean_squared_error, r2_score<br>import matplotlib.pyplot as plt</pre><p>Step 2: Fetch Historical Stock Data</p><pre># Fetch data for Bajaj Finance<br>ticker = &#39;BAJFINANCE.NS&#39;<br>data = yf.download(ticker, start=&#39;2023-05-01&#39;, end=&#39;2023-06-01&#39;)<br>data[&#39;Date&#39;] = data.index<br>data = data[[&#39;Date&#39;, &#39;Open&#39;, &#39;High&#39;, &#39;Low&#39;, &#39;Close&#39;, &#39;Volume&#39;]]<br>data.reset_index(drop=True, inplace=True)</pre><p>Step 3: Preprocess the Data</p><pre># Prepare the data<br>data[&#39;Day&#39;] = data[&#39;Date&#39;].dt.day<br>features = [&#39;Open&#39;, &#39;High&#39;, &#39;Low&#39;, &#39;Volume&#39;, &#39;Day&#39;]<br>X = data[features]<br>y = data[&#39;Close&#39;]<br><br># Split data into training and testing sets<br>X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2, random_state=42)</pre><p>Step 4: Train the Random Forest Model</p><pre># Train Random Forest Regressor<br>model = RandomForestRegressor(n_estimators=100, random_state=42)<br>model.fit(X_train, y_train)</pre><p>Step 5: Evaluate the Model.</p><pre># Predict and evaluate<br>y_pred = model.predict(X_test)<br>mae = mean_absolute_error(y_test, y_pred)<br>mse = mean_squared_error(y_test, y_pred)<br>r2 = r2_score(y_test, y_pred)<br><br>print(f&quot;Mean Absolute Error: {mae}&quot;)<br>print(f&quot;Mean Squared Error: {mse}&quot;)<br>print(f&quot;R2 Score: {r2}&quot;)</pre><p>Step 6: Plot the Predictions.</p><pre># Plot the predictions<br>plt.figure(figsize=(12, 6))<br>plt.plot(data[&#39;Date&#39;], data[&#39;Close&#39;], label=&#39;Actual Prices&#39;)<br>plt.plot(data[&#39;Date&#39;][X_test.index], y_pred, &#39;ro&#39;, label=&#39;Predicted Prices&#39;)<br>plt.xlabel(&#39;Date&#39;)<br>plt.ylabel(&#39;Stock Price&#39;)<br>plt.title(&#39;Random Forest Prediction of Bajaj Finance Stock Prices&#39;)<br>plt.legend()<br>plt.show()</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*K-97s3iVTDeEch0gRy3FFg.png" /><figcaption>Plot using Random Forest Machine Learning Model</figcaption></figure><ul><li><strong>MAE</strong>: On average, predictions are off by about 84.21 units, indicating the model’s average prediction error.</li><li><strong>MSE</strong>: The squared error average is 21066.71, which reflects larger errors more significantly and suggests there is room for improvement in model accuracy.</li><li><strong>R² Score</strong>: With a score of 0.7843, the model explains 78.43% of the variance in stock prices, indicating a strong fit but not perfect.</li></ul><h3>Conclusion</h3><p>Random Forest is a powerful algorithm for predicting stock prices due to its robustness and ability to handle complex, non-linear relationships in data. While it can provide valuable insights, remember that stock price prediction is inherently uncertain, and no model can guarantee accurate forecasts. Always use such models with caution and consider their limitations.</p><p><strong>Thank you for Reading</strong></p><p>If you like this post:</p><ul><li>Please show your support by following and with a clap 👏 or several claps!</li><li>Feel free to share this guide with your friends.</li><li>Your feedback is invaluable — it inspires and guides my future posts.</li><li>Or drop me a message: <a href="https://www.linkedin.com/in/ajaykumar-dev/">https://www.linkedin.com/in/ajaykumar-dev/</a></li></ul><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=88d1b2f93047" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Data Science: Understanding Decision Tree Machine Learning model with Bajaj Finance Stock Data.]]></title>
            <link>https://medium.com/@nikaljeajay36/understanding-decision-tree-machine-learning-model-with-bajaj-finance-stock-data-8098d8dcf7a3?source=rss-ea4e86d061b1------2</link>
            <guid isPermaLink="false">https://medium.com/p/8098d8dcf7a3</guid>
            <category><![CDATA[decision-tree-classifier]]></category>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[scatter-plots]]></category>
            <dc:creator><![CDATA[Ajaykumar Dev]]></dc:creator>
            <pubDate>Sat, 22 Jun 2024 08:24:08 GMT</pubDate>
            <atom:updated>2024-06-22T08:25:32.474Z</atom:updated>
            <content:encoded><![CDATA[<p>Investing in the stock market requires analyzing historical data to predict future trends. Machine learning models, like decision trees, can help make these predictions more precise. In this blog, we’ll explore how decision trees work using Bajaj Finance stock data as our case study.</p><p><strong>Disclaimer:</strong> This blog is intended for educational purposes only. The provided code should not be used for actual stock market price prediction. We do not assume any responsibility for any financial losses incurred as a result of using this code for trading or investment purposes.</p><p>Now, Let’s first understand how Decision Tree Works,</p><p>Dataset is as below,</p><pre>Customer Age Income Bought (Target)<br>1 25 50000 Yes<br>2 30 70000 No<br>3 35 90000 Yes<br>4 20 40000 No<br>5 40 60000 Yes<br>6 45 80000 Yes<br>7 50 100000 No</pre><h4>1. Starting with the Root Node</h4><ul><li><strong>Objective</strong>: Predict whether a customer will buy the product (Bought = Yes/No).</li><li><strong>Root Node</strong>: Initially, the entire dataset is the root node.</li><li><strong>Decision</strong>: Choose the feature that best splits the data into subsets that are most homogenous with respect to the target variable (Bought).</li></ul><h4>2. Choosing Splitting Criteria</h4><ul><li><strong>Impurity Measure</strong>: To decide which feature and value to split on, decision trees use impurity measures like Gini impurity or entropy.</li><li><strong>Example</strong>: Calculate Gini impurity for age and income to find the best split.</li></ul><h4>3. Building the Tree</h4><ul><li><strong>Recursive Process</strong>: The tree recursively splits the dataset into subsets based on the chosen feature and threshold.</li><li><strong>Decision</strong>: Determine the best feature and threshold that minimizes impurity (e.g., Gini impurity).</li></ul><h4>4. Leaf Nodes and Predictions</h4><ul><li><strong>Stopping Criteria</strong>: Stop splitting further based on conditions like maximum depth, minimum samples per leaf, or when no further information gain is possible.</li><li><strong>Leaf Nodes</strong>: Terminal nodes where predictions are made. Each leaf node represents a prediction for the target variable (Bought = Yes/No).</li></ul><h3>Example Decision Tree Construction</h3><p>Let’s construct a decision tree step-by-step for our example dataset:</p><h4>Step 1: Root Node</h4><ul><li>Calculate Gini impurity or entropy for each feature (age and income).</li><li>Suppose age &lt; 30 provides the best split based on Gini impurity.</li></ul><h4>Step 2: Splitting the Dataset</h4><p><strong>Left Subtree (Age &lt; 30)</strong>:</p><p>Further split based on income:</p><p><strong>Income &lt; 60000</strong>: Predict ‘Yes’ (Buy)</p><p><strong>Income &gt;= 60000</strong>: Predict ‘No’ (Don’t Buy)</p><p><strong>Right Subtree (Age &gt;= 30)</strong>:</p><p>Further split based on income:</p><p><strong>Income &lt; 80000</strong>: Predict ‘Yes’ (Buy)</p><p><strong>Income &gt;= 80000</strong>: Predict ‘No’ (Don’t Buy)</p><h4>Why Use Decision Trees?</h4><ol><li><strong>Interpretability</strong>: Decision trees are easy to interpret and visualize.</li><li><strong>Non-Linear Relationships</strong>: They capture non-linear relationships between features and the target variable.</li><li><strong>Minimal Data Preparation</strong>: Requires less data preprocessing compared to other models.</li></ol><h4>Preparing the Data</h4><p>To start, we’ll fetch the last month’s stock data for Bajaj Finance using the yfinance library.</p><pre>import yfinance as yf<br>import pandas as pd<br>import numpy as np<br>import matplotlib.pyplot as plt<br>from sklearn.model_selection import train_test_split<br>from sklearn.tree import DecisionTreeRegressor<br>from sklearn.metrics import mean_absolute_error, mean_squared_error, r2_score<br>from datetime import datetime, timedelta<br><br># Fetch stock data for Bajaj Finance for the last month<br>end_date = datetime.now()<br>start_date = end_date - timedelta(days=30)<br>ticker = &#39;BAJFINANCE.NS&#39;<br>stock_data = yf.download(ticker, start=start_date, end=end_date)<br><br># Prepare the data<br>stock_data[&#39;Date&#39;] = stock_data.index<br>stock_data[&#39;Date&#39;] = stock_data[&#39;Date&#39;].dt.strftime(&#39;%d-%b-%Y&#39;)<br>features = np.arange(len(stock_data)).reshape(-1, 1)<br>X = features<br>y = stock_data[&#39;Close&#39;].values<br><br># Split the data<br>X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2, random_state=42)</pre><h4>Training the Decision Tree Model</h4><p>Next, we’ll train a decision tree regressor using the prepared data.</p><pre># Initialize and train the Decision Tree model<br>model = DecisionTreeRegressor()<br>model.fit(X_train, y_train)<br><br># Predict on the test set<br>y_pred = model.predict(X_test)</pre><h4>Evaluating the Model</h4><p>To understand how well our model performs, we will calculate the Mean Absolute Error (MAE), Mean Squared Error (MSE), and R-squared score (R²).</p><pre># Calculate evaluation metrics<br>mae = mean_absolute_error(y_test, y_pred)<br>mse = mean_squared_error(y_test, y_pred)<br>r2 = r2_score(y_test, y_pred)<br><br>print(f&quot;Mean Absolute Error: {mae}&quot;)<br>print(f&quot;Mean Squared Error: {mse}&quot;)<br>print(f&quot;R2 Score: {r2}&quot;)</pre><h4>Visualizing the Results</h4><p>Finally, let’s visualize the actual vs. predicted stock prices.</p><pre># Plotting the results<br>plt.figure(figsize=(10, 6))<br>plt.plot(stock_data[&#39;Date&#39;], y, label=&#39;Actual Prices&#39;)<br>plt.scatter(stock_data[&#39;Date&#39;].iloc[X_test.flatten()], y_test, color=&#39;red&#39;, label=&#39;Test Data&#39;)<br>plt.scatter(stock_data[&#39;Date&#39;].iloc[X_test.flatten()], y_pred, color=&#39;green&#39;, label=&#39;Predicted Prices&#39;)<br>plt.xticks(rotation=45)<br>plt.title(&quot;Decision Tree - Actual vs Predicted Stock Prices&quot;)<br>plt.xlabel(&#39;Date&#39;)<br>plt.ylabel(&#39;Stock Price&#39;)<br>plt.legend()<br>plt.tight_layout()<br>plt.show()<br><br>O/P<br>[*********************100%%**********************]  1 of 1 completed<br>Mean Absolute Error: 123.5<br>Mean Squared Error: 35803.41893949509<br>R2 Score: 0.6334858596212294</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*RF8x8Wcnf2FFXt80Z1BxYw.png" /></figure><ul><li><strong>MAE (123.5)</strong>: Indicates moderate average error.</li><li><strong>MSE (35803.42)</strong>: Suggests there are some large errors.</li><li><strong>R² (0.63)</strong>: Shows the model explains 63% of the variance, which is decent but could be improved.</li></ul><h4>Conclusion</h4><p>In this blog, we demonstrated how to use a decision tree regressor to predict stock prices using historical data from Bajaj Finance. While decision trees offer a straightforward and interpretable approach, it’s important to consider their limitations, such as overfitting, which can be mitigated by techniques like pruning or using ensemble methods like Random Forests.</p><p>By continuously evaluating and refining our models, we can improve the accuracy of our predictions, making more informed investment decisions. Remember, while machine learning models can provide valuable insights, they should be used in conjunction with other analysis methods and domain knowledge for best results.</p><p><strong>Thank you for Reading</strong></p><p>If you like this post:</p><ul><li>Please show your support by following and with a clap 👏 or several claps!</li><li>Feel free to share this guide with your friends.</li><li>Your feedback is invaluable — it inspires and guides my future posts.</li><li>Or drop me a message: <a href="https://www.linkedin.com/in/ajaykumar-dev/">https://www.linkedin.com/in/ajaykumar-dev/</a></li></ul><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=8098d8dcf7a3" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Data Science: Guide to Encoding Nominal Categorical Features.]]></title>
            <link>https://medium.com/@nikaljeajay36/data-science-guide-to-encoding-nominal-categorical-features-bf3e622b1133?source=rss-ea4e86d061b1------2</link>
            <guid isPermaLink="false">https://medium.com/p/bf3e622b1133</guid>
            <category><![CDATA[encoding]]></category>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[data-science]]></category>
            <dc:creator><![CDATA[Ajaykumar Dev]]></dc:creator>
            <pubDate>Sat, 22 Jun 2024 01:51:56 GMT</pubDate>
            <atom:updated>2024-06-22T01:51:56.499Z</atom:updated>
            <content:encoded><![CDATA[<p>In the world of machine learning, data preprocessing is a critical step that can significantly impact the performance of your models. One common preprocessing task is encoding categorical features, especially nominal categorical features. Nominal features are those where the categories have no inherent order or ranking, such as colors, types of animals, or brands. In this blog post, we’ll explore various methods for encoding nominal categorical features and discuss their pros and cons.</p><h3>Understanding Nominal and Ordinal Data</h3><p>Before diving into encoding techniques, it’s essential to understand the difference between nominal and ordinal data.</p><h3>Nominal Data</h3><p>Nominal data consists of categories that have no inherent order. Examples include colors (red, blue, green), types of animals (cat, dog, bird), or brands (Nike, Adidas, Puma). These categories are merely labels and cannot be logically ordered or ranked.</p><h3>Ordinal Data</h3><p>Ordinal data, on the other hand, consists of categories that have a specific order or ranking. Examples include survey responses (very dissatisfied, dissatisfied, neutral, satisfied, very satisfied) or education levels (high school, bachelor’s, master’s, PhD). While the order is meaningful, the intervals between the categories are not necessarily equal.</p><h3>Why Encode Categorical Features?</h3><p>Machine learning algorithms typically require numerical input, so categorical data must be converted into a numerical format. Proper encoding ensures that the model can interpret and learn from the data effectively. Let’s dive into some popular encoding techniques.</p><h3>One-Hot Encoding</h3><p>One-hot encoding is one of the most widely used methods for encoding nominal data. It creates a new binary column for each category in the original feature. Each row will have a 1 in the column corresponding to its category and 0s in all other columns.</p><pre>import pandas as pd<br>from sklearn.preprocessing import OneHotEncoder<br><br># Sample data<br>data = {&#39;Color&#39;: [&#39;Red&#39;, &#39;Blue&#39;, &#39;Green&#39;]}<br>df = pd.DataFrame(data)<br><br># One-Hot Encoding<br>encoder = OneHotEncoder(sparse_output=False)<br>encoded_data = encoder.fit_transform(df[[&#39;Color&#39;]])<br><br># Convert to DataFrame for better readability<br>encoded_df = pd.DataFrame(encoded_data, columns=encoder.get_feature_names_out([&#39;Color&#39;]))<br>print(encoded_df)<br><br>O/P<br>   Color_Blue  Color_Green  Color_Red<br>0         0.0          0.0        1.0<br>1         1.0          0.0        0.0<br>2         0.0          1.0        0.0<br><br>Process finished with exit code 0</pre><p>Pros:</p><ul><li>Simple and easy to implement.</li><li>Preserves the categorical nature of the data.</li></ul><p>Cons:</p><ul><li>Can lead to high dimensionality if the feature has many categories.</li></ul><h3>Label Encoding</h3><p>Label encoding assigns a unique integer to each category. However, this method is generally not recommended for nominal data because it introduces an arbitrary ordinal relationship between categories.</p><pre>from sklearn.preprocessing import LabelEncoder<br>import pandas as pd<br># Sample data<br>data = {&#39;Color&#39;: [&#39;Red&#39;, &#39;Blue&#39;, &#39;Green&#39;]}<br>df = pd.DataFrame(data)<br><br># Label Encoding<br>label_encoder = LabelEncoder()<br>df[&#39;Color_Encoded&#39;] = label_encoder.fit_transform(df[&#39;Color&#39;])<br>print(df)<br><br>O/P<br>   Color  Color_Encoded<br>0    Red              2<br>1   Blue              0<br>2  Green              1</pre><p>Pros:</p><ul><li>Simple and easy to implement.</li><li>Does not increase dimensionality.</li></ul><p>Cons:</p><ul><li>Introduces an ordinal relationship that may not exist.</li></ul><h3>Binary Encoding</h3><p>Binary encoding converts categories into binary numbers and then splits these binary numbers into separate columns. This method is useful for high cardinality features as it reduces the dimensionality compared to one-hot encoding.</p><pre>import category_encoders as ce<br>import pandas as pd<br><br># Sample data<br>data = {&#39;Color&#39;: [&#39;Red&#39;, &#39;Blue&#39;, &#39;Green&#39;]}<br>df = pd.DataFrame(data)<br><br># Binary Encoding<br>binary_encoder = ce.BinaryEncoder(cols=[&#39;Color&#39;])<br>df_encoded = binary_encoder.fit_transform(df)<br>print(df_encoded)<br><br>O/P<br>   Color_0  Color_1<br>0        0        1<br>1        1        0<br>2        1        1<br><br>Process finished with exit code 0</pre><p>Pros:</p><ul><li>Reduces dimensionality.</li><li>Suitable for high cardinality features.</li></ul><p>Cons:</p><ul><li>More complex to implement.</li><li>May not be as interpretable as one-hot encoding.</li></ul><h3>Feature Hashing</h3><p>Feature hashing, also known as the hashing trick, uses a hash function to convert categories into a fixed number of columns. This method is efficient when dealing with a large number of categories.</p><pre>from sklearn.feature_extraction import FeatureHasher<br><br># Sample data<br>data = [{&#39;Color&#39;: &#39;Red&#39;}, {&#39;Color&#39;: &#39;Blue&#39;}, {&#39;Color&#39;: &#39;Green&#39;}]<br><br># Feature Hashing<br>hasher = FeatureHasher(input_type=&#39;dict&#39;, n_features=3)<br>hashed_features = hasher.transform(data)<br>print(hashed_features.toarray())<br><br>O/P<br>[[-1.  0.  0.]<br> [ 0. -1.  0.]<br> [ 0.  0. -1.]]<br><br>Process finished with exit code 0</pre><p>Pros:</p><ul><li>Efficient for large datasets.</li><li>Fixed number of output columns.</li></ul><p>Cons:</p><ul><li>Can lead to hash collisions, where different categories are mapped to the same column.</li><li>Less interpretable.</li></ul><h3>Conclusion</h3><p>Each encoding technique has its strengths and is best suited for specific scenarios. One-hot encoding is generally preferred for nominal data due to its simplicity and effectiveness in preserving the categorical nature of the data without introducing any ordinal relationships. However, for high cardinality features, binary encoding or feature hashing might be more appropriate.Choosing the right encoding method depends on the nature of your data and the requirements of your machine learning model. By understanding the pros and cons of each method, you can make an informed decision and improve the performance of your models.</p><p><strong>Thank you for Reading</strong></p><p>If you like this post:</p><ul><li>Please show your support by following and with a clap 👏 or several claps!</li><li>Feel free to share this guide with your friends.</li><li>Your feedback is invaluable — it inspires and guides my future posts.</li><li>Or drop me a message: <a href="https://www.linkedin.com/in/ajaykumar-dev/">https://www.linkedin.com/in/ajaykumar-dev/</a></li></ul><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=bf3e622b1133" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Data Science: Understanding Feature Scaling in Machine Learning]]></title>
            <link>https://medium.com/@nikaljeajay36/data-science-understanding-feature-scaling-in-machine-learning-6b290f76668f?source=rss-ea4e86d061b1------2</link>
            <guid isPermaLink="false">https://medium.com/p/6b290f76668f</guid>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[data-science]]></category>
            <dc:creator><![CDATA[Ajaykumar Dev]]></dc:creator>
            <pubDate>Sat, 22 Jun 2024 01:20:40 GMT</pubDate>
            <atom:updated>2024-06-22T01:20:40.316Z</atom:updated>
            <content:encoded><![CDATA[<p>Feature scaling is a crucial preprocessing step in machine learning that involves normalizing the range of independent variables or features in a dataset. This process ensures that all features contribute equally to the learning process and prevents features with larger scales from dominating the model’s performance. In this blog, we will explore the importance of feature scaling, the different methods available, and provide practical examples to illustrate each method.</p><h3>Why Feature Scaling is Important</h3><p>Machine learning models often rely on distance calculations and optimization algorithms that are sensitive to the scale of the data. Without feature scaling, features with larger ranges can disproportionately influence the model, leading to biased results. Here are some key reasons why feature scaling is essential:</p><ol><li>Improved Algorithm Performance: Algorithms like K-Nearest Neighbors (KNN), K-means clustering, and Support Vector Machines (SVM) use distance metrics that are affected by the scale of the features. Scaling ensures that all features contribute equally to the distance calculations.</li><li>Faster Convergence: Gradient descent-based algorithms, such as linear regression and neural networks, converge more quickly when features are scaled. This is because scaling helps maintain consistent step sizes for all features during optimization.</li><li>Numerical Stability: Scaling prevents numerical instability in calculations, such as matrix operations, by avoiding significant scale disparities between features.</li><li>Equal Contribution: Ensures that each feature contributes equally to the learning process, preventing features with larger scales from dominating the model’s performance.</li></ol><h3>Common Methods of Feature Scaling</h3><h3>1. Normalization (Min-Max Scaling)</h3><p>Normalization rescales the range of features to a fixed range, typically [0, 1].</p><p>Formula:<br>𝑥′=𝑥−min⁡(𝑥)/ max⁡(𝑥)−min⁡(𝑥)​</p><p>Example:<br>Suppose we have a feature with the following values: [10, 20, 30, 40, 50].</p><ul><li>Minimum value (min): 10</li><li>Maximum value (max): 50</li></ul><p>Normalized Values:</p><ul><li>For 10: 𝑥′=10−10/50−10=0</li><li>For 20: 𝑥′=20−10/50−10=0.25</li><li>For 30: 𝑥′=30−10/50−10=0.5</li><li>For 40: 𝑥′=40−10/50−10=0.75</li><li>For 50: 𝑥′=50−10/50−10=1</li></ul><pre>from sklearn.preprocessing import MinMaxScaler<br>import numpy as np<br><br># Original data<br>data = np.array([10, 20, 30, 40, 50]).reshape(-1, 1)<br><br># Initialize the MinMaxScaler<br>scaler = MinMaxScaler()<br><br># Fit the scaler to the data and transform it<br>scaled_data = scaler.fit_transform(data)<br><br>print(&quot;Original data:&quot;)<br>print(data.flatten())<br>print(&quot;\nScaled data:&quot;)<br>print(scaled_data.flatten())<br><br>O/P<br>Original data:<br>[10 20 30 40 50]<br><br>Scaled data:<br>[0.   0.25 0.5  0.75 1.  ]<br><br>Process finished with exit code 0</pre><h3>2. Standardization (Z-score Normalization)</h3><p>Standardization transforms the data to have a mean of zero and a standard deviation of one.</p><p>Formula:<br>𝑥′=(𝑥−𝑥ˉ ) / 𝜎</p><p>ˉ​Example:<br>Suppose we have a feature with the following values: [10, 20, 30, 40, 50].</p><ul><li>Mean (μ): 30</li><li>Standard deviation (σ): 15.81 (approx)</li></ul><p>Standardized Values:</p><ul><li>For 10: 𝑥′=10−30 / 15.81=−1.26</li><li>For 20: 𝑥′=20−30 / 15.81=−0.63</li><li>For 30: 𝑥′=30−30 / 15.81=0</li><li>For 40: 𝑥′=40−30 / 15.81=0.63</li><li>For 50: 𝑥′=50−30 / 15.81=1.26</li></ul><pre>from sklearn.preprocessing import StandardScaler<br>import numpy as np<br><br># Sample data<br>data = np.array([10, 20, 30, 40, 50]).reshape(-1, 1)<br><br># Initialize the StandardScaler<br>scaler = StandardScaler()<br><br># Fit the scaler to the data and transform it<br>data_standardized = scaler.fit_transform(data)<br><br>print(&quot;Original data:&quot;)<br>print(data.flatten())<br>print(&quot;\nStandardized data:&quot;)<br>print(data_standardized.flatten())<br><br>O/P<br>Original data:<br>[10 20 30 40 50]<br><br>Standardized data:<br>[-1.41421356 -0.70710678  0. 0.70710678  1.41421356]<br><br>Process finished with exit code 0</pre><h3>3. Scaling to Unit Length</h3><p>This method scales the components of a feature vector such that the entire vector has a length of one.</p><p>Formula:<br>𝑥′=𝑥 / ||x||</p><p>​Example:<br>Suppose we have a feature vector: [3, 4].</p><ul><li>Euclidean length (||x||): 5</li></ul><p>Scaled Values:</p><ul><li>For 3: 𝑥′=3/5=0.6</li><li>For 4: 𝑥′=4/5=0.8</li></ul><pre>from sklearn.preprocessing import normalize<br>import numpy as np<br><br># Sample data<br>data = np.array([10, 20, 30, 40, 50]).reshape(1, -1)<br><br># Normalize the data to unit length using the Euclidean norm<br>data_scaled = normalize(data, norm=&#39;l2&#39;)<br><br>print(&quot;Original data:&quot;)<br>print(data.flatten())<br>print(&quot;\nScaled data:&quot;)<br>print(data_scaled.flatten())<br><br>O/P<br>Original data:<br>[10 20 30 40 50]<br><br>Scaled data:<br>[0.13483997 0.26967994 0.40451992 0.53935989 0.67419986]<br><br>Process finished with exit code 0</pre><h3>4. Robust Scaling</h3><p>Robust scaling uses the median and the interquartile range for scaling, making it less sensitive to outliers.</p><p>Formula:<br>𝑥′=𝑥−median(𝑥) / IQR(𝑥)</p><p>Example:<br>Suppose we have a feature with the following values: [10, 20, 30, 40, 50].</p><ul><li>Median: 30</li><li>Interquartile Range (IQR): 20 (Q3 — Q1, where Q3 = 40 and Q1 = 20)</li></ul><p>Robust Scaled Values:</p><ul><li>For 10: 𝑥′=10−30/20=−1</li><li>For 20: 𝑥′=20−30/20=−0.5</li><li>For 30: 𝑥′=30−30/20=0</li><li>For 40: 𝑥′=40−30/20=0.5</li><li>For 50: 𝑥′=50−30/20=1</li></ul><pre>from sklearn.preprocessing import RobustScaler<br>import numpy as np<br><br># Sample data<br>data = np.array([10, 20, 30, 40, 50]).reshape(-1, 1)<br><br># Initialize the RobustScaler<br>scaler = RobustScaler()<br><br># Fit the scaler to the data and transform it<br>data_scaled = scaler.fit_transform(data)<br><br>print(&quot;Original data:&quot;)<br>print(data.flatten())<br>print(&quot;\nRobust scaled data:&quot;)<br>print(data_scaled.flatten())<br><br>O/P<br>Original data:<br>[10 20 30 40 50]<br><br>Robust scaled data:<br>[-1.  -0.5  0.   0.5  1. ]<br><br>Process finished with exit code 0</pre><h3>When to Perform Feature Scaling</h3><p>Feature scaling should be performed before applying machine learning algorithms that are sensitive to the scale of data, such as:</p><ul><li>Gradient Descent-Based Algorithms: Ensures consistent step sizes for all features, leading to faster convergence.</li><li>Distance-Based Algorithms: Ensures that all features contribute equally to distance calculations, improving the performance of algorithms like KNN, K-means, and SVM.</li><li>Principal Component Analysis (PCA): Ensures that the components maximize the variance correctly, without being biased by the scale of the features.</li></ul><h3>Conclusion</h3><p>Feature scaling is an essential step in the machine learning pipeline that helps improve model performance, ensures numerical stability, and guarantees that all features contribute equally to the learning process. By understanding and applying the appropriate scaling techniques, you can enhance the effectiveness of your machine learning models and achieve more accurate results.</p><p><strong>Thank you for Reading</strong></p><p>If you like this post:</p><ul><li>Please show your support by following and with a clap 👏 or several claps!</li><li>Feel free to share this guide with your friends.</li><li>Your feedback is invaluable — it inspires and guides my future posts.</li><li>Or drop me a message: <a href="https://www.linkedin.com/in/ajaykumar-dev/">https://www.linkedin.com/in/ajaykumar-dev/</a></li></ul><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=6b290f76668f" width="1" height="1" alt="">]]></content:encoded>
        </item>
    </channel>
</rss>