<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Mahesh Jadhav on Medium]]></title>
        <description><![CDATA[Stories by Mahesh Jadhav on Medium]]></description>
        <link>https://medium.com/@ursmaheshj?source=rss-5113a4c6566c------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/1*QeW0ypGc29Wb97p0F444JA.jpeg</url>
            <title>Stories by Mahesh Jadhav on Medium</title>
            <link>https://medium.com/@ursmaheshj?source=rss-5113a4c6566c------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Sun, 22 Nov 2026 01:54:10 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@ursmaheshj/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[DOSE Chemicals in the Brain: How They Affect You and How to Control Them]]></title>
            <link>https://ursmaheshj.medium.com/dose-chemicals-in-the-brain-how-they-affect-you-and-how-to-control-them-2f83cd680bf8?source=rss-5113a4c6566c------2</link>
            <guid isPermaLink="false">https://medium.com/p/2f83cd680bf8</guid>
            <category><![CDATA[dopamine]]></category>
            <category><![CDATA[psychology]]></category>
            <category><![CDATA[healthy-lifestyle]]></category>
            <category><![CDATA[mental-health]]></category>
            <category><![CDATA[brain]]></category>
            <dc:creator><![CDATA[Mahesh Jadhav]]></dc:creator>
            <pubDate>Sat, 04 Jul 2026 20:06:10 GMT</pubDate>
            <atom:updated>2026-07-04T20:06:10.126Z</atom:updated>
            <content:encoded><![CDATA[<p>The human brain is a miracle of biology, coordinating everything from heartbeat to happiness. At the center of this complex system are neurotransmitters, the chemical messengers that influence how we feel, think, and behave. Among these, four stand out for their powerful impact on our mood and motivation:</p><p><strong>Dopamine, Oxytocin, Serotonin, and Endorphins</strong> — collectively known as the <strong>DOSE chemicals</strong>.</p><p>Let’s explore what each of these does, and how they affect your emotions.</p><p><strong>🧠 1. Dopamine: The Reward Molecule</strong></p><p>Dopamine is central to the brain’s reward system. It is released when we experience pleasure, like eating delicious food, achieving a goal, or receiving praise. It also plays a role in motivation and attention.</p><p><strong>How to Boost It Naturally:</strong></p><ul><li>Set and achieve small goals</li><li>Celebrate wins (even tiny ones)</li><li>Eat foods rich in tyrosine (e.g., almonds, bananas, eggs)</li><li>Get regular exercise</li></ul><p><strong>💞 2. Oxytocin — The Love Hormone</strong></p><p>Oxytocin is released during physical touch, bonding, and social interactions. It nurtures trust, empathy, and emotional connection.</p><p><strong>How to Boost It Naturally:</strong></p><ul><li>Hug someone or cuddle a pet</li><li>Practice acts of kindness</li><li>Spend quality time with loved ones</li><li>Engage in meaningful conversations</li></ul><p><strong>🌤️ 3. Serotonin — The Mood Stabilizer</strong></p><p>Serotonin helps regulate mood, sleep, appetite, and digestion. It is often linked to feelings of well-being and happiness.</p><p><strong>How to Boost It Naturally:<br></strong>1. Get sunlight exposure (20 minutes a day)<br>2. Practice mindfulness or meditation<br>3. Exercise regularly</p><p><strong>🏃 4. Endorphins: The Natural Painkillers</strong></p><p>Endorphins are released in response to stress or discomfort, acting as natural painkillers. They also create feelings of pleasure and euphoria.</p><p><strong>How to Boost It Naturally:</strong></p><ul><li>Engage in aerobic exercise</li><li>Laugh often</li><li>Listen to music you love</li><li>Eat dark chocolate or spicy foods</li></ul><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=2f83cd680bf8" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[MongoDB| NoSQL database with 25 basic commands]]></title>
            <link>https://ursmaheshj.medium.com/mongodb-nosql-database-with-25-basic-commands-3b9af48f39de?source=rss-5113a4c6566c------2</link>
            <guid isPermaLink="false">https://medium.com/p/3b9af48f39de</guid>
            <category><![CDATA[nosql]]></category>
            <category><![CDATA[technology]]></category>
            <category><![CDATA[programming]]></category>
            <category><![CDATA[database]]></category>
            <category><![CDATA[mongodb]]></category>
            <dc:creator><![CDATA[Mahesh Jadhav]]></dc:creator>
            <pubDate>Tue, 22 Apr 2025 04:54:34 GMT</pubDate>
            <atom:updated>2025-04-22T05:26:54.313Z</atom:updated>
            <content:encoded><![CDATA[<h3>MongoDB | NoSQL database with 25 basic commands</h3><p>MongoDB is a powerful, flexible, and scalable NoSQL database that has revolutionized the way developers handle data. Unlike traditional relational databases, MongoDB uses a JSON like document-oriented approach, making it ideal for handling large volumes of unstructured data. Let’s dive into some key aspects of MongoDB and understand why it’s a popular choice for modern applications.</p><h3>Key Features of MongoDB</h3><ol><li><strong>Document-Oriented Storage:</strong> MongoDB stores data in BSON (Binary JSON) format. This means you can store complex data structures and nested documents easily.</li><li><strong>Scalability:</strong> MongoDB has ability to scale horizontally. It can spread data across multiple servers to manage huge amounts of information without slowing down.</li><li><strong>Flexibility:</strong> MongoDB is schema-less, so you don’t need to define a strict structure for your data. You can add or remove fields as needed, making it perfect for projects where the data structure might evolve over time.</li><li><strong>High Performance:</strong> MongoDB provides high read and write throughput, making it suitable for applications that require real-time data processing.</li><li><strong>Rich Query Language:</strong> MongoDB offers a powerful query language that supports ad-hoc queries, indexing, and aggregation. You can perform complex searches and data manipulations with ease.</li></ol><h3>MongoDB counterparts to relational databases</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*5FS7udVQuU0PbJBAA_EjlQ.png" /><figcaption>MongoDB counterparts to Relational databases</figcaption></figure><h3>Must know tools in MongoDB ecosystem</h3><ol><li><strong>Atlas</strong>: A fully managed cloud database service that simplifies deploying, scaling, and managing MongoDB databases in the cloud.</li><li><strong>Mongosh</strong>: Shell environment which allows advanced features to communicate with MongoDB database.</li><li><strong>Compass</strong>: GUI interface that connect with MongoDB and performs advanced operations.</li></ol><h3>MongoDB architecture</h3><p>1. <strong><em>Database</em></strong>: A container for collections in MongoDB.<br>2. <strong><em>Collection</em></strong>: A group of MongoDB documents.<br>3. <strong><em>Document</em></strong>: A JSON-like object containing key-value pairs.<br>4. <strong><em>Field</em></strong>: An individual key-value pair within a document.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/792/1*rYMPFKLR8AQmC7iLYPTA4Q.png" /></figure><h3>Top 25 commands in MongoDB</h3><blockquote>1. Shows all databases present</blockquote><pre>&gt;show dbs</pre><blockquote>2. Show current database working on</blockquote><pre>&gt;db</pre><blockquote>3. Switch database or create a new one</blockquote><pre>&gt;use db</pre><blockquote>4. To drop current database</blockquote><pre>&gt;db.dropDatabase()</pre><blockquote>5. To show all collections from current database</blockquote><pre>&gt;show collections</pre><blockquote>6. Create a new collection</blockquote><pre>&gt;db.createCollection(&quot;CollectionName&quot;)</pre><blockquote>7. To drop a collection</blockquote><pre>&gt;db.collectionName.drop()</pre><blockquote>8. Insert a single document (one row) into a collection</blockquote><pre>&gt;db.collectionName.insertOne({name:&#39;Mahesh&#39;,lang:&#39;Python&#39;})</pre><blockquote>9. Insert more than one document (many rows) into a collection</blockquote><pre>&gt;db.collectionName.insertMany([{name:&#39;Mahesh&#39;,lang:&#39;Python&#39;},{name:&#39;John&#39;,lang:&#39;Java&#39;}])</pre><blockquote>10. View all documents (rows) from a collection</blockquote><pre>&gt;db.collectionName.find()</pre><blockquote>11. View filtered documents from collection based on field:values</blockquote><pre>&gt;db.collectionName.find({language:&#39;Python&#39;})</pre><blockquote>12. To return only specific no. of documents</blockquote><pre>&gt;db.collectionName.find().limit(3)</pre><blockquote>13. Sort documents based on a field (Ascending:1 and Descending:-1)</blockquote><pre>&gt;db.collectionName.find().sort({name:1})  #sorts in ascending order<br>&gt;db.collectionName.find().sort({name:-1})  #sorts in descending order</pre><blockquote>14. To view data in an understandable, easy to read, pretty format</blockquote><pre>&gt;db.collectionName.find().pretty()</pre><blockquote>15. Get count of all documents</blockquote><pre>&gt;db.collectionName.find().count()</pre><blockquote>16. Return only first row that matches the given condition</blockquote><pre>&gt;db.collectionName.findOne({name:&#39;mahesh&#39;})</pre><blockquote>17. Update first document that matches the given condition</blockquote><pre>&gt;db.collectionName.updateOne({name:&quot;Mahesh&quot;},{$set:{name:&quot;Mahesh J&quot;,age:25}})</pre><blockquote>18. Update all documents that matches the given condition</blockquote><pre>&gt;db.collectionName.updateMany({name:&quot;Mahesh&quot;},{$set:{name:&quot;Mahesh J&quot;}})</pre><blockquote>19. Update document that matches the given condition else create a new one</blockquote><pre>&gt;db.collectionName.updateOne({name:&quot;Mahesh&quot;},{$set:{name:&quot;Mahesh J&quot;}},{upsert:true})</pre><blockquote>20. Increment the value of specified field for each matched document</blockquote><pre>&gt;db.collectionName.updateOne({name:&quot;Mahesh&quot;},{$inc:{age:5}})<br>&gt;db.collectionName.updateOne({name:&quot;Mahesh&quot;},{$inc:{age:-5}})  #decrease when given negative values</pre><blockquote>21. Rename field name for matched document</blockquote><pre>&gt;db.collectionName.updateOne({name:&quot;Mahesh&quot;},{$rename:{language:&quot;lang&quot;}})</pre><blockquote>22. Remove all documents that match given condition</blockquote><pre>&gt;db.collectionName..remove({name:&quot;Mahesh&quot;})</pre><blockquote>23. Delete a single document that matches given condition</blockquote><pre>&gt;db.collectionName.deleteOne({age:26})</pre><blockquote>24. Delete all documents that matches given condition</blockquote><pre>&gt;db.collectionName.deleteMany({dept:&#39;sales&#39;})</pre><blockquote>25. Using aggregate function to group, sort, perform calculations, analyze data, and much more.</blockquote><pre>&gt;db.collectionName.aggregate([{$match:{age:{$gt:25}}}])  #returns documents that have age more than 25<br>&gt;db.collectionName.aggregate([{$match:{age:{$gt:25}}},{$group:{_id:&quot;$category&quot;,totallikes:{$sum:&quot;$likes&quot;}}}])</pre><p><em>If you’re interested in learning more about MongoDB, check out the </em><a href="https://www.mongodb.com/blog/1"><em>MongoDB Blog</em></a><em> for the latest updates, tutorials, and best practices.</em></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=3b9af48f39de" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Internet and cookies]]></title>
            <link>https://ursmaheshj.medium.com/internet-and-cookies-425f46bbe0ac?source=rss-5113a4c6566c------2</link>
            <guid isPermaLink="false">https://medium.com/p/425f46bbe0ac</guid>
            <category><![CDATA[internet]]></category>
            <category><![CDATA[cookies]]></category>
            <category><![CDATA[web]]></category>
            <category><![CDATA[networking]]></category>
            <category><![CDATA[web-development]]></category>
            <dc:creator><![CDATA[Mahesh Jadhav]]></dc:creator>
            <pubDate>Sat, 01 Feb 2025 12:03:44 GMT</pubDate>
            <atom:updated>2025-02-01T12:14:43.415Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="Internet or browser cookies" src="https://cdn-images-1.medium.com/max/688/1*p-53F9WawI29-LK9CYdpvg.png" /><figcaption>Cookies</figcaption></figure><p>To understand it in an effective way Assume you don’t know anything about internet, cookies and how websites work.</p><p><em>let’s start from beginning to the era where there was nothing at all.</em></p><p>Early-stage computers were large and immobile and in order to make use of information stored in a computer, one had to either travel to the site of the computer or have magnetic computer tapes sent through the conventional postal system.</p><p>In around 1960s The Cold War was at its height and huge tensions existed between North America and the Soviet Union which motivated the U.S. Defense Department to find a way to exchange information incase of a nuclear attack using distributed network. This eventually led to the formation of the ARPANET (Advanced Research Projects Agency Network).</p><p>After the creation of ARPANET, more networks of computers began to join the network, and the need arose for an agreed set of rules for handling data. A new communications protocol was established called Transfer Control Protocol/Internetwork Protocol (TCP/IP). This allowed different kinds of computers on different networks to “talk” to each other. ARPANET and the Defense Data Network officially changed to the TCP/IP standard on January, 1983. All networks could now be connected by a universal language ultimately</p><p>This was just about how internet was born we are yet to explore the websites and browsers.</p><p>The Internet was originally conceived and developed to meet the demand for automated information-sharing between scientists in universities and institutes around the world.</p><p>Tim Berners-Lee, an English computer scientist, was working for the huge European nuclear research hub CERN. CERN researchers were using different computer platforms for work and they badly needed something that would facilitate data sharing among them. In 1989, Tim proposed the idea of a ‘web of information’ . It relied on ‘hyperlinks’ to connect documents together. Written in Hypertext Markup Language (HTML), a hyperlink can point to any other HTML page or file that sits on top of the internet.</p><p>Later In 1990, Berners-Lee developed Hypertext Transfer Protocol (HTTP) and designed the Universal Resource Identifier (URI) system. HTTP is the language computers use to communicate HTML documents over the internet, and the URI, also known as a URL, provides a unique address where the pages can be easily found.</p><p>Berners-Lee also created a piece of software that could present HTML documents in an easy-to-read format. He called this software a ‘browser’ the ‘<a href="https://worldwideweb.cern.ch/browser/">WorldWideWeb</a>’ (not to confuse with the World Wide Web). Later, to avoid confusion, the browser was renamed to Nexus. The first web browser was written on a NeXT computer.</p><p>The first website at CERN — and in the world — was dedicated to the World Wide Web project itself (<a href="https://info.cern.ch/hypertext/WWW/TheProject.html">here</a>) and was hosted on Berners-Lee’s NeXT computer.</p><p>First Browser: <a href="https://worldwideweb.cern.ch/browser/">https://worldwideweb.cern.ch/</a><br>First Website: <a href="https://info.cern.ch/hypertext/WWW/TheProject.html">https://info.cern.ch/hypertext/WWW/TheProject.html</a></p><p>With world wide web internet rapidly got a fame which led more and more businesses to transfer onto internet including financial organization, entertainment industries, E-commerce applications to connect and communicate with their users.</p><p>Still the websites was not advanced. they were struggling to remember who their users were or what they did in previous website visits. Every time a user loaded a new page, a website would treat them like a stranger it had never seen before. That made it impossible to build basic web features we take for granted today, like the shopping carts that follow us from page to page across e-commerce sites.</p><p>To resolve this problem <a href="http://en.wikipedia.org/wiki/Lou_Montulli">Lou Montulli</a> at Netscape found a solution.<br><strong>Cookie</strong> — a small text file passed back and forth between a person’s computer and a single website — as a way to help websites remember visitors.</p><p>Cookies, often referred to as HTML cookies, HTTP cookies, Internet cookies, or browser cookies, are<strong> stored on the user’s computer via web browsers</strong>. Essentially, cookies allow each site to recognize a user as the same person that has visited the site before.</p><p>In the early days, cookies were primarily used to enhance user experience by remembering login details and user preferences. However, as the internet grew, so did the ways in which cookies are used.</p><blockquote>Types of cookies based on source</blockquote><ol><li><strong>First-party cookies:</strong> <br>These are set by the website you are currently visiting. They help the website remember your activities and preferences across multiple visits, like showing you links to pages you recently visited on that site.</li><li><strong>Third-party cookies:</strong> <br>These are set by other parties, not the website you are visiting. They are often used by advertisers and companies to track your activity across different websites for targeted advertising, which raises privacy concerns.</li></ol><blockquote>Types of cookies based on expiry</blockquote><ol><li><strong>Temporary cookies:</strong> <br>Also known as ‘<em>Session cookies</em>’, that expire when you close your browser. They help websites remember your actions and preferences during a single browsing session, like keeping items in your shopping cart as you navigate between pages.</li><li><strong>Persistent cookies:</strong> <br>These cookies stay on your browser for a longer period, even after you close it. They have an expiration date and help websites remember your information, settings, and login details for future visits, providing a better and faster user experience.</li></ol><blockquote>Types of cookies based on purpose</blockquote><ol><li><strong>Strictly Necessary Cookies:</strong><br>These essential cookies are crucial for a website’s functionality. They enable basic features like signing in, adding items to a shopping cart, checking out, and making payments. Because they are necessary for the website to operate, they are exempt from cookie consent requirements.</li><li><strong>Performance Cookies:<br></strong>Also known as statistics cookies, these cookies help websites remember users to enhance their experience. They collect anonymous data on how visitors use the site, including the types of pages visited and any issues encountered. This information is used to improve the website’s performance and understand user interests for better communication and service delivery.</li><li><strong>Functional Cookies:<br></strong>Functional or preference cookies ensure that a website operates correctly and enhances its functionality. They remember user credentials for automatic login and site preferences like language and region. While they do not track browsing activity on other websites, they can be set by third-party providers whose services are used by the website.</li><li><strong>Advertising Cookies:<br></strong>Also known as targeting or tracking cookies, these are used to track users’ online activities and behaviors to build profiles of their interests. This information is used to show relevant advertisements on other websites. Typically set by third-party advertising networks, these cookies are often persistent and can uniquely identify a user’s browser, device, location, and browsing habits.</li></ol><h3>Cookies and User Privacy</h3><p>As cookies became widely used by websites, there have been ongoing concerns about user privacy. These concerns have intensified as ad networks increasingly use personal data collected from cookies to target ads. To address these privacy issues, below data protection laws and directives have been implemented, which regulates provisions to govern the use of cookies and protect user privacy:</p><p>1. General Data Protection Regulation (GDPR)<br> 2. ePrivacy Directive (ePD)<br> 3. California Consumer Privacy Act (CCPA)<br> 4. Lei Geral de Proteção de Dados (LGPD)<br> 5. Digital Personal Data Protection Act (DPDPA)</p><p><em>Source:</em><br>→ <a href="https://en.wikipedia.org/wiki/HTTP_cookie">https://en.wikipedia.org</a><br>→ <a href="https://www.scienceandmediamuseum.org.uk/objects-and-stories/short-history-internet#:~:text=The%20origins%20of%20the%20internet%20are%20rooted%20in%20the%20USA,America%20and%20the%20Soviet%20Union">https://www.scienceandmediamuseum.org.uk</a><br>→ <a href="https://cookieyes.com/blog/internet-cookies/#:~:text=The%20history%20of%20cookies%20can,text%20files%20to%20store%20information.">https://cookieyes.com</a><br>→ <a href="https://home.web.cern.ch/science/computing/birth-web#:~:text=The%20first%20website%20at%20CERN,%3A%20info.cern.ch.">https://home.web.cern.ch</a><br>→ <a href="https://www.internetsociety.org/internet/history-internet/brief-history-internet/">https://www.internetsociety.org</a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=425f46bbe0ac" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[How to extract logs from AWS CloudWatch using python and Boto3]]></title>
            <link>https://ursmaheshj.medium.com/how-to-extract-logs-from-aws-cloudwatch-using-python-and-boto3-d4b3e03a5741?source=rss-5113a4c6566c------2</link>
            <guid isPermaLink="false">https://medium.com/p/d4b3e03a5741</guid>
            <category><![CDATA[programming]]></category>
            <category><![CDATA[python-programming]]></category>
            <category><![CDATA[aws]]></category>
            <category><![CDATA[cloudwatch]]></category>
            <category><![CDATA[python]]></category>
            <dc:creator><![CDATA[Mahesh Jadhav]]></dc:creator>
            <pubDate>Wed, 29 Jan 2025 17:23:55 GMT</pubDate>
            <atom:updated>2025-01-29T17:23:55.181Z</atom:updated>
            <content:encoded><![CDATA[<p>AWS is a popular cloud service that offers CloudWatch, a powerful tool for monitoring, storing, and accessing log files from various resources like EC2 instances, CloudTrail, ECS, and Route 53.</p><p>Knowing how to extract these logs for analysis and examination is essential. In this blog, we will explore the same and get the data we need.</p><figure><img alt="https://www.webiny.com/docs/messages/cloudwatch" src="https://cdn-images-1.medium.com/max/1024/1*AvEVVnAw-mLtIawURDpBIQ.png" /><figcaption>AWS CloudWatch logs</figcaption></figure><p><em>Lets consider we want to extract only the messages which starts with “</em>START RequestId:<em>” from last 7 days logs.</em></p><ol><li>Install and import required libraries:</li></ol><pre>import boto3 <br>from datetime import datetime, timedelta <br>import time</pre><p>2. Create a client with ‘logs’:</p><pre>client = boto3.client(&#39;logs&#39;)</pre><p>3. Define log group and timeframe:</p><pre>log_group_name = &#39;/aws/lambda/wby-graphql-af86eaa&#39;  # Replace with your log group name <br>end_time = int(datetime.now().timestamp() * 1000)  # Current time in milliseconds <br>start_time = int((datetime.now() - timedelta(days=7)).timestamp() * 1000)  # Time 7 days ago in milliseconds</pre><p>4. Define the pattern to extract from logs:</p><pre>filter_pattern = &#39;&quot;START RequestId:&quot;&#39;  # Replace with your filter pattern like &quot;Response of API: {}&quot;</pre><p>5. Get paginator by providing required fields:</p><pre>paginator = client.get_paginator(&#39;filter_log_events&#39;)</pre><p>6. Create a response iterator using paginator</p><pre>response_iterator = paginator.paginate(<br>    logGroupName=log_group_name,<br>    filterPattern=filter_pattern, <br>    startTime=start_time,<br>    endTime=end_time<br>)</pre><p>7. Create a list of required messages by iterating over response iterator</p><pre>logs = []<br>for response in response_iterator:<br>    for event in response[&#39;events&#39;]:<br>        message = event[&#39;message&#39;]<br>        logs.append(message)</pre><h3>Full Code:</h3><p>Below code contains full code on how to extract data from AWS CloudWatch service.</p><pre>import time <br>import boto3<br>from datetime import datetime, timedelta<br><br># Initialize a session using Amazon CloudWatch<br>client = boto3.client(&#39;logs&#39;)<br><br># Define the log group name and the time range for which the logs are to be retrieved<br>log_group_name = &#39;/aws/lambda/wby-graphql-af86eaa&#39;  # Replace with your log group name<br>end_time = int(datetime.now().timestamp() * 1000)  # Current time in milliseconds<br>start_time = int((datetime.now() - timedelta(days=7)).timestamp() * 1000)  # Time 7 days ago in milliseconds<br><br># Function to get logs from CloudWatch<br>def get_logs(log_group_name, start_time, end_time):<br>    &quot;&quot;&quot;<br>    Retrieves logs from the specified log group within the given time range.<br><br>    Parameters:<br>    log_group_name (str): The name of the log group.<br>    start_time (int): The start time in milliseconds.<br>    end_time (int): The end time in milliseconds.<br><br>    Returns:<br>    list: A list of log messages.<br>    &quot;&quot;&quot;<br>    filter_pattern = &#39;&quot;START RequestId:&quot;&#39;  # Replace with your filter pattern<br>    paginator = client.get_paginator(&#39;filter_log_events&#39;)<br>    print(f&quot;paginator: {paginator}&quot;)<br>    <br>    # Paginate through the log events<br>    response_iterator = paginator.paginate(<br>        logGroupName=log_group_name,<br>        filterPattern=filter_pattern, <br>        startTime=start_time,<br>        endTime=end_time<br>    )<br>    <br>    logs = []<br>    for response in response_iterator:<br>        for event in response[&#39;events&#39;]:<br>            message = event[&#39;message&#39;]<br>            logs.append(message)<br>    <br>    return logs<br><br>if __name__ == &#39;__main__&#39;:<br>    # Measure the time taken to retrieve logs<br>    st = time.time()<br>    <br>    # Retrieve logs<br>    logs = get_logs(log_group_name, start_time, end_time)<br>    print(f&quot;Logs: {logs}&quot;)<br>    <br>    # Calculate elapsed time<br>    et = time.time()<br>    elapsed_time = et - st<br>    print(f&quot;Elapsed time: {elapsed_time} seconds&quot;)</pre><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=d4b3e03a5741" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Django QuerySet Field Lookups: Unlocking Advanced Data Filtering Techniques for Your Django…]]></title>
            <link>https://medium.com/django-unleashed/django-queryset-field-lookups-unlocking-advanced-data-filtering-techniques-for-your-django-1eedf0c2781b?source=rss-5113a4c6566c------2</link>
            <guid isPermaLink="false">https://medium.com/p/1eedf0c2781b</guid>
            <category><![CDATA[python]]></category>
            <category><![CDATA[python-web-developer]]></category>
            <category><![CDATA[django-framework]]></category>
            <category><![CDATA[full-stack]]></category>
            <category><![CDATA[django]]></category>
            <dc:creator><![CDATA[Mahesh Jadhav]]></dc:creator>
            <pubDate>Sun, 02 Jun 2024 07:06:06 GMT</pubDate>
            <atom:updated>2024-06-02T07:17:07.119Z</atom:updated>
            <content:encoded><![CDATA[<h3>Django QuerySet Field Lookups: Unlocking Advanced Data Filtering Techniques for Your Django Application</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*A8jYHk2RX8aMRzZ9-LSaow.png" /><figcaption>Django QuerySet Field Lookups</figcaption></figure><p>Field lookups are how you specify the meat of an SQL <strong>WHERE </strong>clause. They’re specified as keyword arguments to the <strong>QuerySet</strong> methods <a href="https://docs.djangoproject.com/en/dev/ref/models/querysets/#django.db.models.query.QuerySet.filter"><strong>filter()</strong></a>, <a href="https://docs.djangoproject.com/en/dev/ref/models/querysets/#django.db.models.query.QuerySet.exclude"><strong>exclude()</strong></a> and <a href="https://docs.djangoproject.com/en/dev/ref/models/querysets/#django.db.models.query.QuerySet.get"><strong>get()</strong></a>. They help us to apply conditions while retrieving the records.</p><p><strong>syntax</strong>: (<em>fieldname__lookuptype = value)</em><strong><br>eg</strong>: <em>model.objects.filter(id__gt=5)</em></p><p>To understand the field lookups in depth and how to use them in your Django application lets create below model in <em>models.py</em> file.</p><pre>#models.py <br>class Player(models.Model):<br>    jerseyno = models.IntegerField(null=False,unique=True)<br>    name = models.CharField(max_length=50)<br>    city = models.CharField(max_length=100)<br>    totalruns = models.IntegerField()<br>    birthdate = models.DateField()<br>    testdebut = models.DateTimeField()</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/904/1*pn1unVjafFOj2syxu1Zfzw.png" /></figure><p>Considering we have above data in our database based on our <em>Player </em>model, lets try to retrieve the data using field lookups.</p><blockquote><em>1. </em><strong><em>exact</em></strong><em>: Matches the exact values in field.</em></blockquote><p><em>eg: Player.objects.filter(city__exact=’Delhi’)</em></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/647/1*Ut8R4CoMRSnQqrKBUyL4uQ.png" /></figure><blockquote><em>2. </em><strong><em>iexact</em></strong><em>: Matches the exact values in field but ignores the case sensitivity.</em></blockquote><p><em>eg: Player.objects.filter(city__iexact=’Delhi’)</em></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/700/1*n5Yyg89jBFEZuLQnS_q6sg.png" /></figure><blockquote><em>3.</em><strong><em> contains</em></strong><em>: Matches field values that contains given value in it. Use ‘</em><strong><em>icontains</em></strong><em>’ to ignore case sensitivity.</em></blockquote><p><em>eg: Player.objects.filter(city__contains=’hi’)</em></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/714/1*e_U8rRqtAtpvkb1U8wOoxQ.png" /></figure><blockquote><em>4. </em><strong><em>in</em></strong><em>: Matches field value in list of values.</em></blockquote><p><em>eg: Player.objects.filter(jerseyno__in=[7,10,18])</em></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/741/1*pVOnj700WF0fdyQ4iWoNHw.png" /></figure><blockquote><em>5. </em><strong><em>gt</em></strong><em>: Matches all the field values which are greater than given value. Use ‘</em><strong><em>gte’ </em></strong><em>for greater than or equals to.</em></blockquote><p><em>eg: Player.objects.filter(totalruns__gt=18820)</em></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/739/1*1_CMxHd9XflJL0b9CFHpbg.png" /></figure><p><em>eg: Player.objects.filter(totalruns__gte=18820)</em></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/742/1*G4NkXEnBZpWkG3jnp-1jCA.png" /></figure><blockquote><em>6. </em><strong><em>lt</em></strong><em>: Matches all the field values which are less than given value. Use ‘</em><strong><em>lte’ </em></strong><em>for less than or equals to.</em></blockquote><p><em>eg: Player.objects.filter(totalruns__lt=18820)</em></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/710/1*urLXzS0Jsqf26W4hXgUmiQ.png" /></figure><p><em>eg: Player.objects.filter(totalruns__lte=18820)</em></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/721/1*q-PHcJol_SuLZyNkPyUmTg.png" /></figure><blockquote><em>7. </em><strong><em>startswith</em></strong><em>: Matches field values that starts with given value. Use ‘</em><strong><em>istartswith</em></strong><em>’ to ignore case sensitivity.</em></blockquote><p>eg: <em>Player.objects.filter(name__startswith=’M’)</em></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/648/1*GjvK1XaMrsX--TWOQRLBhQ.png" /></figure><blockquote><em>8. </em><strong><em>endswith</em></strong><em>: Matches field values that ends with given value. Use ‘</em><strong><em>iendswith</em></strong><em>’ to ignore case sensitivity.</em></blockquote><p>eg: <em>Player.objects.filter(name__endswith=’i’)</em></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/657/1*yqE8AnBabaGx6yjSI_xcaQ.png" /></figure><blockquote><em>9. </em><strong><em>range</em></strong><em>: Matches field dates that comes under given range. Works with both date and datetime.</em></blockquote><p><em>eg: Player.objects.filter(birthdate__range=(‘1980–01–01’,’1985–01–01&#39;))</em></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/713/1*vI-FvaWbYk-rBawvr8liag.png" /></figure><blockquote><em>10. </em><strong><em>date</em></strong><em>: Matches field dates with given date. Works only with datetime field.</em></blockquote><p>eg: <em>Player.objects.filter(testdebut__date=</em>’2005–12–02&#39;<em>)</em></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/648/1*DLq3cKcHqlUuXo_02JEkIw.png" /></figure><blockquote><em>11. </em><strong><em>year</em></strong><em>: Matches year from field dates with given year. Works with both date and datetime.</em></blockquote><p><em>eg: Player.objects.filter(birthdate__year=1981)</em></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/710/1*M3ynuW_CkDVWXH9hYODRqQ.png" /></figure><blockquote><em>12. </em><strong><em>month</em></strong><em>: Matches month from field dates with given month. Works with both date and datetime.</em></blockquote><p>eg: <em>Player.objects.filter(birthdate__month=4)</em></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/740/1*72RRawZeZGcZpGePYenjTA.png" /></figure><blockquote><em>13. </em><strong><em>day</em></strong><em>: Matches day from field dates with given date. Works well with both date and datetime.</em></blockquote><p><em>eg: Player.objects.filter(testdebut__day=20)</em></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/647/1*m5mxJ5iBEv6hGRp778Do3g.png" /></figure><blockquote><em>14. </em><strong><em>week</em></strong><em>: Matches week of field dates with given week number (Normally 52 weeks). Works with both date and datetime fields.</em></blockquote><p><em>eg: Player.objects.filter(birthdate__week=18)</em></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/686/1*nYu1ycRNOd5ytOe2F-NNsA.png" /></figure><blockquote><em>15. </em><strong><em>week_day</em></strong><em>: Matches week day of field date with given week day (usually starts with sunday as 1). Works well with both date and datetime field.</em></blockquote><p><em>eg: Player.objects.filter(birthdate__week_day=3)</em></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/742/1*rsig-5X1Jn6l4XWwwg6cog.png" /></figure><blockquote><em>16. </em><strong><em>quarter</em></strong><em>: Matches quarter of field dates with given quarter. Works with both date and datetime.</em></blockquote><p><em>eg: Player.objects.filter</em>(birthdate__quarter=4)</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/699/1*0yjIvhBUtKzkNTGbSwFivA.png" /></figure><blockquote><em>17. </em><strong><em>time</em></strong><em>: Matches time of field with given time. Only works with datetime field.</em></blockquote><p><em>eg: Player.objects.filter(testdebut__time=</em>’09:33:55&#39;<em>)</em></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/645/1*GNX618d5tC0idmnr6QKnlA.png" /></figure><blockquote><em>18. </em><strong><em>hour</em></strong><em>: Matches hour of the field time with given hour. Only works with datetime field.</em></blockquote><p><em>eg: Player.objects.filter</em>(testdebut__hour=10)</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/740/1*vIKZ845TRSRKM9zyyg0S2Q.png" /></figure><blockquote><em>19. </em><strong><em>minute</em></strong><em>: Matches minute of the field time with given minute. Only works with datetime field.</em></blockquote><p><em>eg: Player.objects.filter</em>(testdebut__minute=49)</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/740/1*MHpQsS8FH3ujFrJtzKj-Jw.png" /></figure><blockquote><em>20. </em><strong><em>second</em></strong><em>: Matches second of the field time with given second. Only works with datetime field.</em></blockquote><p><em>eg: Player.objects.filter(testdebut__second=55)</em></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/646/1*qHKYx6aIvxKRgBBH3jO4sg.png" /></figure><blockquote><em>21. </em><strong><em>isnull</em></strong><em>: Matches null field values. Return all not null values if False.</em></blockquote><p><em>eg: Player.objects.filter(jerseyno__isnull=False)</em></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/742/1*ovc54ZRHD1HDS6xoVMeQrw.png" /></figure><blockquote><em>22. </em><strong><em>regex</em></strong><em>: Matches the field value to described regular expression. Use ‘</em><strong><em>iregex</em></strong><em>’ to ignore case sensitivity.</em></blockquote><p><em>eg: Player.objects.filter(name__regex=’^.a’)</em></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/743/1*Ruos3iAFeAe2Ud85vkLxKg.png" /></figure><blockquote><em>23. </em><strong><em>Combining lookups</em></strong><em>: We can combine conditional lookups like gt, gte, lt, lte along with other field lookups.</em></blockquote><p><em>eg: Player.objects.filter</em>(testdebut__day__lt=15)</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/721/1*IhI5RxWJa-MXI1XEviwwcw.png" /></figure><p><em>eg: Player.objects.filter(birthdate__year__lte=1981)</em></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/741/1*8u16-cgl-4Nkdv7_3p1qZQ.png" /></figure><p><em>eg: Player.objects.filter</em>(birthdate__month__gt=7)</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/700/1*L_AX_KjLCRKRXW1dsToLAg.png" /></figure><p><em>eg: Player.objects.filter</em>(birthdate__month__gte=7)</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/710/1*C5hELCEWG2sTmqvBjre1bg.png" /></figure><blockquote><em>24. </em><strong><em>Combining QuerySets</em></strong><em>: We can combine two or more QuerySets to produce required results. The latter QuerySet will apply lookup on results of earlier QuerySet.</em></blockquote><p>eg: <em>Player.objects.filter(jerseyno__in=[7,10,45]).filter(birthdate__month=4)</em></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/742/1*VHGPZas04gBDSbVFpt45FA.png" /></figure><p>Above all lookups can be used with both get and exclude methods, Just note that get method will raise an exception if it returns more than 1 record.</p><blockquote>If you want to learn more about QuerySets and FieldLookups head over to Django documentation <a href="https://docs.djangoproject.com/en/5.0/ref/models/querysets/#field-lookups">here</a>.</blockquote><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=1eedf0c2781b" width="1" height="1" alt=""><hr><p><a href="https://medium.com/django-unleashed/django-queryset-field-lookups-unlocking-advanced-data-filtering-techniques-for-your-django-1eedf0c2781b">Django QuerySet Field Lookups: Unlocking Advanced Data Filtering Techniques for Your Django…</a> was originally published in <a href="https://medium.com/django-unleashed">Django Unleashed</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Exploring the Most Popular Regression Metrics for Data Scientist]]></title>
            <link>https://ursmaheshj.medium.com/exploring-the-most-popular-regression-metrics-for-data-scientist-cf89500f2360?source=rss-5113a4c6566c------2</link>
            <guid isPermaLink="false">https://medium.com/p/cf89500f2360</guid>
            <category><![CDATA[model-performance]]></category>
            <category><![CDATA[regression-metric]]></category>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[data]]></category>
            <category><![CDATA[machine-learning]]></category>
            <dc:creator><![CDATA[Mahesh Jadhav]]></dc:creator>
            <pubDate>Fri, 23 Jun 2023 03:44:29 GMT</pubDate>
            <atom:updated>2023-06-23T04:33:15.988Z</atom:updated>
            <content:encoded><![CDATA[<p>While applying regression on real-world data it is necessary to make sure our model is optimized to predict correct output and make fewer errors. Below are some of the popular metrics used to evaluate the model’s performance.</p><h3>MAE (Mean Absolute Error)</h3><p>It is basically the mean of absolute error between the actual value and the predicted value. We take absolute error to ignore the negative sign of values for calculation.</p><figure><img alt="Mean Absolute Error" src="https://cdn-images-1.medium.com/max/831/0*3QGjVF0PHlZzgsay" /><figcaption>Mean Absolute Error</figcaption></figure><p>Advantages:</p><ul><li>Error unit same as output variable which makes it easy to interpret.</li><li>Robust to outliers.</li></ul><p>Disadvantages:</p><ul><li>It is not differentiable.</li></ul><h3>MSE (Mean Squared Error)</h3><p>Instead of taking the absolute value of the error, MSE uses squared values of the error to eliminate negative signs.</p><figure><img alt="Mean Squared Error" src="https://cdn-images-1.medium.com/max/744/0*qaZPfjx9som4F0vs" /><figcaption>Mean Squared Error</figcaption></figure><p>Advantages:</p><ul><li>Graph is differentiable.</li><li>Can be used as a loss function.</li></ul><p>Disadvantages:</p><ul><li>Error unit different from output variable</li><li>Not robust to outliers.</li></ul><h3>RMSE (Root Mean Squared Error)</h3><p>It is same as the MSE but at the end of the calculation we will take the square root of the error to make its unit same as the output feature(y).</p><figure><img alt="Root Mean Squared Error" src="https://cdn-images-1.medium.com/max/744/0*JWmAsgWPlrXp55LG" /><figcaption>Root Mean Squared Error</figcaption></figure><p>Advantages:</p><ul><li>square root makes error unit same as output variable (y)</li></ul><p>Disadvantages:</p><ul><li>Not robust to outliers</li></ul><h3>R2 Score</h3><p>It is also called the coefficient of determination or goodness of fit. It helps to evaluate the model’s performance by comparing square of prediction error made by the regression model to square of mean error.</p><figure><img alt="R2 Score" src="https://cdn-images-1.medium.com/max/744/0*BXcpu0me2HDkaZpZ" /><figcaption>R2 Score</figcaption></figure><p>If R2 Score goes towards Zero (0) Model performance decreases.</p><p>R2 Score may get affected by irrelevant features added to the dataset.</p><h3>Adjusted R2 Score</h3><p>Adjusted R2 score does not get affected by any irrelevant column added and try to give the best results on each database being used.</p><figure><img alt="Adjusted R2 Score" src="https://cdn-images-1.medium.com/max/744/0*vfIHIHpBQfUjJiOi" /><figcaption>Adjusted R2 Score</figcaption></figure><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=cf89500f2360" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Handling Missing Data | Data science | Machine learning]]></title>
            <link>https://ursmaheshj.medium.com/effective-strategies-for-handling-missing-data-a215056a07e3?source=rss-5113a4c6566c------2</link>
            <guid isPermaLink="false">https://medium.com/p/a215056a07e3</guid>
            <category><![CDATA[towards-data-science]]></category>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[handling-missing-values]]></category>
            <category><![CDATA[missing-data]]></category>
            <dc:creator><![CDATA[Mahesh Jadhav]]></dc:creator>
            <pubDate>Mon, 05 Jun 2023 04:41:28 GMT</pubDate>
            <atom:updated>2023-06-05T07:57:34.948Z</atom:updated>
            <content:encoded><![CDATA[<h3>How to Handle Missing Data | Machine learning | Data science</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*ip2tYhtwCWVGo3Yen5fX7w.png" /><figcaption>Handling Missing Data</figcaption></figure><p>Handling missing data is one of the important tasks that a data scientist needs to be an expert in. When working on real-world projects, it is common to encounter missing data, and dealing with it requires careful planning to avoid bias and ensure accurate analysis and efficient model training.</p><p>Missing data can be categorized into three types -</p><p>1. <strong>MCAR</strong> — Missing completely at Random.<br>2. <strong>MAR</strong> — Missing at Random.<br>3. <strong>MNAR</strong> — Missing not at Random.</p><p>There are mainly two ways to handle the missing data, either remove the missing values or impute the missing values based on some calculations.</p><h3><strong>Remove missing values</strong>:</h3><p>The missing values are simply removed from the dataset. We use this method if the missing data is either MCAR or MAR and constitutes less than 5% of the total available data.</p><blockquote><strong>Advantages and Disadvantages:<br></strong>1. Easy to implement.<br>2. Preserves distribution if Data is MCAR.<br>3. Excluded data may contain important data and result in decreased model performance.<br>4. Unable to handle missing data in production.</blockquote><ol><li><strong>Listwise deletion: </strong><br>Also known as CCA (Complete Case Analysis), It discards rows where values in any of the columns are missing.</li><li><strong>Pairwise deletion: </strong><br>Also known as ACA (Available Case Analysis), It minimizes the data loss compared to Listwise deletion by ignoring the missing values based on the correlation strength of the relationship between two variables.</li><li><strong>Dropping Column:</strong> <br>If any of the columns contain a large proportion of missing values and show no correlations with the target variable, then instead of removing rows, we can drop the entire column to simplify the dataset.</li></ol><h3><strong>Imputing missing values</strong>:</h3><p>In this technique, we predict the most appropriate value to replace missing values using different statistical methods.</p><h4>Univariate Imputation:</h4><p>Missing values are predicted based on the information available within that specific variable. Values are calculated with different methods based on the data type of that specific variable.</p><blockquote><strong>Advantages and Disadvantages:<br></strong>1. Overcomes data loss issue.<br>2. It may change shape of the distribution of data.<br>3. Change in covariance and correlation between data.<br>4. It may identify unnecessary outliers.</blockquote><p><strong>Numerical Imputations:</strong></p><ol><li><strong>Mean:<br></strong>Missing values are replaced with the mean of the column. Works best on normally distributed data</li><li><strong>Median:<br></strong>Missing values are replaced with the median of the column. Works best on skewed distribution.</li><li><strong>End of Distribution:<br></strong>missing values are replaced with far-end values or extreme far-end values. Based on distributions we use the below formulas.<br><em>Normal</em>: <em>(mean — 3σ) or (mean + 3σ)</em><br><em>Skewed</em>: <em>(Q1–1.5IQR) or (Q3 + 1.5IQR)</em></li><li><strong>Random:<br></strong>Missing values are replaced with a random value selected from the available unique values present in that column. This technique helps preserve the variance and distribution of the data but can be memory-heavy during deployment.</li></ol><p><strong>Categorical Imputations:</strong></p><ol><li><strong>Mode:<br></strong>Here, missing values are replaced with the most recurring value, which is the mode of that specific column.</li><li><strong>Arbitrary:<br></strong>If missing values are MNAR or constitute more than 5% of the total data then we can replace it with custom text like “MISSING”.</li></ol><h4>Multivariate Imputation:</h4><p>In this technique, missing values are predicted based on the relationship between the data present in another column using different concepts like correlation, covariance, and Euclidean distance between two points.</p><blockquote><strong>Advantages and Disadvantages:<br></strong>1. Gives the most accurate predictions for missing data.<br>2. More no of calculations are required which may slow down the process.<br>3. Memory heavy in case of deployment on production.</blockquote><ol><li><strong>KNN Imputer:<br></strong>Missing values are predicted with the help of the K-nearest neighbor algorithm using Euclidean distance. where <em>k</em> is the no. of nearest neighbors to be taken in the calculation.<br><em>Euclidean distance(x,y) = sqrt(weight * sq. distance from present coordinates) </em><br>where, <em>weight = Total no of coordinates / no of present coordinates</em></li><li><strong>Iterative Imputer:<br></strong>Also known as MICE (Multivariate Imputation by Chained Equations), this technique helps to predict missing values using a machine learning (ML) model. The process involves filling in missing values using SimpleImputer with any chosen strategy. The missing feature column is designated as the output variable y, while the other feature columns are treated as input variables X. A regressor is fitted on (X, y) for known y values. Subsequently, the regressor is used to predict the missing values of y. This iterative process is performed for each feature, and repeated for a maximum of max_iter imputation rounds. The results obtained from the final imputation round are returned.</li></ol><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=a215056a07e3" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Create Your Own Dataset: A Step-by-Step Guide to Web Scraping using Selenium and Python]]></title>
            <link>https://ursmaheshj.medium.com/create-your-own-dataset-a-step-by-step-guide-to-web-scraping-using-selenium-and-python-b4a6da73b71d?source=rss-5113a4c6566c------2</link>
            <guid isPermaLink="false">https://medium.com/p/b4a6da73b71d</guid>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[selenium]]></category>
            <category><![CDATA[data-cleaning]]></category>
            <category><![CDATA[web-scraping]]></category>
            <dc:creator><![CDATA[Mahesh Jadhav]]></dc:creator>
            <pubDate>Sat, 13 May 2023 09:59:51 GMT</pubDate>
            <atom:updated>2023-05-17T04:37:01.325Z</atom:updated>
            <content:encoded><![CDATA[<h3>Create Your Own Dataset: A Step-by-Step Guide to Web Scraping using Selenium and Python for Data Collection and Analysis</h3><p>The first and most important requirement in the field of data science is data. However, it’s not always possible to find the required datasets on websites like <a href="https://www.kaggle.com/ursmaheshj">Kaggle </a>or <a href="https://datahub.io/">Datahub</a>. If you’re lucky enough to work for a company, you might have access to data through APIs or Databases. But, most of the time data may not be readily available, and you need to figure out how to create your own datasets from scratch. In such cases, <strong>web scraping</strong> can be a useful tool to gather the required data effectively.</p><p>Web scraping is a technique that extracts information from websites or web pages. This process uses automated software tools to scan the HTML of a website and gather various types of information, such as text, images, and links. The collected data can be examined, manipulated, and used for a variety of purposes like data mining, market research, and machine learning.</p><p>Nowadays, there are many tools available to scrape websites, but some of them may not work if websites generate content dynamically using JavaScript. This is why <a href="https://selenium-python.readthedocs.io/"><strong>Selenium</strong></a><strong> </strong>is becoming increasingly popular as it offers customized configurations and behaves like a real person interacting with the website through a browser.</p><h4>Problem statement:</h4><blockquote>“Conduct an analysis of all the leading IT companies operating in India.<strong>”</strong></blockquote><p>There are various websites available from which you can obtain this data, but we will be using one of the best among them, <a href="https://www.ambitionbox.com/about-us"><strong><em>AmbitionBox</em></strong></a>. It is a career advisory and job search platform based in India. That also provides reviews and details about companies.</p><p><strong>Let’s explore the <em>AmbitionBox </em>website using the browser.</strong></p><ul><li><a href="https://www.ambitionbox.com/list-of-companies?IndustryName=it-services-and-consulting&amp;sort_by=popularity&amp;page=1">https://www.ambitionbox.com/list-of-companies?IndustryName=it-services-and-consulting&amp;sort_by=popularity&amp;page=1</a></li></ul><figure><img alt="alternate image name" src="https://cdn-images-1.medium.com/proxy/1*0siZKUavTzjUZvKhcjztDg.png" /><figcaption>AmbitionBox webpage</figcaption></figure><p><strong>Observing the webpage carefully, let’s see how we can extract the data.</strong></p><ol><li>AmbitionBox provides details for approximately 14,588 companies, but for our purposes, we will be focusing mainly on IT companies.</li><li>Each company is assigned a separate box, which contains all the details regarding the company.</li><li>Each page on the website has around 30 boxes, meaning that one page represents data about 30 companies.</li><li>As there are around 333 pages available, we will have access to data on around 9,990 companies.</li></ol><blockquote>You can get below code in the form of jupyter notebook <a href="https://github.com/ursmaheshj/WebScraping_with_Selenium_and_Python"><strong>here</strong></a> and the extracted dataset is available <a href="https://www.kaggle.com/datasets/ursmaheshj/top-it-companies-in-india"><strong>here</strong></a><strong>.</strong></blockquote><p><strong>Importing all required libraries</strong></p><pre>import pandas as pd<br>from bs4 import BeautifulSoup<br>from selenium import webdriver<br>from selenium.webdriver.chrome.options import Options<br>from selenium.webdriver.chrome.service import Service<br>from time import sleep<br>import re</pre><pre># Setting up the driver path where we have installed the<br># chromedriver<br><br>DRIVER_PATH = &#39;D:\Applications\Chromium\chromedriver_win32&#39;</pre><pre># Creating a Service object using the specified driver_path which<br># enables communication between the Selenium WebDriver and the<br># browser.</pre><pre>s=Service(DRIVER_PATH)</pre><h3>Extracting the data</h3><p>In the next code block, we will use Selenium to iterate through all web pages and extract all the data using BeautifulSoup along with regular expressions. The extracted data will be saved into a dataset. Additionally, we will print the page number after reading each page so that if any issues occur, we can identify the page where the problem occurred.</p><pre>%%time</pre><pre># Create an empty Database</pre><pre>company_df = pd.DataFrame(columns = [ &#39;name&#39;,&#39;rating&#39;,&#39;review&#39;,&#39;tags&#39;,&#39;company_type&#39;,&#39;headquarters&#39;,&#39;age&#39;,&#39;total_emp&#39;,&#39;about&#39;<br>])</pre><pre># setup options parameter for selenium session</pre><pre>options = Options()<br>options.add_argument(&#39;--disable-blink-features=AutomationControlled&#39;)<br>options.add_argument(&quot;--window-size=1920,1200&quot;)</pre><pre># Iterate for loop on all pages</pre><pre>for p in range(1,334):<br>    try:<br>        driver = webdriver.Chrome(service=s,options=options)<br>        url=f&quot;https://www.ambitionbox.com/list-of-companies?IndustryName=it-services-and-consulting&amp;sort_by=popularity&amp;page={p}&quot;<br>        driver.get(url)<br>        soup = BeautifulSoup(driver.page_source,&#39;html.parser&#39;)<br>        companies = soup.find_all(&quot;div&quot;, class_=&quot;company-content-wrapper&quot;)<br><br>        for i in companies:<br>            name = None<br>            rating = None<br>            review = None<br>            tags = None<br><br>            company_type = None<br>            headquarters = None<br>            age = None<br>            total_emp = None<br><br>            about = None<br><br>            try:<br><br>                name = i.find(&#39;h2&#39;,class_=&#39;company-name bold-title-l&#39;).text.strip()<br>                rating = i.find(&#39;p&#39;,class_=&#39;rating&#39;).text.strip()<br>                review = i.find(&#39;a&#39;,class_=&#39;review-count sbold-Labels&#39;).text.strip()<br>                tags = &#39;,&#39;.join([t.find(&#39;a&#39;,class_=&#39;ab_chip body-medium&#39;).text.strip() for t in i.find(&#39;ul&#39;,class_=&#39;chips-block&#39;)])<br><br>                names = i.find(&#39;div&#39;,class_=&#39;company-basic-info&#39;)<br><br>                for n in names.find_all(&#39;p&#39;):<br>                    try:<br>                        if n.find(&#39;i&#39;,class_=&#39;icon-domain&#39;): company_type = n.text.strip()<br>                        if n.find(&#39;i&#39;,class_=&#39;icon-pin-drop&#39;): headquarters = n.text.strip()<br>                        if n.find(&#39;i&#39;,class_=&#39;icon-access-time&#39;): age = n.text.strip()<br>                        if n.find(&#39;i&#39;,class_=&#39;icon-supervisor-account&#39;): total_emp = n.text.strip()<br>                    except:<br>                        continue<br><br>                about = i.find(&#39;p&#39;,class_=&#39;description&#39;).text.strip()<br>                <br>            except:<br>                continue<br>            finally:<br>                company_df.loc[len(company_df)] = [name,rating,review,tags,company_type,headquarters,age,total_emp,about]<br>                <br>        <br>        sleep(1)<br>    except Exception as e:<br>        print(f&quot;Something went wrong: {e}&quot;)<br>    finally:<br>        driver.quit()<br>        print(p, end=&#39;|&#39;)<br>    <br>print(&#39;done&#39;)</pre><pre>1|2|3|4|5|6|7|8|9|10|11|12|13|14|15|16|17|18|19|20|21|22|23|24|25|26|27|28|29<br>|30|31|32|33|34|35|36|37|38|39|40|41|42|43|44|45|46|47|48|49|50|51|52|53|54|<br>55|56|57|58|59|60|61|62|63|64|65|66|67|68|69|70|71|72|73|74|75|76|77|78|79|80<br>|81|82|83|84|85|86|87|88|89|90|91|92|93|94|95|96|97|98|99|100|101|102|103|104<br>|105|106|107|108|109|110|111|112|113|114|115|116|117|118|119|120|121|122|123|<br>124|125|126|127|128|129|130|131|132|133|134|135|136|137|138|139|140|141|142|<br>143|144|145|146|147|148|149|150|151|152|153|154|155|156|157|158|159|160|161|<br>162|163|164|165|166|167|168|169|170|171|172|173|174|175|176|177|178|179|180|<br>181|182|183|184|185|186|187|188|189|190|191|192|193|194|195|196|197|198|199|<br>200|201|202|203|204|205|206|207|208|209|210|211|212|213|214|215|216|217|218|<br>219|220|221|222|223|224|225|226|227|228|229|230|231|232|233|234|235|236|237|<br>238|239|240|241|242|243|244|245|246|247|248|249|250|251|252|253|254|255|256|<br>257|258|259|260|261|262|263|264|265|266|267|268|269|270|271|272|273|274|275|<br>276|277|278|279|280|281|282|283|284|285|286|287|288|289|290|291|292|293|294|<br>295|296|297|298|299|300|301|302|303|304|305|306|307|308|309|310|311|312|313|<br>314|315|316|317|318|319|320|321|322|323|324|325|326|327|328|329|330|331|332|<br>333|done<br>CPU times: total: 31.3 s<br>Wall time: 44min 14s</pre><p>Let’s explore the dataset created</p><pre>company_df.head()</pre><figure><img alt="png" src="https://cdn-images-1.medium.com/proxy/1*6iMKqHIakkINh_INYhADEg.png" /></figure><h3>Manipulating the Data</h3><p>We have obtained the data, but it is not in a format that will be useful for analysis. Therefore, we need to clean the data and retain only the necessary information.<br>To ensure that we do not lose our original data in case anything goes wrong, we will perform these manipulations on a copy of the original dataset.</p><pre>df = company_df.copy()</pre><h4>1. review</h4><p>This column represents numerical data, but Pandas has assigned it as a string. We will create a function that extracts only numerical data from the string.</p><pre>def transform_review(data):<br>    try:<br>        if re.search(r&#39;\d+k&#39;,data):<br>            return float(re.search(r&#39;\d+\.\d+|\d+&#39;,data).group()) * 1000<br>        elif re.search(r&#39;\d+&#39;,data):<br>            return float(re.search(r&#39;\d+&#39;,data).group())<br>        else:<br>            return 0<br>    except:<br>        return 0</pre><pre>df[&#39;review&#39;] = df[&#39;review&#39;].apply(transform_review)</pre><pre>df.head(2)</pre><figure><img alt="png" src="https://cdn-images-1.medium.com/proxy/1*WP_WM8Dg7tms90mHWyoN-A.png" /></figure><h4>2. headquarters</h4><p>headquarters column consist of both the actual headquarters location and the number of offices that the company has. We can separate this information into two distinct columns, namely “headquarter” and “total_offices”.</p><ul><li>headquarter</li></ul><pre>def transform_headquarter(data):<br>    try:<br>        return data.split(&#39;+&#39;) [0].strip()<br>    except:<br>        return None</pre><pre>df[&#39;headquarter&#39;] = df[&#39;headquarters&#39;].apply(transform_headquarter)</pre><ul><li>total_offices</li></ul><pre>def transform_total_offices(data):<br>    try:<br>        if re.search(r&#39;\d+&#39;,data):<br>            data = data.split(&#39;+&#39;) [1].strip()<br>            return int(re.search(r&#39;\d+&#39;,data).group()) + 1<br>        else:<br>            return 1<br>    except:<br>        return None</pre><pre>df[&#39;total_offices&#39;] = df[&#39;headquarters&#39;].apply(transform_total_offices)</pre><p>Now that we have successfully separated and added our columns, we can delete the unnecessary column ‘headquarters’.</p><pre>del df[&#39;headquarters&#39;]</pre><pre>df.head(2)</pre><figure><img alt="png" src="https://cdn-images-1.medium.com/proxy/1*ULsHZtLvjpBi2VFkUzbaOw.png" /></figure><h4>3. age</h4><p>This simply displays the age of the company from the year of its establishment. We can convert this numerical data into a date format and label it as ‘founded_year’, to gain a better understanding.</p><pre>def transform_age(data):<br>    try:<br>        return int(pd.to_datetime(&#39;today&#39;).date().year - int(re.search(r&#39;\d+&#39;,data).group()))<br>    except:<br>        return None</pre><pre>df[&#39;founded_year&#39;] = df[&#39;age&#39;].apply(transform_age)</pre><pre>del df[&#39;age&#39;]</pre><pre>df.head(2)</pre><figure><img alt="png" src="https://cdn-images-1.medium.com/proxy/1*O53YYJT8wb4uYP_YV1krsw.png" /></figure><h4>4. total_emp</h4><p>Due to the mixture of text and numbers, the data is displayed as string values instead of numeric values. We will convert this data into numeric format. Please note that this is not the actual number of employees, but instead represents the minimum count of employees working in India.</p><pre>def transform_total_emp(data):<br>    try:<br>        values = list(map(int,re.findall(r&#39;\d+\.\d+|\d+&#39;,data)))<br>        units = re.findall(r&#39;Lakh|k&#39;,data)<br>        if len(values)==0:<br>            return 0<br>        elif len(values)==1:<br>            if units[0] == &#39;Lakh&#39;:<br>                return values[0] * 100000<br>            elif units[0] == &#39;k&#39;:<br>                return values[0] * 1000<br>            else:<br>                return values[0]<br>        elif len(values)==2:<br>            if len(units)==0:<br>                return (values[0]+values[1])/2<br>            elif len(units)==1:<br>                return values[0]+(values[1]*1000)/2<br>            elif len(units)==2:<br>                if units[0] == units[1] == &#39;Lakh&#39;:<br>                    return (values[0]+values[1])/2 * 100000<br>                elif units[0] == units[1] == &#39;k&#39;:<br>                    return (values[0]+values[1])/2 * 1000<br>                else:<br>                    return ((values[0]*1000) + (values[1]*100000))/2 <br>        else:<br>            return 0<br>    except:<br>        return None</pre><pre>df[&#39;total_emp&#39;] = df[&#39;total_emp&#39;].apply(transform_total_emp)</pre><pre>df.head(2)</pre><figure><img alt="png" src="https://cdn-images-1.medium.com/proxy/1*uCE49cd3GJuopu_IifEXRw.png" /></figure><pre>df.info()</pre><pre>&lt;class &#39;pandas.core.frame.DataFrame&#39;&gt;<br>Int64Index: 9990 entries, 0 to 9989<br>Data columns (total 10 columns):<br> #   Column         Non-Null Count  Dtype  <br>---  ------         --------------  -----  <br> 0   name           9990 non-null   object <br> 1   rating         9990 non-null   object <br> 2   review         9990 non-null   float64<br> 3   tags           9990 non-null   object <br> 4   company_type   7344 non-null   object <br> 5   total_emp      9101 non-null   float64<br> 6   about          6699 non-null   object <br> 7   headquarter    9171 non-null   object <br> 8   total_offices  9166 non-null   float64<br> 9   founded_year   7675 non-null   float64<br>dtypes: float64(4), object(6)<br>memory usage: 858.5+ KB</pre><p>Data has been cleaned successfully let’s correct the datatype of columns to store it in an efficient way</p><pre>col_dtypes = {<br>                &#39;name&#39;: str,<br>                &#39;rating&#39;: int,<br>                &#39;review&#39;: float,<br>                &#39;tags&#39;: str,<br>                &#39;company_type&#39;: str,<br>                &#39;total_emp&#39;: float,<br>                &#39;about&#39;: str,<br>                &#39;headquarter&#39;: str,<br>                &#39;total_offices&#39;: int,<br>                &#39;founded_year&#39;: int,<br>            }<br>df = df.astype(col_dtypes,errors=&#39;ignore&#39;)</pre><p><strong>Let’s check the entire dataset once again.</strong></p><pre>df.head()</pre><figure><img alt="png" src="https://cdn-images-1.medium.com/proxy/1*9s8vA5jCvQcZCOn0TV-dCQ.png" /></figure><p>We have all the necessary details, so now we can save the dataset in the desired format.</p><pre>df.to_csv(&#39;AmbitionBox_Dataset.csv&#39;,index=False)</pre><blockquote>You can get the original python jupyter notebook <a href="https://github.com/ursmaheshj/WebScraping_with_Selenium_and_Python"><strong><em>here</em></strong></a><strong><em> </em></strong>and the extracted dataset is available <a href="https://www.kaggle.com/datasets/ursmaheshj/top-it-companies-in-india"><strong><em>here</em></strong></a><strong>.</strong></blockquote><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=b4a6da73b71d" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Why do we need DATA? Types of Data in statistics]]></title>
            <link>https://ursmaheshj.medium.com/why-do-we-need-data-types-of-data-65bb237e9c11?source=rss-5113a4c6566c------2</link>
            <guid isPermaLink="false">https://medium.com/p/65bb237e9c11</guid>
            <category><![CDATA[statistics]]></category>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[data-analysis]]></category>
            <category><![CDATA[artificial-intelligence]]></category>
            <dc:creator><![CDATA[Mahesh Jadhav]]></dc:creator>
            <pubDate>Sun, 23 Apr 2023 11:15:33 GMT</pubDate>
            <atom:updated>2023-06-04T06:02:38.808Z</atom:updated>
            <content:encoded><![CDATA[<h3>What is DATA? Types of Data in statistics</h3><p>As technology continues to advance machine learning is becoming incredibly popular and effective. But, without data, machine learning is as useless as mobile without battery. It is the data that gives machine learning its power. And, as we are realizing this, the value of data is increasing day by day, and so is the need to learn and implement statistics to better understand and interpret the data</p><h3><em>What actually does the </em>DATA<em> mean?</em></h3><p>According to <a href="https://en.wikipedia.org/wiki/Data">Wikipedia</a>, <strong>data</strong> is a collection of discrete values that convey information, describing the quantity, quality, fact, statistics, other basic units of meaning, or simply sequences of symbols that may be further interpreted.</p><p><strong>Data</strong> can be seen as the smallest units of factual information that can be used as a basis for calculation, reasoning, or discussion</p><p>The word “<strong>data</strong>” was first used to mean “transmissible and storable computer information” in 1946.</p><h3>Why do we need Data?</h3><p>Without data, we would just make wild guesses and jump to the wrong conclusion. Data plays an important role in helping us make informed decisions and understand the world around us.</p><p>Data allows us to identify trends, patterns, and relationships that would otherwise be invisible and difficult to understand.</p><p>In short, data is essential to make correct decisions, conduct research, test hypotheses, and validate theories.</p><blockquote>As we understand what DATA is and why we need it, let’s understand where and how we get IT.</blockquote><h3>Collecting Data</h3><p>Collecting data can be a complex and time-consuming process. There are many different ways of collecting data that can be used depending on the factors like type of data needed and the scope of research. These data collection techniques can be divided into two types.</p><ul><li><strong>Primary Data Collection</strong>: Process of collecting data directly from the resource or original research methods.<br><em>e.g. surveys, interviews, observations, and experiments.</em></li><li><strong>Secondary Data Collection:</strong> Process of gathering data from existing sources.<br><em>e.g. government reports, academic studies, and business reports.</em></li></ul><blockquote>Data<em> is useless until it is analyzed and understood. Understanding the characteristics and properties of data is crucial to know the nature of the data.</em></blockquote><h3><em>Types of </em>DATA</h3><p>By dividing data into different types, we can better understand its nature and derive meaningful insights from it.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Q2Nr6L3AylzLiVAYs7o1wg.jpeg" /><figcaption>Types of data in statistics</figcaption></figure><h4><strong>Numerical Data/Quantitative Data</strong></h4><p>Numerical Data represents numbers or numerical values. It gives quantitative information about a specific thing like how much or how many. Numerical data can be divided into two types based on the nature of the data.</p><ul><li><strong>Continuous Data:</strong> Continuous data is a type of numerical data that can be measured and take on any value within a given specific range, with an infinite number of possible values between any two values. Continuous data can be represented as a range of values instead of specific individual values.<br><strong><em>e.g. </em></strong><em>Height, Weight, Temperature.</em></li><li><strong>Discrete Data:</strong> Discrete data is a kind of numerical data that consists of discrete individual values. These values can be counted and shown using whole numbers or integers. Unlike continuous data, discrete data can only have a finite number of possible values, and they can’t be divided into smaller parts that are meaningful.<br><strong><em>e.g.</em></strong><em> Number of students in a class, Sales happened in one year, Age.</em></li></ul><h4><strong>Categorical Data/Qualitative Data</strong></h4><p>Categorical Data is also known as Qualitative Data because it describes the qualities or characteristics of the subject. Categorical data mostly consists of textual information which represents the features or nature of the subject.</p><ul><li><strong>Nominal Data:</strong> Nominal Data represents the type of qualitative information that can not be ordered or ranked. Nominal data can be grouped into categories.<br><strong><em>e.g.</em></strong><em> Gender, Hair color, Country.</em></li><li><strong>Ordinal Data:</strong> Ordinal Data represents the type of qualitative information that follows a natural order or ranking. It is often used to represent subjective ratings or opinions.<br><strong><em>e.g.</em></strong><em> Customer ratings(1–5), Education levels(school, college, graduate).</em></li></ul><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=65bb237e9c11" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Unlocking the Power of Machine Learning: A Beginner’s Guide to Understanding Algorithms and Models]]></title>
            <link>https://ursmaheshj.medium.com/unlocking-the-power-of-machine-learning-a-beginners-guide-to-understanding-algorithms-and-models-72167d30f2ce?source=rss-5113a4c6566c------2</link>
            <guid isPermaLink="false">https://medium.com/p/72167d30f2ce</guid>
            <category><![CDATA[artificial-intelligence]]></category>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[towards-data-science]]></category>
            <category><![CDATA[data-scientist]]></category>
            <category><![CDATA[data-science]]></category>
            <dc:creator><![CDATA[Mahesh Jadhav]]></dc:creator>
            <pubDate>Tue, 18 Apr 2023 04:51:49 GMT</pubDate>
            <atom:updated>2023-05-13T11:24:01.455Z</atom:updated>
            <content:encoded><![CDATA[<p>Oh man, learning machine learning can be quite an adventure! As I dive into its depths, I realized just how complex and confusing it can be. With the plethora of models and algorithms involved, understanding each one can feel like a never-ending journey. And don’t even get me started on the challenge of perfectly understanding each individual algorithm, as soon as I began to understand a new algorithm, I found myself forgetting the earlier ones. It was like trying to figure out the timeline of the Marvel Cinematic Universe. So, I asked myself, “What can I do to make sense of this madness?” And that’s when I had an Idea.</p><p>I decided to take an eagle’s-eye view of the situation and created a map of algorithms depicting machine learning, complete with all its subsets and famous models. This gave me a broader understanding of which algorithm belongs to what model. I also wrote down the description of each algorithm as briefly as possible, giving me an instant idea of its purpose.</p><p>This approach actually worked! It helped me to understand and grasp the machine learning models more easily, and I’m confident it can do the same for you too. So, take a deep breath, put on your earbuds, and let’s tackle machine learning together!</p><p><strong>Machine learning </strong>is a branch of <a href="https://towardsdatascience.com/introductory-guide-to-artificial-intelligence-11fc04cea042"><em>Artificial Intelligence (AI)</em></a> and computer science that involves statistical and mathematical approaches to find patterns and insights into the data which is then used for building algorithms and models that can learn and make predictions without external instructions. There are many machine learning algorithms that can be broadly classified into 4 types <em>Supervised learning, Unsupervised Learning, Semi-Supervised learning, </em>and<em> Reinforcement learning</em> as shown below.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*eMj6PTmHUw2RlR8RIZTagg.jpeg" /><figcaption>Types of Machine Learning Models</figcaption></figure><h3>Supervised Machine Learning</h3><p>It is a type of machine learning model where algorithms are trained on labeled data to learn the relationships between feature matrix and target variable so that it can make accurate predictions on new, unseen data.</p><h4>Regression Model</h4><p>It provides a set of statistical processes to describe the relationship between independent variables and dependent (target) variables.</p><ul><li><strong>Linear</strong> <strong>Regression:</strong> It is used to predict the value of a continuous dependent variable based on one or more independent variables with the help of a best fit linear line that minimizes the difference between predicted values and the actual values.</li><li><strong>Logistic Regression:</strong> It uses a sigmoid function to estimate the probability of an event occurring based on a given dataset of independent variables. It is primarily used for binary classification problems where the output variable can only take two values (0 | 1).</li></ul><h4>Classification Model</h4><p>Supervised machine learning method where algorithm tries to predict or classify data into predefined categories based on characteristics.</p><ul><li><strong>Decision Tree:</strong> A specific type of probability tree where each node represents a Yes|No type of question and the branch represents outcome of test that enables the algorithm to make decisions.</li><li><strong>Random Forest:</strong> It creates multiple decision trees using a random subset of training data and a random subset of features, which are then combined to make a final prediction.</li><li><strong>KNN (K-Nearest Neighbor):</strong> It is a non-parametric supervised learning method which calculates the K closest neighbors of a given data point in the training set, and then identifies the majority class among those neighbors as the predicted class for that data point. K’s value is a hyperparameter that can be adjusted to improve algorithm’s efficiency.</li><li><strong>SVM (Support Vector Machine):</strong> SVM tries to separate data by drawing a plane between them. We choose the plane in such a way that it creates the largest possible gap between the two types of data. This gap is called the margin. The points closest to the plane are called support vectors. By using these support vectors, we can classify new data into one of the two groups.</li><li><strong>Naive Bayes Classification:</strong> It predicts outcomes based on a set of input features. It uses probability theory to determine the likelihood of an outcome based on the presence or absence of certain features. The “naive” part of its name comes from the assumption that each feature is independent of the others, which simplifies the calculations.</li></ul><h3>Unsupervised Machine Learning</h3><p>Type of Machine learning technique where the algorithm learns to recognize patterns in the data without the use of labeled examples or external guidance.</p><h4>Clustering Models</h4><p>These are machine learning models used to group similar data points together based on their features. They don’t rely on labeled data to make predictions. Instead, they try to find patterns and relationships in the data on their own.</p><ul><li><strong>K - Means Clustering:</strong> This algorithm partitions the dataset into K-Clusters where K is a predefined number. Its goal is to optimize the sum of squared distance between data points and their assigned cluster centroid.</li><li><strong>Hierarchical Model:</strong> This method starts by considering each item as its own cluster, and then repeatedly merges the closest pairs of clusters until all items are in a single group. The resulting tree can be cut at different levels to obtain different levels of clustering. It is useful for exploring relationships between items in a dataset and identifying natural groupings using a tree-like structure called a dendrogram.</li><li><strong>DBSCAN:</strong> Density-based spatial clustering of applications with noise does not require the number of clusters to be specified beforehand. Instead, it determines clusters based on the density of points in the data set. Points that are close together and have high density are considered part of the same cluster, while points that are isolated or have low density are classified as noise.</li><li><strong>GMM (Gaussian Mixture Model):</strong> It assumes that the data comes from a mixture of many Gaussian distributions. Each of these distributions represents a different group or cluster in the data. The model tries to estimate the parameters of these distributions (such as mean and variance) and the probability of each data point belonging to each distribution</li><li><strong>Spectral Clustering Model:</strong> It creates a similarity graph of the data set, where each data point is represented as a node in the graph and the edges between the nodes represent their pairwise similarity. Spectral clustering then uses the graph’s spectral properties to identify clusters in the data by finding the eigenvectors (principal components) of the graph’s Laplacian matrix to project data into lower-dimensional space, where it is easier to identify clusters.</li><li><strong>Mean-Shift Clustering:</strong> It works by finding the centroids or mean points of each cluster by shifting a window around the data space to the areas of higher density. The window shifts towards the steepest ascent until it reaches a peak where the density is highest, and this peak is then designated as the centroid of the cluster. The window is again moved to a new peak and this process is repeated until all centroids are found.</li><li><strong>SOM (Self-Organizing Map):</strong> It is a type of artificial neural network that works by mapping a high-dimensional data set onto a low-dimensional grid of neurons in a way that preserves the topological relationship between the data points. Each neuron in the grid represents a cluster or a group of similar data points.</li></ul><h4>Dimensionality Reduction Models</h4><p>These are techniques in machine learning that simplify complex data sets by reducing the number of features or variables while retaining the most important information. They help to remove noise and redundant information from the data, making it easier to analyze and visualize.</p><ul><li><strong>PCA (Principal Component Analysis):</strong> It transforms the data set into a new coordinate system, where the new axes represent principal components that capture the maximum variance in the data. These principal components are linear combinations of the original features, and they are ordered by the amount of variance they explain in the data. By projecting the data onto the first few principal components, PCA can reduce the dimensionality while retaining most important information.</li><li><strong>NMF (Non-negative Matrix Factorization): </strong>It works by factorizing a non-negative matrix (V) into two lower-rank non-negative matrices W and H, such that V=WH. Here, V is a data matrix with rows representing samples and columns representing features, W is a basis matrix with rows representing basis vectors, and H is a coefficient matrix with columns representing coefficients.</li><li><strong>LDA (Linear Discriminant Analysis):</strong> It works by finding a linear combination of features that maximizes the separation between different classes or categories in the data. By projecting the data onto this new linear subspace, LDA can reduce the dimensionality of the data while retaining the most important information for classification.</li><li><strong>ICA (Independent Component Analysis):</strong> It finds a set of independent components that represent the underlying sources of variation in a complex data set. By separating the data into its independent components, ICA can reduce the dimensionality of the data and extract the most important features or signals.</li><li><strong>Autoencoder:</strong> It works by compressing the input data into a lower-dimensional representation, and then reconstructing it back to its original form. By minimizing the difference between the input and output data, the autoencoder can learn a compact and informative representation of the input data.</li></ul><h3>Semi-Supervised Learning</h3><p>A machine learning approach where a model is trained on a combination of labeled and unlabeled data. In this approach, the labeled data is used to teach the model to recognize patterns and make predictions, while the unlabeled data is used to improve the model’s generalization and robustness.</p><ul><li><strong>Self-Training Model:</strong> The model is trained on a small set of labeled data, and then used to predict the labels of a larger set of unlabeled data. The predicted labels are then added to the labeled dataset, and the model is retrained on the expanded dataset. This process is repeated iteratively, with the model learning from its own predictions and gradually improving its accuracy on the unlabeled data.</li><li><strong>Co-Training Model:</strong> Two separate models are trained on different subsets of data, one with labeled data and the other with unlabeled data. The models exchange information by using their own predictions as additional labeled data for the other model. This iterative process is repeated, with the models learning from each other’s predictions and gradually improving their accuracy on the unlabeled data.</li><li><strong>Generative Model:</strong> The model learns to identify patterns in the labeled data and then uses those patterns to generate new examples. These generated examples are then used to improve the model’s performance on the unlabeled data.</li><li><strong>Label Propagation:</strong> It learns from both labeled and unlabeled data to make predictions on the unlabeled data. The model first assigns labels to the labeled data and then propagates these labels to the unlabeled data based on the similarity between data points. The model iteratively updates the labels of the unlabeled data until a certain convergence criterion is met.</li><li><strong>Semi-Supervised Clustering:</strong> The model uses the labeled data to learn the initial structure of the clusters and then groups the unlabeled data points based on their similarity to the labeled data points. The model iteratively refines the clustering results using both labeled and unlabeled data until the goal is reached.</li></ul><h3>Reinforcement Learning</h3><p>It is a machine learning technique that involves training an agent to make decisions in an environment to maximize a reward. The agent learns through trial and error, receiving feedback in the form of rewards or penalties based on its actions. The goal is for the agent to learn the optimal policy, or sequence of actions, that will result in the highest total reward over time.</p><h4>Model-Based Learning</h4><p>It involves learning a model of the environment in addition to learning the optimal policy. The model is a representation of the dynamics of the environment, which the agent can use to predict the next state and reward given the current state and action. The agent can then use these predictions to plan ahead and select the action that will lead to the highest expected reward.</p><ul><li><strong>MPC (Model Predictive Control):</strong> In this algorithm the agent solves an optimization problem at each time step, using the predictive model to simulate future states and rewards. The solution to the optimization problem gives the best sequence of actions to take in the immediate future. MPC is useful when the environment is complex and has a long time horizon, as it allows the agent to plan ahead and make optimal decisions based on predicted outcomes.</li><li><strong>DP (Dynamic Programming): </strong>It involves learning a value function or a policy by solving a system of Bellman equations. It works by iteratively updating the value of each state based on the values of its neighboring states, until convergence. This method is best suited for environments with known dynamics, where the optimal policy can be computed analytically.</li><li><strong>iLQR (iterative Linear Quadratic Regulator):</strong> It is used to solve optimal control problems. It involves iteratively solving a set of linear-quadratic subproblems to find a sequence of control inputs that minimizes a cost function while satisfying a set of constraints. It is often used in robotics and control systems to find optimal control policies for complex systems.</li><li><strong>Dyna-Q:</strong> It is a model-based reinforcement learning algorithm used to find the optimal action policy in a Markov decision process (MDP). It combines model-free and model-based methods to make predictions and learn from experience. It maintains a Q-table to approximate the optimal action-value function and also learns a model of the environment to plan future actions.</li></ul><h4>Model-Free Learning</h4><p>technique where an agent learns to make decisions through trial and error without having explicit knowledge of the environment or a model of it. The agent learns by directly interacting with the environment and updating its policy based on the observed rewards.</p><ul><li><strong>Q-Learning:</strong> It enables an agent to learn to make optimal decisions in an environment by finding the best actions to take based on the current state. The algorithm updates its Q-values, which represent the expected reward for taking a particular action in a particular state, through trial and error by exploring the environment and receiving rewards. The agent continues to learn and improve its decision-making skills over time by adjusting its Q-values based on the rewards it receives, and ultimately, it aims to learn the optimal policy for maximizing long-term rewards.</li><li><strong>SARSA</strong> <strong>(State-Action-Reward-State-Action):</strong> It learns from experiences (trial and error). The agent interacts with the environment, and at each time step, it observes the current state, takes an action, receives a reward, and observes the next state. SARSA algorithm updates its Q-values based on the current observed state, the action taken, the reward received, and the next state and the action that the agent chooses. Unlike Q-Learning, SARSA uses the same policy to select actions during learning and execution, making it an on-policy algorithm.</li><li><strong>DQN (Deep Q-network): </strong>This algorithm combines the Q-learning algorithm with deep neural networks. It is used to solve problems where the state space is too large to be handled by traditional Q-learning. The DQN algorithm works by approximating the Q-value function using a deep neural network and uses it to update the Q-values iteratively. The neural network is trained on the experiences collected from the environment using a technique called experience replay.</li><li><strong>Monte Carlo Method:</strong> It uses repeated random sampling to estimate the value of an action in a given state. It does not require knowledge of the transition probabilities between states or the rewards associated with taking actions. Instead, it learns by randomly exploring the environment and accumulating rewards over time to update the Q-values of each action-state pair.</li><li><strong>Policy Gradient Method:</strong> It involves training a neural network to directly output the policy that maximizes the expected reward. Unlike value-based methods such as Q-learning, policy gradient methods do not estimate the value of each action, but instead optimize the policy directly. The algorithm uses stochastic gradient descent to update the parameters of the policy network to maximize the expected reward.</li><li><strong>Actor-Critic Method:</strong> Type of reinforcement learning where there are two models working together: the actor, which decides on actions to take, and the critic, which evaluates the actions taken by the actor. The actor model determines the best action to take in a given state, and the critic model evaluates how good the chosen action is. By working together, the actor-critic method aims to learn an optimal policy that maximizes rewards over time.</li></ul><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=72167d30f2ce" width="1" height="1" alt="">]]></content:encoded>
        </item>
    </channel>
</rss>