<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Moosa Ali on Medium]]></title>
        <description><![CDATA[Stories by Moosa Ali on Medium]]></description>
        <link>https://medium.com/@moosa-ali?source=rss-246f2656ec1a------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/1*zeCu6DgvKgV_bElY_s6oTg.png</url>
            <title>Stories by Moosa Ali on Medium</title>
            <link>https://medium.com/@moosa-ali?source=rss-246f2656ec1a------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Sat, 21 Nov 2026 21:03:26 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@moosa-ali/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[ChatGPT, Bard and the Problem With Large Language Models]]></title>
            <link>https://moosa-ali.medium.com/chatgpt-bard-and-the-problem-with-large-language-models-644694cd3d3d?source=rss-246f2656ec1a------2</link>
            <guid isPermaLink="false">https://medium.com/p/644694cd3d3d</guid>
            <category><![CDATA[chatbots]]></category>
            <category><![CDATA[chatgpt]]></category>
            <category><![CDATA[google]]></category>
            <category><![CDATA[nlp]]></category>
            <category><![CDATA[ai]]></category>
            <dc:creator><![CDATA[Moosa Ali]]></dc:creator>
            <pubDate>Sun, 19 Mar 2023 15:21:49 GMT</pubDate>
            <atom:updated>2023-03-19T15:21:49.856Z</atom:updated>
            <content:encoded><![CDATA[<h4>Use with caution!</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*ITqVTMuMgHrs0pQGAya4Fg.png" /></figure><p>AI has finally taken over the common man, with large language models (LLM) taking the charge. With the introduction of applications like <a href="https://openai.com/blog/chatgpt">OpenAIs ChatGPT</a> and <a href="https://blog.google/technology/ai/bard-google-ai-search-updates/">Google’s BARD</a>, conversational AI has become the talk of the town.</p><p>These chatbots accept your questions and queries and reply like any human would. People have been using these for all sorts of cool stuff like asking them about the weather or write a program in their favorite language and even complete essays. However, what most people don’t realize is that like every AI, LLMs have their limitations.</p><p>In this article I’ll be explaining some of the short comings of modern-day conversational AI and why you should use it with caution. But first, let’s take an overview of the two popular models.</p><h4>What are ChatGPT and BARD?</h4><p>ChatGPT was first announced on Nov 30, 2022 and soon became a global sensation. It gathered 100 million users in only a span of 2 months, creating a record for the <a href="https://www.reuters.com/technology/chatgpt-sets-record-fastest-growing-user-base-analyst-note-2023-02-01/">fastest growing</a> application. ChatGPT has been toyed around my millions of users every day since its launch and the results it provides are no less than a shock. Some of its most popular applications are</p><ul><li><strong>All-in-one writer: </strong>ChatGPT can construct full essays, reports, summarize text, write business proposals and the list just goes on.</li><li><strong>All-in-one code writer: </strong>You can ask ChatGPT to write a piece of code in your favorite programming language. It literally takes it a few seconds to generate fully working code.</li><li><strong>Code debugger: </strong>ChatGPT can review your faulty code and provide a list (and fixes) of everything wrong with it. <a href="https://arxiv.org/abs/2301.08653">A study</a> found that its debugging accuracy was on par with some famous deep-learning models like CoCoNut and Codex, but with the added benefit of an easier interface.</li><li><strong>Acting as an SQL engine: </strong>Yes! you can ask ChatGPT to mimic an SQL engine. In easier terms, you can provide it with SQL queries and ask it to compute and return the expected results.</li></ul><p>Some really impressive use-cases no doubt.</p><p><a href="https://blog.google/technology/ai/bard-google-ai-search-updates/">BARD</a>, on the other hand, in Googles response to ChatGPT. It was first announced by Google’s CEO, Sundar Pichai in early Feb, 2023. It is built on top of the Google’s own language model, <a href="https://blog.google/technology/ai/lamda/">LaMDA</a> and will be released as a direct competitor to ChatGPT.</p><p>Bard is currently only released to a closed group of testers and the model used is a light-weight version of LaMDA. Bard one-ups ChatGPT by harnessing the internet to provide up to date information and relevant contexts.</p><h4>Where Does the Problem Lie?</h4><p>With all the hype that has been going around, there little room left for any skepticism. However, I am here to tell you that not all that glitters is gold.</p><p>A large language model, such as ChatGPT develops its knowledge base from the internet. At first glance, this seems alright until you realize that the internet is filled with, both, facts and opinions. Many of these opinions are incorrect or even controversial.</p><p>When a language model is trained on all such information, it is difficult to imagine which of it will retain. The development teams can only do a limited job at fact checking and sooner or later an anomaly is bound to leak into the training dataset.</p><h4>The BARD Blunder</h4><p>A prime example of such misinformation was seen in BARD’s initial demo that caused Google to lose $100 Billion in shares.</p><p>Google released a short advert, showcasing its conversational AI model and its capabilities. However, the hype and excitement were short lived as the demo was not as accurate as it seemed. It showed a conversational prompt where a person asked the AI about the new discoveries made by the James Webb Space Telescope (Screenshot below).</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/768/0*44Oz4eUIf4yKOwGn.png" /><figcaption>Credits: Google</figcaption></figure><p>At first glance the result seems impressive as the model returns some points with high confidence. However, soon some astrounauts started pointing out the mistake in the third bullet point. They claimed that the JWST was, in fact, not the first telescope to capture an exoplanet.</p><p>To understand why the model gave such an answer, I decided to query the same question on Google(Screenshots below).</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*wTsHNole-BQgzgCrTgxk_w.png" /><figcaption>Google Search Results for “Which telescope took the first image of an exoplanet?”</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*nSRzFJZ9BNpNnqumCSjYyw.png" /><figcaption>Google Search Results for “Which telescope took the first image of an exoplanet?”</figcaption></figure><p>As you can see that the most prominent answer here is the James Webb Space Telescope. Upon closer inspection, we see that all the articles are actually referring to the telescope capturing ITS OWN very first exoplanet and not the first of all time. This is a neat mix up of words due to the structure of the question.</p><p>Since the knowledge graph-based search engine could not return what I was looking for, it is no shock that the AI bot made the same blunder. Google BARD is trained on this same data scrapped off the internet and it is very likely that it learned this misleading information as it ranks top in Google search.</p><p>For further clarification I asked ChatGPT the same question.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/837/1*NUDRPxVeMERzwpc34mXvEA.png" /></figure><p>OpenAI’s product seems to realize that an exoplanet was pictured long before the JWST.</p><blockquote>*A mildly important note here. It is possible that OpenAI tweaked their model to give a different answer after Google BARDs disaster. *</blockquote><p>So is this information correct? Let’s find out.</p><p><a href="https://en.wikipedia.org/wiki/Grant_Tremblay">Grant Tremblay</a>, an astrophysicist who pointed out BARDs mistake seemed to have a different opinion.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/458/1*43-QnhMBr4OhBNtOshV_vQ.png" /></figure><p>He pointed out that the first image was captured by <a href="https://arxiv.org/pdf/astro-ph/0409323.pdf">Chauvin et al.</a> in 2004.</p><p>So does ChatGPT also not know any better? Let’s try a different search query. This time we’ll omit the requirement for the picture being taken from a telescope.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/784/1*ADTRHQg0dNOop_nZSBU1cw.png" /></figure><p>Well well well…</p><p>Now the OpenAI variant seems to agree that the first picture was captured in 2004 and it explains that it was via the <strong>Very Large Telescope</strong> in Chile.</p><p>Now there might be technical explanations behind these different answers but I am not an expert in astrophysics, nor do I have the knowledge to confirm or deny any of these claims.</p><p>To a layman, whatever he or she reads, will become the truth. The sole purpose of mentioning all of this is to demonstrate the information ambiguity that might plague large language models.</p><p>There are also examples where people have fooled ChatGPT into confidently outputting misinformation using a series of carefully crafted prompts.</p><p>The point is that while most people have deemed conversational AI as the game changer, it should be used with caution.</p><h4>How Dangerous Can Misinformation Get?</h4><p>Up till now ChatGPT is only used as a toy by majority of its users. People ask it all sorts of weird, silly and challenging questions and enjoy the light-hearted conversation that ensues.</p><p>However, many others have started using it in their day-to-day tasks and this is where the danger lies.</p><p>Imagine students tutoring themselves by asking AI bots subject related questions. If the bot continues to output factually incorrect statements, the student&#39;s education could be affected and result in poor grades.</p><p>Similarly, if used to automate medical related tasks, any misinformation could be the difference between life and death.</p><h4>So We Say No to Artificial Intelligence?</h4><p>Absolutely not. The only way forward is to embrace technology, not resist it. Like all innovations, conversational AI tools are not yet perfect, but this does not mean that we ditch them entirely.</p><p>All good things require time, before they are adequately polished. All we, as users, need to understand is the proper usage. Some useful tips are</p><ul><li>Do not blindly implement AI into mission-critical workflows</li><li>Double-check any information from actual experts</li><li>For now, use artificial intelligence as a second opinion, not as the single source of truth</li></ul><h4>Final Words</h4><p>As an AI engineer myself, I am proud to say mankind is witnessing another revolution. The pace at which AI applications are hitting the market is unprecedented. This is no longer a matter of decades, but years and soon it might just be months.</p><p>Some of you might be afraid for your jobs but being resistant will not get you anywhere. It is important to realize the importance of these breakthroughs and start leveraging them for your advantage. As the famous saying goes</p><blockquote>AI might not replace you, but a person using AI will</blockquote><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=644694cd3d3d" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Best Libraries for Fuzzy Matching In Python]]></title>
            <link>https://medium.com/codex/best-libraries-for-fuzzy-matching-in-python-cbb3e0ef87dd?source=rss-246f2656ec1a------2</link>
            <guid isPermaLink="false">https://medium.com/p/cbb3e0ef87dd</guid>
            <category><![CDATA[nlp]]></category>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[education]]></category>
            <category><![CDATA[python]]></category>
            <category><![CDATA[programming]]></category>
            <dc:creator><![CDATA[Moosa Ali]]></dc:creator>
            <pubDate>Sun, 14 Aug 2022 18:54:04 GMT</pubDate>
            <atom:updated>2022-08-15T10:47:08.966Z</atom:updated>
            <content:encoded><![CDATA[<h4>String matching made simple</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*schtwb4M3jQWa-dHrqu3Lw.png" /></figure><p>Data scientists have to deal with all sorts of data during their day-to-day tasks. One of the biggest challenges to data processing comes with unstructured data. Data is called unstructured when it can not be represented in a tabular (structured) manner. <a href="https://writersbyte.com/uncategorized/why-is-big-data-so-important/">Unstructured data</a> includes text, audio, video files, etc.</p><p>Text data is processed via the use of special techniques. One such technique is fuzzy matching. Fuzzy matching refers to finding similarities between strings, these similarity metrics can be used for multiple purposes. Fuzzy matching can be done via multiple algorithms, each with its pros and cons. Let’s discuss these algorithms below.</p><h4>Common Fuzzy Matching Algorithms</h4><p>Fuzzy matching is used to check whether two strings are the same or different and, in the case of the latter, by what factor they are dissimilar. Fuzzy matching is often referred to as <strong>edit-distance, </strong>i.e., how much does string A need to be edited to match string B? There are multiple methods of calculating the similarity metric. Let’s discuss three of the most common ones.</p><ul><li><strong>Levenshtein Distance</strong></li></ul><p>Levenshtein distance was first used in 1965 by Vladimir Levenshtein. It decides the similarity between 2 strings by computing the minimum number of single-character edits required to convert the first string to the second string. Levenshtein considers the following operations as an edit:</p><ol><li>Addition</li><li>Deletion</li><li>Substitution</li></ol><p>The smaller the no. of required edits is, the closer the two strings are.</p><ul><li><strong>Damerau–Levenshtein distance</strong></li></ul><p>The Damerau-Levenshtein works in exactly the same way as the conventional Levenshtein algorithm but offers one slight improvement. Apart from the three edits mentioned above, Damerau-Levenshtein also takes into account transposition. Transposition corresponds to swapping two adjacent characters and is considered a single operation. Damerau stated that these 4 operations account for 80% of all human misspellings.</p><ul><li><strong>Jaro-Winkler Distance</strong></li></ul><p>The Jaro-Winkler distance algorithm was proposed in 1990. It builds upon the Jaro similarity algorithm, which uses the following equation to calculate the similarity.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/203/1*vIztsIBmptmnBwa3lDhENQ.png" /><figcaption>Jaro similarity equation</figcaption></figure><ul><li>m = number of matching characters</li><li>t is half of the number of transpositions</li><li>|s1| and |s2| are the lengths of each string, respectively.</li></ul><p>The Jaro similarity returns a score between 0 and 1, where 0 represents no match, and 1 represents that the strings are exactly alike. Jaro-Winkler modifies the formula by applying more weightage to the first <em>i</em> matching characters. I will not go into too much detail, but the final form is represented by the equation below.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/352/1*aj55epqWjrK0xaATqikWog.png" /></figure><p>Before moving on, if you find this content helpful do consider supporting me on <a href="https://medium.com/ko-fi.com/moosaali9906">Ko-Fi</a>.</p><figure><a href="http://ko-fi.com/moosaali9906"><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*RAVsw7Mcvsh8aJL3LuG6bg.png" /></a><figcaption>Support me on ko-Fi</figcaption></figure><h4>Fuzzy matching libraries in python</h4><p>Python has a lot of implementations for fuzzy matching algorithms. I have compiled a small list of some of the best libraries available for open-source use.</p><ol><li><strong>FuzzyWuzzy</strong></li></ol><p>FuzzyWuzzy uses the Levenshtein distance implementation of fuzzy matching to give you a ratio score between the provided sentences.</p><p>Install the library using the following command.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*uDKWb3UJYbzK5pMgW5-tSQ.png" /><figcaption>fuzzywuzzy installation</figcaption></figure><p>Once installed, the library can be used simply with one line as shown below.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*RDJJUxeJxscHSfag2RAvmA.png" /><figcaption>Testing fuzzywuzzy</figcaption></figure><p>The resulting ratio comes out to be 90, meaning the 2 sentences are 90% similar.</p><p>The library also comes with an additional package that improves the calculation speed up to 10x. Learn more about FuzzyWuzzy from their <a href="https://pypi.org/project/fuzzywuzzy/">documentation</a>.</p><p>2.<strong> Rapidfuzz</strong></p><p>RapidFuzz is the exact same implementation as FuzzyWuzzy but is mostly written in C++ and offers additional algorithmic improvements over it. Its C++ implementation makes this library extremely fast, and it offers some additional benefits over FuzzyWuzzy.</p><ul><li>It is an MIT license, so are you free to choose whichever license for your project.</li><li>It includes additional implementations such as Jaro-Winkler, which are not present in FuzzyWuzzy</li></ul><p>You can install this library using the following command.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*HqEwSOttCI29PXHQXHafaw.png" /><figcaption>Installing RapidFuzz</figcaption></figure><p>Its functionality is exactly the same as FuzzyWuzzy but with the added benefit of speed.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*g5DG9rxqFFWH7rElp7DQNg.png" /></figure><p>The ratio comes out to be <strong>80.7%</strong></p><p>RapidFuzz also offers more functionality, similar to FuzzyWuzzy. Read more about them in the <a href="https://pypi.org/project/rapidfuzz/">official documentation</a>.</p><p>3. <strong>Jaro-Winkler</strong></p><p>The Jaro-Winkler distance also has multiple implementations in python, but apparently, many of those are incorrect. According to one user on<a href="https://stackoverflow.com/questions/48428810/jellyfish-vs-pyjarowinkler"> StackOverflow</a>, the results produced by <em>pyjarowinkler</em>, a very old and popular python library, seem incorrect. I tested the same problem with the<a href="https://pypi.org/project/jaro-winkler/"> jaro-winkler</a> library and got matching results to other correct implementations.</p><p>Install the library using the following command.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*wic8fV-vR845R5hURSUf0Q.png" /><figcaption>installing jaro-winkler</figcaption></figure><p>Calculate the distance using the “jaro_winkler_metric” function. See the application below.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*sYI5gnEXfFIboSBbzhMCwA.png" /><figcaption>testing jaro-winkler library</figcaption></figure><p>The similarity comes out as <strong>78.3 %</strong></p><p>Jaro-Winkler distance has a very important advantage over Levenshtein or Damerau–Levenshtein when calculating ratios or percentages. The latter-mentioned algorithms provide misleading results for smaller strings. Let’s see an example below.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*vhu2hpslhDPLGrCDsfqIPQ.png" /><figcaption>Testing for shorter strings</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/193/1*3fcDu2Mwo97GNQSpPe_nKg.png" /><figcaption>Result</figcaption></figure><p>Even if the strings only differ by one character, the ratio comes out to be 67% which is a very significant number. Now let’s compare it with the Jaro-Winkler distance.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*SXKct0OUFu9Jnz0k2658pw.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/598/1*ux4nDdN-d6-hXEGc5WAHzA.png" /><figcaption>Results for both the algorithms</figcaption></figure><p>The Jaro-Winkler algorithm returns a similarity percentage of <strong>82.2% , </strong>which is a much more reasonable number.</p><h4>Applications of Fuzzy Matching</h4><p>String matching plays an important part in everyday applications. When you search something on google, and it corrects your typo with the famous <em>“Including results for: …”.</em></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/739/1*n_9uny7yp28XO7661wXHUQ.png" /><figcaption>Google String Matching</figcaption></figure><p>In the background, Google actually runs your query through an edit-distance algorithm and matches it with every word in a pre-defined dictionary. Another example of the same functionality is the auto-correct feature on your phone.</p><p>It is also used in many search algorithms that pop up suggestions of terms that resemble the actual query. This makes the search feature more advanced as you can find the correct match even if you do not input the exact terms.</p><p>A more application of fuzzy-matching is in the de-duplication of records in a database. Many times our records contain similar data, so tricks like ‘groupby’ do not work. Fuzzy matching helps with picking out all records within the defined threshold.</p><h4>Concluding Thoughts</h4><p>Fuzzy matching is a simple yet very useful <a href="https://writersbyte.com/datascience/applied-data-science-with-python-and-pandas/">data processing</a> technique. It is used in web applications in different aspects and is also included in many data processing pipelines. Python offers some amazing libraries that implement some form of fuzzy matching. These libraries offer simple APIs to calculate the string matching score and can be utilized in your applications.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=cbb3e0ef87dd" width="1" height="1" alt=""><hr><p><a href="https://medium.com/codex/best-libraries-for-fuzzy-matching-in-python-cbb3e0ef87dd">Best Libraries for Fuzzy Matching In Python</a> was originally published in <a href="https://medium.com/codex">CodeX</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Let’s Understand Smart Contracts in Blockchain]]></title>
            <link>https://medium.com/thecapital/lets-understand-smart-contracts-in-blockchain-64617075d27c?source=rss-246f2656ec1a------2</link>
            <guid isPermaLink="false">https://medium.com/p/64617075d27c</guid>
            <category><![CDATA[web3]]></category>
            <category><![CDATA[blockchain]]></category>
            <category><![CDATA[smart-contracts]]></category>
            <category><![CDATA[cryptocurrency]]></category>
            <category><![CDATA[bitcoin]]></category>
            <dc:creator><![CDATA[Moosa Ali]]></dc:creator>
            <pubDate>Sat, 25 Jun 2022 19:24:57 GMT</pubDate>
            <atom:updated>2022-06-27T09:19:05.628Z</atom:updated>
            <content:encoded><![CDATA[<h4>And maybe build one…</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*lKj6DQzz3feWTg_dhz-Wyg.png" /></figure><p>A smart contract is essentially a set of legal rules. Just like a legal contract defines proceedings of a certain transaction or event in legal terms, a smart contract defines what rules transactions will follow on decentralized applications (Dapps). In a smart contract, these rules are present in the form of code, and this code is immediately executed once a smart contract is deployed on a blockchain.</p><p>If you don’t understand what a blockchain is, I highly recommend you start with the following article.</p><p><a href="https://writersbyte.com/featured-post/how-does-blockchain-technology-work-in-web3/">How does Blockchain technology work in Web3? - WritersByte</a></p><h4>Structure of a Smart Contract in Solidity.</h4><p>Solidity is one of the most common languages which is used in writing smart contracts. It is supported by the most common blockchains such as Ethereum and Solana. Solidity is a relatively new language that specializes in building smart contracts.</p><p>A smart contract must have the following contents:</p><ol><li>License Identifier</li><li>Compiler Version</li><li>Contract code</li></ol><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*QtehCgbCOcuXrDzfi4w0pA.png" /></figure><p>License Identifiers are placed on top of a Solidity smart contract in the form of a comment. These were made compulsory in solidity version <strong>0.6.8.</strong> These licenses help specify whether your code is open-source or not and helps tools like <a href="https://etherscan.io/">Etherscan</a> in the verification of the code. A License Identifier is written like the following:</p><pre>//SPDX-License-Identifier: MIT</pre><p>The above statement means that our code has an MIT license. A full list of supported licenses can be found <a href="https://spdx.org/licenses/">here</a>.</p><p>Compiler Version is specified right after a license identifier. Solidity has many versions to date and every smart contract requires you to specify which solidity version is to be used to compile that contract. It is specified using the following statement:</p><pre>pragma solidity ^0.8.0;</pre><p>The above statement means the code can be compiled using any version under the 0.8.x version. Choosing the correct version of the solidity compiler is very important, especially when importing contracts from other oracles.</p><p>Contract code is where your script is placed. This is where you write the actual code and set all the rules and regulations for your contract. If you are familiar with any Object Oriented Programming language, then you will find Contracts very similar to classes in OOP.</p><p>You have different contracts that define different sets of rules for your DApp and you can also import one contract into another (just like inheritance).</p><p>A contract looks like the following:</p><pre>Contract myFirstContract {</pre><pre>    // Your code here</pre><pre>}</pre><h4>Sample Contract</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*ArBw5PXq-6nepmlKVXII5Q.png" /></figure><p>Just like every other program, you can define functions in smart contracts. These functions are where the rules of the contract are defined. Let&#39;s look at the following sample contract. This contract is simply a deposit pool for your cryptocurrency.</p><pre>//SPDX-License-Identifier: MIT</pre><pre>pragma solidity ^0.8.0;</pre><pre>contract depositPool {</pre><pre>   // Mapping to keep track of total amount for each user</pre><pre>   mapping(address =&gt; uint256) public userToAmount;</pre><pre>   function deposit() public payable {</pre><pre>       //transfer balance to contract</pre><pre>        updateBalanceRecord(msg.value, msg.sender);</pre><pre>        }</pre><pre>    function updateBalanceRecord(uint256 _amount, address _user) internal {</pre><pre>        userToAmount[_user] += _amount;</pre><pre>        }</pre><pre>    function checkContractBalance() view public returns(uint256) {</pre><pre>        //returns the total balance held by the contract</pre><pre>        return address(this).balance;</pre><pre>    }</pre><pre>}</pre><p>This contract allows you to call the deposit the function which will enable you to deposit some WEI into the contract itself. Furthermore, we have a mapping that helps track which user has deposited how much. We also have a function checkContractBalancethat is used to check the total balance of the smart contract.</p><p>This sample contract shows how we can configure functionalities in a smart contract. There are so much more functionalities that you can add to a smart contract e.g. you can control:</p><ul><li>A minimum amount to deposit.</li><li>Who can call which functions</li></ul><p>You can also add logical functionality in case you want to do some processing on the deposited amount.</p><h4>Conclusion</h4><p>Smart Contracts are used to define functionality for your decentralized applications. Within a smart contract, you define rules and functionalities which will steer your application.</p><p>This was just a simple introduction to smart contracts which is why I did not go into the details of the functionalities and any advanced concepts. If already understand the basics of blockchain, you can try running this code in the <a href="https://remix.ethereum.org/">remix </a>IDE.</p><p><strong>Check out our new platform </strong>👉<strong> </strong><a href="https://thecapital.io/"><strong>https://thecapital.io/</strong></a></p><p><a href="https://twitter.com/thecapital_io">https://twitter.com/thecapital_io</a></p><iframe src="https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Fupscri.be%2Ff%2F3c1144%3Fas_embed%3Dtrue&amp;dntp=1&amp;display_name=Upscribe&amp;url=https%3A%2F%2Fupscri.be%2F3c1144&amp;key=d04bfffea46d4aeda930ec88cc64b87c&amp;type=text%2Fhtml&amp;schema=upscri" width="800" height="400" frameborder="0" scrolling="no"><a href="https://medium.com/media/3b6b127891c5c8711ad105e61d6cc81f/href">https://medium.com/media/3b6b127891c5c8711ad105e61d6cc81f/href</a></iframe><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=64617075d27c" width="1" height="1" alt=""><hr><p><a href="https://medium.com/thecapital/lets-understand-smart-contracts-in-blockchain-64617075d27c">Let’s Understand Smart Contracts in Blockchain</a> was originally published in <a href="https://medium.com/thecapital">The Capital</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[How does a Blockchain work?]]></title>
            <link>https://medium.com/coinmonks/how-does-a-blockchain-work-eb1d1d227a5c?source=rss-246f2656ec1a------2</link>
            <guid isPermaLink="false">https://medium.com/p/eb1d1d227a5c</guid>
            <category><![CDATA[cryptocurrency]]></category>
            <category><![CDATA[bitcoin]]></category>
            <category><![CDATA[blockchain-technology]]></category>
            <category><![CDATA[ethereum]]></category>
            <category><![CDATA[blockchain]]></category>
            <dc:creator><![CDATA[Moosa Ali]]></dc:creator>
            <pubDate>Fri, 27 May 2022 20:23:44 GMT</pubDate>
            <atom:updated>2022-06-02T11:54:48.354Z</atom:updated>
            <content:encoded><![CDATA[<h4>Blockchain, cryptocurrency, mining, decentralization, and more…</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*KaN7IGqXd9IrSwnrIq-84g.png" /></figure><p>Blockchain technology has been around for about a decade now. It started with the advent of the great Bitcoin back in 2009 by Satoshi Nakamoto. Ever since then Blockchain, Cryptocurrency as Web3 have been getting a lot of hype, and today (In 2022) these are all that everyone talks about.</p><p>If you wanna understand web3 more, you can look at the following article.</p><p><a href="https://writersbyte.com/technology/web3-ai-cryptocurrency-and-the-metaverse/">Web3: AI, Cryptocurrency, and the Metaverse - WritersByte</a></p><p>But do people really understand what a Blockchain is? Do people know what Bitcoin is? Probably Not. For most people, it is nothing more than just a tradeable asset that many used to become millionaires. However, we’re here to talk about the theory behind this Blockchain and how it made cryptocurrency possible.</p><h4>What is a Blockchain?</h4><p>A blockchain is like a database system, only more complex and completely decentralized. I believe it is quite obvious by the comparison to a database system that a blockchain is used to store data, but what makes it different? Well to understand that we need to break it into…<em>blocks….</em>😉.</p><ul><li>Blocks</li></ul><p>The illustration below represents what a single block looks like in a blockchain.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*KOQsUAAB0PIjWXKu2B2nkg.png" /></figure><p>Every block contains:</p><ol><li>A Header.</li></ol><p>This <strong>Header </strong>contains vital information such as the <strong><em>hash of the previous block in the chain, timestamp of when the block was created, a nonce (</em></strong><em>this is used in blockchain mining</em><strong><em>), etc.</em></strong></p><p>2. A Body</p><p>The <strong>Body </strong>of the blockchain contains information regarding transactions that have occurred on that blockchain. If the transaction is a transfer of cryptocurrency then the block body will contain the <strong>address from which the transaction was initiated, the address to which the amount was transferred, and the amount which was transferred.</strong></p><ul><li>Linking up the blocks</li></ul><p>As more and more transactions happen, new blocks are added to the blockchain so this creates….. <em>a chain of blocks. </em>This chain structure of this database ensures that it can not be tampered with (we’ll soon see how this is possible).</p><p>If you find this content helpful, consider supporting me on Ko-Fi: ko-fi.com/moosaali9906</p><figure><a href="http://ko-fi.com/moosaali9906"><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*7FccJCZ7zQBkZI9fmEVEuQ.png" /></a></figure><h4>Block Mining</h4><p>Unless you’ve been living under a rock, you must have heard about blockchain mining by someone somewhere. Everyone seems to be jumping on the mining bandwagon but I am quite sure not a lot of them know what mining really is (except that it gives you free cryptocurrency).</p><p><strong><em>Mining</em></strong> is the process of creating a new block on the blockchain. A block isn’t <em>JUST </em>created, it takes a complex process to finish this process. Every time a block is to be created, heavy computation power is needed to solve what is called ‘the blockchain problem’.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*lE19o15LIW8TizeMskyyLQ.png" /></figure><p>I will discuss mining in detail in a different article. For now, you just need to understand that this “<em>problem” </em>is computationally expensive and takes some time to solve (15 sec for Ethereum and 10 mins for Bitcoin). This computation expense is the reason why blockchain technology is considered harmful to the environment(<em>you know because computers run and use electricity and electricity requires fossil fuel burning yada yada yada!!</em>).</p><h4>Gas Fee</h4><p>This is perhaps the perfect time to introduce “<strong><em>gas fee”. </em></strong>If you are into crypto trading or NFT trading, you must be familiar with this fee that you must pay every time you carry out a transaction (Buy/Sell crypto or NFTs). Ever wondered what this fee is for?</p><p>Well, every time you make a transaction, a block is to be created on the blockchain. This requires mining the chain and hence some computation power must be expended. This gas fee amount is awarded to the node(computer) which solves the problem first and allows the transaction to be carried out.</p><p>This is how blockchain miners earn cryptocurrency. <em>It’s really free money since all you gotta do is let your computer do the work and receive the gas fee as a reward.</em></p><h4>Blockchain is decentralized?</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*FKKi-mEENbkftQe237O5qw.png" /></figure><p>Whenever blockchain is mentioned, the term “<em>decentralized</em>” always tags along. This is because a blockchain’s biggest highlight is that it is a decentralized network.</p><p>In conventional web applications, there is a database present on any 1 system. This means that hackers can easily alter this database and you lose all the data. Blockchain being decentralized means that an exact copy of the entire blockchain is present on every node(machine) which is mining that blockchain. This means that even if a hacker is able to manipulate a chain on a block and add or remove some transaction, the blockchains on every other node would remain unaltered and the malicious change would be caught.</p><h4>Proof of work VS Proof of stake</h4><p>These are the last concepts that I will be introducing in this article. Both of these concepts are used to assign mining processes to nodes of a blockchain.</p><ol><li>Proof of Work</li></ol><p>POW was introduced with the bitcoin blockchain (<em>the OG blockchain</em>). In this concept whenever a transaction is to be verified, it is broadcasted to all the nodes and every node will rush to calculate the appropriate hash for that transaction. The node which does so first gets the gas fee as a reward.</p><p>The problem with ‘POW’ is that a lot of resources are wasted in calculating the hash since every node jumps into the race but only one provides the answer. So the resources utilized by all the others are completely wasted.</p><p>2. Proof of Stake</p><p>This algorithm is an improvement over the POW algorithm. Blockchains utilizing the Proof of Stake methodology require every node on its network to deposit a fee before being able to join the blockchain. This fee ensures that the joiner has a ‘<strong><em>stake</em></strong>’ in the Blockchain project.</p><p>Whenever a transaction is to be verified, the blockchain algorithm randomly picks a node on the network to solve the problem. Well not exactly <em>randomly, </em>every blockchain has a different way of picking a node. Some may actually pick randomly, some give preference to nodes with a higher stake in the blockchain.</p><p>While POW has no penalty for anyone trying to carry out a fraudulent transaction, POS actually has some punishments in store. Since the nodes have deposited a fee, the network is able to take away some, if not all, of that fee if a node is accused of fraud.</p><p>Since POS requires people to actually spend money, there are fewer nodes on the network and since only one node is picked at a time to mine a block, no resources are wasted and it is better for the environment.</p><blockquote>Join Coinmonks<a href="https://t.me/coincodecap"> Telegram Channel</a> and<a href="https://www.youtube.com/c/coinmonks/videos"> Youtube Channel</a> learn about crypto trading and investing</blockquote><h3>Also, Read</h3><ul><li><a href="https://coincodecap.com/bookmap-review-2021-best-trading-software">Bookmap Review</a> |<a href="https://coincodecap.com/crypto-exchange-usa"> 5 Best Crypto Exchanges in the USA</a></li><li>The Best Crypto<a href="https://medium.com/coinmonks/hardware-wallets-dfa1211730c6"> Hardware wallet</a> |<a href="https://medium.com/coinmonks/bitbns-review-38256a07e161"> Bitbns Review</a></li><li><a href="https://coincodecap.com/crypto-exchange-in-singapore">10 Best Crypto Exchange in Singapore</a> |<a href="https://coincodecap.com/buy-axs-token"> Buy AXS</a></li><li><a href="https://coincodecap.com/red-dog-casino-review">Red Dog Casino Review</a> |<a href="https://coincodecap.com/swyftx-review"> Swyftx Review</a> |<a href="https://coincodecap.com/coingate-review"> CoinGate Review</a></li><li><a href="https://coincodecap.com/best-crypto-to-invest-in-india-in-2021">Best Crypto to Invest in India</a> |<a href="https://coincodecap.com/wazirx-p2p"> WazirX P2P</a> |<a href="https://coincodecap.com/hi-dollar-review"> Hi Dollar Review</a></li></ul><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=eb1d1d227a5c" width="1" height="1" alt=""><hr><p><a href="https://medium.com/coinmonks/how-does-a-blockchain-work-eb1d1d227a5c">How does a Blockchain work?</a> was originally published in <a href="https://medium.com/coinmonks">Coinmonks</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[No Medium Partner Program? No problem]]></title>
            <link>https://moosa-ali.medium.com/no-medium-partner-program-no-problem-5d6dca416998?source=rss-246f2656ec1a------2</link>
            <guid isPermaLink="false">https://medium.com/p/5d6dca416998</guid>
            <category><![CDATA[education]]></category>
            <category><![CDATA[medium]]></category>
            <category><![CDATA[productivity]]></category>
            <category><![CDATA[writing]]></category>
            <category><![CDATA[self-improvement]]></category>
            <dc:creator><![CDATA[Moosa Ali]]></dc:creator>
            <pubDate>Mon, 23 May 2022 18:18:00 GMT</pubDate>
            <atom:updated>2022-05-23T18:18:00.095Z</atom:updated>
            <content:encoded><![CDATA[<h4>Time to buy a Ko-Fi I guess...</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*tYE4n1ydG0AvXMn_-ADaXw@2x.png" /></figure><p>About a year ago I joined Medium in hopes of trying to teach what I learn. I started writing about Data Science, Artificial Intelligence, and General Python programming in order to give back to the community which had taught me so much. I started coding algorithms and compiled material for all who wished to learn.</p><p>Initially, it was a hobby but soon after it started to feel like a burden as writing programming articles is much more difficult as compared to normal blogs. Implementing the code first, then penning down the explanations, and finally designing relevant illustrations to make sure that the content is as helpful to the reader as it can get.</p><h4>Then what?</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*McQYIWWM9rlbhLtnzegCdw.jpeg" /><figcaption>Picture from Unsplash.com</figcaption></figure><p>The effort seemed a lot but all of it became worth it as soon as I received a message of gratitude from a random reader, thanking me for the tutorials. After writing for a few months I discovered that medium actually has a partner program where you can get paid for writing articles!!. Seemed so amazing at first, I would be helping the community AND be receiving something in return.</p><p>Sadly, as I approached to apply for this partner program….</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/708/1*ZHx2NQnp4E1lHpwoegqAuA.png" /><figcaption>My country is not among the one’s supported 😞</figcaption></figure><p>It was quite demotivating news for me, not going to lie. After all that effort, some compensation would have been quite a mood booster. Anyways, I didn’t let this bring me down. I continued to write and continued my efforts in teaching people.</p><h4>So What Now?</h4><p>I recently realized that there are other ways you can allow others to support you. So I discovered <a href="http://ko-fi.com/moosaali9906">Ko-Fi</a> by reading some other articles on medium. It is a platform that allows you to receive support in the form of donations.</p><figure><a href="http://ko-fi.com/moosaali9906"><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*RAVsw7Mcvsh8aJL3LuG6bg.png" /></a></figure><p>So if you’re like me, a <strong><em>“Medium Partner Program affectee” </em></strong>then perhaps Ko-Fi can help you too. Simply create and set up your account and then share your profile link. People can send you donations as they please.</p><h4>Ending note…</h4><p>So if you have been my active follower or ever has any of my writing helped you do consider supporting me on my <a href="http://ko-fi.com/moosaali9906">Ko-Fi</a> Page.</p><p>Just follow the following link: <a href="https://ko-fi.com/moosaali9906">Ko-fi.com/moosaali9906</a></p><p>And if you like to learn about Data Science, Artificial Intelligence, or Programming in general do check out my other articles.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=5d6dca416998" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Boruta Feature Selection Explained in Python]]></title>
            <link>https://medium.com/geekculture/boruta-feature-selection-explained-in-python-7ae8bf4aa1e7?source=rss-246f2656ec1a------2</link>
            <guid isPermaLink="false">https://medium.com/p/7ae8bf4aa1e7</guid>
            <category><![CDATA[features]]></category>
            <category><![CDATA[artificial-intelligence]]></category>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[python]]></category>
            <category><![CDATA[machine-learning]]></category>
            <dc:creator><![CDATA[Moosa Ali]]></dc:creator>
            <pubDate>Sat, 14 May 2022 19:15:14 GMT</pubDate>
            <atom:updated>2022-06-10T10:27:28.022Z</atom:updated>
            <content:encoded><![CDATA[<h4>Implementation and explanation from scratch</h4><figure><img alt="Boruta Feature Selection" src="https://cdn-images-1.medium.com/max/1024/1*Te999PiAT-bUHkSRFr7FlQ.png" /></figure><p>This article aims to explain, the very popular, Boruta feature selection algorithm. Boruta automates the process of feature selection as it automatically determines any thresholds and returns features that are most meaningful in your dataset. Boruta works on the “all-relevant” principle as it provides you with ALL the features that are relevant to your Machine Learning problem.</p><h4>Need for Feature Selection?</h4><p>Datasets can contain features that may be completely irrelevant to your problem. These features increase the size of your dataset, add complexity to the artificial intelligence model, and have either, no impact on the output, or worsen the results. It is important to identify these features and remove them before moving on to the training stages.</p><p>You can find a little more detail on Feature selection in the following <a href="https://writersbyte.com/featured-post/feature-selection-algorithms-for-machine-learning/">article</a>.</p><p><a href="https://writersbyte.com/featured-post/feature-selection-algorithms-for-machine-learning/">Feature Selection Algorithms for Machine Learning - WritersByte</a></p><h4>Boruta Algorithm</h4><p>This algorithm was first introduced as a package for R. It comprises the following steps:</p><p>Create copies of the original features by randomly shuffling the features(Shadow Features).</p><ol><li>Concatenate these shadow features to the original dataset.</li></ol><p>2. Train this new dataset using the Random Forest Classifier.</p><p>3. Check feature importance for the highest-rated Shadow feature.</p><p>4. All original features that are more important than the <strong><em>most important shadow feature </em></strong>are the ones that we want to keep.</p><p>5. Repeat 3 and 4 for some iterations (20 is a reasonable number) and keep track of the features that appear as important in every iteration.</p><p>6. Use binomial distribution to finalize which features provide enough confidence to be kept in the final list.</p><p>Before moving on, if you are finding this article helpful, do consider supporting me on Ko-Fi.</p><figure><a href="https://ko-fi.com/moosaali9906"><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*RAVsw7Mcvsh8aJL3LuG6bg.png" /></a></figure><h4>Implementation</h4><p>You can find the complete code in the repository: <a href="https://github.com/Moosa-Ali/Boruta-Feature-Selection-Implementation/blob/main/Boruta-feature-selection-implementation.ipynb">Boruta Feature Selection</a></p><p>Before any algorithm we obviously need some sort of data to perform feature selection. For this purpose, we will use the same dataset which we used in the last article about <a href="https://writersbyte.com/featured-post/feature-selection-algorithms-for-machine-learning/?swcfpc=1">feature selection</a>.</p><p>This dataset can be found at the following link on <a href="https://www.kaggle.com/datasets/fedesoriano/stroke-prediction-dataset?rvi=1">Kaggle</a>.</p><h4>Load and process data</h4><pre># important libraries</pre><pre>import pandas as pd<br>import numpy as np<br>from tqdm.notebook import tqdm<br>import scipy as sp<br>from sklearn.ensemble import RandomForestClassifier<br>import matplotlib.pyplot as plt<br>from sklearn.utils import shuffle</pre><p>We will use all of the above libraries above.</p><pre>data =  pd.read_csv(&quot;healthcare-dataset-stroke-data.csv&quot;)<br>data.head()</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Mzln5PGJ-Gu5IJz7YPvyqw.png" /><figcaption>Heart Stroke Dataset</figcaption></figure><p>A lot of useless data, nothing we haven’t seen before.</p><p>Time for a cleanup. For advanced classes on data cleaning and processing, refer to the following <a href="https://writersbyte.com/featured-post/applied-data-science-with-python-and-pandas/?swcfpc=1">post</a>.</p><p><a href="https://writersbyte.com/featured-post/applied-data-science-with-python-and-pandas/?swcfpc=1">Applied Data Science with Python and Pandas - WritersByte</a></p><pre># converting to numeric</pre><pre>data[&quot;gender&quot;] = pd.factorize(data[&quot;gender&quot;])[0]<br>data[&quot;ever_married&quot;] = pd.factorize(data[&quot;ever_married&quot;])[0]<br>data[&quot;work_type&quot;] = pd.factorize(data[&quot;work_type&quot;])[0]<br>data[&quot;Residence_type&quot;] = pd.factorize(data[&quot;Residence_type&quot;])[0]<br>data[&quot;smoking_status&quot;] = pd.factorize(data[&quot;smoking_status&quot;])[0]</pre><pre># additional cleaning</pre><pre>data.dropna(inplace =True)<br>data.drop(&quot;id&quot;, axis =1, inplace = True)<br>data.reset_index(inplace=True, drop=True)<br>data.head()</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*-g8rppRf2uqS_J2XvrO_8g.png" /><figcaption>Cleaned data for feature selection</figcaption></figure><p>All freshened up.</p><pre># seperate input and output variables</pre><pre>X = data.drop(&quot;stroke&quot;, axis = 1)<br>y = data[&quot;stroke&quot;]</pre><p>Separating inputs and outputs.</p><ol><li>Creating Shadow features</li></ol><p>For this, we just need to shuffle the original features and concatenate them to the original dataset.</p><pre>for col in X.columns:<br>    X[f&quot;shadow_{col}&quot;] = X[col].sample(frac=1).reset_index(drop=True)</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*dvgZ9TCVywOpB3qPvqsyTw.png" /><figcaption>Shadow features concatenated</figcaption></figure><p>2. Calculate Importance</p><pre>def get_important_features(X, y):</pre><pre># Initiliaze Random Forest CLassifier<br>rf = RandomForestClassifier(max_depth=20)</pre><pre># Fit Random Forest on provided data<br>rf.fit(X,y)</pre><pre># Create dictionary of feature importances<br>importances = {feature_name: f_importance for feature_name, f_importance in zip(X.columns, rf.feature_importances_)}</pre><pre># Isolate importances of Shadow features<br>only_shadow_feat_importance = {key:value for key,value in importances.items() if &quot;shadow&quot; in key}</pre><pre># get importance level of most important shadow feature<br>highest_shadow_feature = list(dict(sorted(only_shadow_feat_importance.items(), key=lambda item: item[1], reverse=True)).values())[0]</pre><pre># get original feature which fulfill boruta selection criteria<br>selected_features = [key for key, value in importances.items() if value &gt; highest_shadow_feature]</pre><pre>return selected_features</pre><p>This function trains a <strong>Random Forest classifier</strong> on our heart stroke dataset. The classifier returns the importance it assigns to each feature in the variable `feature_importances_`.</p><p>We then create a dictionary of each feature along with its importance and single out the most important shadow feature.</p><p>Finally, it returns the dictionary with all original features which have an importance score greater than the singled out shadow feature.</p><p>Now since one trial isn’t enough, we need to run multiple trials to make sure we get satisfactory results.</p><h4>Multiple Trials</h4><pre>TRIALS = 50</pre><pre>feature_hits = {i:0 for i in data.columns}</pre><pre>for _ in tqdm(range(TRIALS)):</pre><pre>    imp_features = get_important_features(X, y)</pre><pre>        for key, _ in feature_hits.items():</pre><pre>            if key in imp_features: feature_hits[key] += 1</pre><pre>print(feature_hits)</pre><p>The results of our 50 runs are as follows</p><pre>{&#39;gender&#39;: 0,  &#39;age&#39;: 50,  &#39;hypertension&#39;: 0,  &#39;heart_disease&#39;: 0,  &#39;ever_married&#39;: 0,  &#39;work_type&#39;: 0,  &#39;Residence_type&#39;: 0,  &#39;avg_glucose_level&#39;: 50,  &#39;bmi&#39;: 1,  &#39;smoking_status&#39;: 0,  &#39;stroke&#39;: 0}</pre><p>Age and avg_glucose_level come at as important, 50 times and BMI comes out important in 1 trial. Now to justify whether BMI appearing important in just 1 trial makes it important or not, we will use a binomial distribution.</p><h4>Binomial Distribution</h4><p>The following line of code returns us the probabilities according to a binomial distribution.</p><pre># Calculate the probability mass function<br>pmf = [sp.stats.binom.pmf(x, TRIALS, .5) for x in range(TRIALS + 1)]</pre><p>A binomial distribution with a probability 0.5 has a bell-shaped curve with 5% of the overall probability in the tails.</p><p>First, we need a function that gives us the number of iterations that form the tail.</p><pre># trails_in_green_zone</pre><pre>def get_tail_items(pmf):<br>   total = 0<br>       for i, x in enumerate(pmf):<br>           total += x<br>           if total &gt;= 0.05:<br>               break</pre><pre>return i</pre><p>The rules are simple. If the no. of iterations fall in the right tail, we call it the green zone (features that must be kept). If it falls in between the bell shape, we call it the blue zone(Features that can be played around with) and if they are in the right tail, it is called the red zone (features that should be dropped).</p><p>Let’s visualize the distribution we created.</p><pre># plot the binomial distribution</pre><pre>plt.plot([i for i in range(TRIALS + 1)], pmf,&quot;-o&quot;)<br>plt.title(f&quot;Binomial distribution for {TRIALS} trials&quot;)<br>plt.xlabel(&quot;No. of trials&quot;)<br>plt.ylabel(&quot;Probability&quot;)<br>plt.grid(True)</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/473/1*Tx99YpE2hWg3ZXeDi4phjg.png" /><figcaption>Binomial distribution for 50 trials</figcaption></figure><h4>Final Selection</h4><p>Now we just need to code the rules which we discussed above, about deciding which features fall into the Green, Blue, and Red zone.</p><pre># select features from n number of trials</pre><pre>def choose_features(feature_hits, TRIALS, thresh):</pre><pre>    #define boundries<br>    green_zone_thresh = TRIALS - thresh<br>    blue_zone_upper = green_zone_thresh<br>    blue_zone_lower = thresh</pre><pre>    green_zone = [key for key, value in feature_hits.items() if    value &gt;= green_zone_thresh]</pre><pre>    blue_zone = [key for key, value in feature_hits.items() if (value &gt;= blue_zone_lower and value &lt; blue_zone_upper)]</pre><pre>    return green_zone, blue_zone</pre><p>Now run the above functions in the following order</p><pre>thresh = get_tail_items(pmf)<br>green, blue = choose_features(feature_hits, TRIALS, thresh)</pre><pre>green,blue</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/335/1*-Fv1_ePlpMoH0VX_kSNY0Q.png" /><figcaption>Important Features according to our Boruta Algorithm</figcaption></figure><p>As we can see these are exactly the features that we got when we ran the Python implementation in Boruta in <a href="https://writersbyte.com/featured-post/feature-selection-algorithms-for-machine-learning/">the other article</a>.</p><p>If you enjoyed this, do check out my other blog posts at:</p><p><a href="http://www.writersbyte.com/">www.writersbyte.com</a></p><p>Also, don’t forget to buy me a Kofi.</p><figure><a href="http://ko-fi.com/moosaali9906"><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*RAVsw7Mcvsh8aJL3LuG6bg.png" /></a></figure><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=7ae8bf4aa1e7" width="1" height="1" alt=""><hr><p><a href="https://medium.com/geekculture/boruta-feature-selection-explained-in-python-7ae8bf4aa1e7">Boruta Feature Selection Explained in Python</a> was originally published in <a href="https://medium.com/geekculture">Geek Culture</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Feature Selection Algorithms for Machine Learning]]></title>
            <link>https://medium.com/codex/feature-selection-algorithms-for-machine-learning-35507679eab?source=rss-246f2656ec1a------2</link>
            <guid isPermaLink="false">https://medium.com/p/35507679eab</guid>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[python]]></category>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[artificial-intelligence]]></category>
            <category><![CDATA[ai]]></category>
            <dc:creator><![CDATA[Moosa Ali]]></dc:creator>
            <pubDate>Sat, 07 May 2022 18:46:18 GMT</pubDate>
            <atom:updated>2022-06-10T10:18:01.676Z</atom:updated>
            <content:encoded><![CDATA[<h4>Choosing the right ones</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*MKMLi7DqtYxM87Fz5Qrhbw.png" /></figure><p>Feature Selection is an optional, yet important preprocessing step for your Machine Learning model. It is a common practice to feed your Machine Learning model data as you receive it. This is a very common rookie mistake, data that you receive from whatever source will always contain errors. There are 2 important steps that you must carry out on this data before moving on to training the models.</p><ol><li>Data Cleaning</li><li>Feature filtering/ Selection</li></ol><p>I have talked about Data Cleaning multiple times in other articles, if you would like to learn more about it then you can check the following <a href="https://writersbyte.com/featured-post/applied-data-science-with-python-and-pandas/?swcfpc=1">blog post</a>.</p><p><a href="https://writersbyte.com/featured-post/applied-data-science-with-python-and-pandas/?swcfpc=1">Applied Data Science with Python and Pandas - WritersByte</a></p><h4>Feature Selection</h4><p>However, in this post, we will talk about selecting important input features from your dataset such that it has minimum impact on the outputs and reduces the complexity of the data.</p><p>First, let&#39;s talk about how correlation amongst input variables affects a Machine Learning model.</p><h4>Correlation between features</h4><p>Suppose you have a dataset with the following 2 features and the output labeled Y as shown in the diagram below.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*3_YCOX3yHxNA---EAZUp1A.png" /><figcaption>Data set with redundant features</figcaption></figure><p>variables x_2 and x_3 have exactly the same value throughout the dataset. A Machine Learning model learns the given pattern in the input variables corresponding to each output variable. Now for variables that have a high correlation(x_2 and x_3 in the above diagram), one of these variables is not providing any valuable information to our model. This is because they both have the same pattern, it is as if they are the same variable.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Qm80lts4S4E2MGJFYjwodg.png" /><figcaption>Reduced data set with only important features</figcaption></figure><p>Since one of these variables adds no additional information to the model we can remove the feature altogether. What’s the harm in keeping such a variable you may ask?</p><p>well…</p><ol><li>It increases the dimensionality of the data unnecessarily. This increases training times and we may run into the problem of <a href="https://en.wikipedia.org/wiki/Curse_of_dimensionality">curse of dimensionality</a>.</li><li>Although in most models, keeping the feature may not make the model any “worse” however such highly correlated variables affect different models differently. These might cause to add confusion to the model and hence reduce performance.</li></ol><p>If you find my work helpful, consider supporting me on Kofi. Click on the image below.</p><figure><a href="http://ko-fi.com/moosaali9906"><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*RAVsw7Mcvsh8aJL3LuG6bg.png" /></a><figcaption>Click on image to show your support</figcaption></figure><h4>Algorithms</h4><p>At this point it may seem quite simple, just remove a feature at random amongst the 2. However in real life, you will rarely find variables that show EXACTLY the same pattern/values across the dataset, hence there are a few more things to consider before removing a variable or deciding whether two variables are even highly correlated or not.</p><p>This is why there are specialized algorithms that decide for you, which variables to keep and which to remove. We will talk about 2 such algorithms.</p><h4>Boruta Feature Selection</h4><p>Boruta Feature selection algorithm was first introduced as a package for R. It is a very useful algorithm that defines its own thresholds and provides you with the most accurate features from the provided dataset.</p><p>A complete explanation and implementation of Boruta can be found <a href="https://writersbyte.com/programming/boruta-feature-selection-explained-in-python/?swcfpc=1">here</a>:</p><p><a href="https://writersbyte.com/programming/boruta-feature-selection-explained-in-python/?swcfpc=1">Boruta Feature Selection Explained in Python - WritersByte</a></p><p>Boruta shuffles the provided input features (each feature column separately) and then concatenates these (called shadow features) with the original data. After this, the complete data set is trained using a Random Forest classifier. This classifier returns feature importance for the entire input. Boruta then sets the threshold as the <strong>strongest shuffled(shadow) feature.</strong></p><p>Any real feature which has an importance level lower than the most important shuffled feature is dropped. Boruta has a python package that helps you calculate the features. Below is a demonstration of how it works.</p><pre># install the package<br>!pip install boruta</pre><pre># import important libraries<br>import pandas as pd<br>from boruta import BorutaPy<br>from sklearn.ensemble import RandomForestRegressor<br>import numpy as np</pre><p>Now we load the dataset and clean it up a little bit like removing NaN values and converting categorical variables to numerical representation.</p><pre>#load data<br>heart_data = pd.read_csv(&quot;healthcare-dataset-stroke-data.csv&quot;)</pre><pre># converting to numeric</pre><pre>heart_data[&quot;gender&quot;] = pd.factorize(heart_data[&quot;gender&quot;])[0]<br>heart_data[&quot;ever_married&quot;] = pd.factorize(heart_data[&quot;ever_married&quot;])[0]<br>heart_data[&quot;work_type&quot;] = pd.factorize(heart_data[&quot;work_type&quot;])[0]<br>heart_data[&quot;Residence_type&quot;] = pd.factorize(heart_data[&quot;Residence_type&quot;])[0]<br>heart_data[&quot;smoking_status&quot;] = pd.factorize(heart_data[&quot;smoking_status&quot;])[0]</pre><pre># additional cleaning<br>heart_data.dropna(inplace =True)<br>heart_data.drop(&quot;id&quot;, axis =1, inplace = True)</pre><pre>heart_data.head()</pre><p>The Final dataset looks like the one below.</p><figure><img alt="Dataset after cleaning" src="https://cdn-images-1.medium.com/max/860/1*W7me8CikqhphBlJmRJF9aA.png" /><figcaption>Heart Stroke dataset</figcaption></figure><p>Now let’s run the Boruta algorithm.</p><pre>X = heart_data.drop(&quot;stroke&quot;, axis = 1)<br>y = heart_data[&quot;stroke&quot;]</pre><pre># we will use the randomforest algorithm<br>forest = RandomForestRegressor(n_jobs = -1,max_depth = 10)<br># initialize boruta<br>boruta = BorutaPy(estimator = forest, n_estimators = &#39;auto&#39;,max_iter = 50,)</pre><pre># Boruta accepts np.array <br>boruta.fit(np.array(X), np.array(y))</pre><pre># get results<br>green_area = X.columns[boruta.support_].to_list()<br>blue_area = X.columns[boruta.support_weak_].to_list()<br>print(&#39;Selected Features:&#39;, green_area)<br>print(&#39;Blue area features:&#39;, blue_area)</pre><figure><img alt="Result of the Boruta Algorithm" src="https://cdn-images-1.medium.com/max/372/1*EVtVgiNLxzkn7oMfVh0LOQ.png" /><figcaption>Result of the Boruta algortihm</figcaption></figure><p>So out of the 10 original features, Boruta believes that only the 2 features returned are the most important features to make any reasonable decision.</p><h4>mRMR Feature Selection</h4><p>MRMR stands for <em>Maximum Relevance Minimum Redundancy. </em>While Boruta looks amongst the features to find the most important ones, <strong>MRMR makes sure that the features selected are not only the ones that provide minimum correlation between the input features but also have a high correlation with the output variable</strong>.</p><p>This algorithm was first introduced in the following <a href="https://bmcbioinformatics.biomedcentral.com/articles/10.1186/s12859-016-1423-9">paper</a>.</p><p>MRMR works iteratively, it first asks you how many features you want to keep, and then for every iteration it calculates 1 feature that is most relevant to the output variable and least related to any of the features in our dataset. Once a feature is selected it is removed from the original dataset and the next iteration begins until K (the number of features we require) iterations are completed.</p><p>I will explain the details of the algorithm in a separate post. For now, let’s look at its python implementation.</p><p>Install the python package using the following command</p><pre>!pip install mrmr_selection</pre><p>You can find the complete documentation for this package at their official <a href="https://github.com/smazzanti/mrmr">Github repository here</a>.</p><p>The usage is quite straightforward.</p><pre>from mrmr import mrmr_classif<br>selected_features = mrmr_classif(X=X, y=y, K=2)</pre><p>I have set K as 2 just to see if the selected features match with what we are returned by Boruta.</p><pre>print(selected_features)</pre><figure><img alt="Features returned by MRMR with K=2" src="https://cdn-images-1.medium.com/max/230/1*Py-fpLS5z1-SPiIZO3FuBg.png" /><figcaption>Features returned by MRMR with K=2</figcaption></figure><p>And well yes we have the exact same features as what we got from the Boruta algorithm above. However, what makes MRMR flexible is that if you believe that 2 features might not be enough to get you a better result then you can choose to use as many as you want.</p><p>Let’s carry out a few more runs.</p><pre># top 4 features<br>top_4 = mrmr_classif(X=X, y=y, K=4)<br># top 6 features<br>top_6 = mrmr_classif(X=X, y=y, K=6)</pre><pre>print(&quot;Best 4 features:&quot;, top_4)<br>print(&quot;Best 6 features:&quot;, top_6)</pre><figure><img alt="Features returned by MRMR for k = 4 and k = 6" src="https://cdn-images-1.medium.com/max/884/1*nxyY0M-PwqPDMC448G-j7w.png" /><figcaption>Features returned by MRMR for k = 4 and k = 6</figcaption></figure><h4>Conclusion</h4><p>Feature selection is a live saver when you are low on memory resources and, at times, can even help improve the performance of your model. It is an essential step in the process of building your machine learning model.</p><p>Consider supporting me on Kofi. Click on the image below.</p><figure><a href="http://ko-fi.com/moosaali9906"><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*RAVsw7Mcvsh8aJL3LuG6bg.png" /></a><figcaption>Click on the image to show your support</figcaption></figure><p>To learn more about Machine Learning you can check out the following articles.</p><ul><li><a href="https://writersbyte.com/featured-post/naive-bayes-python-implementation-and-understanding/">Naive Bayes Python Implementation and Understanding - WritersByte</a></li><li><a href="https://writersbyte.com/datascience/implementing-multi-variable-linear-regression-algorithm-in-python/?swcfpc=1">Multi-variable Linear Regression Python Implementation.</a></li></ul><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=35507679eab" width="1" height="1" alt=""><hr><p><a href="https://medium.com/codex/feature-selection-algorithms-for-machine-learning-35507679eab">Feature Selection Algorithms for Machine Learning</a> was originally published in <a href="https://medium.com/codex">CodeX</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Naive Bayes Python Implementation and Understanding]]></title>
            <link>https://becominghuman.ai/naive-bayes-python-implementation-and-understanding-7e44a2943b29?source=rss-246f2656ec1a------2</link>
            <guid isPermaLink="false">https://medium.com/p/7e44a2943b29</guid>
            <category><![CDATA[programming]]></category>
            <category><![CDATA[tutorial]]></category>
            <category><![CDATA[python]]></category>
            <category><![CDATA[naive-bayes]]></category>
            <category><![CDATA[machine-learning]]></category>
            <dc:creator><![CDATA[Moosa Ali]]></dc:creator>
            <pubDate>Sun, 30 Jan 2022 15:11:19 GMT</pubDate>
            <atom:updated>2022-02-01T15:53:11.664Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*ApxUjIwTAqIoUgPpj4yRsA.png" /></figure><p>Naive Bayes is a Machine Learning Classifier that is based on the Bayes Theoram of conditional probability. In this article, we will be understanding conditional probability (Bayes Theoram) and then moving on to how it translates to the Naive Byes classifier. we will be understanding the mathematics behind this classifier and then finally coding it in Python.</p><h4>Bayes Theoram</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/217/1*QbwSP2CEhq-Y-waUGZmxYw.jpeg" /><figcaption>source: Wikipedia</figcaption></figure><p>Named after the statistician Thomas Bayes, this theorem is also known as the theorem of conditional probability. It allows us to calculate the probability of a particular event GIVEN a set of prior conditions. For example, the probability that it will rain tomorrow GIVEN that it rained yesterday.</p><p>The formula for calculating conditional probability is shown below.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*ocQdGDVi94LHHDZkVc7kwQ.png" /></figure><p>The term on the left-hand side is read as <em>‘the probability of event A occurring given that event B has occurred’. </em>The term on the right-hand side is the probability of both events occurring together divided by the probability of event B occurring. The formula is quite straightforward. I will not be delving into its derivation or intuition as this article is not about Bayes Theorem but rather Naive Bayes Classifier.</p><figure><a href="https://aijobsboard.com/?fbclid=IwAR0GcsLs8A6CFM-fynrZMM3sTgo_Zpbto2CjglJm0Dmi6otC6YWc7CDFpQk"><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*dAlwXfVrtYspdCzmC8iADw.png" /></a><figcaption>Big Data Jobs</figcaption></figure><h4>The Naive Bayes Classifier</h4><p>Since we are classifying our data into discrete labels, just like any other classifier, for Naive Bayes we will have a set of input features as well as their corresponding output class. A Naive Bayes classifier calculates probability using the following formula.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Ct6U2fvyd1dHD0A3Ws8vEQ.png" /></figure><p>The left side means, what is the probability that we have y_1 as our output given that our inputs were {x_1 ,x_2 ,x_3}. Now let’s suppose that our problem had a total of 2 classes i.e. {y_1, y_2}. We will now use the above formula twice first to calculate the probability of y_1 occurring and then for y_2 occurring. Whichever has a higher probability will be our predicted class.</p><p>This is how Naive Bayes is used for classification.</p><h3>Trending AI Articles:</h3><blockquote><a href="https://becominghuman.ai/why-corporate-ai-projects-fail-part-1-4-3b820041ab00">1. Why Corporate AI projects fail?</a></blockquote><blockquote><a href="https://becominghuman.ai/how-ai-will-power-the-next-wave-of-healthcare-innovation-695a2196aae8">2. How AI Will Power the Next Wave of Healthcare Innovation?</a></blockquote><blockquote><a href="https://becominghuman.ai/machine-learning-by-using-regression-model-f0c7993a66c8">3. Machine Learning by Using Regression Model</a></blockquote><blockquote><a href="https://becominghuman.ai/top-data-science-platforms-in-2021-other-than-kaggle-a1b4609c06b0">4. Top Data Science Platforms in 2021 Other than Kaggle</a></blockquote><h4>Naive Bayes in Python</h4><p>Let&#39;s start coding it in Python.</p><pre>import pandas as pd<br>import numpy as np<br>from sklearn.model_selection import train_test_split</pre><p>Our necessary libraries for playing with data. If you do not know how to work around with Pandas then you might want to read about it <a href="https://writersbyte.com/featured-post/applied-data-science-with-python-and-pandas/?swcfpc=1">here first</a>.</p><p><a href="https://writersbyte.com/featured-post/applied-data-science-with-python-and-pandas/?swcfpc=1">Applied Data Science with Python and Pandas - WritersByte</a></p><p>Now let’s load our dataset. I have used the Heart disease prediction data set which can be found on <a href="https://www.kaggle.com/cdabakoglu/heart-disease-classifications-machine-learning">Kaggle</a>.</p><pre>data = pd.read_csv(&#39;heart-disease-data/heart.csv&#39;) #Read the dataset<br>data.head()</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/611/1*HCxxNJ24hdd7nfefBS2y3A.png" /><figcaption>Head of the dataset</figcaption></figure><p>For a Naive Bayes Classifier, we need discrete variables since we can not use continuous variables in calculating probabilities. So we need to drop some columns here such as <strong>cholesterol</strong> and <strong>trestbps.</strong></p><pre>data.drop([&quot;age&quot;, &quot;trestbps&quot;, &quot;chol&quot;, &quot;thalach&quot;, &quot;oldpeak&quot;, &quot;slope&quot;],axis = 1 ,inplace=True) #drop irrelevant columns</pre><pre>data.head()</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/324/1*yCh3pxlH3ue5pMhFiUkKKA.png" /><figcaption>dropped columns</figcaption></figure><pre>X = data[data.keys()[:-1]]<br>y = data[data.keys()[-1]]</pre><pre>X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.10, random_state=42)#test-train split</pre><pre>data_train = pd.concat([X_train, y_train],axis = 1) #concat back<br>data_test = pd.concat([X_test, y_test],axis = 1)</pre><p>Now we need to code the helper function that would help us calculate all the necessary probabilities.</p><p>According to the formula we will need to calculate the probability of occurrence of every input feature as well as output feature and their conditional probabilities given each class label.</p><p>First, we will calculate the probability for each input variable.</p><pre>#Calcualting probabilites for inputs independantly</pre><pre>def get_probabilities_for_inputs(n, column_name, data_frame):<br>    <br>    temp = data_frame[column_name] #isolate targetted column<br>    temp = temp.value_counts() #get counts of occurences of each input variable<br>    <br>    return (temp/n) #return probiblity of occurence by dividing with total no. of data points</pre><p>Next, we will calculate conditional probabilities for the input given an output class.</p><pre>#calculating conditional probability</pre><pre>def get_conditional_probabilities(data_frame, n,target, given):<br>    <br>    focused_data = data[[target, given]] #isolate target column an dfocus input column</pre><pre>targets_unique = data[target].unique()#list of unique outputs in data<br>    inputs_unique = data[given].unique()<br>    <br>    groups = focused_data.groupby(by = [given, target]).size().reset_index()<br>    groups[0] = groups[0]/ n<br>    <br>    <br>    for targets in targets_unique:<br>        current_target_length = len(focused_data[focused_data[target] == targets])<br>        groups[0] = np.where(groups[target] == targets, groups[0].div(current_target_length),groups[0])<br>    <br>    return groups</pre><p>Next, we will write down our ‘<em>fit</em>’ function that will calculate and return all the necessary probabilities which we will then use for making classifications.</p><pre>def calculate_probabilities(data):<br>    #splititng input data<br>    x = data[data.keys()[:-1]]<br>    y = data[data.keys()[-1]]<br>    target = y.name<br>    <br>    #get length of dataframe<br>    n = len(data)<br>    <br>    #get probabilities for each individual input and output<br>    f_in = lambda lst: get_probabilities_for_inputs(n, lst, x)<br>    input_probablities = list(map(f_in,x.keys()))<br>    <br>    output_probabilities = get_probabilities_for_inputs(n ,target, y.to_frame())<br>    <br>    #get conditional probabilities for every input against every output<br>    f1 = lambda lst: get_conditional_probabilities(data, n, target,lst)<br>    conditional_probabilities = list(map(f1, data.keys()[:-1]))<br>    <br>    return input_probablities, output_probabilities, conditional_probabilities</pre><p>Now that we have all the necessary calculations done and out of the way, we need to make a function that will give us our output class label by making calculations according to the Naive Bayes formula that we wrote above.</p><pre>def naive_bayes_calculator(target_values, input_values, in_prob, out_prob, cond_prob):<br>    <br>    target_values.sort()#sort the target values to assure ascending order<br>    classes = [] #initialise empty probabilites list<br>    <br>    for target_value in target_values:<br>        num = 1 #initilaise numerator<br>        den = 1 #initialise denominator</pre><pre>#calculate denominator according to the formula<br>        for i,x in enumerate(input_values):<br>            den *= in_prob[i][x]</pre><pre>#calculate numerator according to the formula    <br>        for i, x_1 in enumerate(input_values):<br>            temp_df = cond_prob[i]<br>            num *= temp_df[(temp_df.iloc[:,0] == x_1) &amp; (temp_df.iloc[:,1] == target_value)][0].values[0]<br>        num *= out_prob[target_value]</pre><pre>final_probability = (num/den) #final conditional probability value<br>        <br>        classes.append(final_probability) #append probability for current class in a list<br>        <br>    return (classes.index(max(classes)), classes)</pre><p>Now we have all our functions out of the way, we can move on to running them and storing our calculations.</p><pre>in_prob, out_prob, cond_prob = calculate_probabilities(data_train)#use training data for the initial calculations</pre><p>The three variables will have all the necessary probabilities; <strong><em>probabilities of all the inputs, probabilities for the output class, and conditional probabilities.</em></strong></p><p>Let’s test the calculations that we have made.</p><pre>#testing with dummy data<br>naive_bayes_calculator([1,0], [1,1,0,2,1,3,3],in_prob,out_prob,cond_prob)</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/425/1*ttPJNUjeqXOCM6Susq9iUw.png" /><figcaption>Outputs</figcaption></figure><p>We have our class prediction and the probabilities for each class inside a <a href="https://writersbyte.com/programming/4-key-python-data-structures-e-very-beginner-must-know/?swcfpc=1">tuple</a>.</p><p>Now it&#39;s time to test on our ‘<strong>test data</strong>’.</p><p>The following function takes a set of inputs and returns the predicted class against each in a list.</p><pre>def naive_bayes_predictor(test_data, outputs, in_prob, out_prob, cond_prob):<br>    <br>    final_predictions = [] #initialise empty list to store test predictions<br>    <br>    for row in test_data:<br>        #get prediction for current data<br>        predicted_class, probabilities = naive_bayes_calculator(outputs, row, in_prob, out_prob, cond_prob)<br>        #append to list<br>        final_predictions.append(predicted_class)<br>        <br>        <br>    return final_predictions</pre><p>Now calculate accuracy.</p><pre>test_data_as_list = X_test.values.tolist()<br>unique_targets = y_test.unique().tolist()</pre><pre>predicted_y = naive_bayes_predictor(test_data_as_list,unique_targets,in_prob,out_prob,cond_prob)</pre><pre>print(&quot;Accuracy:&quot;, (np.count_nonzero(y_test == predicted_y)/len(y_test)) *100)</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/226/1*ku5MmvzzTnqgcZlGUH7v8g.png" /><figcaption>Our test accuracy</figcaption></figure><p>An accuracy of <strong>77.4% </strong>is certainly not a bad number considering that we dropped certain important columns and the naivety of the algorithm ignores certain correlations between the input variables.</p><h4>Conclusion</h4><p>Naive Bayes is a very simple classifier granted that you understand basic probability and the concept of inputs and outputs in Machine Learning. The algorithm does have certain shortcomings such as ignoring the dependency of input variables on each other. It is very simple to build and gives good results if your data is according to its requirements.</p><h3>Don’t forget to give us your 👏 !</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/780/0*2lvCls4yjxVMfZSR" /></figure><iframe src="https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Fupscri.be%2F8f5f8b%3Fas_embed%3Dtrue&amp;dntp=1&amp;url=https%3A%2F%2Fupscri.be%2F8f5f8b&amp;key=a19fcc184b9711e1b4764040d3dc5c07&amp;type=text%2Fhtml&amp;schema=upscri" width="800" height="400" frameborder="0" scrolling="no"><a href="https://medium.com/media/c43026df6fee7cdb1aab8aaf916125ea/href">https://medium.com/media/c43026df6fee7cdb1aab8aaf916125ea/href</a></iframe><figure><a href="https://becominghuman.ai/artificial-intelligence-communities-c305f28e674c"><img alt="" src="https://cdn-images-1.medium.com/max/255/1*2f7OqE2AJK1KSrhkmD9ZMw.png" /></a></figure><figure><a href="https://upscri.be/8f5f8b"><img alt="" src="https://cdn-images-1.medium.com/max/255/1*v-PpfkSWHbvlWWamSVHHWg.png" /></a></figure><figure><a href="https://becominghuman.ai/write-for-us-48270209de63"><img alt="" src="https://cdn-images-1.medium.com/max/255/1*Wt2auqISiEAOZxJ-I7brDQ.png" /></a></figure><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=7e44a2943b29" width="1" height="1" alt=""><hr><p><a href="https://becominghuman.ai/naive-bayes-python-implementation-and-understanding-7e44a2943b29">Naive Bayes Python Implementation and Understanding</a> was originally published in <a href="https://becominghuman.ai/">Becoming Human: Artificial Intelligence Magazine</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[pytyhApplied Data Science with Python and Pandas]]></title>
            <link>https://blog.devgenius.io/pytyhapplied-data-science-with-python-and-pandas-419e63302e79?source=rss-246f2656ec1a------2</link>
            <guid isPermaLink="false">https://medium.com/p/419e63302e79</guid>
            <category><![CDATA[data-analysis]]></category>
            <category><![CDATA[pandas-dataframe]]></category>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[python]]></category>
            <category><![CDATA[pandas]]></category>
            <dc:creator><![CDATA[Moosa Ali]]></dc:creator>
            <pubDate>Tue, 18 Jan 2022 12:31:39 GMT</pubDate>
            <atom:updated>2022-01-18T14:46:10.899Z</atom:updated>
            <content:encoded><![CDATA[<h3>Applied Data Science with Python and Pandas</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*fhm9K6ZYNWXMVMmwRyOieA.png" /></figure><p>Data Science is a very important skill that has become a necessity of the 21st century. With the increase in data, the need for data scientists has increased. When it comes to learning Data Science, Python is the go-to programming language. Python comes with a plethora of tools that aid the data science process and one of the most prominent tools is Pandas. In this article, we will be looking at how to process data using the Pandas library such that it can be used for other tasks such as Analysis or Machine Learning.</p><h4>What is Data Science?</h4><p>Surprisingly there is no right answer to this question. People define Data Science only in the way that THEY work with data. In reality, Data Science is a general term that covers all those techniques and procedures that we apply to data. This is why when looking at a Data Science job you will find a variety of tools, as a requirement for the job, that may or may not be related to each other e.g. <strong>SQL, Python, R, or Power BI</strong>.</p><p>A person that works with extracting data and ETL would be proficient with SQL but may or may not know Python. Conversely, a person who builds Machine Learning models will be proficient in Python but may or may not know SQL. However, both of these can call themselves data scientists since they work with data but have different responsibilities.</p><p>In this article, I will be explaining Data Cleaning and filtering as part of the Data Science domain. If you are interested in <a href="https://writersbyte.com/datascience/ai-for-beginners-2/?swcfpc=1">Machine Learning</a> then you can start reading <a href="https://writersbyte.com/datascience/implementing-multi-variable-linear-regression-algorithm-in-python/?swcfpc=1">here</a>.</p><p><a href="https://writersbyte.com/datascience/implementing-multi-variable-linear-regression-algorithm-in-python/?swcfpc=1">Multi-variable Linear Regression Python Implementation.</a></p><h4>Downloading data</h4><p>If you land a data science job then you will be working with real-world collected by whichever organization you work for. You will have to extract that data from an SQL database using …you guessed it…SQL!. However, for the purpose of this tutorial, I will be downloading an open-source dataset from Kaggle.</p><p>We will use the “World Energy Consumption” dataset which you can download from <a href="https://www.kaggle.com/pralabhpoudel/world-energy-consumption">here</a>.</p><h4>Getting started with pandas</h4><p>You will first have to install pandas to start working. It can be done by simply running the following command in your command prompt.</p><pre>pip install pandas</pre><p>If you are unfamiliar with working with the command prompt or just generally installing packages for python, you can start reading <a href="https://writersbyte.com/programming/python-for-absolute-beginners-in-3-easy-steps/?swcfpc=1">here</a>.</p><p><a href="https://writersbyte.com/programming/python-for-absolute-beginners-in-3-easy-steps/?swcfpc=1">Python for Absolute Beginners in 3 easy steps - WritersByte</a></p><p>Once Pandas is installed we are ready to load the data set.</p><p>Import pandas first.</p><pre>import pandas as pd</pre><p>Notice that we have written ‘as pd’ in the import statement above, this is how we alias imports in Python. Since it is difficult to write the compete term Pandas everywhere you use the library, you simply reference it by using the term ‘pd’.</p><p>Pandas allow you to load data from various sources such as excel sheets, CSV(comma separated values) files, text files, and even SQL tables. Our data is in the form of a CSV file so we will use the following function to load our dataset.</p><pre>world_power_data = pd.read_csv()</pre><p>Let&#39;s view what we have in the data.</p><pre>world_power_data.head()</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1012/1*ndBSbNbYuka6br5IR-1IIw.png" /><figcaption>top 5 rows of the dataset</figcaption></figure><p>The <strong>.head()</strong> function provides us with the first 5 rows of the dataset (the head of the dataset). We can enter any number inside the parenthesis to pass it as an argument and the function will return the number of rows equal to the number passed.</p><p>Conversely, we also have a <strong>.tail()</strong> function which does the exact opposite i.e. returns the last 5 rows of the dataset.</p><p>Looks like we have 122 columns in this dataset, let’s get an overview of the data.</p><pre>world_power_data.info()</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/518/1*pzTbF7tvL5JrzmmXIaSEQA.png" /></figure><p>Alright, so we have <strong>119 columns with float values, 1 column with integers, and 2 columns as objects</strong>. An Object data type in Pandas simply means that the column contains values of multiple datatypes.</p><p>We also now know that the dataset has taken up <strong><em>16.2+ MB of RAM.</em></strong></p><p>So this is a very huge dataset.</p><h4>What is a data frame?</h4><p>Before we move on to carry out an analysis of the data, it is important that we understand what a data frame is and what its structure is. A data frame has 3 essential parts.</p><ol><li>Header Row</li><li>Index</li><li>Data</li></ol><figure><img alt="" src="https://cdn-images-1.medium.com/max/512/1*cEhUhTEoTDobM8izFteCjw.png" /><figcaption>Structure of a data frame</figcaption></figure><p>The <strong>Header row </strong>defines the column names of the data. The <strong>Index </strong>gives an identifier to each row and the <strong>data </strong>is the main crux of the data frame, the numerical data which holds meaning.</p><p>We can jump to any row or column of the dataset using the indexes and columns.</p><p><strong>By Index</strong></p><pre>world_power_data.loc[45:50] #return data rows from index no. 45 to 50</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1009/1*DBqHlJbGKIVIi1odpfkD8g.png" /><figcaption>Indexed data frame</figcaption></figure><p><strong>By Column</strong></p><pre>world_power_data[[&quot;country&quot;, &quot;year&quot;, &quot;coal_prod_change_pct&quot;, &quot;coal_prod_change_twh&quot; ,&quot;gas_prod_change_pct&quot;]].head() #show only certain columns</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/577/1*pP0Z7yeE5o_C-rYAF8Lyxw.png" /><figcaption>Specific Columns shown</figcaption></figure><p><strong>Removing 2020 data</strong></p><p>Looks like this dataset contains values till the year 2019 only. Let&#39;s remove all rows with sparse data from the year 2020. We will do this in the following steps.</p><ul><li>Get indexes of the data rows for the year 2020</li></ul><pre>twentytwenty_idx = world_power_data[world_power_data[&#39;year&#39;] == 2020].index</pre><p>Now tell pandas to drop the columns with these indexes.</p><pre>world_power_data.drop(index = twentytwenty_idx, inplace = True)</pre><p>We have the data removed but we have messed up the indexes so for consistency across the entire data frame we need to reset the indexes.</p><pre>world_power_data.reset_index(inplace = True)</pre><h4>Keeping only focus countries</h4><p>Now we want to perform our analysis only on some specific developed countries since these are the ones that are the major energy consumers in the world.</p><pre>#our focus countries<br>focus_countries = [&#39;United States&#39;, &#39;Canada&#39;, &#39;Russia&#39;, &#39;United Kingdom&#39;, &#39;China&#39;]</pre><pre>#filter for focus countries<br>power_data = world_power_data[world_power_data[&quot;country&quot;].isin(focus_countries)]</pre><pre>power_data</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1001/1*N1gq8QnmRCG0P0LXzgdi0w.png" /><figcaption>Focus countries only</figcaption></figure><h4>Filtering data based on year</h4><p>We have data since the year 1900 (which really raises questions on the credibility of this dataset XD). We want to analyze data for only the 21st century so let&#39;s get to filtering.</p><pre>power_data = power_data[power_data[&quot;year&quot;] &gt;= 2000] #return data for only years greater than or equal to 2000</pre><pre>power_data</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1005/1*s4j2KStp5fmTxUl4maYHJQ.png" /><figcaption>Year filtered</figcaption></figure><h4>Column Filtering</h4><p>We have a total of 122 columns in this data set. We are only interested in the data of consumption of energy so we only want to keep those columns.</p><p>We will start by creating a list of all the column names which have the term ‘<strong>consumption</strong>’ in them.</p><pre>#get column names with the word consumption in them<br>consumption_cols = [col for col in power_data.keys()[power_data.keys().str.contains(&#39;consumption&#39;)]]</pre><p>Next, we will still need identifiers for the country name and year.</p><pre>consumption_cols.append(&#39;country&#39;) #append country column<br>consumption_cols.append(&#39;year&#39;) #append year column</pre><pre>consumption_cols</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/297/1*35w0aWgsCFmNRMgwT2Jdgg.png" /><figcaption>focus columns</figcaption></figure><p>Let&#39;s reduce the dataset to only show these columns. Note, since the year and country are appended at the end of the list they will appear as the last columns, this is okay for our analysis but if you want to fix this for aesthetic reasons then you can have several workarounds.</p><pre>power_data = power_data[consumption_cols]</pre><pre>power_data</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1003/1*8_waC0LLv339Q5lSMaZbNQ.png" /><figcaption>Only 15 columns left</figcaption></figure><h4><strong>Total energy consumptions</strong></h4><p>Our next problem statement is to find the total energy consumed by each of these countries over the last 2 decades for each of the energy sources. We will use the group by statement for this purpose, group by is a very useful and important aggregation tool.</p><pre>#group by country and sum the values in the remainig columns<br>total_power_consumed = power_data.groupby([&#39;country&#39;]).sum()</pre><p>The above statement tells pandas to group by the data using the ‘country’ column and then sum the values in the remaining columns. The illustration explains it better.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*zBYQemi9qosuEzqj8QNfvA.png" /><figcaption>Group By</figcaption></figure><p>You can have any type of aggregation you want for example average, count etc. The aggregation of the year column does not make sense and it does not even matter for our case so we can ignore this column or even drop it if we want.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1002/1*a_jLYmMDPQRGJlOVqmWZ7A.png" /><figcaption>grouped by country</figcaption></figure><p>We now have the total power consumed by each country for each resources type. We want this data to be saved separately so that we don’t have to carry out the analysis every time.</p><pre>total_power_consumed.to_csv(&#39;Total_Power_Consumed.csv&#39;) #save as csv</pre><p>The above function saves our data as a separate CSV file.</p><p>Let’s perform some more analysis. Say we want to answer the following questions.</p><p><strong><em>Which resource is used most widely for energy consumption by each country?</em></strong></p><p>Before we answer this question we need to take care of one problem. There is a column in our data named ‘<em>primary_energy_consumption</em>’ which gives a vague number for total energy consumed but since we want to data specifically for individual energy resources. So we will drop this column.</p><pre>total_power_consumed.drop(columns = [&#39;primary_energy_consumption&#39;],inplace = True)</pre><p>Now we will first see which country uses which energy resource most. For this we will sum the dataset column-wise and return the name of the column with the highest sum.</p><pre>total_power_consumed.idxmax(axis = 1) #which resource is consumed most</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/351/1*WoP2lmDlbCF6iPwJKlzC7g.png" /><figcaption>max consumption</figcaption></figure><p>Looks like everyone uses <strong><em>fossil fuel</em></strong> as their primary energy generator.</p><p>Now we will see in what amount are these consumed. For this, we will simply return the max of the column-wise sums.</p><pre>print(total_power_consumed.max(axis = 1))</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/254/1*fI9YCAD4bQrc6Eo99tycTA.png" /></figure><p>So we see that china actually utilizes <strong><em>48k+ TW</em></strong> of power from <strong><em>fossil fuels</em></strong> alone!!</p><p>Now for one final question, let’s answer how the usage trend changed in America over the last two decades for <strong>Solar, Wind, Nuclear, and Fossil Fuel power</strong>.</p><p>First, we filter the data for only US Data.</p><pre>united_states = power_data[power_data[&quot;country&quot;] == &quot;United States&quot;]<br>united_states.reset_index(drop = True, inplace =True)</pre><p>Next, we plot the data for the respective columns. Even though we have specialized tools for plotting Python, however, Pandas itself also allows you to plot data directly from a data frame. If you want to learn plotting in python, you can read below.</p><p><a href="https://writersbyte.com/programming/data-visualization-in-python-using-matplotlib/?swcfpc=1">Data Visualization and Analysis in Python using MatPlotLib - WritersByte</a></p><p>For Pandas, we do the following.</p><pre>united_states.plot(&#39;year&#39;,\<br>                   [&#39;solar_consumption&#39;,&#39;wind_consumption&#39;, &#39;nuclear_consumption&#39;, &#39;fossil_fuel_consumption&#39;],\<br>                   logy = True ,figsize= (16,12), title = &quot;Trend of Energy Consumption in America&quot;)</pre><p>Here we have specified the following items.</p><ol><li>‘year’ should be used for the x-axis.</li><li>Columns used for y-axis values.</li><li>Use logarithmic y-axis (To see trends clearly)</li><li>Set figure size to [16x12]</li><li>Give the plot a title.</li></ol><figure><img alt="" src="https://cdn-images-1.medium.com/max/954/1*MX1YQwtG0KrCJFwnAmaBdQ.png" /></figure><p>Looks like the use of Fossil fuel and Nuclear energy has been quite consistent however there has been a sharp rise in the use of Wind and Solar power. This is most probably because of the increase in demand for electricity.</p><h4>Final Thoughts</h4><p>So what you saw above is just one part of Data Science. This is what we call data processing. You ask yourself what sort of insight you want from the data and then you apply different processing techniques to gain that insight. This article aimed and showed you a handful of techniques that you can apply on your dataset to gain valuable in sight.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=419e63302e79" width="1" height="1" alt=""><hr><p><a href="https://blog.devgenius.io/pytyhapplied-data-science-with-python-and-pandas-419e63302e79">pytyhApplied Data Science with Python and Pandas</a> was originally published in <a href="https://blog.devgenius.io/">Dev Genius</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Understanding Regex with Python]]></title>
            <link>https://medium.com/geekculture/understanding-regex-with-python-a27f176ecc36?source=rss-246f2656ec1a------2</link>
            <guid isPermaLink="false">https://medium.com/p/a27f176ecc36</guid>
            <category><![CDATA[regex]]></category>
            <category><![CDATA[programming]]></category>
            <category><![CDATA[regular-expressions]]></category>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[python]]></category>
            <dc:creator><![CDATA[Moosa Ali]]></dc:creator>
            <pubDate>Tue, 11 Jan 2022 17:56:20 GMT</pubDate>
            <atom:updated>2022-01-12T07:25:07.356Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*1eIsiXKCfEj_xAKGe3zWiw.png" /></figure><p>When working with data we rarely come across clean and processed data. Before this data can be used anywhere, we need to apply a lot of processing to it in order to make it of any use. One of these techniques is pattern matching in strings to keep only the desired pattern or, conversely, remove unwanted patterns. We find these patterns in strings by using <strong><em>Regular Expressions</em></strong> or <strong><em>Regex</em></strong> for short. Regex is an internationally accepted set of rules to determine the formatting of a string.</p><p><a href="https://writersbyte.com/featured-post/regex-101-for-python-data-science/?swcfpc=1">Regex 101 for Python Data Science - WritersByte</a></p><p>In this article, we will explore what these expressions are and how we can utilize them in Python to cleanse our data. To learn about other data processing techniques <a href="https://writersbyte.com/datascience/5-data-processing-techniques-for-data-science-beginners-python-and-pandas/?swcfpc=1">click here</a>.</p><h3>Understanding Regular Expressions: Regex 101</h3><p>Regex are a set of rules defined to find patterns in strings.</p><p>Regex are defined by a set of characters. There are ordinary characters and then there are certain special characters. Together these are used to define your regular expression.</p><h4>Ordinary Characters</h4><p>These are the basic alphabetic, numeric, or symbolic characters that we use every day. These are mostly used in an exact match scenario. For example, if I want to find all the exclamation marks within a piece of text then my regex would be simply <strong>‘!’</strong>. The same applies to matching entire words or sentences as it is.</p><h4>Special Characters</h4><p>What we discussed above is string matching but that’s not really what regex are used for. These expressions are created to match complex patterns within the text and that is where we use the special characters. The table below represents some of the most common special characters used.</p><p><a href="https://writersbyte.com/datascience/5-data-processing-techniques-for-data-science-beginners-python-and-pandas/?swcfpc=1">5 Data Processing Techniques for Data Science Beginners | Python and Pandas - WritersByte</a></p><p>Character Explanation</p><ul><li><strong>. (Dot)</strong>: Matches any character except a newline character</li><li><strong>* (Asterisk)</strong>: Matches 0 or more characters specified by the preceding character</li><li><strong>+ (Plus)</strong>: Matches at least 1 or more occurrences of the term specified by the preceding character</li><li><strong>^ (Caret)</strong>: Matches the start of a string</li><li><strong>$ (Dollar)</strong>: Matches the end of a string</li><li><strong>? (Question mark)</strong>: Causes the resulting RE to match 0 or 1 repetitions of the preceding RE. &#39;ab?&#39; will match either ‘a’ or ‘ab’.</li><li><strong>{m}</strong>: matches exactly m occurrences of the preceding regex</li><li><strong>{m,n}</strong>: matches at least m and almost n occurrences of the preceding regex</li></ul><p>Let’s look at a few examples, we will use the following text from <em>Harry Potter and the philosophers stone</em>.</p><blockquote>Yet Harry Potter was still there, asleep at the moment, but not for long. His Aunt Petunia was awake and it was her shrill voice that made the first noise of the day.</blockquote><blockquote>“Up! Get up! Now!”</blockquote><blockquote>Harry woke with a start. His aunt rapped on the door again.</blockquote><blockquote>-Harry Potter and the philosophers stone.</blockquote><p>If you want to match an exact string then all you need to do is pass in that exact string as the <em>expression</em>. For example, if you want to find Harry Potter in the above text then you will write “<strong>Harry Potter</strong>” as the expression.</p><p>You can specify regex to find only a specific set of characters or numbers. You can also specify it to find only alphabetic characters or only numbers in a particular text. This is done by using the square brackets ‘[]’. The square braces enclose the set of characters we want e.g. if we want only alphabetic characters we will write it as:</p><pre>[A-Za-z]</pre><p>This specifies 2 ranges of characters; from A to Z and from a to z (making sure the match is case insensitive). We can specify to match all characters in a text as well.</p><p>Let’s say we want to match the <strong>entire string that is between quotation marks</strong>. Our expression will look like this:</p><pre>&quot;.*&quot;</pre><p>The above expression can be read as :</p><ul><li><strong>“: Look for a quotation mark</strong></li><li><strong>. : Look for any character afterward</strong></li><li><strong>*: Refers to the preceding character. Look for any number of occurrences of the character specified by the preceding character (dot in our case — so it will look for any number of occurrences of any character)</strong></li><li><strong>“: Look for another quotation mark to end the search</strong></li></ul><p>So the regex above will basically look for a quotation mark, then any character after if until another quotation mark is found.</p><p>All of this might seem difficult to understand. Regular Expressions can be very difficult to understand and the only real way to understand them is to practice them. So instead of just writing them, we will now see them in action in Python. If you are new to python, you might want to start reading this article first.</p><p><a href="https://writersbyte.com/programming/python-for-absolute-beginners-in-3-easy-steps/?swcfpc=1">Python for Absolute Beginners in 3 easy steps - WritersByte</a></p><h3>Regex in Python</h3><p>Read till the end for a bonus tip (you don’t want to miss out on this.)</p><p>You can try out the entire code used in this tutorial by cloning the following repository. <a href="https://github.com/Moosa-Ali/Regular-Expressions">Click here</a>.</p><p>If you want to learn to build a Jupyter notebook as in the git directory, the <a href="https://writersbyte.com/featured-post/how-to-use-jupyter-notebook-for-python-programming/?swcfpc=1">read here</a></p><p>First, we need to import the Python library that is designed to resolve Regular Expressions.</p><pre>import re</pre><p>Now let’s find some text where we need matches.</p><pre>txt = &#39;&#39;&#39;We have contacted Mr. Jhon Doe and havbe confirmed that he will be joining us for the meeting this evening.<br>If you would like to contact him yourself you can call him on +1-415-5552671 or email him at jhon.doe@outlook.com.<br>We also have the contact details of his assistant, you can contact him in case Mr. Doe does not respond.<br>The assistants email id is jack_smith12@outlook.com.&#39;&#39;&#39;</pre><p>The above text has a lot of information. We don’t need it all. Let’s say we just want to extract the contact information from it.</p><p>Let’s start with the <strong>email</strong>.</p><pre>email_regex = &#39;\S+@{1}\S+(.com){1}&#39;</pre><pre>x = re.search(email_regex, txt) #regex to find an email address</pre><pre>print(email_regex) #print out the result</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/519/0*CZ5gddYf6YNQuk9Q.png" /><figcaption>Regex object</figcaption></figure><p>The above object shows the match that was found with the regex as well as the indexes on which the match was found.</p><p>Let’s first see how to read this regex.</p><ul><li><strong>\S</strong>: Finds a non-whitespace character</li><li><strong>+</strong>: Specifies to find 1 or more non-whitespace occurrences</li><li><strong>@</strong>: Exact match specifies to find a ‘@’ symbol.</li><li><strong>{1}</strong>: specifies to find only 1 ‘@’ symbol.</li><li><strong>\S</strong>: Again specifies to find non-whitespace characters.</li><li><strong>+</strong>: Find at least 1 non-whitespace character.</li><li><strong>(.com)</strong>: Find an exact match for .com</li><li><strong>{1}</strong>: Find exactly one occurrence for ‘.com’</li></ul><p>The <em>re.search()</em> function only returns the first occurrence of the match. The purpose of this function is no more than to know if a match is present or not. The <em>&#39;re</em>&#39; library has many other useful functions that we can utilize.</p><h4>Find all occurrences</h4><p>To return all matches of the provided regex we use the <strong><em>re.findall()</em></strong> function.</p><pre>emails = re.findall(r&#39;\S+@{1}\S+(?:\.com)&#39;, txt) #finding all emails</pre><pre>print(emails)</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/418/0*kc81uf8NONBf8kGu.png" /><figcaption>Emails found in the text</figcaption></figure><p>There is one slight difference to take note of here. I changed the regex a little bit when using it in the re.findall() function. The reason for this is that the ‘findall()‘ function returns whatever group it finds within the regex. When we specified the (.com) at the end, it treated it as a group and returned only this group instead of the entire match. When we place the ‘<strong>?:</strong>‘ inside the parenthesis, it specifies it as a non-capturing group (according to the documentation) and the function returns the entire match.</p><h4>Substitute Expressions</h4><p>We can use regular expressions to remove certain parts of a string by specifying its pattern.</p><p>This is done using the re.sub() function.</p><pre>substituted_string = re.sub(r&#39;\S+@{1}\S+(.com){1}&#39;, &#39;&#39;, txt) #remove emails from the given text.</pre><pre>print(substituted_string)</pre><pre>We have contacted Mr. Jhon Doe and havbe confirmed that he will be joining us for the meeting this evening. If you would like to contact him yourself you can call him on +1-415-5552671 or email him at . We also have the contact details of his assistant, you can contact him in case Mr. Doe does not respond. The assistants email id is .</pre><p>As you can see in the text above, the emails have been replaced with a blank string. This is a good way of removing unwanted entities from your text.</p><p>Another good use of this is to redact documents. Say we want to forward the above text to someone but want to hide the emails. We can simply do it as.</p><pre>redacted = re.sub(&#39;\S+@{1}\S+(.com){1}&#39;, &#39;&lt;email&gt;&#39;, txt) #place emails tags in text</pre><pre>print(redacted)</pre><pre>We have contacted Mr. Jhon Doe and havbe confirmed that he will be joining us for the meeting this evening. If you would like to contact him yourself you can call him on +1-415-5552671 or email him at &lt;email&gt;. We also have the contact details of his assistant, you can contact him in case Mr. Doe does not respond. The assistants email id is &lt;email&gt;.</pre><p>Our receiver now knows that a certain portion is to contain an email address, but the actual address is hidden from them.</p><h4>Multiple expressions in a single line.</h4><p>We specify Python to find multiple expressions by using the same regex. This is done by separating the two expressions using the | (bar) symbol. The complete expression will look like this</p><pre>(exp1)|(exp2)|(exp3)</pre><p>We can specify as many expressions as we want. The bar is read as an OR and Python is told to find expression 1 OR expression 2 OR expression 3.</p><p>Let’s try it out. We can see that our text also has a phone number. Let’s redact all contact information from the text.</p><pre>re.findall(&#39;\S+@{1}\S+[.com]{1}|\+[0-9]{1}-[0-9]{3}-[0-9]{7}&#39;, txt)</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/566/0*6OzD_wGxAPjeRH11.png" /><figcaption>Contact information from the text</figcaption></figure><p>Seems like our multi-regex string is working ok.</p><pre>#tagging all contact information as confidential<br>redacted = re.sub(&#39;\S+@{1}\S+[.com]{1}|\+[0-9]{1}-[0-9]{3}-[0-9]{7}&#39;, &#39;&lt;confidential&gt;&#39;, txt)</pre><pre>print(redacted)</pre><pre>We have contacted Mr. Jhon Doe and havbe confirmed that he will be joining us for the meeting this evening. If you would like to contact him yourself you can call him on &lt;confidential&gt; or email him at &lt;confidential&gt; We also have the contact details of his assistant, you can contact him in case Mr. Doe does not respond. The assistants email id is &lt;confidential&gt;</pre><h3>BONUS TIP!!</h3><p>If you want to practice regular expressions and want to see what your particular regex does then r<a href="https://regexr.com/">egexr.com</a> is a very helpful tool.</p><h3>Final Thoughts</h3><p>Regex are very helpful if you work with data a lot. They help you in making sure that your data is in the right shape and structure. Other than data science, these are also widely used in development especially when you want to ensure user input, from the front end, to be in the right format. Regular expressions may seem a little difficult to understand at first but they get more clear with practice.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=a27f176ecc36" width="1" height="1" alt=""><hr><p><a href="https://medium.com/geekculture/understanding-regex-with-python-a27f176ecc36">Understanding Regex with Python</a> was originally published in <a href="https://medium.com/geekculture">Geek Culture</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
    </channel>
</rss>