<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Dede Kurniawan on Medium]]></title>
        <description><![CDATA[Stories by Dede Kurniawan on Medium]]></description>
        <link>https://medium.com/@dedekurniawann?source=rss-9f804bd8d19f------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/1*ftb4KpFD9DPIxgsqYp0eVA.jpeg</url>
            <title>Stories by Dede Kurniawan on Medium</title>
            <link>https://medium.com/@dedekurniawann?source=rss-9f804bd8d19f------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Sun, 22 Nov 2026 18:19:49 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@dedekurniawann/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[Time and Happiness]]></title>
            <link>https://dedekurniawann.medium.com/time-and-happiness-ac7d34989f3c?source=rss-9f804bd8d19f------2</link>
            <guid isPermaLink="false">https://medium.com/p/ac7d34989f3c</guid>
            <category><![CDATA[life]]></category>
            <category><![CDATA[happiness]]></category>
            <category><![CDATA[time]]></category>
            <category><![CDATA[diary]]></category>
            <dc:creator><![CDATA[Dede Kurniawan]]></dc:creator>
            <pubDate>Sat, 19 Sep 2026 01:18:17 GMT</pubDate>
            <atom:updated>2026-09-19T01:18:17.382Z</atom:updated>
            <content:encoded><![CDATA[<h4>A reflection when I waste my time and feel sad of something that I didn’t even understand</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*honNwolcwxDfqnwaSTGJDw.png" /><figcaption>Image from <a href="http://canva.com/">Canva</a></figcaption></figure><p>Sometimes I wonder whether a long life is a blessing or, on the contrary, a burden. If we have a lot of money, we might think that a long life is a blessing. However, when we’re poor, a long life might be a burden. For me, human life is a perfect balance, neither too long nor too short.</p><p>I feel that when we have too much time, we tend to take what we have for granted. When we’re extremely happy, we might always wish to feel that happiness forever. But if we were to experience that happiness forever, would the feeling remain the same, or would we eventually grow bored?</p><blockquote>I finally realized that we can feel happiness because we understand that it won’t last forever. Because we know it won’t last forever, we naturally come to appreciate whatever is happening to us at the moment.</blockquote><p>Does regret arise because we can’t sustain happiness for a long time? I feel that, in fact, regret is part of happiness itself. This is actually a recurring cycle. People will continue to experience fluctuating emotions. I think it’s precisely because of these fluctuations that happiness exists. Plus, when we realize that we don’t have much time, we’ll truly do our best.</p><p>Realizing that our time is limited is mandatory. But can we be aware of this all the time? Of course not; we’ll fall back into a cycle where we’ll waste time again.</p><blockquote>We realize how important time is only after we’ve wasted it.</blockquote><p>If we hadn’t wasted it, I don’t think people would understand just how important time is. I truly don’t understand, when I see wise people, how they can be so calm and fully aware of all this. It’s truly amazing.</p><p>The human mind is truly complex to understand. We need to experience something firsthand before we can truly understand it. Can’t we grasp just how important time and happiness are simply by listening to others’ advice? It really does feel different when we experience something ourselves compared to when we hear about it from others. Perhaps this is simply how human beings are designed. We were created to experience life’s ups and downs and learn from them.</p><p>I hope I won’t have any regrets about having wasted time. I truly want to be happy, but I also won’t shy away from sorrow or pain when they come. I will accept it and welcome it in the same way like happiness.</p><p>I realized that sorrow and pain are part of happiness. We would never experience happiness without sorrow and pain.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=ac7d34989f3c" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[From Biology to Data Science and Finally to Bioinformatician]]></title>
            <link>https://dedekurniawann.medium.com/from-biology-to-data-science-and-finally-to-bioinformatician-4b0bd7d289b1?source=rss-9f804bd8d19f------2</link>
            <guid isPermaLink="false">https://medium.com/p/4b0bd7d289b1</guid>
            <category><![CDATA[programming]]></category>
            <category><![CDATA[biology]]></category>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[bioinformatics]]></category>
            <category><![CDATA[self-improvement]]></category>
            <dc:creator><![CDATA[Dede Kurniawan]]></dc:creator>
            <pubDate>Sat, 27 Dec 2025 07:57:39 GMT</pubDate>
            <atom:updated>2025-12-27T15:22:16.661Z</atom:updated>
            <content:encoded><![CDATA[<h4>When I was trying to find a suitable role in the Data Science World, I suddenly found myself in the Bioinformatics world.</h4><figure><img alt="Dna sequence in hand. Wireframe dna code molecules structure mesh" src="https://cdn-images-1.medium.com/max/1024/1*XXG2xlNBVQGGrF2-B150-Q.jpeg" /><figcaption>Imgae by <a href="http://www.freepik.com/">Freepik</a></figcaption></figure><p>This story began when I was still an undergraduate student. When I first entered the biology department, I felt uncertain whether I had truly chosen the right major for my future career. Thanks to that doubt, I was driven to study <strong>data science</strong>. At that time, specifically in 2021, data science was a popular field. Before the artificial intelligence (AI) explosion, data scientists were once nominated as “<a href="https://hbr.org/2012/10/data-scientist-the-sexiest-job-of-the-21st-century"><strong>The Sexiest Job of the 21st Century</strong></a>” by Harvard Business Review.</p><p>As a biology student at that time, I spent most of my time studying data science rather than biology itself. Then I started studying Python programming, statistics, linear algebra, machine learning, and deep learning. The more I learn about data science, the more enthusiastic I become. Since then, I even started writing a series of articles on <a href="https://dedekurniawann.medium.com/">Medium</a> about data science topics. Writing articles every week helped my learning progress. I truly understood what I was learning when I rewrote the insights I gained in a simple and easy-to-understand writing style.</p><blockquote>“If you can’t explain it simply, you don’t understand it well enough.” — Albert Einstein</blockquote><p>My enthusiasm for learning data science was truly exploding. Until, at one point, I began to realize that data science is a vast world. In the world of data science, I also need to choose a specific path, for example, finance, healthcare, economics, etc. If I had to choose finance as that specific path, I felt I had no chance and would definitely lose in the competition. At this point, I started to lose motivation and thought about returning to the world of biology. Then, my enthusiasm vanished instantly when a groundbreaking AI chatbot was released at that time, 2022 (<a href="https://chatgpt.com/">ChatGPT</a>). I was truly worried that <a href="https://www.tryexponent.com/blog/data-analyst-ai-skills">data scientists might be replaced by AI</a> in the future. Maybe not entirely, but a data scientist who can only write code without creativity would certainly be replaced. I felt I was only excited about writing code with minimal creative thinking as a data scientist.</p><p>Since that moment, I felt truly confused. I questioned myself, “Will everything I’ve learned so far be in vain?” Then I found a way to use what I knew about data science in the field of biology. I saw a big gap in the biology world, especially when it came to the use of computer techniques to understand complex biological data. In an era where data growth is so fast, I felt that data science knowledge is extremely useful to solve biological problems.</p><blockquote>Data is an important resource in this era, perhaps even the “<strong>new oil.</strong>” Just like crude oil, data without proper processing becomes useless. The true value of data emerges when we successfully unravel its complexity and extract useful insights from it.</blockquote><p>Supported by my interest in molecular biology, I became more confident in choosing this path. A path where there is harmony between the wet lab and the dry lab, which is <strong>Bioinformatics</strong>. Bioinformatics may not be as popular a field as data science, but I truly found my identity in bioinformatics. I even feel so confident that I believe even AI cannot replace me in this field. “<strong><em>Bioinformatics is not just about writing code to understand biological data; it is also about the creativity to solve complex problems in biology</em></strong>”. This field also requires a strong foundation in both wet lab and dry lab work.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/960/1*EuxhCMqT1UQ0B8ViXYVIVg.png" /><figcaption>When Biology meets Programming.</figcaption></figure><p>Just like data science, bioinformatics is also an incredibly vast field. I have to select a particular path in bioinformatics as well. At the end of the day, I decided to focus on <a href="https://www.thermofisher.com/sg/en/home/life-science/sequencing/sequencing-learning-center/next-generation-sequencing-information/ngs-basics/what-is-next-generation-sequencing.html">Next Generation Sequencing</a> (NGS) and Multi-<a href="https://doi.org/10.3389/fmed.2022.911861">OMICS</a>. My interest in this path was greatly influenced by my supervisor at Genbinesia Indonesia, <a href="https://scholar.google.com/citations?user=BBtlZXMAAAAJ&amp;hl=en">Maulana M. Nashrulloh, Ph.D.</a> I developed my skills under his guidance by co-authoring multiple books, including <strong>eDNA metabarcoding</strong>, <strong>mitogenomics</strong>, and <strong>bacterial genome assembly</strong>. Furthermore, I also had the opportunity to serve as an instructor for several technical training sessions, such as <a href="https://www.instagram.com/p/C_BXyZCyTW2/">fundamentals of molecular biology and DNA barcoding</a>.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*9HK39isNE8LcpZ_lH1XdTg.png" /><figcaption>Some of my published books with my Genbinesia supervisor.</figcaption></figure><p>Because my bioinformatics technical skills were improving rapidly, I truly felt I was in the right field. At that point, an internship opportunity came at the <a href="https://en.genomics.cn/">Beijing Genomics Institute</a> (BGI), a top institution in life sciences and genomics. In short, I was accepted into the program and went to China to improve my bioinformatics skills. At this institution, my technical abilities grew exponentially. Working directly on top-tier projects proved to be an effective way to gain valuable experience. One of the major projects I worked on at BGI was the <a href="https://db.cngb.org/10kp/"><strong>Ten Thousand Plant Genome Project (10KP)</strong></a>. Through this project, I learned several technical skills from this research, such as <a href="https://github.com/openwdl/wdl">Workflow Description Language (WDL)</a>, Linux command line, genome assembly, genome annotation, and comparative genomics.</p><p>During my time at BGI, I learned an incredible amount in a very short period. I was fortunate to have a supportive supervisor (<a href="https://scholar.google.com/citations?user=DNmk68oAAAAJ&amp;hl=en">Dr. Sunil Kumar Sahu, Ph.D.</a>) who gave me a lot of flexibility. My professor didn’t just limit me to bioinformatics projects that required a lot of coding; he also allowed me to explore wet lab experiments and even work in the botanical garden. In just one year at BGI, I also achieved several achievements. The achievements I’m quite proud of from that time are winning a 3-minute thesis competition and publishing an <a href="https://doi.org/10.1016/j.plgene.2025.100530">SCI Q2 paper</a>.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*LQKPe3-LmveQx8F_R4FIFQ.png" /><figcaption>My achievements during my undergraduate periode at BGI.</figcaption></figure><p>If I continue to be in a supportive environment like this, I will likely keep growing. With my supervisor, supportive colleagues, and prestigious projects, I might achieve even greater things in the future. Publishing a paper in a top-tier journal like <a href="http://www.cell.com/"><strong>Cell</strong></a>, <a href="http://www.nature.com/"><strong>Nature</strong></a>, or <a href="http://www.science.org/"><strong>Science</strong></a><strong> </strong>is one of my goals. Staying at BGI for my postgraduate studies seems like the right choice.</p><p>Moving forward, on my personal Medium blog, I will write more about the world of bioinformatics. Unlike before, when my focus was on data science, bioinformatics seems to suit me better. It won’t be limited to just bioinformatics; I might also write about interesting topics in biology and the application of AI in this field.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=4b0bd7d289b1" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[When Machine Learning is Applied to Biological Research]]></title>
            <link>https://levelup.gitconnected.com/when-machine-learning-is-applied-to-biological-research-d860c683a70e?source=rss-9f804bd8d19f------2</link>
            <guid isPermaLink="false">https://medium.com/p/d860c683a70e</guid>
            <category><![CDATA[bioinformatics]]></category>
            <category><![CDATA[research]]></category>
            <category><![CDATA[biology]]></category>
            <category><![CDATA[data]]></category>
            <category><![CDATA[machine-learning]]></category>
            <dc:creator><![CDATA[Dede Kurniawan]]></dc:creator>
            <pubDate>Sun, 26 Nov 2023 03:51:49 GMT</pubDate>
            <atom:updated>2023-11-27T02:32:18.210Z</atom:updated>
            <content:encoded><![CDATA[<h4>An overview of machine learning and its applications in biological research</h4><figure><img alt="Scientis" src="https://cdn-images-1.medium.com/max/1024/1*pj5jxiUajQYdR8IqT5VYtA.jpeg" /><figcaption>Photo by <a href="https://www.pexels.com/photo/researchers-wearing-hair-nets-and-coveralls-reading-a-report-8539423/">Karolina Grabowska</a> from Pexels</figcaption></figure><p>The growing amount of biological data drives the use of machine learning in biological research. Most biological data is generated from next-generation sequencing technology. Sayers et al. (2023) reported that the number of public datasets in <a href="https://www.ncbi.nlm.nih.gov/genbank/">GenBank</a> is around 19.6 trillion base pairs [1]. The increasing complexity of biological data requires analysis techniques, such as machine learning. Machine learning is very useful when we have large-scale data and complexity.</p><p>In this article, I will give a brief explanation of what machine learning is. Then, I will give some examples of the application of machine learning in biological research.</p><p><a href="#5968">1. What is machine learning?</a><br><a href="#7d0c">2. How does machine learning work? </a><br><a href="#9b83">3. Use of machine learning in biological research </a><br><a href="#e3ab">4. Challenges of applying machine learning in biological research</a></p><h3>What is machine learning?</h3><p>The human brain has limitations in recognizing patterns. This happens when we are faced with a large amount of data. Machine learning is a study to identify patterns and relationships in complex data [2]. Machine learning uses computational methods to simulate how humans learn by exploring patterns in data [3]. Just like humans, machine learning models are constantly learning to improve their performance in completing tasks.</p><p>Machine learning is divided into 2 approaches, including supervised learning and unsupervised learning [4].</p><ol><li><strong>Supervised learning</strong> is a type of algorithm that learns labeled data. Unsupervised learning is widely used to solve regression and classification tasks. Linear regression, support vector machine (SVM), decision tree, and random forest are examples of supervised learning algorithms.</li><li><strong>Unsupervised learning</strong> is a type of algorithm that learns from unlabeled data. Unsupervised learning is widely used for clustering tasks. Examples of unsupervised learning algorithms include K-means clustering and PCA.</li></ol><figure><img alt="Application of supervised and unsupervised learning" src="https://cdn-images-1.medium.com/max/1024/1*qk5qqiqe7Ebt8RUUgTT6UQ.png" /><figcaption>Application of supervised and unsupervised learning to various types of biological data [3]</figcaption></figure><p>Sometimes, the above two approaches are combined into one, <strong>Semi-supervised learning</strong> [4]. Semi-supervised learning is a combination of supervised and unsupervised learning. This algorithm uses a small amount of labeled data combined with a large amount of unlabeled data [4].</p><h3>How does machine learning work?</h3><p>The main capability of machine learning is to make accurate predictions. Machine learning learns patterns in a data set to make predictions. A data set consists of several data points that are described by a feature. Features in a data set can be gene expression patterns, nucleotide sequences, protein sequences, and drug concentrations. For example, in taxonomic classification research, we may want to predict the name of a particular species based on nucleotide sequences. In such an example, the nucleotide sequence acts as the feature, while the species name acts as the prediction target (output).</p><p>In general, the machine learning workflow starts from collecting raw data. The raw data must be cleaned to improve the quality of the data. The prediction result of the machine learning model depends on how good the data quality is. Next is the selection of the right features based on the type of problem to be solved. The types of features available will be used as a consideration in choosing the right machine learning model. The machine learning model will be trained based on the features that have been determined. After the training is complete, the machine learning model can be used to generate a prediction.</p><figure><img alt="Classical machine learning workflow" src="https://cdn-images-1.medium.com/max/1024/1*dllYO9RpzHSNhwAlXC9e6Q.jpeg" /><figcaption>General classical machine learning workflow [6]</figcaption></figure><h3>Use of machine learning in biological research</h3><p>Greener et al. (2020) explained that there are two purposes why machine learning is used in biology. First, machine learning is used to make accurate predictions when experimental data availability is lacking. Second, machine learning is used to improve understanding of the complexity of biology [4].</p><p>Here are some examples of biological research that have applied machine learning.</p><h4>1. Prediction of antibiotic resistance</h4><p>Moradigaravand et al. (2018) used various ML algorithms to detect microbial antibiotic resistance. Their algorithms included logistic regression, random forest, gradient-boosted decision trees, and neural networks. They used whole genome sequence data from 1936 E. coli strains to predict resistance to 11 compounds across four classes of antibiotics [7].</p><h4>2. Prediction of dental implant health status</h4><p>Ghensi et al. (2020) used a random forest algorithm to predict the health status of dental implants. They used a microbiome metagenomic dataset on dental plaque as a feature. Meanwhile, the health condition of dental implant patients is used as a prediction target [8].</p><h4>3. Species interaction network prediction</h4><p>Strydom et al. (2021) reviewed a simple neural network algorithm to predict species interactions. They successfully developed an accurate model with limited data. Predicting species interaction networks can provide conservation-related information to protect species, communities, and ecosystems [9].</p><h4>4. Cancer cell viability prediction</h4><p>Angermueller et al. (2016) reviewed the use of supervised learning in predicting cancer cell viability. They used somatic cell variants, drug chemical makeup, and drug concentration as features. The algorithms they used include support vector machine and random forest [6].</p><h4>5. Taxonomic classification of microbes</h4><p>Parks et al. (2011) used the naive Bayes algorithm to classify microbial species. They used microbial genome sequence data as features and taxonomic labels as prediction targets. They proved that the NB algorithm has an advantage in accuracy [10].</p><h4>6. Colorectal cancer prediction</h4><p>Thomas et al. (2019) used a random forest algorithm to identify the relationship between gut microbiome and colorectal cancer. The ML model they generated had an AUC score of 0.84 which was validated on data that was not included in the training set [11].</p><h4>7. Prediction of human immunodeficiency virus type 1 (HIV-1) presence</h4><p>Solis-Reyes et al (2018) developed open-source software to detect the presence or absence of (HIV-1) based on genome sequence data. They used various algorithms for classification tasks, such as support vector machine, logistic regression, decision tree, random forest, naive Bayes, etc [12].</p><h4>8. Machine learning-based biological sequence analysis</h4><p>Liu (2019) built a platform, BioSeq-Analysis, for machine learning-based biological sequence analysis. on the BioSeq-Analysis platform, users only need to upload benchmark data sets. Then, BioSeq-Analysis will generate optimal predictors and report performance measures [13].</p><h4>9. Drug side effect prediction</h4><p>Zhang et al. (2015) adopted a multi-label learning technique to predict drug side effects. They used an ensemble method, namely the “feature selection-based multi-label k-nearest neighbor method (FS-MLKNN)” [14].</p><h4>10. Identification of genes causing autism disease</h4><p>Asif et al. (2018) predicted Autism Spectrum Disorder (ASD) candidate genes based on gene ontology. They used different classification methods, such as naive Bayes, decision trees, random forest, and linear and radial SVM. They reported that the random forest algorithm has higher accuracy with an AUC value reaching 0.80 [15].</p><h3>Challenges of applying machine learning in biological research</h3><p>The application of machine learning in the field of biology provides promising potential. However, there are still some challenges that must be resolved by researchers. Here are the challenges of applying machine learning in biological research [3]:</p><ol><li>It is difficult to represent diverse biological data in a format that is easily understood by machine learning algorithms.</li><li>Often the amount of data available is only for certain domains, such as medical biology.</li><li>The difficulty of combining different types of data, for example in multi-omics data.</li><li>Reproducibility is affected by different data processing paths even when using the same dataset.</li><li>The lack of statistical background for biologists makes it difficult to interpret machine learning models.</li></ol><h3>Conclusion</h3><p>The increasing amount of biological data presents opportunities and challenges for applying machine learning to biological research. The two main goals of applying machine learning to biological research are to make accurate predictions and to understand the complexity of biological data. Machine learning has been widely applied in biological research, such as predicting antibiotic resistance, predicting colon cancer, classifying microbial taxonomy, identifying disease-causing genes, etc. However, there are still some challenges in applying machine learning to biological research.</p><h3>References</h3><ol><li>Sayers, E. W., Cavanaugh, M., Clark, K., et al. 2023. GenBank 2023 update. <em>Nucleic acids research.</em> <em>51</em>(D1): D141–D144.</li><li>Asnicar, F., Thomas, A.M., Passerini, A. <em>et al.</em> 2023. Machine learning for microbiologists. <em>Nature Reviews Microbiology.</em></li><li>Auslander, N., Gussow, A. B., &amp; Koonin, E. V. 2021. Incorporating Machine Learning into Established Bioinformatics Frameworks<em>. International Journal of Molecular Sciences. 22(6): 2903.</em></li><li>Greener, J.G., Kandathil, S.M., Moffat, L. <em>et al.</em> 2022. A guide to machine learning for biologists. <em>Nature Reviews Molecular Cell Biology</em>. 23: 40–55.</li><li>Angermueller, C., Pärnamaa, T., Parts, L., &amp; Stegle, O. (2016). Deep learning for computational biology. <em>Molecular Systems Biology</em>. 12(7): 878.</li><li>Moradigaravand D, Palm M, Farewell A, Mustonen V, et al. 2018. Prediction of antibiotic resistance in <em>Escherichia coli</em> from large-scale pan-genome data. <em>PLoS Computational Biology</em>. 14(12): e1006258.</li><li>Ghensi, P., Manghi, P., Zolfo, M., et al. 2020. Strong oral plaque microbiome signatures for dental implant diseases identified by strain-resolution metagenomics<em>. Npj Biofilms and Microbiomes. 6(1): 1–12.</em></li><li>Strydom, T., Catchen, M. D., Banville, F., et al. 2021. A roadmap towards predicting species interaction networks (across space and time). <em>Philosophical Transactions of the Royal Society B: Biological Sciences.</em> 376(1837).</li><li>Parks, D. H., MacDonald, N. J., &amp; Beiko, R. G. 2011.<em> </em>Classifying short genomic fragments from novel lineages using composition and homology<em>. BMC Bioinformatics. 12(1): 328.</em></li><li>Thomas, A. M., Manghi, P., Asnicar, F., et al. 2019. Metagenomic analysis of colorectal cancer datasets identifies cross-cohort microbial diagnostic signatures and a link with choline degradation. <em>Nature Medicine.</em> <em>25</em>(4): 667–678.</li><li>Solis-Reyes, S., Avino, M., Poon, A., &amp; Kari, L. (2018). <em>A</em>n open-source k-mer-based machine learning tool for fast and accurate subtyping of HIV-1 genomes<em>. PLOS ONE. </em>13(11): e0206409.</li><li>Liu B. 2019. BioSeq-Analysis: a platform for DNA, RNA, and protein sequence analysis based on machine learning approaches. <em>Briefings in Bioinformatics.</em> 20(4): 1280–1294.</li><li>Zhang, W., Liu, F., Luo, L., &amp; Zhang, J. 2015. Predicting drug side effects by multi-label learning and ensemble learning. <em>BMC bioinformatics.</em> 16, 365.</li><li>Asif M, Martiniano HFMCM, Vicente AM, Couto FM. 2018. Identifying disease genes using machine learning and gene functional similarities, assessed through Gene Ontology. <em>PLoS ONE</em>. 13(12): e0208626.</li></ol><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=d860c683a70e" width="1" height="1" alt=""><hr><p><a href="https://levelup.gitconnected.com/when-machine-learning-is-applied-to-biological-research-d860c683a70e">When Machine Learning is Applied to Biological Research</a> was originally published in <a href="https://levelup.gitconnected.com/">Level Up Coding</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Basic Processing Biological Sequences Using Native Python and Biopython]]></title>
            <link>https://medium.datadriveninvestor.com/basic-processing-biological-sequences-using-native-python-and-biopython-4cae93786547?source=rss-9f804bd8d19f------2</link>
            <guid isPermaLink="false">https://medium.com/p/4cae93786547</guid>
            <category><![CDATA[biopython]]></category>
            <category><![CDATA[rna]]></category>
            <category><![CDATA[biology]]></category>
            <category><![CDATA[dna]]></category>
            <category><![CDATA[python]]></category>
            <dc:creator><![CDATA[Dede Kurniawan]]></dc:creator>
            <pubDate>Sun, 29 Oct 2023 15:07:23 GMT</pubDate>
            <atom:updated>2023-11-30T09:27:52.782Z</atom:updated>
            <content:encoded><![CDATA[<h4>What are biological sequences? How are biological sequences processed?</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Z80JICMMk0QbqSskPCrBeA.jpeg" /><figcaption>Image by Freepik.com</figcaption></figure><p>When working with biological sequence data, we may encounter low-quality data. Even while the data in databases like the NCBI is of excellent quality, there are instances where it is not. One of the many causes of low-quality data is typos. If we leave this bad data uncorrected, it will mess up the analysis results. <strong>The ability to process and analyze nucleotide sequences is fundamental in bioinformatics </strong>[1]. In this article, we will learn about the different types of biological sequences. Next, we will do some basic sequence processing.</p><h3>What are Biological Sequences?</h3><p>Genetic information in eukaryotic organisms is generally stored in the form of DNA [1]. The information in DNA is very important in providing instructions to the cell. DNA must be transcribed into RNA in order for the information encoded in it to be utilized. Messages containing genetic information are transported out of the nucleus via RNA (mRNA). Some parts of mRNA that contain information will be translated into amino acids (proteins). Proteins play an important role in almost all structural and regulatory functions of the body [2].</p><figure><img alt="The central dogma of biology" src="https://cdn-images-1.medium.com/max/559/1*2i_yzPV0k8SCI0tkChzXAw.png" /><figcaption>The central dogma of biology (Image by Author)</figcaption></figure><p>So, <strong>biological sequences include DNA, RNA, and proteins</strong>. Let’s talk a little more about these biological sequences.</p><h3>DNA, RNA, and Protein</h3><p>Nucleotides are the basic units of DNA and RNA. In computing, DNA and RNA sequences are represented as strings with 4 alphabets. DNA is represented with the letters “A” (Adenine), “C” (Cytosine), “G” (Guanine), and “T” (Thymine). Meanwhile, RNA is represented with “A”, “C”, “G”, and “U” (Uracil). Each nitrogenous base is always in a pair connected by hydrogen bonds. “A ”will pair with “T”/“U”, while “G ”will pair with “C”. Often, DNA and RNA nucleotide sequences are represented by these nitrogenous bases.</p><p>Another important biological sequence is proteins. Proteins are represented in amino acid sequences. There are 20 different amino acids that makeup proteins. IUPAC organizes these 20 types of amino acids into 20 different alphabet symbols [3].</p><figure><img alt="IUPAC amino acid code" src="https://cdn-images-1.medium.com/max/444/1*kjERKrpDYBb3ph7oy62QYQ.png" /><figcaption>IUPAC amino acid code [3]</figcaption></figure><p>So, for DNA and RNA sequences, we will only work with 4 alphabets. The four alphabets that represent nucleotides include “A”, “C”, “G”, “T” or “U”. Meanwhile, for proteins, we will work with 20 alphabets representing amino acids [4].</p><p>After understanding the types of sequences, we will move on to discuss sequence processing. First, we will see how to process a sequence using Python. Then, we will use the Biopyhton library [7] to perform sequence processing.</p><h3>Basic Sequence Processing Using Python</h3><p>Before writing a program to work with biological sequences, we need to represent them in a particular data structure or data type. There are various ways to represent sequences, such as storing them in .FASTA files. However, the most common and simple method is to store sequences as text or strings [1, 4]. A string is a collection of characters or letters arranged in sequence.</p><blockquote>You can see the entire Python code in this article on 👉 <a href="https://colab.research.google.com/drive/1lgBnAuCrHBCV1QN2vr1DHHFt1WDQXcvd?usp=sharing">Google Colabolatory</a></blockquote><h4>1. DNA sequence validation</h4><p>The first basic sequence processing is DNA sequence validation. Sometimes, we will encounter a DNA sequence that contains characters other than “ATGC”. DNA sequences that contain characters other than “ATGC” are invalid sequences.</p><pre>def dna_validation(sequence):<br>    &quot;&quot;&quot;function to validate the dna sequence&quot;&quot;&quot;<br><br>    seq = sequence.upper()<br>    valid = seq.count(&quot;A&quot;) + seq.count(&quot;G&quot;) + seq.count(&quot;C&quot;) + seq.count(&quot;T&quot;)<br>    if valid == len(seq):<br>        return True<br>    else:<br>        return False<br><br>dna_validation(&quot;ACCUCTGCU6&quot;)<br># False<br><br>dna_validation(&quot;TAGTTAGTTT&quot;)<br># True</pre><p>We can use the function to confirm DNA sequences. When we include the right DNA sequence, the function will return True and vice versa.</p><h4>2. Counting the types of DNA nitrogenous bases</h4><p>DNA consists of 4 types of nitrogenous bases represented as “A”, “T”, “G”, and “C”. We will create a function to calculate the nitrogenous base content in a DNA sequence.</p><p>The result of the function is a dictionary data type, where the “key ”is the nitrogenous base symbol and the “value ”is the amount of nitrogenous base content.</p><pre>def count_base(dna_seq):<br>    &quot;&quot;&quot;function to count the number of nitrogenous bases&quot;&quot;&quot;<br><br>    tempfreq = {&quot;A&quot;:0, &quot;G&quot;:0, &quot;C&quot;:0, &quot;T&quot;:0}<br>    for nuc in dna_seq:<br>        tempfreq[nuc] += 1<br>    return tempfreq<br><br>count_base(&quot;TCATTCTGACTGCAACGGGCAATAT&quot;)<br># {&#39;A&#39;: 7, &#39;G&#39;: 5, &#39;C&#39;: 6, &#39;T&#39;: 7}</pre><h4>3. Calculating GC content</h4><p>GC content is the percentage of guanine and cytosine nitrogenous bases in a DNA or RNA sequence. You might be wondering why we need to measure GC content. Measuring GC content can provide useful insights, one of which is in designing primers. Primers with a GC content of around 40% — 60% are generally preferred because they have a melting point that is suitable for PCR [5].</p><pre>def gc_content(sequence):<br>    &quot;&quot;&quot;function to calculate the GC content&quot;&quot;&quot;<br><br>    result = round((sequence.count(&quot;G&quot;) + sequence.count(&quot;C&quot;))/len(sequence) * 100)<br>    return f&quot;GC content: {result}%&quot;<br><br>gc_content(&quot;TCATTCTGACTGCAACGGGCAATAT&quot;)<br># GC content: 44%</pre><p>The function will calculate the GC content by first calculating the total amount of GC. Then, the total GC content will be divided by the total nitrogenous bases in a sequence. The result of the calculation is multiplied by 100 to calculate the percentage (%) of GC content in a sequence.</p><h4>4. Making DNA complemented and reverse complemented</h4><p>In eukaryotic organisms, DNA is double-stranded. This means that nucleotide sequences are always paired. For example, complement A is T and complement C is G.</p><pre>def dna_complement(sequence):<br>    &quot;&quot;&quot;function to make the complement of a DNA sequence&quot;&quot;&quot;<br><br>    complement = {&quot;A&quot;: &quot;T&quot;, &quot;T&quot;: &quot;A&quot;,<br>                 &quot;G&quot;: &quot;C&quot;, &quot;C&quot;:&quot;G&quot;}<br>    return &quot;&quot;.join([complement[s] for s in sequence])<br><br>dna_complement(&quot;TCATTCTGACTGCAACGGGCAATAT&quot;)<br># AGTAAGACTGACGTTGCCCGTTATA</pre><p>On the other hand, reverse complement is a sequence from the 5&#39; end to the 3&#39; end. An easy way to do reverse complement is to reverse the original sequence first, then each nucleotide is replaced with its complement.</p><pre>def reverse_conplement(sequence):<br>    &quot;&quot;&quot;function to reverse the DNA sequence&quot;&quot;&quot;<br>    print(sequence)<br>    print(&quot;&quot;.join(&quot;|&quot; for s in range(len(sequence))))<br>    print(dna_complement(sequence[::-1]))<br><br>reverse_conplement(&quot;TCATTCTGACTGCAACGGGCAATAT&quot;)<br># TCATTCTGACTGCAACGGGCAATAT<br># |||||||||||||||||||||||||<br># ATATTGCCCGTTGCAGTCAGAATGA</pre><h4>5. Transcription</h4><p>Transcription is the process of rewriting the information in DNA into RNA (precisely mRNA) [6]. Assuming the sequence we input is a DNA template, then we will get the RNA sequence obtained by replacing “T” with “U” [1].</p><pre>def transcription(dna_seq):<br>    &quot;&quot;&quot;function to translate DNA sequences to RNA&quot;&quot;&quot;<br><br>    rna = dna_seq.replace(&quot;T&quot;, &quot;U&quot;)<br>    return rna<br><br>transcription(&quot;TCATTCTGACTGCAACGGGCAATAT&quot;)<br># UCAUUCUGACUGCAACGGGCAAUAU</pre><h4>6. Translation</h4><p>The translation is the process of converting information from mRNA into amino acids (proteins) [6]. Translation is done by taking 3 nucleotide letters as the key to find the corresponding amino acid code. The amino acid code must first be written down in the form of a Python dictionary [4]. The key in the dictionary will be the codon (64 in total) and the corresponding value is the amino acid code (according to IUPAC).</p><pre>RNA_Codons = {<br>    # &#39;M&#39; - START, &#39;*&#39; - STOP<br>    &quot;GCU&quot;: &quot;A&quot;, &quot;GCC&quot;: &quot;A&quot;, &quot;GCA&quot;: &quot;A&quot;, &quot;GCG&quot;: &quot;A&quot;,<br>    &quot;UGU&quot;: &quot;C&quot;, &quot;UGC&quot;: &quot;C&quot;,<br>    &quot;GAU&quot;: &quot;D&quot;, &quot;GAC&quot;: &quot;D&quot;,<br>    &quot;GAA&quot;: &quot;E&quot;, &quot;GAG&quot;: &quot;E&quot;,<br>    &quot;UUU&quot;: &quot;F&quot;, &quot;UUC&quot;: &quot;F&quot;,<br>    &quot;GGU&quot;: &quot;G&quot;, &quot;GGC&quot;: &quot;G&quot;, &quot;GGA&quot;: &quot;G&quot;, &quot;GGG&quot;: &quot;G&quot;,<br>    &quot;CAU&quot;: &quot;H&quot;, &quot;CAC&quot;: &quot;H&quot;,<br>    &quot;AUA&quot;: &quot;I&quot;, &quot;AUU&quot;: &quot;I&quot;, &quot;AUC&quot;: &quot;I&quot;,<br>    &quot;AAA&quot;: &quot;K&quot;, &quot;AAG&quot;: &quot;K&quot;,<br>    &quot;UUA&quot;: &quot;L&quot;, &quot;UUG&quot;: &quot;L&quot;, &quot;CUU&quot;: &quot;L&quot;, &quot;CUC&quot;: &quot;L&quot;, &quot;CUA&quot;: &quot;L&quot;, &quot;CUG&quot;: &quot;L&quot;,<br>    &quot;AUG&quot;: &quot;M&quot;,<br>    &quot;AAU&quot;: &quot;N&quot;, &quot;AAC&quot;: &quot;N&quot;,<br>    &quot;CCU&quot;: &quot;P&quot;, &quot;CCC&quot;: &quot;P&quot;, &quot;CCA&quot;: &quot;P&quot;, &quot;CCG&quot;: &quot;P&quot;,<br>    &quot;CAA&quot;: &quot;Q&quot;, &quot;CAG&quot;: &quot;Q&quot;,<br>    &quot;CGU&quot;: &quot;R&quot;, &quot;CGC&quot;: &quot;R&quot;, &quot;CGA&quot;: &quot;R&quot;, &quot;CGG&quot;: &quot;R&quot;, &quot;AGA&quot;: &quot;R&quot;, &quot;AGG&quot;: &quot;R&quot;,<br>    &quot;UCU&quot;: &quot;S&quot;, &quot;UCC&quot;: &quot;S&quot;, &quot;UCA&quot;: &quot;S&quot;, &quot;UCG&quot;: &quot;S&quot;, &quot;AGU&quot;: &quot;S&quot;, &quot;AGC&quot;: &quot;S&quot;,<br>    &quot;ACU&quot;: &quot;T&quot;, &quot;ACC&quot;: &quot;T&quot;, &quot;ACA&quot;: &quot;T&quot;, &quot;ACG&quot;: &quot;T&quot;,<br>    &quot;GUU&quot;: &quot;V&quot;, &quot;GUC&quot;: &quot;V&quot;, &quot;GUA&quot;: &quot;V&quot;, &quot;GUG&quot;: &quot;V&quot;,<br>    &quot;UGG&quot;: &quot;W&quot;,<br>    &quot;UAU&quot;: &quot;Y&quot;, &quot;UAC&quot;: &quot;Y&quot;,<br>    &quot;UAA&quot;: &quot;*&quot;, &quot;UAG&quot;: &quot;*&quot;, &quot;UGA&quot;: &quot;*&quot;<br>}<br><br>def translation(rna_seq, init_pos=0):<br>    &quot;&quot;&quot;function to convert rna sequences into amino acids&quot;&quot;&quot;<br><br>    return [RNA_Codons[rna_seq[pos: pos +3]] for pos in range(init_pos, len(rna_seq)-2, 3)]<br><br>rna = transcription(&quot;TCATTCTGACTGCAACGGGCAATAT&quot;)<br>translation(rna)<br># [&#39;S&#39;, &#39;F&#39;, &#39;*&#39;, &#39;L&#39;, &#39;Q&#39;, &#39;R&#39;, &#39;A&#39;, &#39;I&#39;]</pre><p>What we have done above is a basic step in sequence processing. In the real world, you may encounter complicated codes to process a nucleotide sequence. To deepen your understanding of sequence processing, you can practice on the Rosalind website. Rosalind is a platform for learning bioinformatics through problem-solving.</p><h3>Performing Sequence Processing with the Biopython Library</h3><p>Some of you might be struggling to write native Python code to perform sequence processing. If you don’t want to have trouble writing native Python code, you can use <strong>Biopython</strong>. Biopython is an open-source computational molecular biology library [7]. By using this library, you can simply call functions to process nucleotide sequences. Furthermore, Biopython can also be used to perform statistical analysis, visualization, and build models from biological data.</p><p>Before using Biopython, make sure you install it first.</p><pre># using pip<br>pip install biopython<br><br># using conda<br>conda install -c conda-forge biopython</pre><p>Unlike the native Python code, we will represent nucleotide sequences in Biopython’s Seq object. How is this different from representing sequences in string as we did above? The difference lies in the methods that we can apply to the object. In general, Biopython Seq objects support the same methods as regular strings. However, there are additional biologically relevant methods, such as translate() and reverse_complement().</p><p>We will first import the module.</p><pre>from Bio.Seq import Seq<br>from Bio.SeqUtils import gc_fraction</pre><p>There are 2 modules that we will use, namely Bio.Seq and Bio.SeqUtils. The first module will be used to create the Seq object. Meanwhile, we will use the second module to calculate GC content.</p><figure><img alt="Biopython code [1]" src="https://cdn-images-1.medium.com/max/636/1*5tmp0oR06fe_4QyAA0KIwA.png" /></figure><p>The example above is the basic foundation of sequence processing using Biopython. There are still a lot of Biopython capabilities that we haven’t tried yet. You can practice on your own by reading the official documentation of Biopython.</p><figure><img alt="Biopython code [2]" src="https://cdn-images-1.medium.com/max/630/1*mvN4rTnRT0oo6HjhVcfdUw.png" /></figure><h4>Key-take away</h4><ul><li>Biological sequences consist of DNA, RNA, and protein sequences.</li><li>DNA sequences are represented with the letters “AGCT”, while RNA sequences are represented with the letters “AGCU”.</li><li>Unlike DNA and RNA sequences, protein sequences are represented by 20 different alphabets (IUPAC standards).</li><li>Some of the basic sequence processing we have done includes DNA validation, calculating nitrogenous base types, calculating GC content, and transcription and translation.</li><li>Biopython is a very useful library in assisting the processing of DNA, RNA, and protein sequences.</li></ul><h4>References:</h4><ol><li>Rocha, M. &amp; P. G. Ferreira. Bioinformatics Algorithms Design and Implementation in Python. Academic Press. London.</li><li>Bosserhoff, A. &amp; M. Next Generation Sequencing and Data Analysis. Springer. 1–14.</li><li>IUPAC codes. Available at: <a href="https://www.bioinformatics.org/sms/iupac.html">https://www.bioinformatics.org/sms/iupac.html</a> (Accessed: 29 October 2023).</li><li>Stevens, T. J. &amp; W. Boucher. 2015. Python Programming for Biology: Bioinformatics and Beyond. Cambridge University Press</li><li>Staff, B.T.B. (2021). _PCR Primer Design Tips_, _Behind the Bench_. Available at: <a href="https://www.thermofisher.com/blog/behindthebench/pcr-primer-design-tips/">https://www.thermofisher.com/blog/behindthebench/pcr-primer-design-tips/</a> (Accessed: 29 October 2023).</li><li>Urry, L. A., Cain, M. L., Minorsky, P. V., Wasserman, S. A., &amp; Reece, R. B. (2017). Campbell Biology. 11th Edition. Pearson. United States.</li><li>Cock PA, Antao T, Chang JT, Chapman BA, Cox CJ, Dalke A, Friedberg I, Hamelryck T, Kauff F, Wilczynski B and de Hoon MJL. (2009). Biopython: freely available Python tools for computational molecular biology and bioinformatics. Bioinformatics. 25: 1422–1423.</li></ol><p>Subscribe to DDIntel <a href="https://www.ddintel.com/">Here</a>.</p><p>Have a unique story to share? Submit to DDIntel <a href="https://datadriveninvestor.com/ddintelsubmission">here</a>.</p><p>Join our creator ecosystem <a href="https://join.datadriveninvestor.com/">here</a>.</p><p><a href="https://ddintel.datadriveninvestor.com/"><em>DDIntel</em> </a>captures the more notable pieces from our <a href="https://www.datadriveninvestor.com/"><em>main site</em></a> and our popular <a href="https://medium.datadriveninvestor.com/"><em>DDI Medium publication</em></a>. Check us out for more insightful work from our community.</p><p>DDI Official Telegram Channel: <a href="https://t.me/+tafUp6ecEys4YjQ1">https://t.me/+tafUp6ecEys4YjQ1</a></p><p>Follow us on <a href="https://www.linkedin.com/company/data-driven-investor"><em>LinkedIn</em></a>, <a href="https://twitter.com/@DDInvestorHQ"><em>Twitter</em></a>, <a href="https://www.youtube.com/c/datadriveninvestor"><em>YouTube</em></a>, and <a href="https://www.facebook.com/datadriveninvestor"><em>Facebook</em></a>.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=4cae93786547" width="1" height="1" alt=""><hr><p><a href="https://medium.datadriveninvestor.com/basic-processing-biological-sequences-using-native-python-and-biopython-4cae93786547">Basic Processing Biological Sequences Using Native Python and Biopython</a> was originally published in <a href="https://medium.datadriveninvestor.com/">DataDrivenInvestor</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[DNA Barcoding: A Powerful Tool for Species-Level Identification]]></title>
            <link>https://dedekurniawann.medium.com/dna-barcoding-a-powerful-tool-for-species-level-identification-f3e89d39e734?source=rss-9f804bd8d19f------2</link>
            <guid isPermaLink="false">https://medium.com/p/f3e89d39e734</guid>
            <category><![CDATA[biology]]></category>
            <category><![CDATA[dna]]></category>
            <category><![CDATA[bioinformatics]]></category>
            <category><![CDATA[diversity]]></category>
            <dc:creator><![CDATA[Dede Kurniawan]]></dc:creator>
            <pubDate>Fri, 20 Oct 2023 02:34:30 GMT</pubDate>
            <atom:updated>2023-10-29T15:29:33.224Z</atom:updated>
            <content:encoded><![CDATA[<h4>Getting started with DNA barcoding, its history, and DNA barcoding workflow in species identification</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*2TEuqiYSQmBRX-yTowqnmw.jpeg" /><figcaption>Image by <a href="https://www.freepik.com/free-photo/dna-representation-concept_65671000.htm#query=DNA&amp;position=47&amp;from_view=search&amp;track=sph">Freepik</a></figcaption></figure><p>Indonesia is a country with <strong>mega biodiversity</strong>. According to the Global Biodiversity Index (2022), Indonesia ranks second in the world with the highest biodiversity with a score of 418.78 (below Brazil) [1]. The high diversity allows that there are still many flora and fauna that still need to be explored. We need accurate tools in species identification for biodiversity exploration. DNA barcoding has proven a powerful tool in species-level identification [2]. In this article, we will dig deeper into DNA barcoding. I will explain the definition, history, and workflow of DNA barcoding.</p><h3>What is DNA Barcoding?</h3><p>DNA barcoding is a molecular-level species identification tool. This identification is done using highly conserved genes. So, what exactly is DNA barcoding? <strong>DNA barcoding</strong> is a species identification tool based on certain DNA sequences from an organism [3]. Meanwhile, Wilson et al. (2018) explain that DNA barcoding refers to using short, standardized DNA sequences as markers to recognize species [4]. In short, DNA barcoding focuses on using information from one or more regions in the DNA sequence to identify species.</p><figure><img alt="DNA barcoding" src="https://cdn-images-1.medium.com/max/823/1*3ahb8_tO6aNfP7BDRyWgEg.png" /><figcaption>Representative DNA barcoding for species identification (<a href="https://www.agri-pulse.com/articles/15847-california-may-soon-have-a-dna-barcode-library-for-invasive-insects">source</a>)</figcaption></figure><p>A specific region of DNA sequence used as an identification tool is called a “<strong>DNA barcode</strong>”. DNA barcodes used as markers must meet several criteria: containing significant species-level genetic variability and divergence, universal PCR primers that are easy to sequence, and short DNA sequences (400–600 bp) to facilitate extraction and amplification capabilities [5]. Specific DNA regions that are widely used in DNA barcoding for plant identification include <em>rbcL</em>, <em>matK</em>, <em>trnH-psbA</em>, and <em>ITS</em> [6].</p><h3>Brief History of DNA Barcoding</h3><p>DNA barcoding was first proposed by Hebert et al. (2003) using the mitochondrial cytochrome oxidase subunit I (COI) gene as a marker for animal identification [5]. The term “DNA barcoding” was introduced as short gene sequences of DNA regions used as species identification markers. Soon after, DNA barcoding was widely used by researchers to identify various organisms from the Plantae, Fungi, Archaea, Bacteria, Viruses, Protista, and Ciliates [2]. The success of this research opened new doors in the field of taxonomy.</p><p>One of the most significant contributions of DNA barcoding is as an environmental law enforcement tool. DNA barcoding has been actively used to prevent illegal trade in biodiversity [7]. In addition to these contributions, DNA barcoding continues to be used in various fields, such as to identify protected, invasive, and endangered species, as well as to test the identity and purity of botanical products [6].</p><p>As taxonomic projects using DNA barcoding increase, there are global initiatives that assist researchers in organizing DNA barcoding databases. A milestone was created when the International Barcode of Life (iBOL) project was launched. iBOL is an international project (participated by 25 countries) that aims to build a DNA barcode reference library to serve as the basis for an identification system [3].</p><h3>DNA Barcoding Workflow for Species-Level Organism Identification</h3><p>Species identification using DNA barcoding is not simple. It requires several processes until an organism can finally be identified. DNA barcoding procedures include sampling, extraction of sample DNA, selection of barcode genes for amplification with universal primers, amplification by PCR, sequencing using Sanger or high-throughput sequencing [2], and analysis of matching sample DNA barcode sequences with DNA barcode libraries [6]. Reference databases for DNA barcode comparison can be obtained from the Barcode of Life Data System (BOLD) and Genbank [4].</p><figure><img alt="DNA barcoding workflow" src="https://cdn-images-1.medium.com/max/531/1*ZVgtvZNNAR553KaF7XrIPQ.png" /><figcaption>Representation of DNA barcoding workflow [2]</figcaption></figure><h4>Conclusion</h4><p>DNA barcoding is a molecular species identification tool using standardized marker genes. Gene regions used as markers must contain significant species-level genetic variability and divergence, universal PCR primers, and short DNA sequences (400–600 bp). In addition to species identification, DNA barcoding is also used as a means of identifying protected species, invasive species, and endangered species, as well as to test the identity and purity of botanical products.</p><h4>Reference:</h4><ol><li>Global Biodiversity Information Facility (GBIF) .2023. <a href="https://www.gbif.org/">https://www.gbif.org/.</a> Accessed on October 6, 2023.</li><li>Antil, S., Abraham, J. S., Sripoorna, S., Maurya, S., Dagar, J., Makhija, S., Bhagat, P., Gupta, R., Sood, U., Lal, R., &amp; Toteja, R. 2022. DNA barcoding, an effective tool for species identification: A Review. <em>Molecular Biology Reports</em>. 50(1): 761–775.</li><li>Trivedi, S., H. Rehman., S. Saggu., C. Panneerselvam., S.K. Ghosh. 2018. DNA barcoding and molecular phylogeny. Springer International.</li><li>Wilson, J.-J., Sing, K.-W., &amp;amp; Jaturas, N. 2019. DNA barcoding: Bioinformatics workflows for Beginners. <em>Encyclopedia of Bioinformatics and Computational Biology</em>. 985–995.</li><li>Hebert, P. D. N., Cywinska, A., Ball, S. L., &amp; deWaard, J. R. 2003. Biological identifications through DNA barcodes. Proceedings of the Royal Society B. <em>Biological Sciences</em>. 270(1512): 313–321.</li><li>Kress, W. J., Liu, A. Z., Newman, M., &amp; Li, Q.-J. 2005. The molecular phylogeny of <em>Alpinia</em> (Zingiberaceae): a complex and polyphyletic genus of gingers. <em>American Journal of Botany</em>. 92(1): 167–178.</li><li>Asis, A. M., Lacsamana, J. K., &amp; Santos, M. D. (2016). Illegal trade of regulated and protected aquatic species in the Philippines detected by DNA barcoding. <em>Mitochondrial DNA. Part A, DNA mapping, sequencing, and analysis</em>, <em>27</em>(1), 659–666.</li></ol><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=f3e89d39e734" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Why Julia’s Language is Perfect for Scientific Computing?]]></title>
            <link>https://levelup.gitconnected.com/why-julias-language-is-perfect-for-scientific-computing-e2b37f7d58ee?source=rss-9f804bd8d19f------2</link>
            <guid isPermaLink="false">https://medium.com/p/e2b37f7d58ee</guid>
            <category><![CDATA[julia]]></category>
            <category><![CDATA[computer-science]]></category>
            <category><![CDATA[scientific-computing]]></category>
            <category><![CDATA[programming]]></category>
            <dc:creator><![CDATA[Dede Kurniawan]]></dc:creator>
            <pubDate>Thu, 24 Aug 2023 06:14:56 GMT</pubDate>
            <atom:updated>2023-08-29T17:51:05.244Z</atom:updated>
            <content:encoded><![CDATA[<h4>What are you waiting for, learn Julia now!</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*tHQDbtbf-l0LqepVbKWT9A.jpeg" /><figcaption>Photo by <a href="https://www.pexels.com/photo/grayscale-photo-of-a-futuristic-robot-8294608/">Pavel Danilyuk</a> on Pexels</figcaption></figure><p>One of the biggest problems in scientific computing is the <strong>two language problems</strong>. Researchers usually start a scientific application using a productivity language (e.g. Python, R, or Matlab). As the computational complexity increases, researchers will switch the application to a high-performance language, such as C. Converting an existing application to another language is difficult.</p><p>Julia&#39;s programming language is <strong>here to solve this problem</strong>. Julia is one of the languages with high productivity and high performance. It’s like Python, but it runs like C. In this article, I will highlight why Julia is perfect for scientific computing. Let’s get to know more about Julia!</p><h3>A brief introduction to Julia</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/633/1*FP47Zkwg-ACLVdjrFBNoCg.png" /><figcaption>Julia (Image compiled by author)</figcaption></figure><p>Julia is an open-source multipurpose programming language that can run on any operating system. Jeff Besanzon and his friends (Stefan Karpinski, Viral B. Shah, and Alan S. Edelman) are the people behind the creation of this fantastic language. Besanzon et al. explained that there was no particular reason why this programming language was named “Julia”, this name was used because it was good.</p><p>As written in his blog “<a href="https://julialang.org/blog/2012/02/why-we-created-julia/">Why We Created Julia</a>”, Bezanson et al. are greedy humans. They wanted to create the perfect language. Julia is the result of their ambition. It is designed to have the advantages of other programming languages such as Matlab, Lisp, Mathematica, Ruby, Perl, R, Python, and even C. In the end, Julia can support a wide range of work. Ultimately, Julia can support a wide range of work in scientific computing, machine learning, data mining, large-scale linear algebra, distributed and parallel computing.</p><p>The language is approximately <strong>11 years old</strong> since its first official release in February 2012. The biggest release of this programming language was in Julia version 1.6 with long-term support (LTS). To date, Julia 1.9 is the latest version released in May 2023.</p><h3>Why Julia is suitable for scientific computing?</h3><p>Julia is a programming language that was originally designed for scientific computing. Here are some reasons why Julia is suitable for performing scientific computing tasks:</p><h4>1. Julia is easy to learn</h4><p>Julia is a high-level programming language. Unlike low-level languages (e.g., Java or C), high-level languages are easier for beginners to learn and understand. Researchers only need a short time to understand Julia, especially if they are migrating from Python, R, and Matlab.</p><p>Take a look at the simple example program below to see why Julia is easier to understand:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*uAD1YyV9eJFoTv0ZNik-bQ.png" /><figcaption>Comparison of “Hello, World!” program code in Java, C, and Julia (Image by author)</figcaption></figure><p>In a simple program that displays “Hello, World!”, the Julia language syntax is more concise and simple than C and Java. Researchers don’t need to worry about the complexity of the program. With Julia, researchers only need to focus on the problem they want to solve.</p><h4>2. Allows for a high level of abstraction</h4><p>Scientific problems are already very complicated. Researchers need a programming language with a high level of abstraction. Abstraction allows researchers to focus on innovative solutions without worrying about complex technical details. Julia enables a very high level of abstraction.</p><p>For example, a researcher wants to analyze heterogeneous and complex biological data. To understand the data, the researcher must visualize the data. With Julia’s high level of abstraction, researchers can write code that is intuitive and easy to understand. They can use available open-source libraries, such as <a href="https://docs.juliaplots.org/latest/tutorial/">Plots.jl</a>, <a href="https://github.com/JuliaPy/PyPlot.jl">PyPlot.jl</a>, and <a href="https://github.com/JuliaPlots/StatsPlots.jl">StatsPlot.jl</a>.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*Rn0AlnK8fl-iGtL1.png" /><figcaption>Examples of some open-source libraries for performing scientific computing (related to biology topics) [2]</figcaption></figure><h4>3. High performance like C or Fortran</h4><p>One of the common problems of scientific computing is handling large data sizes. The performance of the programming language becomes very important to complete the analysis better and faster. It also provides convenience for researchers. Without high performance, simulating complex mathematical models may not be possible.</p><p>Julia comes with high computational performance. Compared to other dynamic programming languages (e.g., Python, R, and Matlab), Julia excels in terms of performance. The following is a comparison of Julia’s performance with other programming languages:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/470/0*2JVUtCgkv99GjZNR.jpg" /><figcaption>Julia’s performance benchmark compared to other programming languages [3]</figcaption></figure><h4>4. Has many productivity features</h4><p>Programming languages with productivity features allow researchers to solve scientific problems more quickly and efficiently. Julia is designed with various productivity features such as:</p><ul><li><strong>Dynamic typing</strong>: Researchers do not need to explicitly define the type of data types to be used. Julia automatically determines the type of data type.</li><li><strong>Automatic memory management</strong>: Julia relies on Garbage collection to manage memory. Researchers do not need to allocate and free memory manually.</li><li><strong>Rich type annotation</strong>: The researcher can provide a specific description of what data type to use in a variable or function. It uses the :: operator to annotate data types (e.g., square(x::Int64, y::Float64)).</li><li><strong>Multiple dispatch</strong>: The researcher can define multiple functions with the same name and different behavior according to the given data type. In simple terms, functions in Julia language are like templates. When the function is called, the compiler will automatically select and apply the function template according to the given data type.</li></ul><blockquote>“Multiple dispatch is also one of the reasons why Julia can have high performance.”</blockquote><h4>5. Metaprogramming</h4><p>Nowadays, most scientific problems are solved with statically written source code. Once written, the source code is executed by the compiler and is never changed during the execution process. With Julia’s flexibility, researchers can modify the source code during run time. This is called metaprogramming. Metaprogramming has the advantage of automating parts of the scientific problem-solving process.</p><h3>Limitations of Julia&#39;s programming language</h3><p>Nothing is perfect in this world, and Julia is no exception. Even with its various advantages, Julia still has some limitations. Here are the limitations of Julia:</p><ul><li><strong>An emerging language</strong>: Julia is a new programming language (11 years since its release in 2012). The Julia community is also not as large as the Python community. This means that there are currently fewer resources to learn Julia compared to Python.</li><li><strong>It uses large memory</strong>: This is due to the function template feature (multiple dispatch). It provides different types of functions for different types of data which ends up using more memory space.</li><li><strong>Multiple dispatch inhibits modularity</strong>: Since each function can apply to multiple data types, there is no point at which it will be closed in the future. This feature continues to be developed to support modularity.</li><li><strong>The small number of open source libraries</strong>: Python currently has 476465 projects listed on <a href="https://pypi.org/">PyPI</a>, while the number of Julia libraries listed on <a href="https://docs.julialang.org/en/v1/stdlib/Pkg/">Julia Package Manager</a> is around 10000.</li></ul><h3>Conclusion</h3><p>The Julia programming language has great potential to replace the current popular programming languages. Julia has the simplicity of Python but has the high performance of C. Some of the advantages of Julia, such as easy to learn (readable), high level of abstraction, high performance, many productivity features, and metaprogramming. On the other hand, Julia also has some limitations. However, these shortcomings do not reduce the superiority of Julia language for scientific computing.</p><h4>References:</h4><p>[1] Bezanson, J., Edelman, A., Karpinski, S., dkk. (2012) <em>Why we created julia</em>, <em>The Julia Programming Language</em>. Available at: <a href="https://julialang.org/blog/2012/02/why-we-created-julia/">https://julialang.org/blog/2012/02/why-we-created-julia/</a> (Accessed: 24 August 2023).</p><p>[2] Roesch, E., Greener, J. G., MacLean, A. L., Nassar, H., Rackauckas, C., Holy, T. E., &amp; Stumpf, M. P. (2023). Julia for biologists. <em>Nature Methods</em>, <em>20</em>(5), 655–664.</p><p>[3] Gao, K., Mei, G., Piccialli, F., Cuomo, S., Tu, J., &amp; Huo, Z. (2020). Julia language in machine learning: Algorithms, applications, and open issues. <em>Computer Science Review</em>, <em>37</em>, 100254.</p><p>[4] Bezanson, J., Karpinski, S., Shah, V. B., &amp; Edelman, A. (2012). Julia: A Fast Dynamic Language for Technical Computing. <em>ArXiv</em>.</p><p>[5] Bezanson, J., Chen, J., Chung, B., Karpinski, S., Shah, V. B., Vitek, J., &amp; Zoubritzky, L. (2018). <em>Julia: dynamism and performance reconciled by design. Proceedings of the ACM on Programming Languages, 2(OOPSLA), 1–23.</em></p><p>[6] Engheim, E. (2023). <em>Julia as a Second Language: General purpose programming with a taste of data science</em>. Manning Publications.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=e2b37f7d58ee" width="1" height="1" alt=""><hr><p><a href="https://levelup.gitconnected.com/why-julias-language-is-perfect-for-scientific-computing-e2b37f7d58ee">Why Julia’s Language is Perfect for Scientific Computing?</a> was originally published in <a href="https://levelup.gitconnected.com/">Level Up Coding</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Machine Learning: How to Easily Understand Multiple Linear Regression]]></title>
            <link>https://medium.datadriveninvestor.com/machine-learning-how-to-easily-understand-multiple-linear-regression-4b2ac5f161d4?source=rss-9f804bd8d19f------2</link>
            <guid isPermaLink="false">https://medium.com/p/4b2ac5f161d4</guid>
            <category><![CDATA[python]]></category>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[linear-regression]]></category>
            <category><![CDATA[statistics]]></category>
            <category><![CDATA[data-science]]></category>
            <dc:creator><![CDATA[Dede Kurniawan]]></dc:creator>
            <pubDate>Sun, 18 Jun 2023 03:45:17 GMT</pubDate>
            <atom:updated>2023-06-22T07:04:06.863Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*cfvVVL6qoohVwJRwCB3FIw.jpeg" /><figcaption>Photo by Pixabay from <a href="https://www.pexels.com/photo/abstract-accuracy-accurate-aim-262438/">Pexels</a></figcaption></figure><h4>Get started with multiple linear regression and how to implement it using Python</h4><p>In a previous article, I explained simple linear regression. We can use simple linear regression to model the linear relationship between one dependent variable (y) and independent variables (x). On the other hand, modeling the relationship of the dependent variable (y) to two or more independent variables (x) is called <strong>multiple linear regression</strong>. Multiple linear regression allows us to <strong>predict the value of the dependent variable based on several independent variables</strong>. In most prediction problems, multiple linear regression performs better than simple linear regression.</p><p>In this article, I will explain multiple linear regression and how to implement it using Python. To get a better understanding, I suggest you read my previous <a href="https://pub.towardsai.net/machine-learning-the-first-step-is-to-understand-simple-linear-regression-30dde8b4739">article</a>.</p><h3>How multiple linear regression really works?</h3><p>In simple linear regression, we can predict the dependent variable based on one independent variable (feature). For example, we want to predict the price of a laptop based on one feature of its processor speed. We might think that many other features can help us predict the price of a laptop. Imagine we have extra features such as screen size, RAM, and weight. Again, we have to predict the price of the laptop based on these features.</p><figure><img alt="dummy dataset" src="https://cdn-images-1.medium.com/max/794/1*9q-ULPoE3e8A5ig8EZyBsw.png" /><figcaption>Illustration of a dataset containing various features such as processor speed, screen, RAM, and laptop weight</figcaption></figure><p>How does the multiple linear regression model solve this problem? The answer is by <strong>adding weights to each of the features</strong>. If the only feature is the processor speed, our model will predict the laptop price as the feature value multiplied by the appropriate weight and added with a bias. For models that have many features, <strong>we only need to multiply each feature by its corresponding weight</strong> to get more accurate prediction results. Therefore, the model equation for predicting laptop price looks like this:</p><blockquote><strong>price </strong>= 100 + 50(processor speed) + 20(screen) + 15(ram) — 15(weight of laptop)</blockquote><p>In the equation, why are there positive weights and negative weights? This is because the features with positive weights (processor speed, screen size, RAM) are positively correlated with the price of the laptop. In other words, the price of a laptop gets higher when the processor speed gets higher and the screen size gets bigger and the memory capacity gets bigger. Meanwhile, a feature with a negative weight (weight of laptop) will make the price of the laptop decrease as the value of the feature increases.</p><p>If <strong>the weight of a feature is higher</strong> (either positive or negative), we can assume that <strong>the feature is very important</strong> in affecting the price of the laptop. However, what if the weight is 0? <strong>A feature that has a weight of 0 indicates that the feature is not relevant</strong> to the price of the laptop [1].</p><p>What we did above is a simple description of how multiple linear regression works. In general, multiple linear regression has the following equation:</p><figure><img alt="Multiple linear regression equation" src="https://cdn-images-1.medium.com/max/1024/1*upzPlviEOkhxIsICUsc1-g.png" /><figcaption>Multiple linear regression equation (Image by author)</figcaption></figure><p>In this equation:</p><ul><li><strong>y</strong> is the predicted value or dependent variable</li><li><strong>x</strong> is the features or independent variable</li><li><strong>α</strong> is the bias</li><li><strong>β</strong> is the weight of each feature</li></ul><p>The multiple linear regression equation is actually the same as the simple linear regression equation. It’s just that multiple linear regression uses weight for each feature.</p><h3>Important note!</h3><p>Base &amp; Wolf (2014) [2] explain that there are several important assumptions in performing regression analysis. Some of the assumptions that you must confirm are as follows:</p><ul><li><strong>Linearity</strong>: The relationship between the dependent variable and the independent variable must be linear. That is, each unit change in the value of the independent variable leads to the same change in the dependent variable.</li></ul><figure><img alt="Relationship between variables" src="https://cdn-images-1.medium.com/max/1024/1*MekOTIQnA610upa_K8g_yQ.png" /><figcaption>Relationship between variables: linear (positively correlated), linear (negatively correlated), non-linear relationship (Image by author)</figcaption></figure><ul><li><strong>Homoscedasticity</strong>: This assumption is related to the distribution of errors/residuals from the model. Homoscedasticity means that a constant (equal) variance of the errors should be maintained.</li></ul><figure><img alt="Homoscedasticity vs. heteroscedasticity" src="https://cdn-images-1.medium.com/max/520/1*L2A6pGA6Fdy5xrLNAATkQQ.png" /><figcaption>Homoscedasticity vs. heteroscedasticity (Image by author)</figcaption></figure><ul><li><strong>Multivariate Normality</strong>: The standard regression model assumes that the residuals/errors should be normally distributed. This means that the residual values should be centered around a mean value of zero.</li><li><strong>Absence of Multicollinearity</strong>: This implies that there is no multicollinearity in the data. Multicollinearity refers to a situation where one independent variable has a strong correlation with another independent variable.</li></ul><p>Also, you should make sure not to use too many features. Using too many features will only cause the regression model to be inaccurate. Moreover, the features have no significant correlation or effect on the dependent variable. However, choosing appropriate features for the model can result in a satisfactory level of performance [3].</p><h3>How to perform multiple linear regression using Python?</h3><p>After understanding multiple linear regression, the next step is to apply it. Here I will use the California housing price dataset. You can download the data set on Kaggle.</p><h4>Import necessary library</h4><p>There are several commonly used libraries such as Numpy, Pandas, Matplotlib, Seaborn, and Scikit-learn.</p><pre>import pandas as pd<br>import numpy as np<br>import matplotlib.pyplot as plt<br>import seaborn as sns<br>from sklearn.model_selection import train_test_split<br>from sklearn.linear_model import LinearRegression<br>from sklearn.metrics import r2_score</pre><ul><li><strong>Pandas </strong>is used to manipulate data frame.</li><li><strong>Matplotlib </strong>and <strong>Seaborn </strong>are used to create data visualizations</li><li><strong>Scikit-learn</strong> is used to build and evaluate models</li></ul><h4>Load and read the data</h4><p>After downloading the data set, the next step is to load it using the pd.read_csv() function. We will also see an overview of the features of the dataset.</p><pre>df = pd.read_csv(&quot;housing.csv&quot;)<br>df.head()</pre><figure><img alt="Output" src="https://cdn-images-1.medium.com/max/1024/1*Ph02C1xCsPxTnVhOnLBPRA.png" /><figcaption>Output</figcaption></figure><p>Here’s information about each of these features:</p><ol><li>longitude: A measure of how far west a house is. A higher value means it’s further west</li><li>latitude: A measure of how far north a house is. Higher values mean further north</li><li>housing_median_age: The average age of houses in a block. Lower numbers indicate newer buildings</li><li>total_rooms: The total number of rooms in a block</li><li>total_bedrooms: Total number of bedrooms in a block</li><li>population: Total number of people living in a block</li><li>households: Total number of households or groups of people living in a housing unit</li><li>median_income: The average income of households within a housing block (measured in tens of thousands of US Dollars)</li><li>median_house_value: Average house value for households within a block (measured in US Dollars)</li><li>ocean_proximity: Location of the house to the ocean/sea</li></ol><pre>df.info()</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/644/1*7Ne1OYvUIv2sYalFm_shJA.png" /><figcaption>Output</figcaption></figure><p>So, our dataset has a total of 20640 rows and 10 columns (9 features are input features and the median_house_value is the dependent variable). All the features are of numeric type, except for the ocean_proximity feature which is of object type.</p><h4>Data exploratory</h4><p>We will dig more into the data set. First, we will examine the features that are of category type (ocean_proximity).</p><pre>sns.countplot(data=df, x=&quot;ocean_proximity&quot;)</pre><figure><img alt="Output" src="https://cdn-images-1.medium.com/max/633/1*VKGW3y5O2WllXreQ1rOOFQ.png" /><figcaption>Output</figcaption></figure><p>This feature has 5 unique values and the values are not balanced. Since this feature is not a prediction target, we do not need to perform oversampling or undersampling to balance the values. Next, we will check the correlation between features.</p><pre>corr = df.corr()<br>corr[&quot;median_house_value&quot;].sort_values(ascending=False).to_frame()</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/636/1*7niKeXu97DQrrcaJKrdtmw.png" /><figcaption>Output</figcaption></figure><p>An interesting insight here is that average income is highly correlated to house prices. This makes sense because people will generally buy a house that matches their average income.</p><p>Since we have the longitude and latitude features, it is easier to analyze if we visualize them.</p><pre>df.plot(kind=&quot;scatter&quot;, x=&quot;longitude&quot;, y=&quot;latitude&quot;, alpha=0.5,<br>        s= df[&quot;population&quot;]/100, label=&quot;population&quot;, figsize=(10,7),<br>        c=&quot;median_house_value&quot;, cmap=plt.get_cmap(&quot;jet&quot;), colorbar=True)<br>plt.legend()</pre><figure><img alt="output" src="https://cdn-images-1.medium.com/max/908/1*dRMIQ2jXBVKkKjGrfN-Mrw.png" /><figcaption>output</figcaption></figure><p>The visualization helps us know the location of a particular house. We can see that the price of a house is higher the closer it is to the beach. To compare, you can also look at the California map.</p><p>We also have to check whether the data has missing values or not.</p><pre>df.isna().sum().to_frame()</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/660/1*pTjdtOLYhwTl8S1I3dvlTA.png" /><figcaption>Output</figcaption></figure><p>The feature total_bedrooms has a total of 207 missing values or about 1% of the total rows. since this is a small number, we can drop the rows with missing values. Next, we have to check for outlier conditions in the data set.</p><pre>num_col = [&#39;longitude&#39;, &#39;latitude&#39;, &#39;housing_median_age&#39;, &#39;total_rooms&#39;,<br>           &#39;total_bedrooms&#39;, &#39;population&#39;, &#39;households&#39;, &#39;median_income&#39;,<br>           &#39;median_house_value&#39;]<br><br># visualize the outlier<br>plt.figure(figsize=(15, 10))<br>n = 1<br>for col in num_col:<br>    plt.subplot(4, 3, n)<br>    sns.boxplot(data=df, x=col)<br>    plt.title(&quot;Outlier in &quot; + col, fontsize=16)<br>    n += 1<br>plt.tight_layout()</pre><figure><img alt="Outlier analysis" src="https://cdn-images-1.medium.com/max/1024/1*Wd2SQqWJ0SM9D2JU3BDLwQ.png" /><figcaption>output</figcaption></figure><p>Wow! Our data set has a lot of outliers. Outliers can cause the model to be inaccurate. We need to deal with this problem.</p><h4>Data preprocessing</h4><p>After knowing that the missing values in the data are small, we can drop them.</p><pre>df.dropna(inplace=True)</pre><p>The machine can only read data of type category. Feature ocean_proximity has a value of type category. We will convert it to numeric type using the one-hot encoding method.</p><pre>cat = pd.get_dummies(df[&quot;ocean_proximity&quot;])<br>df = pd.concat([df, cat], axis=1)<br>df.drop(&quot;ocean_proximity&quot;, axis=1, inplace=True)</pre><p>After that, we will separate the dataset into training sets and test sets. We will use the training sets to train the model, while we will use the test sets to evaluate the model.</p><pre>X = df.drop(&quot;median_house_value&quot;, axis=1)<br>y = df[&quot;median_house_value&quot;]<br>X_train, X_test, y_train, y_test = train_test_split(X, y, <br>                                                    test_size = 0.25, <br>                                                    random_state = 0)</pre><p>linear regression models perform poorly against unscaled data. We will scale the data using the RobustScaler method. This is because we also want to reduce the effect of outliers.</p><pre>scaler_X = RobustScaler()<br>X_train_scaled = scaler_X.fit_transform(X_train)<br>X_test_scaled = scaler_X.transform(X_test)</pre><h4>Modeling and evaluation</h4><p>We have analyzed and processed the data until it is ready to be fed to the model. We will train the regression model on the scaled data.</p><pre>model = LinearRegression()<br>model.fit(X_train_scaled, y_train)</pre><p>We can also see the weight of each feature using model.coef_. To see the bias, we can use model.intercept_.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/717/1*SJK69aQLfw0is_x11kJiew.png" /></figure><p>Each coefficient corresponds to each feature and the bias is added as a constant. This is what makes up a model. Then the linear regression model equation can be obtained as follows:</p><blockquote><strong>model </strong>= bias + coef₁*feature₁ + coef₂*feature₂ + … + coefₙ*featureₙ</blockquote><p>We can substitute the weight and bias values into the regression equation to make predictions.</p><blockquote><strong>model </strong>= 245227.1868829204 -104584.66322722(longitude) -99374.66096139(latitude) + … -31674.11792617(near ocean)</blockquote><p>To see the performance of the regression model, we can evaluate it using the R-squared (R²) metric. The R² value shows how well the model fits the data. The value is in the range of 0 to 1, the larger the value the better the model’s performance.</p><pre>y_pred = model.predict(X_test_scaled)<br>r_squared = r2_score(y_test, y_pred)<br>print(f&#39;R-squared: {r_squared}&#39;)<br><br># output<br>R-squared: 0.6472382266013552</pre><p>We get an R² value of about 65% which means that 65% of the variance in the dependent variable can be explained by the independent variable/feature. This shows that our linear regression model does not generalize well to the data. In other words, our model is underfitting. This can be due to the following two reasons:</p><ul><li>The features in the dataset have a poor correlation to the dependent variable.</li><li>The model used is too simple.</li></ul><h3>Summary</h3><p>Multiple linear regression finds the linear relationship between a dependent variable (y) and multiple independent variables (x). It works the same as simple linear regression, except that multiple linear regression has weights for each feature. In most prediction cases, multiple linear regression performs better than simple linear regression. To support the success of linear regression analysis, it is necessary to observe several assumptions such as linearity, homoscedasticity, multivariate normality, and the absence of multicollinearity. One commonly used metric to measure the fit of the model to the data is R-squared (R²).</p><h4>References:</h4><p>[1] Serrano, L. (2021). <em>Grokking Machine Learning</em>. Manning Publications.</p><p>[2] Best, H. &amp; Wolf, C. (2014). <em>The SAGE Handbook of Regression Analysis and Causal Inference</em>. SAGE Publications Ltd.</p><p>[3] Ilseven, E., &amp; Gol, M. (2019). <em>A Comparative Study on Feature Selection Based Improvement of Medium-Term Demand Forecast Accuracy. 2019 IEEE Milan PowerTech.</em></p><p>Subscribe to DDIntel <a href="https://ddintel.datadriveninvestor.com/">Here</a>.</p><p>Visit our website here: <a href="https://www.datadriveninvestor.com/">https://www.datadriveninvestor.com</a></p><p>Join our network here: <a href="https://datadriveninvestor.com/collaborate">https://datadriveninvestor.com/collaborate</a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=4b2ac5f161d4" width="1" height="1" alt=""><hr><p><a href="https://medium.datadriveninvestor.com/machine-learning-how-to-easily-understand-multiple-linear-regression-4b2ac5f161d4">Machine Learning: How to Easily Understand Multiple Linear Regression</a> was originally published in <a href="https://medium.datadriveninvestor.com/">DataDrivenInvestor</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Machine Learning: The First Step is to Understand Simple Linear Regression]]></title>
            <link>https://pub.towardsai.net/machine-learning-the-first-step-is-to-understand-simple-linear-regression-30dde8b4739?source=rss-9f804bd8d19f------2</link>
            <guid isPermaLink="false">https://medium.com/p/30dde8b4739</guid>
            <category><![CDATA[beginner]]></category>
            <category><![CDATA[python]]></category>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[linear-regression]]></category>
            <category><![CDATA[data-science]]></category>
            <dc:creator><![CDATA[Dede Kurniawan]]></dc:creator>
            <pubDate>Wed, 07 Jun 2023 08:40:36 GMT</pubDate>
            <atom:updated>2023-06-09T08:59:59.643Z</atom:updated>
            <content:encoded><![CDATA[<h4>Mastering Simple Linear Regression in Python without a Lot of Math</h4><figure><img alt="Header image" src="https://cdn-images-1.medium.com/max/1024/1*mNu6VmuoONq9HEOk-jCwmg.jpeg" /><figcaption>Photo by Andrea Piacquadio on <a href="https://www.pexels.com/photo/woman-in-red-long-sleeve-writing-on-chalk-board-3769714/">Pexels</a></figcaption></figure><p>In the machine learning universe, who doesn’t know about linear regression? Of course, everyone knows about it. Linear regression is the most fundamental and widely applied concept in daily life. Simple linear regression is used to find the correlation between two variables (explanatory and response variables). In addition, linear regression aims to predict the value of the response variable from the value of the explanatory variable that has never been observed before [1].</p><p>I will explain simple linear regression using a little math in this article. The goal is that you can understand how it works and how to apply it using Python.</p><h3>Get an intuition of simple linear regression</h3><p>Linear regression works by <strong>drawing a line that is as close as possible to the existing data points</strong> [2]. The data points are the values of the explanatory and response variables. Simple linear regression <strong>assumes that the two explanatory and response variables are linearly correlated</strong> so that a straight line can be drawn.</p><figure><img alt="Illustration of a linear regression model that draws a straight line between data points" src="https://cdn-images-1.medium.com/max/1024/1*2s6eG20EfKEyu1flqvWUqA.png" /><figcaption>Illustration of a linear regression model that draws a straight line between data points (Image by author)</figcaption></figure><p>How do we get the regression line? <strong>1) </strong>The first step is to randomly select a straight line. <strong>2)</strong> Next, randomly select data points and adjust the line as close as possible to each data point. <strong>3)</strong> Repeat the second step several times until you get a line that is as close as possible to all data points [2].</p><figure><img alt="Illustration of a linear regression algorithm to determine the best line" src="https://cdn-images-1.medium.com/max/721/1*BM-cehBL5OiBeTZJ3Y20ww.png" /><figcaption>Illustration of a linear regression algorithm to determine the best line that is as close as possible to all data points [2]</figcaption></figure><p>Let’s imagine we have a collection of student data containing the study duration and the score obtained during the exam. We are asked to predict the students’ scores from their study duration. Let’s try to apply simple linear regression thinking to this problem.</p><figure><img alt="dummy dataset" src="https://cdn-images-1.medium.com/max/588/1*iIyMsFJllBhOPsJem3Foig.png" /><figcaption>Student score dataset</figcaption></figure><p>Based on the information from the data, what is the score if a student studies for 7 hours? The answer is 95. You may realize that t<strong>he longer the study duration, the better score they will get</strong>. From the pattern of the data, we can conclude that for every 1 hour of study duration, they get a score of +5. We can think that the score is obtained from a combination of two things: the score obtained by students with &lt;1 hour of study duration and the score obtained by students while studying. Let’s simplify it using a formula:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/541/1*222vjO_2WxfHMFl0bzjuVQ.png" /></figure><p>What we have here is a formula to predict the score that students get from their study duration. Using the formula, we can predict that a student who studies for 7 hours will most likely get a score of 95.</p><figure><img alt="Illustration of predicting students’ scores from their study duration using the concept of simple linear regression" src="https://cdn-images-1.medium.com/max/1024/1*NOL4tXdwvMbf2VzrjjcULQ.png" /><figcaption>Illustration of predicting students’ scores from their study duration using the concept of simple linear regression (Image by author)</figcaption></figure><p>The formula we get is the same as the formula in simple linear regression. In simple linear regression, the formula is like this:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/279/1*DjAiIDd-BR3lZLMiloCmNw.png" /></figure><p>Yes, the simple linear regression formula has actually been learned in middle school. Back then, we probably only used it to find the gradient of a straight line. In simple linear regression, the formula means the following:</p><ul><li><strong>y</strong> → the response/target variable that we are trying to predict from the explanatory variables. In the above problem, this is the “score”.</li><li><strong>x</strong> → the explanatory variable we are using to make a prediction. In the problem above, this is “hours”.</li><li><strong>m</strong> → Slope is the steepness of the regression line that tells us how much we expect the y value to increase when we increase the x value by one unit. In machine learning, the slope is also known as <strong>weight</strong>.</li><li><strong>c</strong> → The intercept tells us where the regression line crosses the vertical y-axis. In machine learning, the intercept is also known as <strong>bias</strong>.</li></ul><figure><img alt="Simple linear regression anatomy" src="https://cdn-images-1.medium.com/max/1024/1*UgSLpWRxyvkbEd-eQphP7g.png" /><figcaption>Simple linear regression anatomy (Image by author)</figcaption></figure><p>Next, how do we know how well our regression line fits the data points? The answer is to <strong>minimize the error in the model prediction to find the best line</strong>. Defining and measuring the error of a model is called <strong>loss function</strong> or <strong>cost function</strong> sometimes also called <strong>error function</strong>. The difference between the predicted value and the observed value is called the <strong>residual (error)</strong>. The measure of the fit of a model to predict all values of the response variable close to the observed values is called the <strong>residual sum of squares (RSS/SSres)</strong> [1].</p><figure><img alt="Residual vs. actual value vs. predicted value in simple linear regression" src="https://cdn-images-1.medium.com/max/501/1*lDsnFbFcEgh3eUxKbXKBoA.png" /><figcaption>Residual vs. actual value vs. predicted value in simple linear regression (Image by author)</figcaption></figure><h3>Perform simple linear regression using Python</h3><p>After getting an intuition about the concept of simple linear regression, we will proceed by performing simple linear regression using Python. You can download the dataset that we will use on <a href="https://www.kaggle.com/datasets/shubham47/students-score-dataset-linear-regression">Kaggle</a>. There are some libraries that we need to import first that can make it easier for us when building a simple regression model.</p><pre>import numpy as np<br>import pandas as pd<br>import matplotlib.pyplot as plt</pre><p>Next, we can read and load the dataset using Pandas.</p><pre># read and load the data<br>df = pd.read_csv(&quot;score.csv&quot;)<br>df.head()</pre><figure><img alt="Output" src="https://cdn-images-1.medium.com/max/646/1*blnOQkQDZ0wb-g9fpjuBlg.png" /><figcaption>Output</figcaption></figure><pre># get information about the data<br>df.info()<br><br># output<br>&lt;class &#39;pandas.core.frame.DataFrame&#39;&gt;<br>RangeIndex: 25 entries, 0 to 24<br>Data columns (total 2 columns):<br> #   Column  Non-Null Count  Dtype  <br>---  ------  --------------  -----  <br> 0   Hours   25 non-null     float64<br> 1   Scores  25 non-null     int64  <br>dtypes: float64(1), int64(1)<br>memory usage: 528.0 bytes</pre><pre># data preparation<br>from sklearn.model_selection import train_test_split<br><br>X = df.iloc[:,:-1].values<br>y = df.iloc[:, 1:].values<br>X_train, X_test, y_train, y_test = train_test_split(X, <br>                                                    y, <br>                                                    test_size=0.3,<br>                                                    random_state=99)</pre><p>The data set has two columns, “Hours” and “Scores”. The “Hours” column will be the explanatory variable, while the “Scores” column will be the response variable. In addition, the dataset only consists of 25 rows and no missing values. We also need to separate the data set into training and test data. The training data is used to build a simple regression model, while the test data is used to evaluate how good our model is. Now is the time to build the regression model!</p><pre># build regression model<br>from sklearn.linear_model import LinearRegression<br><br>model_reg = LinearRegression()<br>model_reg.fit(X_train, y_train)</pre><p>To build and train a simple regression model, we can use Scikit-learn. To see the fit of the regression model to the data set, we can visualize it using Matplotlib.</p><pre># visualize the regression model<br>X_vis = np.array([0, 10]).reshape(-1, 1)<br>y_vis = model_reg.predict(X_vis)<br>plt.scatter(X_train, y_train, label=&quot;Actual values&quot;)<br>plt.plot(X_vis, y_vis, &quot;-r&quot;, label=&quot;Regression line&quot;)<br>plt.title(&quot;Regression analysis&quot;)<br>plt.ylabel(&quot;Score&quot;)<br>plt.xlabel(&quot;Hours&quot;)<br>plt.xlim(0, 10)<br>plt.ylim(0, 100)<br>plt.legend()</pre><figure><img alt="Simple linear regression model" src="https://cdn-images-1.medium.com/max/647/1*_CQsUVnfhkv4SQzS4FTHDw.png" /><figcaption>Simple linear regression model (Output)</figcaption></figure><p>The simple regression model seems to be pretty good at drawing a line linearly between a set of data points. As we discussed above, the linear regression formula is as follows:</p><figure><img alt="Simple linear regression formula" src="https://cdn-images-1.medium.com/max/556/1*aFsN-MsbqYtyi3boAoc47A.png" /><figcaption>Simple linear regression formula (Image by author)</figcaption></figure><p>To know the value of the regression equation produced by the model, we must display the intercept and slope values.</p><pre># input<br>print(f&#39;intercept: {model_reg.intercept_}&#39;)<br>print(f&#39;slope: {model_reg.coef_}&#39;)<br><br># output<br>intercept: [1.91863322]<br>slope: [[9.82609393]]</pre><p>From the intercept and slope values, we can know that the regression equation produced by the model is y = 9.82609393X + 1.91863322. To confirm the results of the slope and intercept values from Scikit-learn, we can also calculate these values using the equation.</p><figure><img alt="Slope and intercept formula" src="https://cdn-images-1.medium.com/max/497/1*9ydg09mgVNySNy3frtLkew.png" /><figcaption>Slope and intercept formula (Image by author)</figcaption></figure><pre># calcullate the slope value<br>cov_xy = np.cov(X_train.flatten(), y_train.flatten())<br>variance_x = np.var(X_train, ddof=1)<br>slope = cov_xy[0][1] / variance_x<br><br># calcullate the intercept value<br>intercept = np.mean(y_train) - slope * np.mean(X_train)<br><br>print(f&quot;slope: {slope}&quot;)<br>print(f&quot;intercept: {intercept}&quot;)<br>print(f&quot;regression formula: y = {slope} X + {intercept}&quot;)</pre><figure><img alt="Output" src="https://cdn-images-1.medium.com/max/639/1*wlKCprLzpWDp-CpUXm7cMQ.png" /><figcaption>Output</figcaption></figure><p>It turns out that the intercept and slope values we calculated are the same as the values from Scikit-learn. Then, we also have to evaluate how good the regression model is. One of the standard metrics for evaluating linear regression models is <strong>R-squared (R²) </strong>[3].</p><pre># evaluate the regression model<br>from sklearn.metrics import r2_score<br><br>y_pred = model_reg.predict(X_test)<br>r_squared = r2_score(y_test, y_pred)<br>print(f&#39;R-squared: {r_squared}&#39;)<br><br># ouput<br>R-squared: 0.9230325919412203</pre><p>The R² value of 0.9230325919412203 indicates that the regression model we have built is quite good. The R² value is in the range of 0 to 1, the accuracy of the regression model is getting better if the R² value is close to 1 [4]. To ensure the R² value generated by Scikit-learn, we can calculate it using the R² formula.</p><figure><img alt="R-squared formula" src="https://cdn-images-1.medium.com/max/1024/1*SscYjDKao_T4H6JuS5CYwA.png" /><figcaption>R-squared formula (Image by author)</figcaption></figure><pre># calcullate the r-aquared value<br>ss_res = sum([(y_i - model_reg.predict(x_i.reshape(-1, 1))[0])**2<br>              for x_i, y_i in zip(X_test, y_test)])<br>ss_tot = sum([(y_i - np.mean(y_test))**2 for y_i in y_test])<br>r_squared = 1 - (ss_res / ss_tot)<br><br>print(f&#39;R-squared: {ss_res} / {ss_tot} = {r_squared}&#39;)<br><br># output<br>R-squared: [250.45194582] / [3254.] = [0.92303259]</pre><p>Our results show the same R² value as the calculation from Scikit-learn. We can also use other evaluation metrics such as MAE, MSE, and MAPE to check how well our regression model is.</p><h3>Conclusion</h3><p>Simple linear regression is the <strong>most fundamental concept</strong> of machine learning. It works by <strong>randomly initiating a line</strong>, <strong>fitting the line</strong> to each data point, and <strong>obtaining a line</strong> that is as close as possible to the entire data point. Besides being used to determine the <strong>correlation</strong> between data, simple linear regression can also be used to <strong>make predictions</strong>. A commonly used metric to evaluate how good a linear regression model is is <strong>R-squared (R²)</strong>.</p><h4>References:</h4><p>[1] Hackeling, G. (2017). <em>Mastering Machine Learning with Scikit-Learn</em>. Packt Publishing.</p><p>[2] Serrano, L. (2021). <em>Grokking Machine Learning</em>. Manning Publications.</p><p>[3] Chicco, D., Warrens, M., &amp; Jurman, G. (2021). <em>The coefficient of determination R-squared is more informative than SMAPE, MAE, MAPE, MSE and RMSE in regression analysis evaluation</em>. PeerJ Computer Science.</p><p>[4] Al-Mosawe, A., Kalfat, R., &amp; Al-Mahaidi, R. (2017). <em>Strength of Cfrp-steel double strap joints under impact loads using genetic programming</em>. Composite Structures.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=30dde8b4739" width="1" height="1" alt=""><hr><p><a href="https://pub.towardsai.net/machine-learning-the-first-step-is-to-understand-simple-linear-regression-30dde8b4739">Machine Learning: The First Step is to Understand Simple Linear Regression</a> was originally published in <a href="https://pub.towardsai.net/">Towards AI</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[How to Improve Our Data Preprocessing Skills with this Several Project]]></title>
            <link>https://levelup.gitconnected.com/how-to-improve-our-data-preprocessing-skills-with-this-several-project-607c6dc9ee56?source=rss-9f804bd8d19f------2</link>
            <guid isPermaLink="false">https://medium.com/p/607c6dc9ee56</guid>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[data-pre-processing]]></category>
            <category><![CDATA[data]]></category>
            <category><![CDATA[projects]]></category>
            <category><![CDATA[machine-learning]]></category>
            <dc:creator><![CDATA[Dede Kurniawan]]></dc:creator>
            <pubDate>Sat, 20 May 2023 08:59:12 GMT</pubDate>
            <atom:updated>2023-05-21T18:17:54.505Z</atom:updated>
            <content:encoded><![CDATA[<h4>Perform data processing using several datasets can improve our skills</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*-zWs0L4M9mt1g2_aRgvKeA.jpeg" /><figcaption>Image by rawpixel.com on <a href="https://www.freepik.com/free-photo/man-giving-business-presentation-using-futuristic-digital-pen_16015122.htm#query=improve&amp;position=1&amp;from_view=search&amp;track=sph">Freepik</a></figcaption></figure><p>Working on various projects can improve our data science skills. We should often work on multiple data processing projects to improve our data processing skills. You don’t have to work on projects with a high level of complexity, simple tasks with a few repetitions are enough. The key is to practice a lot until it becomes a <strong>habit</strong>.</p><blockquote><em>“Success is the product of daily habits, not once in a lifetime transformations.” </em>— James Clear</blockquote><p>In this article, I will share a data processing project using various datasets. I will also explain each step that I have gone through in simple terms to make it easier to understand.</p><h4>Prerequisites</h4><p>There are four libraries that I have used while working on this project. These libraries are <strong>Pandas, Matplotlib, Seaborn, </strong>and <strong>Scikit-learn</strong>. Pandas is used to read and manipulate data, Scikit learn is used to standardize and deal with data imbalance, Matplotlib and Seaborn are used to visualize data.</p><pre>import pandas as pd<br>import matplotlib.pyplot as plt<br>import seaborn as sns</pre><p>I also recommend you read this <a href="https://levelup.gitconnected.com/mastering-data-preprocessing-and-preparation-to-improve-data-quality-4bc9ec6746b3">article</a> first to improve your understanding of Data Preprocessing. Let’s practice!</p><h3>1. Preprocessing the Titanic Dataset</h3><p>Anyone just starting out in machine learning must be familiar with the <a href="https://www.kaggle.com/competitions/titanic">Titanic dataset</a>. It contains passenger data from the world-famous cruise ship disaster that sank in the North Atlantic. Until now, this dataset is still often used for education purposes. You can download this dataset for free on <a href="https://www.kaggle.com/">Kaggle</a>.</p><p>After successfully downloading the data, we can directly load and read the data using Pandas.</p><pre>df = pd.read_csv(&quot;Titanic.csv&quot;)<br>df.head()</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*--foIHt9En1JkJwThkPQeA.png" /><figcaption>Output</figcaption></figure><p>The dataset contains 12 features, such as passenger ID, passenger name, age, gender, and whether they survived or not. Here’s a more detailed description of the 12 features:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/944/1*9eZoGH2jBCD-YrFOT3fVVQ.png" /><figcaption>Description of each feature in the Titanic dataset (Image by author)</figcaption></figure><p>To check information such as the number of rows and columns, the data type of each column, and the data size, we can use df.info().</p><pre>df.info()</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/652/1*78WDP0bfxEI1FnpB9A9NUg.png" /><figcaption>Output</figcaption></figure><p>From the information, we can see that our dataset consists of 12 columns and a total of 891 rows. Our data has some missing values such as in the columns “Age”, “Cabin”, and “Embarked”. For that, we need to determine the exact number of missing values in each of these columns.</p><pre>df.isna().sum().to_frame()</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/674/1*1qfwGwR-19Bcz98XacqYNA.png" /><figcaption>Output</figcaption></figure><p>It can be confirmed that the “Age” column has 177 missing values out of a total of 891 rows (about 19%). Meanwhile, the “Cabin” column has more missing values (687 missing values or about 77%) and the “Embarked” column has only a few missing values (2 rows).</p><ul><li>The best approach we can take to handle missing values in the “Cabin” column is to delete the column. This is a common approach when the column is highly uninformative [1].</li><li>The “Age” column holds information that is important for machine learning model analysis. We will lose a lot of information if we remove rows that have missing values in this column, so the best approach to handle this column is to perform imputation (e.g., mean imputation). Data imputation is a common way to deal with missing values where the missing value’s substitutes are discovered through statistical or machine learning techniques [2].</li><li>Meanwhile, the “Embarked” column only has a few missing values. The best approach to deal with it is to remove the rows that have missing values in this column.</li></ul><pre># handling missing value<br>median_age = df[&quot;Age&quot;].median()<br>df[&quot;Age&quot;].fillna(median_age, inplace=True)<br>df.drop(columns=&quot;Cabin&quot;, inplace=True) <br>df.dropna(axis=0, inplace=True) <br><br># check whether there are still missing values<br>df.isna().sum()</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/645/1*bDKM0CFVRF5ho7AbB44_iA.png" /><figcaption>Output</figcaption></figure><p>Most machine learning algorithms work well on numeric features. Some columns like “Sex”, and “Embarked” have categorical values. Therefore, we have to convert these category values into numeric features.</p><ul><li>The best approach for the “Sex” column is to do label encoding. Label encoding does its job well when the feature values are unordered categorical data. In some cases, label encoding can have the best performance compared to one-hot encoding [3]. Therefore, we will change the category value of “male” to 0 and “female” to 1.</li><li>The “Embarked” column has 3 different category value classes, including “C”, “Q”, and “S”. The best approach we can take with this column is to perform one hot encoding. This is because one-hot encoding is suitable when we have feature values of more than two categories and does not assume any sorting relationship between categories.</li></ul><pre># handling categorical data<br>df.replace([&quot;male&quot;, &quot;female&quot;], [0, 1], inplace=True)<br>embarked_col = pd.get_dummies(df[&quot;Embarked&quot;], prefix=&quot;Embarked&quot;)<br>df = pd.concat([df, embarked_col], axis=1)<br>df.drop(&quot;Embarked&quot;, axis=1, inplace=True)</pre><p>To increase the number of features in the dataset, we can make new features. The columns “Sibsp” and “Parch” can show information about the number of families. Therefore, we can create a new column (“Family_size”) from the combination of these two columns. This process is also called feature engineering.</p><pre># create a new feature<br>df[&quot;Family_size&quot;] = df[&#39;SibSp&#39;] + df[&quot;Parch&quot;] + 1</pre><p>The last step in data processing is to <strong>drop unnecessary features</strong>. Dropping irrelevant features can result in a simpler model that is easier to interpret and usually gives better performance [4]. Features such as “PassengerId”, “Name”, and “Ticket” have unique values for each passenger. We can remove these because they do not have a significant impact on the machine-learning model later.</p><pre># drop unecessary feature<br>df.drop([&quot;PassengerId&quot;, &quot;Name&quot;, &quot;Ticket&quot;], axis=1, inplace=True)</pre><p>At this point, we have finished processing our data. We can take a look at our data by using the df.head() function.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/860/1*QwvMxYBnGKrrQUkXSewWzQ.png" /><figcaption>Output</figcaption></figure><p>The data is ready to be analyzed using machine learning algorithms. Note that data pre-processing is an iterative process, which means that we can repeat this process if we feel that the data is still not suitable for the machine learning algorithm being used.</p><h3>2. Preprocessing the Red Wine Quality Dataset</h3><p>The Wine quality dataset is one of the most popular datasets among machine learning practitioners. This dataset was created by Cortez et al. (2009) which contains physicochemical features of a wine that can describe its quality [5]. You can download this dataset on the <a href="https://archive.ics.uci.edu/ml/datasets/wine+quality">UCI Machine Learning Repository</a> site.</p><pre>df = pd.read_csv(&quot;winequality-red.csv&quot;, sep=&quot;;&quot;)<br>df.head()</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*UimoEjYxcWVqlHDvTSZFPA.png" /><figcaption>Output</figcaption></figure><p>By looking at the top 5 rows in the dataset, we can find out what the features are. The dataset consists of 12 features, such as “fixed acidity”, “pH”, “alcohol”, and the “quality” level of the wine. Here is more detailed information about these features:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/943/1*1UhuhAmHK1conjmOxSGpRQ.png" /><figcaption>Description of each feature in the Red wine quality dataset (Image by author)</figcaption></figure><pre>df.info()</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/644/1*KPBv-xGxHEexyE8ghs7CDg.png" /><figcaption>Output</figcaption></figure><p>We can confirm that the dataset consists of 12 columns (features) and 1599 rows. All features are numeric, consisting of integer and float. There are no missing values for each feature in the dataset. However, we still need to make sure that there are no missing values.</p><pre>df.isna().sum().to_frame()</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/643/1*Xym_r_0d1bcfqbAZgz6A-A.png" /><figcaption>Output</figcaption></figure><p>It turns out that there is no missing value in any of the features. Next, we will see whether there are duplicated values or not.</p><pre>df.duplicated().sum()<br><br># output<br>240</pre><p>It turns out that there are 240 duplicated values (about 15% of the total rows). Our most straightforward approach in dealing with this is to drop the duplicated rows in the entire column. Duplicate data will only increase computational resources and training time when building machine learning models.</p><pre>df.drop_duplicates(inplace=True)</pre><p>Now, we have successfully filtered out the duplicated values and left 1359 rows. In this dataset, we don’t need to think about handling category data. This is because all the features in the dataset are numeric values. Next, what we need to think about is how to handle outliers.</p><p>Outliers can cause our machine-learning models to make inaccurate predictions. This is because outliers can skew the data distribution and make the model learn the wrong patterns. Removing outliers from the dataset before training another model can significantly improve the performance of the resulting model [6].</p><pre># create function<br>def visualize_outlier(df):<br>  &quot;&quot;&quot;<br>  This function will create a box plot visualization on each column<br>  &quot;&quot;&quot;<br><br>  plt.figure(figsize=(15, 10))<br>  n = 1<br>  for col in df.columns:<br>      plt.subplot(4, 3, n)<br>      sns.boxplot(data=df, x=col)<br>      plt.title(&quot;Outlier in &quot; + col, fontsize=16)<br>      n += 1<br>  plt.tight_layout()<br><br># call the function to create a box plot<br>visualize_outlier(df)</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*3tz_ES8KaguJmbOKN91YZQ.png" /><figcaption>output</figcaption></figure><p>Box plot visualization can allow us to see outliers in the data. From the visualization, we can see that almost all the features have outliers.</p><p>With a limited number of rows, we cannot just drop the outliers. We could lose a lot of information. Our approach is to replace the values detected as outliers with the median value of each column. Compared to the mean value, the median value is more resistant to outliers [7].</p><p>The column “quality” will be used for the target variable. Regarding that, we will not do anything to the outliers in this column (there are also fewer outliers in this column).</p><pre># create function<br>def handling_outlier(df, columns):<br>  &quot;&quot;&quot;&quot;<br>  This function is used to detect outliers using the 1.5 IQR method and <br>  replace the outlier value with the missing value. Then, the missing vale<br>  will be replaced with the median value in each column. <br>  &quot;&quot;&quot;<br><br>  for col in columns:<br>      q1 = df[col].quantile(0.25)<br>      q3 = df[col].quantile(0.75)<br>      iqr = q3 - q1<br>      lower_bound = q1 - (1.5 *iqr)<br>      upper_bound = q3 + (1.5 *iqr)<br>      <br>      # replace the outliers with the median of the column<br>      df.loc[(df[col] &lt; lower_bound) | (df[col] &gt; upper_bound), col] = None<br>      df[col].fillna(df[col].median(), inplace=True)<br>          <br>  return df<br><br># handling outliers using the function<br>columns = df.drop(&quot;quality&quot;, axis=1)<br>handling_outlier(df, columns)</pre><p>Now the outliers are handled successfully. To be sure, we can create a box plot visualization for the values in each column. We can use the function we created earlier.</p><pre>visualize_outlier(df)</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*WsS8U2sIcSpF1XIk82Qb0Q.png" /><figcaption>Output</figcaption></figure><p>The box plot this time is better than before the outliers were handled. Although there are still some outliers, we will standardize them later to reduce the impact of the outliers. Now, we need to handle the values in the “quality” column.</p><pre>df[&quot;quality&quot;].value_counts().to_frame()</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/710/1*04fI_X5FHb7hcy0NXAQuuQ.png" /><figcaption>Output</figcaption></figure><p>The “quality” column consists of numeric values (3 to 8) that represent the quality rating of the wine. We will transform the values in this column into binary values of 0 (bad quality) and 1 (good quality). Bad quality includes ratings less than 6.5, while good quality includes ratings greater than 6.5.</p><pre>bins = [2, 6.5, 8]<br>quality_labels = [0, 1]<br>df[&quot;quality&quot;] = pd.cut(df[&quot;quality&quot;], bins=bins, labels=quality_labels)</pre><p>We are almost at the last part of preprocessing for this dataset. Earlier I said that to reduce the impact of outliers again is to standardize. In some cases, Z-score standardization is proven to be more effective and efficient compared to min-max standardization and decimal scaling methods [8]. Before that, we will first divide the dataset into training data and test data.</p><pre>from sklearn.model_selection import train_test_split<br>from sklearn.preprocessing import StandardScale<br><br># split dataset<br>X = df.drop(&quot;quality&quot;, axis=1)<br>y = df[&quot;quality&quot;]<br>X_train, X_test, y_train, y_test = train_test_split(X, y, <br>                                                    test_size = 0.25, <br>                                                    random_state = 0)<br># perform standarization<br>scaler_X = StandardScaler()<br>X_train_scaled = scaler_X.fit_transform(X_train)<br>X_test_scaled = scaler_X.transform(X_test)</pre><p>We use the Scikit-learn library to split the dataset and standardize the features. When standardizing, <strong>we only fit the training data</strong>. This is to ensure that the model does not learn anything from the test data. We can also display the results of the standardization (e.g. in the variable X_train_scaled).</p><pre>scaled = pd.DataFrame(X_train_scaled, columns=X.columns)<br>scaled.head()</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*3R3vUQ3JH1b0tCzyk5p5EA.png" /><figcaption>Output</figcaption></figure><p>Finally, the dataset is ready to be learned by machine learning models.</p><h3>3. Preprocessing the Brain Stroke Dataset</h3><p>Finally, we will preprocess the brain stroke dataset. This dataset contains general information and medical conditions of the patient. based on this information, we can train a machine learning model to predict whether a patient is likely to have a stroke. you can download the dataset at <a href="https://www.kaggle.com/datasets/jillanisofttech/brain-stroke-dataset">kaggle.com</a>.</p><pre>df = pd.read_csv(&quot;brain_stroke_full.csv&quot;)<br>df.rename(columns={&quot;Residence_type&quot;: &quot;residence_type&quot;}, inplace=True)<br>df.head()</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*HxfqyHQ1JEv4PBwZHpwkYA.png" /><figcaption>Output</figcaption></figure><p>The dataset has 12 features, such as “gender”, “age”, and “stroke”. Here is more detailed information about these features:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/878/1*UOP00Y1XBAGKVNWCY2KS-w.png" /><figcaption>Output</figcaption></figure><pre>df.info()</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/645/1*WXmslt1jm-9-z12gVsVZGA.png" /><figcaption>Output</figcaption></figure><p>We can see that the dataset consists of 4981 rows and 11 columns. There are 5 features of the categorical type and 6 other features of the numeric type. It looks like there are no missing values in each of the features. Let’s make sure!</p><pre>df.isna().sum().to_frame()</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/644/1*cqQ0qZH-fYzC4kSYQe6QGA.png" /><figcaption>Output</figcaption></figure><p>It turns out that the dataset does not have any missing values. Next, we also have to check whether there is duplicated data or not.</p><pre>df.duplicated().sum()</pre><p>The code returns 0. It also turns out that there is no duplicated data. It can be said that the dataset is quite clean. We will handle 5 features that are of category type. These features include, “gender”, “ever_married”, “work_type”, “residence_type”, and “smoking_status”.</p><pre>cat_columns = [&quot;gender&quot;, &quot;ever_married&quot;, &quot;work_type&quot;, &quot;residence_type&quot;, &quot;smoking_status&quot;]<br><br>for col in cat_columns:<br>    print(f&quot;Unique value for {col}\t: {df[col].unique()}&quot;)</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/745/1*QWceIhemcqWqR_PWp5tuxQ.png" /><figcaption>Output</figcaption></figure><p>Some columns only have 2 unique values and some have more than 2 unique values. For columns that have 2 unique values (“gender”, “ever_married”, and “residence_type”), we will do label encoding or transform them into binary values 0 and 1. Meanwhile, we will do one-hot encoding for columns that have more than 2 unique values (“work_type” and “smoking_status”).</p><pre># label encoding<br>df[&quot;gender&quot;] = df[&quot;gender&quot;].map({&quot;Male&quot;: 1, &quot;Female&quot;: 0})<br>df[&quot;ever_married&quot;] = df[&quot;ever_married&quot;].map({&quot;Yes&quot;: 1, &quot;No&quot;: 0})<br>df[&quot;residence_type&quot;] = df[&quot;residence_type&quot;].map({&quot;Urban&quot;: 1, &quot;Rural&quot;: 0})<br><br># one hot encoding<br>work_col = pd.get_dummies(df[&quot;work_type&quot;], prefix=&quot;work_type&quot;)<br>smoking_col = pd.get_dummies(df[&quot;smoking_status&quot;], prefix=&quot;smoking_status&quot;)<br>df = pd.concat([df, work_col, smoking_col], axis=1)<br>df.drop([&quot;work_type&quot;, &quot;smoking_status&quot;], axis=1, inplace=True)</pre><p>Finally, all the features in the dataset are of numeric type. Then, we will determine the explanatory and response variables. The column “stroke” will be the response variable, while the rest of the columns will be the explanatory variable.</p><pre>X = df.drop(&quot;stroke&quot;, axis=1)<br>y = df[&quot;stroke&quot;]</pre><p>While downloading this dataset, I noticed that there was information saying that the proportion of the response variable (“stroke”) was not balanced. This is important to check because imbalanced data can affect the prediction results of machine learning models. Classification models trained on imbalanced data will tend to produce high accuracy but low precision and recall for minority classes [9]</p><pre>y.value_counts().plot(kind=&quot;bar&quot;)</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/754/1*dtSH-WOZLE-x6TP_dUrF4A.png" /><figcaption>Output</figcaption></figure><p>It turns out to be true! The dataset is imbalanced, the response variable tends to lean toward class 0 (no stroke). Experiments conducted by Chuang [10] also proved that compared to imbalanced datasets, balanced datasets can practically produce better model classification accuracy. One popular way to overcome data imbalance is data sampling, including undersampling and oversampling. Balanced random undersampling cannot be used for large and highly imbalanced datasets, but is effective for small datasets [11].</p><p>Our approach for this dataset is undersampling using the SMOTE (<em>synthetic minority over-sampling technique</em>) algorithm. This algorithm has also been shown to lead to better classifier performance (in ROC space) [12].</p><pre>from collections import Counter<br>from imblearn.over_sampling import SMOTE<br><br>sm = SMOTE(random_state=0)<br>X_res, y_res = sm.fit_resample(X, y)<br>print(Counter(y_res))<br>y_res.value_counts().plot(kind=&quot;bar&quot;)</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/761/1*nA0yx9n2_8s8y3FD_05W6g.png" /><figcaption>Output</figcaption></figure><p>The data already looks balanced. Remember that the algorithm for balancing imbalanced data depends on the condition of the data. This means that there is no algorithm that always performs well in every type of dataset.</p><pre>X_train, X_test, y_train, y_test = train_test_split(X_res, y_res, <br>                                                    test_size = 0.25, <br>                                                    random_state = 0)<br><br>print(f&quot;X train:{X_train.shape}&quot;)<br>print(f&quot;X test :{X_test.shape}&quot;)<br><br># output<br>X train:(7099, 16)<br>X test :(2367, 16)</pre><p>The impact of undersampling makes our dataset larger than before. The training data has 7099 rows, while the test data has 2367 rows.</p><p>The last step we will do is to reduce the impact of outliers by performing standardizing. In this dataset, we will standardize using RobustScaler.</p><pre>from sklearn.preprocessing import RobustScaler<br><br>scaler_X = RobustScaler()<br>X_train_scaled = scaler_X.fit_transform(X_train)<br>X_test_scaled = scaler_X.transform(X_test)</pre><p>The dataset is ready to be used to train machine learning algorithms.</p><h3>Conclusion</h3><p>Data preprocessing is an important step before training machine learning algorithms. Common steps in data preprocessing include <strong>data cleaning</strong> (handling missing values, duplicates, and irrelevant features), <strong>data transformation</strong> (performing label encoding and one hot encoding), <strong>feature scaling</strong>, and dealing with <strong>imbalanced data</strong>. Different datasets also require different preprocessing treatments.</p><blockquote>“The more we do data preprocessing, the more we will understand what we need to do in this process.”</blockquote><h4>References:</h4><p>[1] Taylor, M., &amp; Amir, N. (1994). The Problem of Missing Clinical Data for Research in Psychopathology: Some Solution Guidelines. <em>The Journal of Nervous and Mental Disease</em>.</p><p>[2] Abidin, N., Ismail, A., &amp; Emran, N. (2018). Performance Analysis Of Machine Learning Algorithms For Missing Value Imputation. International <em>Journal of Advanced Computer Science and Applications</em>.</p><p>[3] Jackson, E., &amp; Agrawal, R. (2019). Performance Evaluation of Different Feature Encoding Schemes on Cybersecurity Logs. <em>2019 SoutheastCon</em>.</p><p>[4] ] I. Guyon, A. Elisseeff. (2003). An introduction to variable and feature selection. <em>Journal of Machine Learning Research,</em> 3(7–8):1157–1182.</p><p>[5] P. Cortez, A. Cerdeira, F. Almeida, T. Matos and J. Reis. Modeling wine preferences by data mining from physicochemical properties. <em>In Decision Support Systems, Elsevier</em>, 47(4):547–553.</p><p>[6] Géron, A. (2019). <em>Hands-on Machine Learning with Scikit-Learn, Keras, and TensorFlow</em>. O’Reilly Media, Inc.</p><p>[7] Blessing, R. (1997). Outlier Treatment in Data Merging. <em>Journal of Applied Crystallography</em>.</p><p>[8] Mohamad, I., &amp; Usman, D. (2013). Standardization and Its Effects on K-Means Clustering Algorithm. <em>Research Journal of Applied Sciences, Engineering, and Technology.</em></p><p>[9] Ashraf, S., Saleem, S., Ahmed, T., Aslam, Z., &amp; Muhammad, D. (2020). Conversion of adverse data corpus to shrewd output using sampling metrics. <em>Visual Computing for Industry, Biomedicine, and Art.</em></p><p>[10] Chuang, P., &amp; Wu, D. (2019). Applying Deep Learning to Balancing Network Intrusion Detection Datasets. <em>2019 IEEE 11th International Conference on Advanced Infocomm Technology (ICAIT).</em></p><p>[11] Thai-Nghe, N., Gantner, Z., &amp; Schmidt-Thieme, L. (2011). A new evaluation measure for learning from imbalanced data. The 2011 International Joint Conference on Neural Networks.</p><p>[12] N. V. Chawla, K. W. Bowyer, L. O.Hall, W. P. Kegelmeyer, “SMOTE: synthetic minority over-sampling technique,” <em>Journal of artificial intelligence research</em>, 321–357, 2002.</p><h3>Level Up Coding</h3><p>Thanks for being a part of our community! Before you go:</p><ul><li>👏 Clap for the story and follow the author 👉</li><li>📰 View more content in the <a href="https://levelup.gitconnected.com/?utm_source=pub&amp;utm_medium=post">Level Up Coding publication</a></li><li>💰 Free coding interview course ⇒ <a href="https://skilled.dev/?utm_source=luc&amp;utm_medium=article">View Course</a></li><li>🔔 Follow us: <a href="https://twitter.com/gitconnected">Twitter</a> | <a href="https://www.linkedin.com/company/gitconnected">LinkedIn</a> | <a href="https://newsletter.levelup.dev/">Newsletter</a></li></ul><p>🚀👉 <a href="https://jobs.levelup.dev/talent/welcome?referral=true"><strong>Join the Level Up talent collective and find an amazing job</strong></a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=607c6dc9ee56" width="1" height="1" alt=""><hr><p><a href="https://levelup.gitconnected.com/how-to-improve-our-data-preprocessing-skills-with-this-several-project-607c6dc9ee56">How to Improve Our Data Preprocessing Skills with this Several Project</a> was originally published in <a href="https://levelup.gitconnected.com/">Level Up Coding</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Mastering Data Preprocessing and Preparation to Improve Data Quality]]></title>
            <link>https://levelup.gitconnected.com/mastering-data-preprocessing-and-preparation-to-improve-data-quality-4bc9ec6746b3?source=rss-9f804bd8d19f------2</link>
            <guid isPermaLink="false">https://medium.com/p/4bc9ec6746b3</guid>
            <category><![CDATA[python]]></category>
            <category><![CDATA[data-quality]]></category>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[data-pre-processing]]></category>
            <dc:creator><![CDATA[Dede Kurniawan]]></dc:creator>
            <pubDate>Sun, 14 May 2023 04:45:54 GMT</pubDate>
            <atom:updated>2023-05-14T14:56:01.644Z</atom:updated>
            <content:encoded><![CDATA[<h4>Understand the principles and practical guide of data preprocessing</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*kb1UCak6ZDQ5OwZDQcIbWw.jpeg" /><figcaption>innovation and data — Images generated by <a href="https://lexica.art/prompt/8d5d8ba3-ee3b-4219-b69b-3c1d9dc3a4b8">Lexica Aperture v2</a> models</figcaption></figure><p>Real-world data is <strong>not always clean</strong> and ready to be analyzed using machine learning algorithms. Many machine learning algorithms do not work well when the data is of poor quality. Always remember <em>“garbage in garbage out”</em>, If we input poor-quality data into the machine learning model, the model’s output will not be of good quality.</p><blockquote><strong>“No quality data, no quality mining results” — “Quality decisions must be based on quality data”</strong></blockquote><p>With these problems, we need to improve the quality of our data by performing data preprocessing. Data preprocessing is an important step in machine learning that is performed to clean, transform, and prepare raw data before it is understood and analyzed by machine learning models. <strong>Data scientists can spend up to 80% of their time preparing data</strong> before analysis or modeling [<a href="https://hbr.org/2018/08/what-data-scientists-really-do-according-to-35-data-scientists">1</a>].</p><p>This article will discuss data preprocessing and how to do it with Python.</p><h3>1. Importing the libraries and Data set</h3><p>There are two popular libraries used when doing data preprocessing and preparation. The two libraries are <a href="https://pandas.pydata.org/">Pandas</a> and <a href="https://scikit-learn.org/stable/index.html">Scikit-learn</a>. Pandas is used to read data and manipulate data. On the other hand, Scikit Learn is used for scaling data, imputing missing values, encoding categorical variables, and splitting data.</p><pre>import pandas as pd<br>import sklearn</pre><p>Then, to read the data, we can use one of the functions in the Pandas library. Pandas have many functions to read various types of data such as .csv, .json, and .xlsx.</p><pre>df = pd.read_csv(&quot;your file path&quot;)</pre><h3>2. Data Cleaning</h3><p>Data cleaning is one of the critical aspects of data preprocessing, which is the process of improving data quality by <strong>removing errors/invalid</strong>, <strong>missing values</strong>, <strong>duplicates</strong>, and <strong>outliers</strong>. In this step, we also <strong>correct inconsistencies</strong> in the data set to ensure our data is accurate. Simply put, this is the process of removing data that does not belong in our data set. This is very important to ensure the quality of data used in machine learning modeling so that the analysis/prediction results become more accurate and reliable.</p><p>In this process, there are several approaches that are widely used to improve data quality by performing data cleaning:</p><h4><strong>Make a copy of the data set</strong></h4><p>Humans often make mistakes whether intentionally or unintentionally. <strong>By making a copy of the original data</strong>, we will <strong>have an original copy</strong> of the data to <strong>repeat the cleaning process if an error occurs</strong> while cleaning the data. This can be used as backup data.</p><pre># make a copy of the entire data set<br>df_copy = df.copy(deep=True)</pre><h4>Remove irrelevant data</h4><p>Not all data (features) are useful when analyzing or predicting using machine learning models. Take a good look at our data and do some <strong>filtering to remove irrelevant data</strong>. Also, we can perform statistical analyses such as calculating correlations to test which data are highly correlated.</p><pre># calculating correlation between variables<br>df.corr(method=&quot;pearson&quot;) </pre><p>Besides removing irrelevant data, we should also remove <strong>duplicate data</strong> and <strong>invalid data</strong>. The term “duplicate data” refers to the same value repeated for the same observation point. Meanwhile, invalid data refers to data that cannot be trusted or cannot be used for analysis due to an error.</p><pre>def remove_duplicates(df):<br>    &quot;&quot;&quot;A function to remove duplicated data on columns or rows&quot;&quot;&quot;<br><br>    duplicates = df[df.duplicated()]<br><br>    if len(duplicates) == 0:<br>        print(&quot;No duplicated data was found!&quot;)<br>    else:<br>        df.drop_duplicates(inplace=True)<br>        print(f&quot;A total of {len(duplicates)} duplicated data were removed&quot;)</pre><h4>Fix structural errors</h4><p>Some examples of structural errors include <strong>inconsistent variable names</strong>, <strong>incongruent naming convention</strong>s, and <strong>spelling mistakes</strong>. Humans may be able to easily recognize these errors, but not machines. Many machine learning models will not recognize these errors and will make incorrect predictions.</p><h4>Deal with missing value</h4><p>Some of the data we expect may in fact be missing or incomplete, this is called a <strong>missing value</strong>. Simply put, a missing value is an <strong>empty cell in a dataset that we want to use for analytical purposes</strong> [2]. Missing values can be caused by several errors, such as human error, sensor malfunction, and missed observations. We can’t just leave this problem alone, we have to deal with it!</p><p>The first thing we have to do is observe and identify it. Missing values are usually represented as empty cells, zero or 0, NAN, null, none, 999, and NA. In Python, a missing value is represented by <strong>NaN</strong> (<strong>Not a Number</strong>).</p><figure><img alt="Deal with missing value" src="https://cdn-images-1.medium.com/max/1024/1*uFAAMInAsKhBPLS9DlHIqA.png" /><figcaption>A <a href="https://www.kaggle.com/competitions/titanic/data">titanic dataset</a> with missing values presented in pandas (Image by author)</figcaption></figure><p>After identifying it, we have to make sure whether the missing value represents something valuable for the machine learning model. If not, there are several approaches we can take to solve this problem:</p><ul><li><strong>Drop missing value</strong>: The quickest approach is to remove rows or columns with missing values in our dataset. This is a quick and effective method if the number of missing values is relatively small and insignificant compared to the size of our dataset. However, if the number of missing values is very large, using this method can reduce the size of the dataset and cause us to lose important information.</li></ul><pre>def remove_missing_values(df, index):<br>    &quot;&quot;&quot;A function to remove duplicated data on columns or rows&quot;&quot;&quot;<br><br>    if index == &quot;col&quot;:<br>        df.dropna(axis=1, inplace=True)<br>        print(&quot;Missing value in column successfully removed&quot;)<br>    elif index == &quot;row&quot;:<br>        df.dropna(inplace=True)<br>        print(&quot;Missing value in row successfully removed&quot;)<br>    else:<br>        print(&quot;The index argument is invalid&quot;)</pre><ul><li><strong>Perform imputation</strong>: The next approach is to impute or fill the missing values in our data with valid values. Some commonly used imputation techniques are replacing the missing values with mean, median, and mode. If we are considering the relationship between variables and want more accurate imputation results, we can use a machine learning algorithm such as KNN.</li></ul><pre>from sklearn.impute import SimpleImputer, KNNImputer<br><br># imputation with mean, median, or mode<br>imputer = SimpleImputer(missing_values=pd.NA, strategy=&quot;mean&quot;) <br>df_imputed = imputer.fit_transform(df)<br><br># imputation with KNN algorithm<br>imputer = KNNImputer(n_neighbors=n)<br>df_imputed = imputer.fit_transform(df)</pre><h4>Remove outliers from the data set</h4><p>In addition to missing values, outliers are also often present in a data set. Outliers are data objects whose values are too different from the overall population and can skew our analysis too far in a certain direction.</p><figure><img alt="The presence of outliers in a data set" src="https://cdn-images-1.medium.com/max/1024/1*xxt_HYoukTJs0zxppU5azQ.png" /><figcaption>The presence of outliers in a data set (Image by author)</figcaption></figure><p>Jafari (2022) [2] describes three important perspectives in dealing with outliers:</p><ul><li>Outliers can be data errors that should be detected and removed.</li><li>Outliers that are not errors can skew the results of analytical tools that are sensitive to the presence of outliers.</li><li>An outlier may be an incorrect entry.</li></ul><p>There are several approaches we can take in detecting outliers, such as <strong>standardization using Z-score</strong>,<strong> visualization using box plots</strong>, and <strong>detection using statistical range</strong> (Q1–1.5*IQR, Q3+1.5*IQR). Once the outliers have been detected, we can immediately remove them from the data set.</p><pre>def remove_outlier(df, col_name):<br>    &quot;&quot;&quot;A function to remove outlier using IQR&quot;&quot;&quot;<br><br>    Q1 = df[col_name].quantile(0.25)<br>    Q3 = df[col_name].quantile(0.75)<br>    IQR = Q3 - Q1<br>    lower_bound = Q1 - 1.5 * IQR<br>    upper_bound = Q3 + 1.5 * IQR<br>    df = df[(df[col_name] &gt; lower_bound) &amp; (df[col_name] &lt; upper_bound)]<br>    return df</pre><p>Apart from using these methods to detect outliers, we can also use Python libraries such as <a href="https://pyod.readthedocs.io/en/latest/"><strong>PyOD</strong></a>. This library is used to detect outliers in data using various outlier detection algorithms such as PCA, LOF (Local Outlier Factor), ABOD (Angle-Based Outlier Detection), etc.</p><h3>3. Handling categorical data</h3><p>Humans can handle different types of values in a data set such as categorical data (e.g. “gender”, “color”, or “region”). However, this is not the case for machines. <strong>Most machine learning algorithms prefer to work with numbers and will work well when all the data we have is numbers</strong>. A simple example is the K-nearest Neighbor (KNN) algorithm. This algorithm will calculate the distance between observations using Euclidean distance [3]. The distance calculation is not possible if the data is a string (e.g. “English”).</p><p>Therefore, we must turn data in the form of text or categories into numbers.</p><h4>Label encoding</h4><p>One technique for handling categorized data is <strong>label encoding</strong>. This technique will <strong>turn each category value into a unique numeric value, ranging from 0 to n-1</strong> (where n is the number of different categories). If we have categorical data in the form of city names, then for each city name, we will give a unique label into a number, for example, 0 for “Chicago”, 1 for “Los Angeles”, and 2 for “New York”.</p><figure><img alt="Label encoding" src="https://cdn-images-1.medium.com/max/1024/1*j9pjNZX92WqoCeG1Oww5UQ.png" /><figcaption>Label encoding can turn each value in the “City” column into a unique numeric value (Image by author)</figcaption></figure><p>However, the biggest problem with label encoding techniques is that machine learning models may <strong>assume an ordering relationship between categories</strong> that we don’t actually want [4]. Therefore, label encoding techniques are only suitable for <strong>unordered categorical data</strong>.</p><pre># label encoding using scikit-learn <br>from sklearn.preprocessing import LabelEncoder<br><br>encoded = LabelEncoder()<br>label_encoded = encoded.fit_transform(df)</pre><p>It is different if our data does have naturally ordered values (e.g. “agree”, “neutral”, and “disagree”), we can handle it with <strong>Ordinal encoding</strong>. When converting ordinal data into numeric values, we must <strong>retain the notion of ordering</strong> [3].</p><h4>One-hot encoding</h4><p>To handle the biggest problem of the label encoding technique, we can use another alternative, <strong>One-hot encoding</strong>. One-hot encoding will <strong>create a binary value (0 and 1) for each category in the data set</strong>. In this technique, <strong>each category value will be a separate column</strong> in the data labeled 0 or 1 depending on whether the category is present or not in a particular row of data.</p><p>For example, in the data category of cities (e.g. “Chicago”, “Los Angeles”, and “New York”), we can create three new columns, and each row of data will contain a value of 1 or 0 in the corresponding column.</p><figure><img alt="One-hot encoding" src="https://cdn-images-1.medium.com/max/1024/1*Rup1r0fILXNK5mB2MNrXuw.png" /><figcaption>One-hot encoding can turn each category value in the “City” column into a binary vector and each category will be separated into different columns (Image by the author).</figcaption></figure><p>The one-hot encoding technique is more often used than label encoding when we have more than two category values and do not assume any ordering relationship between categories. However, one problem with the one-hot encoding technique is that it <strong>can generate a very large number of features if there are many different category values</strong>. Not only that, but it will also produce rows that contain mostly zeros (sparse matrix) [4].</p><pre># one hot encoding using scikit learn<br>from sklearn.preprocessing import OneHotEncoder<br><br>encoded = OneHotEncoder()<br>onehot_encoded = encoded.fit_transform(df)</pre><h3>4. Splitting the datasets into a training set and test set</h3><p>When building a machine learning model, we need to <strong>ensure that the model works well on data that has never been seen before</strong>. If we use all the data we have to train the model, we cannot measure how well the model works on data that has never been seen before. Therefore, we need to divide our data into a training set and a test set.</p><ul><li><strong>Training set</strong>: Data used to build/train machine learning models [5].</li><li><strong>Test set</strong>: The rest of the data is used to assess how well the machine learning model works [5].</li></ul><p>In a supervised learning problem, each training and test set consists of an <strong>exploratory variable</strong> (usually denoted by a capital X) and a <strong>response variable</strong> (usually denoted by a lowercase y). It is important to ensure that no data from the training set is included in the test set. If this happens, it will be difficult to assess whether the model has learned to generalize from the training set or has simply memorized it. Models that memorize the training set are called <strong>overfitting </strong>[6]. Overfitting can occur when the model performs very well in making predictions on training data, but cannot perform well on data that has never been seen before.</p><p>Now, the problem is what is the proportion of data for the training set and the test set?</p><figure><img alt="Splitting data set into training set and test set" src="https://cdn-images-1.medium.com/max/1024/1*Jkt4FOxB-sgIoWlJtGt43Q.png" /><figcaption>A dataset split into a training set and a testing set with a proportion of 75:25 (Image by author)</figcaption></figure><p>The answer is that it depends on how large the data size is. Using 75% for the training set and 25% for the test set is a good rule of thumb [5]. If our data is very large, we should use a smaller proportion for the test set, such as 5% or 10%.</p><pre>from sklearn.cross_validation import train_test_split<br><br>X_train, X_test, y_train, y_test = train_test_split(X, y, <br>                                                    test_size = 0.25, <br>                                                    random_state = 0)</pre><h3>5. Feature Scaling</h3><p>Some machine learning algorithms such as linear regression, KNN, and K-means clustering are <strong>very sensitive to data that has different feature scales</strong>. When we have feature values with different scales, the machine learning model may give disproportionate weight to features with larger scales. This can lead to inaccurate and unstable models. Therefore, we need to perform <strong>feature scaling</strong>.</p><p>Feature scaling is the process of <strong>adjusting the scale of feature values to ensure that features have a similar range of values</strong>. Here are some ways to scale data:</p><figure><img alt="several ways to scale data" src="https://cdn-images-1.medium.com/max/932/1*4VwFHys94en7onEUM0iOMw.png" /><figcaption>There are several ways to scale data [5]</figcaption></figure><h4>Normalization (Min-Max scaling)</h4><p>This is the simplest method. The normalization <strong>method maps feature values into a range between two values, such as 0 and 1</strong> [5, 6]. It works by converting each feature value into a value relative to the minimum and maximum values of the feature. Normalization does not work well when there are outliers in the data. This is because outliers can affect the minimum and maximum values which in turn affect the normalization results.</p><figure><img alt="Normalization (Min-Max scaling)" src="https://cdn-images-1.medium.com/max/302/1*97NuCCmt9Q0Quph4uBbY0g.png" /></figure><pre>from sklearn.preprocessing import MinMaxScaler<br><br>scaler_X = MinMaxScaler()<br>X_train_scaled = scaler_X.fit_transform(X_train)<br>X_test_scaled = scaler_X.transform(X_test)</pre><h4>StandardScaler (Z-score scaling)</h4><p>On the other hand, there is StandardScaler. The StandardScaler method <strong>maps the feature values into a normal distribution with a mean of 0 and a standard deviation of 1</strong> [5]. Just like normalization, it also does not work well when our data has significant outliers. This can negatively impact our standardization by affecting the mean and variance of the features [6].</p><figure><img alt="StandardScaler" src="https://cdn-images-1.medium.com/max/178/1*j6vbjKHUkqraCsiggbulWQ.png" /></figure><pre>from sklearn.preprocessing import StandardScaler<br><br>scaler_X = StandardScaler()<br>X_train_scaled = scaler_X.fit_transform(X_train)<br>X_test_scaled = scaler_X.transform(X_test)</pre><h4>Robust Scaler</h4><p>An alternative when there are significant outliers in the data is to use the Robust scaler. It works the same way as StandardScaler, but<strong> it uses median and interquartile range (IQR) values</strong>. This is what makes the Robust scaler resistant to outliers because it ignores these very different data points.</p><pre>from sklearn.preprocessing import RobustScaler<br><br> scaler = RobustScaler()<br> X_train_scaled = scaler.fit_transform(X_train)<br> X_test_scaled = scaler.transform(X_test)</pre><h3>Key Takeaway</h3><p>One of the keys to a successful machine learning project is to ensure that the data we have is of good quality. Data preprocessing and preparation are ways to improve data quality. There are several things that are commonly done in this process including:</p><ol><li><strong>Data cleaning</strong>: Remove irrelevant data, fix structural errors, deal with missing values, and remove outliers.</li><li><strong>Dealing with categorized data</strong>: Label encoding and One-hot encoding.</li><li><strong>Splitting the data into training sets and test sets</strong>.</li><li><strong>Feature scaling</strong>: Normalization (Min-Max scaling), StandardScaler (Z-score scaling), and Robust Scaler.</li></ol><h3>References:</h3><p>[1] <em>What data scientists really do, according to 35 data scientists</em> (2018) <em>Harvard Business Review</em>. Available at: <a href="https://hbr.org/2018/08/what-data-scientists-really-do-according-to-35-data-scientists">https://hbr.org/2018/08/what-data-scientists-really-do-according-to-35-data-scientists</a> (Accessed: May 8, 2023).</p><p>[2] Jafari, R. (2022). <em>Hands-On Data Preprocessing in Python: Learn how to effectively prepare data for successful data analytics</em>. Packt Publishing.</p><p>[3] Albon, C. (2018). <em>Machine learning with Python cookbook: practical solutions from preprocessing to deep learning</em>. O’Reilly Media, Inc.</p><p>[4] Serrano, L. (2021). <em>Grokking Machine Learning</em>. Manning Publications.</p><p>[5] Müller, A. C., Guido, S. (2016). <em>Introduction to Machine Learning with Python: A Guide for Data Scientists</em>. O’Reilly Media, Inc.</p><p>[6] Hackeling, G. (2017). Mastering Machine Learning with scikit-learn: Apply effective learning algorithms to real-world problems using scikit-learn. Packt Publishing.</p><h3>Level Up Coding</h3><p>Thanks for being a part of our community! Before you go:</p><ul><li>👏 Clap for the story and follow the author 👉</li><li>📰 View more content in the <a href="https://levelup.gitconnected.com/?utm_source=pub&amp;utm_medium=post">Level Up Coding publication</a></li><li>💰 Free coding interview course ⇒ <a href="https://skilled.dev/?utm_source=luc&amp;utm_medium=article">View Course</a></li><li>🔔 Follow us: <a href="https://twitter.com/gitconnected">Twitter</a> | <a href="https://www.linkedin.com/company/gitconnected">LinkedIn</a> | <a href="https://newsletter.levelup.dev/">Newsletter</a></li></ul><p>🚀👉 <a href="https://jobs.levelup.dev/talent/welcome?referral=true"><strong>Join the Level Up talent collective and find an amazing job</strong></a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=4bc9ec6746b3" width="1" height="1" alt=""><hr><p><a href="https://levelup.gitconnected.com/mastering-data-preprocessing-and-preparation-to-improve-data-quality-4bc9ec6746b3">Mastering Data Preprocessing and Preparation to Improve Data Quality</a> was originally published in <a href="https://levelup.gitconnected.com/">Level Up Coding</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
    </channel>
</rss>