<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Mundy Reimer on Medium]]></title>
        <description><![CDATA[Stories by Mundy Reimer on Medium]]></description>
        <link>https://medium.com/@mundyreimer?source=rss-65f1193d859e------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/2*rwiPKm-zSxy0Z5Ncw2lvlg.jpeg</url>
            <title>Stories by Mundy Reimer on Medium</title>
            <link>https://medium.com/@mundyreimer?source=rss-65f1193d859e------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Fri, 20 Nov 2026 06:27:49 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@mundyreimer/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[Analyzing Microbiome Data]]></title>
            <link>https://medium.com/data-science/analyzing-microbiome-data-320728b56b8e?source=rss-65f1193d859e------2</link>
            <guid isPermaLink="false">https://medium.com/p/320728b56b8e</guid>
            <category><![CDATA[data-visualization]]></category>
            <category><![CDATA[bioinformatics]]></category>
            <category><![CDATA[microbiome]]></category>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[genomics]]></category>
            <dc:creator><![CDATA[Mundy Reimer]]></dc:creator>
            <pubDate>Wed, 24 Jun 2020 20:24:30 GMT</pubDate>
            <atom:updated>2020-06-25T21:14:03.180Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*c74j3M2P5y-N0iH_" /><figcaption>Source: <a href="https://wallpapercave.com/dna-background">DNA wallpapercave</a></figcaption></figure><h4>Part 1 — Genomic Data Science Series</h4><p>This article is part of a tutorial series on applying machine learning to bioinformatics data:</p><p>Part 1 — Data Acquisition &amp; Preprocessing</p><p><a href="https://towardsdatascience.com/visualizing-high-dimensional-microbiome-data-eacf02526c3a">Part 2 — Visualizing High-Dimensional Data</a></p><p>To follow along, you can either download our <a href="https://github.com/njparker1993/Microbiome_Machine_Learning/blob/master/Microbiome_Data_Acquisition.ipynb">Jupyter notebook</a>, or continue reading and typing in the following code as you proceed through the walkthrough.</p><h3>Introduction</h3><p>Many of us spend the majority of our time indoors, but few of us realize the extent to which every surface we touch is home to specialized organisms that colonize those spaces — doorknobs, kitchen counters, tables, chairs, bathrooms, refrigerators, etc. When we think of different ecosystems such as deserts, rainforests, grasslands, or coral reefs, we know that different communities of organisms are specialized to live in those areas of the world. Similarly, we can infer that different compositions or groups of microorganisms and fungi may be specialized to live in different locations or surfaces around us. We call these special communities of microorganisms a <a href="https://en.wikipedia.org/wiki/Microbiota"><em>Microbiome</em></a>.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/640/0*IM2a3GfyNLfDB4xC.jpg" /><figcaption>Source: <a href="https://www.geograph.org.uk/photo/2815842">Invisible Worlds Exhibition 2012, Geograph</a></figcaption></figure><p>Now what if we only knew the genetic differences between these communities of organisms? Could we perhaps reverse engineer the location of where these organisms came from? In the following tutorial series, we’ll show you how we go from the genetic sequence data of simple swabs of surfaces to figuring out how to differentiate between the locations of where those swabs came from without knowing the locations beforehand using some data science detective work.</p><p>To investigate the genetic differences between microbiomes we were able to find a data set using <a href="https://qiita.ucsd.edu/static/doc/html/index.html"><em>Qiita</em></a>.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*It0HUUu6lD7_vnha" /><figcaption>Source: <a href="https://qiita.ucsd.edu/">Qiita Documentation</a></figcaption></figure><p>Qiita (pronounced ‘<em>cheetah</em>’) is a multi-omics software package and database that holds microbial ecology datasets and offers a nice interface to search for different types of -omics data, find reference data from papers, and even provide information regarding the file types and associated metadata. It is based on the <a href="https://en.wikipedia.org/wiki/QIIME"><em>QIIME</em></a> bioinformatics pipeline that transforms and processes raw sequencing data to common formats that are friendly to data analysis. Other common bioinformatics pipelines are <em>MOTHUR</em>, <em>DADA2</em>, and <em>USEARCH</em>, etc. For a comparison of the different pipelines, <a href="https://journals.plos.org/plosone/article?id=10.1371/journal.pone.0227434">see here</a>.</p><h3>Acquiring our Data</h3><p>To access the data you’ll need to create an account and then <a href="https://qiita.ucsd.edu/study/description/10423">download the data</a> collected by the <a href="https://caporasolab.us/">Caporaso Lab</a> at the Pathogen &amp; Microbiome Institute from Northern Arizona University, by clicking on the <em>All QIIME maps and BIOMs</em> on the left-hand side.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*bvWd8VzMoT7unwcU" /><figcaption>Source: <a href="https://qiita.ucsd.edu/study/description/10423">Qiita, Caporaso Lab Project</a></figcaption></figure><p>Once done, you’ll need to unzip the file, place it in a folder called <em>data</em>, and then transform the data into a format we can work with. To do that we will first need to install a package called <em>biom-format </em>by going to your terminal and typing:</p><pre>pip install biom-format</pre><p>Once this is done we can then run the following command in our terminal:</p><pre>biom convert -i data/BIOM/60899/all.biom -o data/BIOM/60899/feature_table.tsv — to-tsv</pre><p>This command will convert our .biom files to a more familiar .tsv format, from which we can then go into our <em>Jupyter</em> notebook and read into a <em>Pandas</em> dataframe by:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/82154d9368d7be55783dfd6cbd64bd36/href">https://medium.com/media/82154d9368d7be55783dfd6cbd64bd36/href</a></iframe><p>The <a href="http://biom-format.org/">BIOM file format</a> is commonly found in <em>-omics</em> datasets where it is used to represent <em>OTU</em> tables. OTUs stand for <a href="https://en.wikipedia.org/wiki/Operational_taxonomic_unit"><em>Operational Taxonomic Units</em></a> which can be thought of as clusters of organisms grouped by genetic sequence similarity of a particular region of the genome. In essence, they act as a proxy for “species” at different taxonomic levels when the clusters of organisms are unknown. The observations or rows in this table are OTUs and the corresponding matrix records the number of times each OTU is observed in each sample.</p><p>Since the scientists who originally collected this data most likely took swabs of various surfaces and locations and then sequenced the DNA of the microorganisms and fungi that were picked up by this swab, we don’t necessarily know the composition of the actual species beforehand. All we have are different groups of strings of DNA (composed of <em>A, T, C, Gs</em>) from which we can then compare to a reference database by running the <a href="https://en.wikipedia.org/wiki/BLAST_(biotechnology)"><em>BLAST</em></a> local alignment search algorithm on it.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/600/1*VlrrLHzl607PzAHNKjg6oQ.jpeg" /><figcaption>Source: <a href="https://www.ncbi.nlm.nih.gov/books/NBK62051/">BLAST Glossary</a></figcaption></figure><p>Such a tool can be <a href="https://blast.ncbi.nlm.nih.gov/Blast.cgi">found here</a> offered by the <a href="https://en.wikipedia.org/wiki/National_Institutes_of_Health">NIH</a>. In our case, this step was already done for us.</p><h3>Data Preprocessing</h3><p>Fortunately, our dataset already came with a corresponding metadata file with information regarding sample ids, location where the sequencing was performed, the type of experiment the data came from, the sequencing machine used, the timestamps of the swab collection, the geolocation, and much more. We can open our metadata file in pandas by doing the following:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/d9e8ff23f8f4433f5e1324aa55634fff/href">https://medium.com/media/d9e8ff23f8f4433f5e1324aa55634fff/href</a></iframe><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*Yv1V8V-swvz3ILxe" /><figcaption>Source: Image by author</figcaption></figure><p>We can then create a helper function that takes the output from our .biom file and converts it to something that is easier to work with, which we’ll call a <em>feature table</em>:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/a0d03d98c4c544629b319755c5c89a44/href">https://medium.com/media/a0d03d98c4c544629b319755c5c89a44/href</a></iframe><p>Then we can run our above helper function on our previous raw table and rename our index:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/09b6db3379f4e8c21258080acfcbaf3f/href">https://medium.com/media/09b6db3379f4e8c21258080acfcbaf3f/href</a></iframe><p>We can see that we start with a table that has 1,497 rows and 25,756 columns. Because each column represents a particular genetic sequence or OTU (which corresponds to our features or dimensions), we can see that we are dealing with incredibly high-dimensional data. This is quite common when dealing with metagenomic data sets and presents new challenges that traditional data science techniques may not be able to handle.</p><p>To help us deal with this high dimensionality, we used our helper function to rename every column that was originally labeled with a genetic sequence to a corresponding integer, so that we don’t have really long strings of genetic sequences as column names. Each of these columns contains either frequencies or counts of that particular OTU for each row. Each row represents a sample id that might identify for example a particular swab.</p><p>From here we can undergo a few common practices used to clean bioinformatics data. By making sure each sequence has more than 20,000 reads associated with it and has been seen in at least 3 different samples (in essence, applying a series of thresholds), we can subset our data such that we now have 301 rows and 1,894 columns.</p><p><em>*Note that there is much that can be improved upon at this step, and many scientific papers have been written about this. We are approaching this from a quick-and-dirty perspective.</em></p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/08707ec6f906e5513e92c715d9c3e0a2/href">https://medium.com/media/08707ec6f906e5513e92c715d9c3e0a2/href</a></iframe><figure><img alt="" src="https://cdn-images-1.medium.com/max/876/0*VgRfaUm3mCjmAMA9" /><figcaption>Source: Image by author</figcaption></figure><p>Since we applied thresholds to reduce our feature table, we will also remove those corresponding entries from our metadata table:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/092d8311bd818a71408e5ce4622163fc/href">https://medium.com/media/092d8311bd818a71408e5ce4622163fc/href</a></iframe><p>From here we can now use our metadata to double-check and see that we have samples that come from 3 different cities:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/4105aaadf7aa0e176bf9c15a18b735bc/href">https://medium.com/media/4105aaadf7aa0e176bf9c15a18b735bc/href</a></iframe><figure><img alt="" src="https://cdn-images-1.medium.com/max/480/0*e5MIaHczG4_ft6eJ" /><figcaption>Source: Image by author</figcaption></figure><h3>Next Steps…</h3><p>We have walked through the steps explaining the nature of microbiome metagenomic data, how to obtain it, how to load it, and how to preprocess it and get it into a format from which we can now do analysis on. In <a href="https://medium.com/@njparker_8716/visualizing-high-dimensional-microbiome-data-eacf02526c3a">our next article</a>, we’ll show you how to apply various machine learning algorithms and dimensionality reducing techniques to help visualize and interpret high dimensional data so that we can infer an underlying geographic structure from our data without having to resort to labels provided via our metadata.</p><p>See: <a href="https://towardsdatascience.com/visualizing-high-dimensional-microbiome-data-eacf02526c3a">Part 2 — Visualizing High-Dimensional Data</a></p><p>The two co-authors of this tutorial series are <a href="https://medium.com/u/76a19323c6b6">Nicholas Parker</a> &amp; <a href="https://medium.com/u/65f1193d859e">Mundy Reimer</a>, both of whom are graduates of the <a href="https://www.usfca.edu/arts-sciences/graduate-programs/data-science">Masters in Data Science program from the University of San Francisco</a>.</p><p>If you would like to reach out, you can find us at:</p><p><strong>Mundy Reimer<br></strong><a href="https://www.linkedin.com/in/mundyreimer">Linkedin</a><br><a href="https://twitter.com/MondayRhymer">Twitter</a><br><a href="https://mundyreimer.github.io/">Personal Blog</a></p><p><strong>Nick Parker<br></strong><a href="https://www.linkedin.com/in/nicholas-j-parker">Linkedin</a><br><a href="https://medium.com/@njparker_8716">Personal Blog</a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=320728b56b8e" width="1" height="1" alt=""><hr><p><a href="https://medium.com/data-science/analyzing-microbiome-data-320728b56b8e">Analyzing Microbiome Data</a> was originally published in <a href="https://medium.com/data-science">TDS Archive</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
    </channel>
</rss>