<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Himanshi Allahabadi on Medium]]></title>
        <description><![CDATA[Stories by Himanshi Allahabadi on Medium]]></description>
        <link>https://medium.com/@himanshiallahabadi?source=rss-e28d8673e8a1------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/1*KGlCOGf6sNSdaJzICrGJTA.png</url>
            <title>Stories by Himanshi Allahabadi on Medium</title>
            <link>https://medium.com/@himanshiallahabadi?source=rss-e28d8673e8a1------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Sat, 21 Nov 2026 15:55:53 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@himanshiallahabadi/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[Preparing Molecular Cancer Data for Large-Scale Analysis]]></title>
            <link>https://medium.com/@himanshiallahabadi/preparing-molecular-cancer-data-for-large-scale-analysis-f9d258692e95?source=rss-e28d8673e8a1------2</link>
            <guid isPermaLink="false">https://medium.com/p/f9d258692e95</guid>
            <category><![CDATA[exploratory-data-analysis]]></category>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[cancer]]></category>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[data-visualization]]></category>
            <dc:creator><![CDATA[Himanshi Allahabadi]]></dc:creator>
            <pubDate>Thu, 28 Nov 2024 11:01:17 GMT</pubDate>
            <atom:updated>2024-11-30T09:06:39.825Z</atom:updated>
            <content:encoded><![CDATA[<blockquote>A practical guide for data scientists and ML enthusiasts who want to get hands-on with life sciences data and understand its potential for transforming medicine.</blockquote><h4><strong>The Cancer Genome Atlas (TCGA)</strong></h4><p>Genomics data plays a pivotal role in understanding the molecular basis of diseases, identifying potential therapeutic targets, and advancing research in precision medicine. TCGA is a landmark cancer genomics program that provides researchers with an extensive <strong>repository of</strong> <strong>clinical, genomics and transcriptomics data</strong> from thousands of patients across <strong>33 types of cancer</strong>.</p><h3>Data Search and Retrieval</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*6ouJ28SEKwc5SwKjiuKLRg.png" /></figure><p>Data can be accessed using the <strong>Genomic Data Commons (GDC) API</strong>, which offers multiple endpoints for querying metadata and downloading files. The primary APIs are:</p><ul><li><strong>Files API</strong>: Enables access to file-level metadata and datasets, such as gene expression quantification files.</li><li><strong>Cases API</strong>: Retrieves clinical metadata associated with patient cases, including demographic, diagnostic, treatment and exposure information.</li><li><strong>Projects API</strong>: Provides an overview of the available projects, including information on cancer types, the number of cases, and data types available for each project.</li></ul><p>List of available fields can be found on the <a href="https://docs.gdc.cancer.gov/API/Users_Guide/Appendix_A_Available_Fields/#field-listing-by-endpoint">GDC website</a>. Non-programmatic access to the data is also possible on the <a href="https://portal.gdc.cancer.gov/">GDC web portal</a>.</p><h4>Query for Gene Expression Data</h4><p>Using the <strong>Files API</strong>, we retrieve gene expression quantification files, as well as related metadata for <strong>lung and bronchus cancer</strong>. The metadata includes file_id, file_name, case_id, sample_id, and sample_type—essential for merging gene expression data with clinical metadata.</p><pre>filters = {<br>    &quot;op&quot;: &quot;and&quot;,<br>    &quot;content&quot;: [<br>        {&quot;op&quot;: &quot;in&quot;, &quot;content&quot;: {&quot;field&quot;: &quot;cases.primary_site&quot;, &quot;value&quot;: [&quot;bronchus and lung&quot;]}},<br>        {&quot;op&quot;: &quot;in&quot;, &quot;content&quot;: {&quot;field&quot;: &quot;files.experimental_strategy&quot;, &quot;value&quot;: [&quot;RNA-Seq&quot;]}},<br>        {&quot;op&quot;: &quot;in&quot;, &quot;content&quot;: {&quot;field&quot;: &quot;files.data_type&quot;, &quot;value&quot;: [&quot;Gene Expression Quantification&quot;]}}<br>    ]<br>}<br><br>params = {<br>    &quot;filters&quot;: filters,<br>    &quot;size&quot;: &quot;1000&quot;,  <br>    &quot;fields&quot;: &quot;file_id,file_name&quot;,<br>    &quot;format&quot;: &quot;json&quot;<br>}<br><br>response = requests.post(&quot;https://api.gdc.cancer.gov/files&quot;, json=params)<br>gene_expression_metadata = response.json()</pre><h4>Query for Clinical Metadata</h4><p>Next, we use the <strong>Cases API</strong> to retrieve clinical data, including demographic, diagnostic, and exposure details for lung cancer patients. This metadata is essential for linking the molecular data with patient characteristics and outcomes.</p><pre>filters = {<br>    &quot;op&quot;: &quot;in&quot;,<br>    &quot;content&quot;: {&quot;field&quot;: &quot;primary_site&quot;, &quot;value&quot;: [&quot;bronchus and lung&quot;]}<br>}<br><br>params = {<br>    &quot;filters&quot;: filters,<br>    &quot;fields&quot;: (<br>        &quot;case_id,demographic.gender,demographic.race,demographic.ethnicity,demographic.vital_status,&quot;<br>        &quot;diagnoses.tumor_stage,diagnoses.age_at_diagnosis,exposures.tobacco_smoking_status,&quot;<br>        &quot;exposures.alcohol_history&quot;<br>    ),<br>    &quot;size&quot;: &quot;1000&quot;, <br>    &quot;format&quot;: &quot;json&quot;<br>}<br><br>response = requests.post(&quot;https://api.gdc.cancer.gov/cases&quot;, json=params)<br>clinical_metadata = response.json()</pre><h4>Mapping Identifiers for Merging</h4><ul><li>Each TCGA case represents a single patient (case_id)</li><li>A patient may have multiple samples (sample_id) for different tissue types (e.g., tumour or normal tissue)</li><li>Each sample is linked to files (file_id) containing experimental data.</li></ul><p>To map the gene expression files to their respective patients in the clinical metadata file, we will need to map file_name to case_id. For this, we need some additional metadata. We query the <strong>Files API</strong> again to get the needed metadata (file_name, case_id, sample_id and sample_type).</p><pre>params = {<br>    &quot;filters&quot;: filters,<br>    &quot;fields&quot;: &quot;file_id,file_name,cases.case_id,cases.samples.sample_id,cases.samples.sample_type&quot;,<br>    &quot;size&quot;: &quot;1000&quot;,<br>    &quot;format&quot;: &quot;json&quot;<br>}<br><br>response = requests.post(&quot;https://api.gdc.cancer.gov/files&quot;, json=params)<br>file_metadata = response.json()</pre><p>After merging, cleaning and a few more data preparation steps (check <a href="https://github.com/himanshi4693/cancer_gex/blob/main/TCGA_Lung_Bronchus_Cancer_EDA.ipynb">notebook</a> for details), we get something like this :</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*V6MiD67_Eyvxt59bXVBVkg.png" /><figcaption>Preview of gene expression and clinical data of lung and bronchus cancer patients from TCGA. The actual data is highly dimensional (~58k columns) — the columns above are only for representation.</figcaption></figure><p>The column names starting with ‘ENSG’ represent actual expression counts for different gene_ids. Note that a gene can have multiple isoforms, and gene_id is their unique representation. In other words, gene_id is mapped many-to-one with gene_name.</p><h3>Data Exploration and Visualisation</h3><p>Using this data, we study molecular differences across cancer and normal samples, and explore correlations with clinical variables like patient survival and tumour characteristics.</p><h4>Gene Expression Heatmap</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*ycm7aYRlOu5lEu2dWpN8gw.png" /><figcaption>Heatmap of gene expression counts for 100 genes with the highest variance</figcaption></figure><p>In this representation, normal and tumour tissue samples are labeled with the prefixes ‘N_’ and ‘T_’ on the y-axis. Tumour samples are grouped at the bottom, while normal samples are displayed at the top. The visualisation clearly highlights that several genes are either over-expressed or under-expressed in tumour tissues.</p><h4>T-tests (tumour v/s normal samples)</h4><pre># Subset the data by sample type<br>tumor_data = filt_lung_df[filt_lung_df[&#39;cases.samples.sample_type&#39;] == &#39;Primary Tumor&#39;]<br>normal_data = filt_lung_df[filt_lung_df[&#39;cases.samples.sample_type&#39;] == &#39;Solid Tissue Normal&#39;]<br><br>gene_columns = filt_lung_df.columns[4:]  # Skip metadata columns<br><br># Check which gene columns are missing in either tumor or normal data<br>missing_in_tumor = [gene for gene in gene_columns if gene not in tumor_data.columns]<br>missing_in_normal = [gene for gene in gene_columns if gene not in normal_data.columns]<br><br># Print missing columns<br>print(&quot;Missing in tumor data:&quot;, len(missing_in_tumor))<br>print(&quot;Missing in normal data:&quot;, len(missing_in_normal))<br><br>print(f&quot;Num rows tumor data: {len(tumor_data)}&quot;)<br>print(f&quot;Num rows normal data: {len(normal_data)}&quot;)<br><br># Filter out missing genes from both tumor_data and normal_data<br>common_genes = [gene for gene in gene_columns if gene in tumor_data.columns and gene in normal_data.columns]<br><br># Perform t-test only on common genes<br>p_values = {}<br>for gene in common_genes:<br>    t_stat, p_val = ttest_ind(tumor_data[gene], normal_data[gene], nan_policy=&#39;omit&#39;)<br>    p_values[gene] = p_val<br><br># Convert to DataFrame for easy viewing<br>p_value_df = pd.DataFrame(p_values.items(), columns=[&#39;Gene&#39;, &#39;p_value&#39;])<br>p_value_df[&#39;Significant&#39;] = p_value_df[&#39;p_value&#39;] &lt; 0.05  # Set threshold for significance<br><br>p_value_df</pre><p>The t-tests showed that ~41k of ~58k genes significantly distinguish tumour from normal samples. Gene expression patterns carry critical information, with many genes potentially playing an important role in distinguishing tumour biology from normal tissue.</p><h4>Log Fold Change of Genes</h4><p>Log fold change quantifies the relative difference in gene expression levels between the two sample types. It indicates how strongly a gene’s expression is upregulated or downregulated in one condition compared to the other. This can help to identify genes having a significant impact on cancer progression or suppression.</p><pre># Calculate log2 fold change<br>logFC = {}<br>for gene in p_value_df[&#39;Gene&#39;]:<br>    tumor_mean = tumor_data[gene].mean()<br>    normal_mean = normal_data[gene].mean()<br>    if normal_mean &gt; 0:<br>        logFC[gene] = np.log2(tumor_mean / normal_mean)<br><br># Add log2 fold change to p_value_df<br>p_value_df[&#39;logFC&#39;] = p_value_df[&#39;Gene&#39;].map(logFC)<br><br># Compute -log10(p-value)<br>p_value_df[&#39;-log10(p-value)&#39;] = -np.log10(p_value_df[&#39;p_value&#39;])<br><br># Set thresholds for significance<br>logFC_threshold = 1  # fold change &gt; 2 or &lt; -2<br>p_value_threshold = 0.05  # significance level<br><br># Create the volcano plot<br>plt.figure(figsize=(10, 8))<br>sns.scatterplot(data=p_value_df, x=&#39;logFC&#39;, y=&#39;-log10(p-value)&#39;, hue=&#39;Significant&#39;, palette=&#39;coolwarm&#39;, s=100, edgecolor=&#39;k&#39;)<br><br># Add horizontal and vertical lines to highlight the thresholds<br>plt.axhline(y=-np.log10(p_value_threshold), color=&#39;grey&#39;, linestyle=&#39;--&#39;)<br>plt.axvline(x=logFC_threshold, color=&#39;grey&#39;, linestyle=&#39;--&#39;)<br>plt.axvline(x=-logFC_threshold, color=&#39;grey&#39;, linestyle=&#39;--&#39;)<br><br>plt.title(&#39;Volcano Plot of Differential Gene Expression&#39;, fontsize=16)<br>plt.xlabel(&#39;Log2 Fold Change&#39;, fontsize=14)<br>plt.ylabel(&#39;-Log10 p-value&#39;, fontsize=14)<br>plt.legend(title=&#39;Significance&#39;, loc=&#39;upper left&#39;)<br><br>plt.show()</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/855/1*jJJf3YcjCDm-JSJk4oD-kQ.png" /></figure><p>In the plot above, genes to the left of the first vertical line or to the right of the second have expression levels at least twice as high in tumour samples compared to normal samples. Combining LogFC with statistical significance helps to prioritise genes for further study. Below is a list of top 10 statistically significant genes with the highest fold change (FC), retrieved via the <a href="https://rest.ensembl.org/documentation/info/lookup">Ensembl Lookup API</a>. The final column provides insights from the literature regarding their relevance to cancer.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*YmmazDLwrvojWzMW98Pw3w.png" /></figure><p>This was about retrieving and preparing the dataset for large-scale analysis, along with some initial exploratory data analysis. We’ve set the stage for more advanced machine learning applications, which will be discussed in upcoming posts.</p><p>Code and metadata used in this project can be found in <a href="https://github.com/himanshi4693/cancer_gex/tree/main">this repo</a>.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=f9d258692e95" width="1" height="1" alt="">]]></content:encoded>
        </item>
    </channel>
</rss>