<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Maria Luisa on Medium]]></title>
        <description><![CDATA[Stories by Maria Luisa on Medium]]></description>
        <link>https://medium.com/@info_31460?source=rss-063e088a3632------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/1*d0X3gG0C7FbaZ9hZ3JHC7Q.png</url>
            <title>Stories by Maria Luisa on Medium</title>
            <link>https://medium.com/@info_31460?source=rss-063e088a3632------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Wed, 18 Nov 2026 04:22:14 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@info_31460/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[Scaling out Variant Calling with Spark DeepVariant]]></title>
            <link>https://medium.com/@info_31460/scaling-out-variant-calling-with-spark-deepvariant-e6c2482b577f?source=rss-063e088a3632------2</link>
            <guid isPermaLink="false">https://medium.com/p/e6c2482b577f</guid>
            <category><![CDATA[genomics]]></category>
            <category><![CDATA[bioinformatics]]></category>
            <category><![CDATA[spark]]></category>
            <category><![CDATA[variant-calling]]></category>
            <category><![CDATA[cloud-computing]]></category>
            <dc:creator><![CDATA[Maria Luisa]]></dc:creator>
            <pubDate>Wed, 13 Mar 2024 17:09:30 GMT</pubDate>
            <atom:updated>2024-03-13T17:09:30.732Z</atom:updated>
            <content:encoded><![CDATA[<p>Note: in our commitment to keeping the Internet authentic and useful no generative AI has been used to write this article.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*tcaDjkOy8C5FfiI1nsmBrw.jpeg" /><figcaption>Image credit: unsplash.</figcaption></figure><h3>Introduction</h3><p>Variant Calling is the process by which variants are identified in DNA sequences. In the last couple of years, many different tools have emerged that can solve this task with high quality and reliability.</p><p>One of the leading projects in this area is DeepVariant from Google. Although the accuracy and support for many different sequencer brands is quite decent, one of its main shortcomings is the limitations of leveraging cluster computing to increase throughput.</p><p>This is where Spark DeepVariant comes into play; powered by Apache Spark, Spark DeepVariant adds many new features,</p><ul><li>It leverage SoTA Deep Variant models at scale in a cluster.</li><li>It can use multiple GPUs in multiple nodes or in single node.</li><li>It is faster than open source version, even for single node.</li><li>It comes CVE(Common Vulnerabilities and Exposures) free.</li><li>It’s actively developed.</li><li>It supports fault tolerance through Apache Spark.</li><li>It supports Streaming.</li></ul><p>Now, let’s jump into the basics of using Spark DeepVariant!</p><h3>Basics and Data Loading 💽</h3><p>Using Spark DeepVariant feels like using your favorite Python package, you can access it from within your typical Jupyter notebook, and integrate it with other libraries you like. The only difference is that you can connect it with a multi-node cluster to speed up your results. Let’s take a look at how you would load data into it!</p><p>To start feeding data into Spark DeepVariant, you first need to load your data into an Apache Spark Dataframe. Dataframes in Spark are similar to those in Pandas, with the difference that they deliver data distribution in a cluster, fault tolerance, and many other advantages. So this is what loading some reads into a Dataframe would look like,</p><pre>from sparkdv.datasets.dataset_reader import BamReader<br><br>reads_path = &quot;s3://my_reads/*&quot;<br><br>reads_df = BamReader().\<br>  setTasksPerBamReadsFile(96).\<br>  setTasksGroupSize(1).\<br>  loadReads(spark, reads_path)</pre><p>There are a few things to unpack here; first the utility class we used for lifting the reads data is called BamReader, it will load data from any distributed file system(DFS), and it comes with a number of options, let’s explore some of them:</p><ul><li>TasksPerBamReadsFile: this one controls how many tasks you want to run for each BAM file. Remember you can have hundreds of BAM files in that folder!</li><li>TasksGroupSize: This controls how to group tasks, a task group of certain size is allocated to each Spark executor. If you have multiple small groups you can distribute them more effectively across your cluster’s executors.</li></ul><p>Once you run this code, if you’re using Jupyter, you can dig into the Dataframe’s content like this,</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*NidLi7t6r7xL3DYwL-WaZQ.png" /></figure><p>In this example we are just focusing on groups(according to BAM standard) and taskIds(as we discussed before), but there’s actually much more that you can obtain from this Dataframe,</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*4qTy5TyXQBmQAG1zLzfQgQ.png" /></figure><p>In general, the BamReader is super-lazy, meaning that it will only load the data that you want to see, this results in the ability of being able to handle thousands of files at the same time.</p><p>Now that we have all our reads files loaded into our Dataframe, let’s jump right into the next step that is generating examples.</p><h3>Example Generation</h3><p>At this point we have a single unified point of access to all our reads data files, so the next step is to feed them to the MakeExamples stage. The MakeExamples step will generate the candidate pile-up images that will help determine what type of variant is present at that specific location. Let’s get started with this,</p><pre>from sparkdv.transformers import VariantCallerModel, MakeExamples<br>from pyspark.ml import PipelineModel<br><br>make_examples = MakeExamples()\<br>  .setMode(&quot;calling&quot;)\<br>  .setRefPath(&quot;s3://reference/GRCh38_no_alt_analysis_set.fasta&quot;)\<br>  .setRegions(&quot;chr20&quot;)<br><br>examples = make_examples.transform(reads_df)</pre><p>The definition of the MakeExamples() stage is very simple, after instantiating the component we pass three options,</p><ul><li>We setMode() to “calling” mode.</li><li>We set the reference path to the location of our reference genome.</li><li>We set the regions to “chr20” for chromosome 20.</li></ul><p>And finally, the most important part, calling the transform() method on our reads. After this transformation takes place we can inspect the resulting examples Dataframe, and for doing so, we will focus on two new columns that we get after this this transformation locus and pileup,</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/465/1*p2JIqFf_9eVURBWCDEAg1A.png" /></figure><p>The locus is the location of the variant within the chromosome, and the pipeup is the pileup image corresponding to the generated example, and that is used for running predictions with the model.</p><p>At this point, the only missing piece is the Variant Calling stage, so let’s see how this is done in the next section.</p><h3>Calling Variants at Scale 🚀</h3><p>At this point our dataframe contains a good number of candidate examples that we want to pass through our model to obtain predictions. Remember that the results we are after can be described as three probabilities as follows,</p><ol><li>Homozygous Reference Probability (REF): This probability indicates the likelihood that the individual at the genomic position carries two copies of the reference allele.</li><li>Heterozygous Variant Probability (HET): This probability indicates the likelihood that the individual at the genomic position carries one copy of the reference allele and one copy of the alternate (variant) allele.</li><li>Homozygous Variant Probability (ALT): This probability indicates the likelihood that the individual at the genomic position carries two copies of the alternate (variant) allele.</li></ol><p>So now let’s see how we can define our Variant Caller Stage,</p><pre>variant_caller = VariantCallerModel()\<br>  .load(&quot;./variant_caller_fast&quot;)\<br>  .setUseGPU(False)<br><br>probabilities = variant_caller.transform(examples)<br>probabilities.limit(20).select(&quot;locus&quot;, &quot;probabilities&quot;).show(truncate=False)</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/973/1*CYAA4ls0n30IpcOLg3CTkA.png" /></figure><p>To understand what’s happening here, we start with the VariantCallerModel class, we use that class to load a pretrained checkpoint variant_caller_fast, into memory.</p><p>Finally we call transform() on our examples, remember this is not triggering any actual computation, it’s a lazy method that will just return a Dataframe on which we can call actions on it.</p><p>In the last line, we pick two columns locus, and probabilities, for which we will display a limited number of rows, just in order not to exhaust the resources of the notebook we’re using to run this code.</p><p>And this is it, we’re running predictions! 🎉</p><h3>Using the GPU 🌪️</h3><p>Enabling the GPU to be used on each of your cluster nodes is as easy as setting the useGPU flag to true ,</p><pre>variant_caller = VariantCallerModel()\<br>  .load(&quot;./variant_caller_fast&quot;)\<br>  .setUseGPU(True)</pre><p>The only thing you need, is to make sure that the partitioning of your Dataframe allows you to use the GPU in a way that you don’t exhaust its memory, for example,</p><pre>examples = examples.repartition(12)</pre><p>this could be a good starting point, for a cluster with 12 nodes, but bigger GPUs will allow you to go with other values, like 24 or more.</p><h3>Conclusions and Final Remarks</h3><p>We have seen how easy and interactive it is to actually use Spark Deep Variant; we loaded reads data, generated examples, and called variants on them. We even showed how GPU is used.</p><p>There are other topics such as Streaming support or Databricks support which we haven’t touched on here. Please stay tuned for a future post on these topics.</p><p>If you want to try this product yourself, just get in touch at info@orangenomix.com, and request a trial license. We are looking forward to hearing from you!</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=e6c2482b577f" width="1" height="1" alt="">]]></content:encoded>
        </item>
    </channel>
</rss>