<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Lorena Derezanin 🧬 on Medium]]></title>
        <description><![CDATA[Stories by Lorena Derezanin 🧬 on Medium]]></description>
        <link>https://medium.com/@lorenaderezanin?source=rss-e7786355f4d------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/1*VkroU9nY2NwJyagHpOkOYg.jpeg</url>
            <title>Stories by Lorena Derezanin 🧬 on Medium</title>
            <link>https://medium.com/@lorenaderezanin?source=rss-e7786355f4d------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Wed, 18 Nov 2026 01:05:01 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@lorenaderezanin/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[Struggling to organize and automate your bioinformatics scripts? Snakemake to the rescue! ]]></title>
            <link>https://medium.com/@lorenaderezanin/struggling-to-organize-and-automate-your-bioinformatics-scripts-snakemake-to-the-rescue-a20abb198ebd?source=rss-e7786355f4d------2</link>
            <guid isPermaLink="false">https://medium.com/p/a20abb198ebd</guid>
            <category><![CDATA[bioinformatics]]></category>
            <category><![CDATA[genomics]]></category>
            <category><![CDATA[automation]]></category>
            <category><![CDATA[scalability]]></category>
            <category><![CDATA[data-analysis]]></category>
            <dc:creator><![CDATA[Lorena Derezanin ]]></dc:creator>
            <pubDate>Thu, 11 May 2023 15:35:42 GMT</pubDate>
            <atom:updated>2023-05-16T17:26:55.368Z</atom:updated>
            <content:encoded><![CDATA[<h4>Learn how to create nice and tidy pipelines and save some time</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*EZXZIARroEiIhaaG" /><figcaption>Photo by <a href="https://unsplash.com/@windows?utm_source=medium&amp;utm_medium=referral">Windows</a> on <a href="https://unsplash.com/?utm_source=medium&amp;utm_medium=referral">Unsplash</a></figcaption></figure><p>So you just got the first batch of sequencing data to analyze for your research project. You’re eager to run an exploratory analysis and plot captivating graphs to show to your supervisor. Recently, you took some courses on bioinformatics and data wrangling but the idea of what you need to do is still somewhat foggy.</p><p>Nevertheless, you dive right in and start typing some code! You pull it off by “piping” a bunch of shell commands or running numerous R scripts sequentially under your meticulous supervision. You sacrificed the whole weekend, but it was worth it, your boss is pleased and your lab mates nod appreciatively!</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/540/0*ZncsHkm00FPRYMAh.gif" /><figcaption>Source: <a href="https://giphy.com/">Giphy</a></figcaption></figure><p>[Phew]… You got away with it this time, but you’re increasingly aware that this was a tedious and <strong>not particularly efficient way of keeping scripts easy to execute on demand </strong>(e.g. for your next lab meeting😲). And what about making your analyses reproducible once your publication is out? Ouch. 😬</p><p>I’ve definitely been there and done all of that! During my first year as a PhD student, I was grappling with the basics of shell scripting, figuring out bioinformatics tools, and digging myself out of dependency hell on a daily basis. It seemed to me that there must be a better way to keep my scripts in check, but I didn’t have a clear idea of what to do about it.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/469/0*uuhH285OstC6-nDC.gif" /><figcaption>Source: <a href="https://giphy.com/">Giphy</a></figcaption></figure><p>Things got way better when I stumbled upon <strong>conda</strong>, a flexible package manager, developed to support programming languages like Python and R, and resolve dependency issues of multiple tools piled up together.</p><blockquote>How many times did you need Python 2.7 for one tool, but Python 3.5 and its compatible packages for the other one? <em>🤯</em></blockquote><p>Conda deploys <strong>isolated environments to install tools and run analyses </strong>which saves a lot of headaches. It pairs well with Snakemake to keep your pipelines as low-maintenance as possible. More on that a bit later.</p><blockquote>If you aren’t already familiar with conda, here’s a <a href="https://docs.conda.io/en/latest/">read</a> to get you up to speed!</blockquote><h3>Sure, but what’s Snakemake?</h3><p>In a nutshell, the Snakemake workflow manager is a tool used to create <strong>reproducible </strong>and<strong> scalable</strong> data analysis workflows or pipelines. The nicest thing about Snakemake is that it’s neat and tidy and pipelines are described via a human-readable, Python-based language.</p><blockquote>If you’re not too familiar with coding in Python yet, fear not! Snakemake uses some very basic Python syntax that we’ll explore more closely in follow-up posts.</blockquote><p>What I would like to share with you here is a <strong>quick guide</strong> on how you can get your pipelines up and <strong>running efficiently</strong> with the least fiddling, so you can focus on other urgent stuff. We all know there’s plenty of that in a life of a PhD student and any <strong>tool that saves time is highly valuable</strong>. So read all about one such tool from a formerly struggling fellow student!</p><h3>A tiny Snakemake pipeline</h3><p>Snakemake pipelines are built with sequential steps called rules. Each rule requires an <strong>input</strong>, an<strong> output</strong>, and a <strong>script to execute</strong> in order to fulfill a task of that particular rule.</p><p>Here is an example of a minimal set of components in a Snakemake pipeline:</p><pre># declare a simple rule<br>rule analyze_data:<br>    input:<br>        &quot;file1.txt&quot;,<br>        &quot;file2.txt&quot;<br>    output:<br>        &quot;output.txt&quot;<br>    shell:<br>        &quot;python analyze_data.py file1.txt file2.txt &gt; output.txt&quot;</pre><p>Now let’s replace the dummy script with a real and familiar example. As bioinformaticians we will or already have worked with sequencing data that requires <strong>removal of adapter sequences</strong> that are added before sequencing.</p><p>There’s a debate about the necessity of adapter removal on <a href="https://www.biostars.org/p/212136/">Biostars forum</a>, but we’ll pretend this step is completely nonnegotiable for us here for the sake of mastering <em>the way of Snek</em> 🐍.</p><p>Our tiny pipeline is using paired-end reads in the form of compressed fastq files as <strong>input </strong>(input.r1and input.r2) for adapter removal with <a href="https://github.com/FelixKrueger/TrimGalore"><em>trim_galore</em></a><em>. </em>Of course, you can modify the script and use another tool if you’d like.</p><pre># declare a rule to trim illumina adapters from paired-end reads<br>rule trim_galore:<br>    input:<br>        r1=&quot;reads/sample_A.1.fastq.gz&quot;, <br>        r2=&quot;reads/sample_A.2.fastq.gz&quot;<br>    output:<br>        &quot;trimmed/sample_A.1_val_1.fq.gz&quot;,<br>        &quot;trimmed/sample_A.1.fastq.gz_trimming_report.txt&quot;,<br>        &quot;trimmed/sample_A.2_val_2.fq.gz&quot;,<br>        &quot;trimmed/sample_A.2.fastq.gz_trimming_report.txt&quot;<br>    shell:<br>        &quot;trim_galore --paired --illumina -o {output} {input.r1} {input.r2}&quot;</pre><p>The <strong>output</strong> of this tiny pipeline consists of validated trimmed reads and reports. The output will land in a directory very sensibly named trimmed. In this case, we assume <em>trim_galore</em> is already installed on our machines and ready to be called to action.</p><h4>Placeholders</h4><p>In the <strong>shell</strong> section we are calling the command to execute the trimming task. Here we are using <strong>placeholders </strong><strong>{input.r1}</strong>, <strong>{input.r2}</strong>and <strong>{output}</strong>.</p><p>These placeholders are going to be <strong>replaced by Snakemake with the full name </strong>of<strong> </strong>each input file<strong> </strong>for {input.r1}and{input.r2}, and with the full name of the output file for the {output}<strong>before running the command</strong>.</p><p>You may have noticed that the name of our sample is rather dull and not particularly logical. In the real world, samples we deal with are named something more like “ERR3989340_S1_L001_R1_001.fastq.gz”. And on top of that, there is often inconsistency in sample naming across different pools or batches of samples.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/500/0*ZEFAAtpmdza1XmjL" /><figcaption>Source: <a href="https://giphy.com/">Giphy</a></figcaption></figure><h3>Batch processing of large datasets</h3><p>When attempting to analyze samples in bulk, we certainly spend quite some time preparing samples before processing! To simplify this step, we can <strong>generalize </strong>the <em>trim_galore</em> rule with <strong>wildcards </strong>to process multiple samples at once.</p><h4>Wildcards</h4><p>Similar to <a href="https://tldp.org/LDP/GNU-Linux-Tools-Summary/html/x11655.htm">wildcards in shell</a>, where asterisk(*), {curly brackets}or [square brackets] are used for pattern matching, <strong>wildcards in Snakemake can be replaced by any text in order to make up the filename.</strong></p><blockquote>Using the same wildcard in the input and output informs Snakemake how to match real names of input files to output files.</blockquote><p>Here’s an updated version of the script with {sample} wildcard.</p><pre># list of sample names<br>SAMPLES=[&quot;Sample_A&quot;, &quot;Sample_B&quot;, &quot;Sample_C&quot;]<br><br># update the rule with sample wildcards<br>rule trim_galore:<br>    input:<br>        r1=expand(&quot;reads/{sample}.1.fastq.gz&quot;, sample=SAMPLES),<br>        r2=expand(&quot;reads/{sample}.2.fastq.gz&quot;, sample=SAMPLES)<br>    output:<br>        &quot;trimmed/{sample}.1_val_1.fq.gz&quot;,<br>        &quot;trimmed/{sample}.1.fastq.gz_trimming_report.txt&quot;,<br>        &quot;trimmed/{sample}.2_val_2.fq.gz&quot;,<br>        &quot;trimmed/{sample}.2.fastq.gz_trimming_report.txt&quot;<br>    shell:<br>        &quot;trim_galore --paired --illumina -o {output} {input.r1} {input.r2}&quot;</pre><p>This little trick with wildcards will enable us to <strong>pass multiple samples as paired input files </strong>from the reads directory. Wildcards allow rules to be written in a more general way, while still being applicable to specific input and output files.</p><p>Let thatexpand function simmer in the back of your head for now, we’ll learn all about it in the upcoming post.</p><h4>Running rules in parallel</h4><p>Depending on the available resources (CPUs and RAM) on your machine or remote cluster, Snakemake will run as many <strong>parallel executions </strong>of the rule <em>trim_galore</em> as possible.</p><p>For example, if you have 50 samples (100 paired R1 and R2 files) in the reads directory - that will lead to 50 instances of this rule to run in parallel, each instance occupying one core. It’s good practice to specify the upper limits, e.g. max. CPUs (--cores) so that the Snakemake run doesn’t eat up all the available resources. 🤓</p><blockquote>The use of wildcards along with out-of-the-box parallel execution of tasks is what makes Snakemake <strong>scalable </strong>when applied on large datasets.</blockquote><p>With that, we’re well underway toward the ultimate automation of complex tasks and repetitive data analyses! We’re basically unstoppable! 🙌🏻</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/480/0*HajhddwYY_NSIU71" /><figcaption>Source: <a href="https://giphy.com/">Giphy</a></figcaption></figure><p>Now let’s add a couple more rules to our Snakemake pipeline. Following the trimming step, we will add a rule to <strong>map our reads to the reference genome </strong>using <a href="https://github.com/bwa-mem2/bwa-mem2"><em>bwa-mem2</em></a><em>.</em></p><p>As our Snakemake pipeline will now grow from a 1-rule to a 2-rule one, we also need to declare <strong>the final output of the pipeline</strong>. We’re adding this in the form of rule allat the beginning of the pipeline where we specify that the bam file containing mapped reads is our final output.</p><pre><br>SAMPLES=[&quot;Sample_A&quot;, &quot;Sample_B&quot;, &quot;Sample_C&quot;]<br><br># add rule all, declare final output<br>rule all:<br>    input:<br>        expand(&quot;mapped/{sample}.bam&quot;, sample=SAMPLES)<br><br><br>rule trim_galore:<br>    input:<br>        r1=&quot;reads/{sample}.1.fastq.gz&quot;, <br>        r2=&quot;reads/{sample}.2.fastq.gz&quot;<br>    output:<br>        &quot;trimmed/{sample}.1_val_1.fq.gz&quot;,<br>        &quot;trimmed/{sample}.1.fastq.gz_trimming_report.txt&quot;,<br>        &quot;trimmed/{sample}.2_val_2.fq.gz&quot;,<br>        &quot;trimmed/{sample}.2.fastq.gz_trimming_report.txt&quot;<br>    shell:<br>        &quot;trim_galore --paired --illumina -o {output} {input.r1} {input.r2}&quot;<br><br><br># add rule bwa mem to map reads to a ref genome<br>rule bwa_mem2:<br>    input:<br>        reads=[<br>        &quot;trimmed/{sample}.1_val_1.fq.gz&quot;, <br>        &quot;trimmed/{sample}.2_val_2.fq.gz&quot;<br>        ],<br>        ref=[<br>            &quot;genome.fasta&quot;, <br>            &quot;genome.amb&quot;, <br>            &quot;genome.ann&quot;, <br>            &quot;genome.bwt.2bit.64&quot;, <br>            &quot;genome.pac&quot;<br>        ]<br>    output:<br>        &quot;mapped/{sample}.bam&quot;<br>    threads: 8<br>    shell:<br>        &quot;bwa-mem2 mem -t {threads} {input.ref} {input.reads} &gt; {output}&quot;</pre><p>Now we’ll add rule sortto our pipeline. To sort mapped reads, we’re going to use <a href="https://lomereiter.github.io/sambamba/"><em>sambamba</em></a> tool.</p><pre><br>SAMPLES=[&quot;Sample_A&quot;, &quot;Sample_B&quot;, &quot;Sample_C&quot;]<br><br># update rule all<br>rule all:<br>    input:<br>        expand(&quot;mapped/{sample}.sorted.bam&quot;, sample=SAMPLES)<br><br><br>rule trim_galore:<br>    input:<br>        r1=&quot;reads/{sample}.1.fastq.gz&quot;, <br>        r2=&quot;reads/{sample}.2.fastq.gz&quot;<br>    output:<br>        &quot;trimmed/{sample}.1_val_1.fq.gz&quot;,<br>        &quot;trimmed/{sample}.1.fastq.gz_trimming_report.txt&quot;,<br>        &quot;trimmed/{sample}.2_val_2.fq.gz&quot;,<br>        &quot;trimmed/{sample}.2.fastq.gz_trimming_report.txt&quot;<br>    shell:<br>        &quot;trim_galore --paired --illumina -o {output} {input.r1} {input.r2}&quot;<br><br><br>rule bwa_mem2:<br>    input:<br>        reads=[<br>            &quot;trimmed/{sample}.1_val_1.fq.gz&quot;, <br>            &quot;trimmed/{sample}.2_val_2.fq.gz&quot;<br>        ],<br>        ref=[<br>            &quot;genome.fasta&quot;, <br>            &quot;genome.amb&quot;, <br>            &quot;genome.ann&quot;, <br>            &quot;genome.bwt.2bit.64&quot;, <br>            &quot;genome.pac&quot;<br>        ]<br>    output:<br>        &quot;mapped/{sample}.bam&quot;<br>    threads: 8<br>    shell:<br>        &quot;bwa-mem2 mem -t {threads} {input.ref} {input.reads} &gt; {output}&quot;<br><br><br># declare rule for sorting bam file<br>rule sambamba_sort:<br>    input:<br>        &quot;mapped/{sample}.bam&quot;<br>    output:<br>        &quot;mapped/{sample}.sorted.bam&quot;<br>    threads: 8<br>    shell:<br>        &quot;sambamba sort -t {threads} --out {output} {input}&quot;</pre><p>You notice we have now updated the rule allto match <strong>our new final output</strong>, a file containing <em>sorted mapped reads</em>. One thing that got me confused when I started working with Snakemake, is that the final output of the pipeline is referred to as an input. Here’s why is that so:</p><blockquote>Snakemake works backwards to figure out if the input of the ‘rule all’ (your final output file) can be generated from the intermediary input files of each rule, all the way back to the very first <strong>input</strong> you gave it.</blockquote><p>It’s not particularly intuitive, I know. 😅</p><p>Still, this is a good move by Snakemake! Because <strong>it checks for inconsistencies in your inputs and outputs</strong>, e.g. <strong>if the input files exist</strong> and <strong>if wildcards can be filled to match inputs of </strong><strong>rules</strong>. If all is correct, only then it will run the shell command and execute the tasks.</p><h4>Give it a try</h4><p>To <strong>run this pipeline</strong>, you’ll need to copy and paste the script in a simple text file named Snakefileand adapt the paths to your directories and files. The recommended way to <a href="https://snakemake.readthedocs.io/en/stable/getting_started/installation.html"><strong>download Snakemake is via Conda</strong></a><strong> </strong>so Snakemake can better deal with software dependencies.</p><p>Once you have Snakemake installed on your machine, you can run the following command in your Terminal, in the directory where the Snakefile lives:</p><pre>snakemake --cores 10 --verbose -s Snakefile</pre><p>This would be a minimal command to charm the Snek and get it to do tasks for you. But before you run it, read a bit further to learn <strong>how to deploy this command in a conda environment</strong> to make your life easier.</p><h3>What about reproducibility?</h3><p>One of the advantages of Snakemake pipelines is that they’re <strong>executable consistently across different computing systems</strong>. What does that mean?🤔</p><p>As mentioned eariler, <strong>conda</strong> is a manager for installing tools, such as Snakemake into <strong>isolated environments. </strong>For example, you could create an environment called “snakemake” and install the <a href="https://anaconda.org/bioconda/snakemake">current version of Snakemake</a> tool in it. Besides Snek, all of its dependencies (e.g. specific version of Python and helper scripts) will get installed. Here’s how to do that:</p><pre># create conda environment named &quot;snakemake&quot;<br>conda create -n snakemake<br><br># activate the environment<br>conda activate snakemake<br><br># install snakemake tool in this env from bioconda channel<br>conda install -c bioconda snakemake<br><br># now go ahead and run that command that calls snakemake and runs the pipeline<br>snakemake --cores 10 --verbose -s Snakefile</pre><blockquote>Such a self-contained environment is <strong>independent</strong> from software installed “outside” of it on your machine. Well, for the most part, to be precise.</blockquote><p>This snakemake environment also contains <strong>a recipe how to recreate it on a different machine with exact versions of tools you used.</strong></p><pre># here is the recipe with software versions I used<br># you can save this to a file &quot;env.yaml&quot;<br>name: snakemake<br>channels:<br>  - conda-forge<br>  - default<br>  - bioconda<br>  - defaults<br>dependencies:<br>  - snakemake-minimal=7.19.1<br>  - python=3.11.0</pre><p>So your collaborator, or another curious researcher, or your archnemesis can try to <strong>reproduce your work</strong>, and possibly trigger an intergalactic war.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/680/0*K0WoHInkyv6wpOFl.gif" /><figcaption>Source: <a href="https://tenor.com/">Tenor</a></figcaption></figure><h3>Conclusion</h3><p>We’re done! For now, at least! Congrats, you just learned some basic concepts around Snakemake. 🎉🎉🎉</p><p>Here’s a quick recap:</p><ol><li><strong><em>Scalability </em></strong>— with Snakemake we can <strong>scale up </strong>from a simple one-step pipeline to a more complex multi-step one while using<strong> wildcards </strong>and <strong>in-built parallel job execution.</strong></li><li><strong><em>Reproducibility </em></strong>— with the help of <strong>conda</strong>, we can create an <strong>isolated environment </strong>to run Snakemake pipeline and make the analysis <strong>reproducible </strong>and <strong>also avoid software dependency hell.</strong></li></ol><p>In the <strong>follow-up post</strong>, you will read about:</p><ul><li><strong>useful parameters</strong> that you can specify when running Snakemake</li><li><strong>how to create a recipe for the environment</strong></li><li><strong>how to execute on a cluster</strong></li><li>use <strong>wrapper scripts</strong></li><li>and also the general <strong>pros and cons</strong> of Snakemake</li></ul><p>If you find this post helpful and you happen to know other <strong>fellow struggling bioinformaticians</strong>, please do <strong>share</strong> it with them, so we can get to that <strong>ultimate pipeline automation</strong> and take over the world!</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/400/0*xngBoTcjFqYDThiD.gif" /><figcaption>Source: <a href="https://getyarn.io/">Yarn</a></figcaption></figure><p><em>Thank you for reading my first Medium post! 🙌🏻 Let me know what you think about it. How can I improve it? Does it need more GIFs? Surely, it needs more GIFs!</em> 😇</p><p><em>If you’re interested in bioinformatics, genomics and beyond, follow me on </em><a href="https://medium.com/@lorenaderezanin"><strong><em>Medium</em></strong></a><em> or find me on</em><strong><em> </em></strong><a href="https://www.linkedin.com/in/derezanin-lorena/"><strong><em>LinkedIn</em></strong></a><em>.</em><strong><em> </em></strong><em>See you around!</em></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=a20abb198ebd" width="1" height="1" alt="">]]></content:encoded>
        </item>
    </channel>
</rss>