<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Jillian Rowe on Medium]]></title>
        <description><![CDATA[Stories by Jillian Rowe on Medium]]></description>
        <link>https://medium.com/@jilliandrowe?source=rss-87b832c70819------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/1*pxZv-rRWDsehL2U2PeQWRw.png</url>
            <title>Stories by Jillian Rowe on Medium</title>
            <link>https://medium.com/@jilliandrowe?source=rss-87b832c70819------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Sun, 22 Nov 2026 21:18:28 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@jilliandrowe/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[Nextflow Hackathon 2026: Adding QC Reports to nf-core/cellpainting (and Why Data Visualization…]]></title>
            <link>https://medium.com/data-science-collective/nextflow-hackathon-2026-adding-qc-reports-to-nf-core-cellpainting-and-why-data-visualization-27a76a939d23?source=rss-87b832c70819------2</link>
            <guid isPermaLink="false">https://medium.com/p/27a76a939d23</guid>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[bioinformatics]]></category>
            <category><![CDATA[high-content-screening]]></category>
            <category><![CDATA[ai]]></category>
            <dc:creator><![CDATA[Jillian Rowe]]></dc:creator>
            <pubDate>Thu, 30 Apr 2026 00:00:22 GMT</pubDate>
            <atom:updated>2026-05-08T07:31:54.787Z</atom:updated>
            <content:encoded><![CDATA[<h3>Nextflow Hackathon 2026: Adding QC Reports to nf-core/cellpainting (and Why Data Visualization Matters for AI)</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1000/0*60LbaH70-UbYPYuA.png" /></figure><h3>Overview</h3><p>I just got back from the Nextflow 2026 Hackathon, and it was exactly what my software engineer soul needed. Two days of focused open-source work, surrounded by people who get genuinely excited about workflow orchestration and reproducible pipelines.</p><p>More importantly one of the projects was based around <strong>cellpainting</strong>.</p><h3>What is Cell Painting?</h3><p><a href="https://en.wikipedia.org/wiki/Cell_painting">Cell Painting</a> is a high-throughput, image-based assay used in drug discovery and biology to analyze cellular changes by staining cells with fluorescent dyes. It highlights up to eight major components — such as the nucleus, ER, and mitochondria — to generate “morphological profiles,” allowing researchers to see how cells respond to drugs or genetic changes.</p><h3>The Hackathon: Open Source is Good for the Soul</h3><p>I submitted a PR to add QC reporting modules to the <a href="https://nf-co.re/cellpainting/dev/">nf-core/cellpainting pipeline</a>, and along the way, I got to play with the Nextflow Quilt plugin for data management. This isn’t just about adding some notebooks to a pipeline. This is about <strong>enriching our data for the AI agents that are enroute</strong>. (As a side note I am in full on existential crisis over my coding myself out of jobs with AI agents. I am a problem.)</p><p>I’ve been wanting to contribute to nf-core for a while. The Cell Painting pipeline was a perfect target because:</p><ul><li>I love cellpainting</li><li>It’s actively being developed</li><li>It handles high-content microscopy data (pretty cells!)</li><li>It was missing some QC reporting that I knew would be useful</li><li>The maintainers were <em>at the hackathon</em> to help review</li></ul><p>The hackathon goal: add automated reporting modules that generate metadata summaries and descriptive statistics from the CytoTable parquet outputs.</p><p>The self-serving goal: Indoctrinate the other bioinformatics pipeline engineers into the cult that is <em>generating lots of reports</em>.</p><h3>What I Built: QC Modules for CytoTable Outputs</h3><p>The Cell Painting pipeline uses <a href="https://github.com/cytomining/CytoTable">CytoTable</a> to convert CellProfiler CSV outputs into analysis-ready parquet files. These parquet files contain morphological features extracted from microscopy images — things like cell size, nuclear intensity, cytoplasm texture, etc.</p><p>But once you have a parquet file, you need to know what’s in it. How is it structured? How do the column names map back to our sample sheet? Do the distributions look reasonable? Are there any obvious data quality issues?</p><p>That’s where my PR comes in. I added two new reporting modules:</p><h4>1. Metadata Extraction Module (cytotable/metadata)</h4><p>Extracts structural information about the parquet file:</p><ul><li>File info (name, size, path)</li><li>DataFrame overview (rows, columns, total data points)</li><li>Column structure by compartment (cells, nuclei, cytoplasm)</li><li>Metadata columns list</li><li>Data types summary</li></ul><p>Outputs:</p><ul><li>Marimo app (.py)</li><li>Structured metadata JSON (.json)</li><li>Optional HTML report</li><li>Optional Jupyter notebook export (.ipynb)</li></ul><h4>2. Descriptive Statistics Module (cytotable/descriptivestats)</h4><p>Generates comprehensive stats and visualizations:</p><ul><li>Summary statistics for ALL numeric columns (count, mean, std, min, quartiles, max)</li><li>High-quality plots (300 DPI PNG):</li><li>Cell, nuclear, and cytoplasm size distributions</li><li>DNA intensity distributions</li><li>Missing values analysis</li><li>Feature categorization by compartment and measurement type</li></ul><p>Outputs:</p><ul><li>Marimo app (.py)</li><li>Statistics parquet file (.parquet)</li><li>Optional PNG plots (300 DPI)</li><li>Optional HTML report</li><li>Optional Jupyter notebook export (.ipynb)</li></ul><p>Both modules use <a href="https://marimo.io/">marimo</a> to create reactive Python notebooks. They’re reproducible, automated, and generate both human-readable reports and machine-readable outputs.</p><p>Here’s what the workflow looks like now:</p><pre>CELLPROFILER_ANALYSIS → CYTOTABLE → CYTOTABLE_METADATA<br>                                   → CYTOTABLE_DESCRIPTIVESTATS</pre><h3>Plot Twist: Discovering Marimo at the Hackathon</h3><p>Here’s where the hackathon paid dividends. I initially built these modules using Papermill and Jupyter notebooks. Standard stuff. Works fine, but notebook make for terrible git diffs. I was asked if I’d heard of Marimo.</p><p>I had not!</p><p><a href="https://marimo.io/">Marimo</a> is a reactive Python notebook framework that generates .py files instead of .ipynb files.</p><p>Learning about <a href="https://marimo.io/">Marimo</a> led me down a fun cascade. I learned about <a href="https://seqera.io/containers/">Seqera containers</a>. No need to build my own containers!</p><h3>Reports, reports, reports</h3><p>As a <em>quick side rant</em> I am a big believer in generating reports for all the files. A big part of my job is to take on pipelines from scientists. The scientists are the domain experts and they drive the design of the pipelines.</p><p>Understanding the file types, the structure, and some basic descriptive statistics is always helpful. For me, for the scientists.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1000/0*7AVrjwuR1ZPn0kqL.png" /></figure><p>Reports are an excellent communication tool. The screenshot above was produced with the Nextflow powered marimo module. The html report is <em>very</em> nicely displayed in Quilt. We have plenty of tech problems, but we have considerably more people problems. Make demos. Build visualizations. Go talk to some humans and not just the AI!</p><p><strong>The Module</strong></p><p>Running the module in the Nextflow pipeline looks like this:</p><pre>script:<br>&quot;&quot;&quot;<br>marimo export html ${prefix}_stats.py \\<br>    -o ${prefix}_stats.html \\<br>    -- \\<br>    --parquet-file ${parquet_file} \\<br>    --output-prefix ${prefix}<br>&quot;&quot;&quot;</pre><p>The marimo export html command executes the app and generates the HTML report in one shot. The HTML reports is an artifact that a scientist can view, or that we can display in an interface like <a href="https://www.quilt.bio/">Quilt.bio</a>.</p><h3>The Quilt Side: Why Data Cataloging Matters</h3><p>The OSS version of Quilt is a real fancy S3 browser that displays html, ipynb along with csv and parquets as nicely rendered tables.</p><p>I ran the updated pipeline using the <strong>Nextflow Quilt plugin</strong>, which pushes all outputs to a Quilt data package on S3. The Quilt UI is a tool I love handing to scientists.</p><p>Look at this:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1000/0*p4p1vsW-Pi_hjvCf.png" /></figure><p>Quilt’s homepage showing cellpainting package</p><p>Pushing our Nextflow pipeline results gives us a <strong>data catalog</strong> that:</p><ul><li>Renders Jupyter notebooks in-browser</li><li>Displays Markdown READMEs with full formatting</li><li>Shows HTML reports directly</li><li>Previews parquet files and CSVs with interactive tables</li><li>Displays images and plots</li><li>Provides a search and query interface</li></ul><p>And here’s the stats report, rendered beautifully:</p><figure><img alt="Quilt’s descriptive statistics report" src="https://cdn-images-1.medium.com/max/1000/0*zzorhdaQ4ffZr1fC.png" /></figure><p>See those plots? Those size distributions? The summary statistics? All rendered directly in the browser. No downloading files from S3. No setting up Cyberduck! No “let me spin up a Jupyter server.” Just navigate to the package and explore.</p><h3>How Quilt Packages Map to Nextflow Modules</h3><p>Here’s a peek at the generated packages:</p><figure><img alt="Quilt’s Nextflow packages" src="https://cdn-images-1.medium.com/max/1000/0*cTrHzK9t_HSohFd9.png" /></figure><p>Each run creates a versioned package with all outputs organized by stage.</p><p>One of the clever things the Quilt plugin does is automatically organize outputs by module. The Quilt package structure mirrors the Nextflow module structure, which makes it trivial to understand what came from where.</p><p>Here’s the mapping:</p><pre>Quilt Package Name                    ←→  Nextflow Module Path<br>─────────────────────────────────────────────────────────────────────────<br>cellprofiler/illumination_correction  ←→  modules/local/cellprofiler/illuminationcorrection/<br>cellprofiler/assay_development        ←→  modules/local/cellprofiler/assaydevelopment/<br>cellprofiler/analysis                 ←→  modules/local/cellprofiler/analysis/<br>cytotable/default_suffix              ←→  modules/local/cytotable/<br>multiqc/default_suffix                ←→  modules/nf-core/multiqc/</pre><p>And the new QC reporting modules I added will also show up in their own packages:</p><pre>cytotable_metadata/default_suffix     ←→  modules/local/reports/cytotable/metadata/<br>cytotable_descriptivestats/...        ←→  modules/local/reports/cytotable/descriptivestats/</pre><p>This means when you’re browsing the Quilt catalog, you can immediately trace each output back to the exact Nextflow module, or the custom output path, that generated it.</p><p>The package structure looks like this in the Quilt UI:</p><pre>📦 your-bucket/cellpainting-run-2026-04-30/<br>├── 📁 cellprofiler/<br>│   ├── 📁 illumination_correction/<br>│   │   └── 📄 Batch1_Plate1_IllumAGP.npy<br>│   │   └── 📄 Batch1_Plate1_IllumDNA.npy<br>│   │   └── ...<br>│   ├── 📁 assay_development/<br>│   │   └── 📄 QC_output_A01_site1.png<br>│   │   └── ...<br>│   └── 📁 analysis/<br>│       └── 📄 Batch1_Plate1_A01_1_Image.csv<br>│       └── 📄 Batch1_Plate1_A01_1_Cells.csv<br>│       └── ...<br>├── 📁 cytotable/<br>│   └── 📄 Batch1_Plate1_A01_1.parquet<br>│   └── 📄 Batch1_Plate1_A02_1.parquet<br>│   └── ...<br>├── 📁 cytotable_metadata/<br>│   └── 📄 Batch1_Plate1_A01_1_metadata.py<br>│   └── 📄 Batch1_Plate1_A01_1_metadata.json<br>│   └── 📄 Batch1_Plate1_A01_1_metadata.html<br>│   └── ...<br>├── 📁 cytotable_descriptivestats/<br>│   └── 📄 Batch1_Plate1_A01_1_stats.py<br>│   └── 📄 Batch1_Plate1_A01_1_descriptive_stats.parquet<br>│   └── 📄 Batch1_Plate1_A01_1_size_distributions.png<br>│   └── ...<br>└── 📁 multiqc/<br>    └── 📄 multiqc_report.html<br>    └── 📄 multiqc_data/</pre><p>Each package is versioned, so you can track changes across pipeline runs. Click on a package, and you see all the files from that stage. Click on a file, and Quilt renders it (if it’s a supported format) or lets you download it.</p><h3>Future Work — Context Enriching for AI Agents</h3><p>I saw this post from Simon at <a href="https://www.quilt.bio/">Quilt</a> and it got me thinking about my work on AI agents.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1000/0*c9Jvf-UwQt5mH9Gc.png" /></figure><p>How you never want to just leave the AI without adult supervision, but that you want to give it as much context as you can. Of course, you can’t fit multi GiB files into the context window, but you could fit a lot of enriched metadata about each file into the context window!</p><p>We’re rapidly moving toward a world where <strong>AI agents will be the primary consumers of our data</strong>. Not just human scientists manually exploring CSVs. Not just dashboards with pre-baked visualizations. But autonomous agents that need to:</p><ul><li>Understand what data is available</li><li>Determine if it’s high quality</li><li>Extract relevant features</li><li>Make decisions about next steps</li></ul><p>For this to work, our data needs to be <strong>enriched</strong>. It needs:</p><h3>1. Structured Metadata</h3><p>The metadata JSON tells an agent exactly what’s in the parquet file without opening it:</p><pre>{<br>  &quot;file_info&quot;: {<br>    &quot;filename&quot;: &quot;2021_04_26_Batch1_BR00117035_A01_1.parquet&quot;,<br>  },<br>  &quot;dataframe_info&quot;: {<br>    &quot;rows&quot;: 1543,<br>    &quot;columns&quot;: 2847,<br>    &quot;total_data_points&quot;: 4391721<br>  },<br>  &quot;column_structure&quot;: {<br>    &quot;Cells&quot;: 1421,<br>    &quot;Nuclei&quot;: 712,<br>    &quot;Cytoplasm&quot;: 714<br>  }<br>}</pre><p>An agent can read this and immediately know: “This is a Cell Painting dataset with 1543 cells, extracted features from three compartments, and it’s about 12 MB.”</p><h3>2. Statistical Summaries</h3><p>The descriptive stats parquet gives the agent a numerical understanding of the data:</p><ul><li>Which features have high variance?</li><li>Are there any suspicious outliers?</li><li>What’s the dynamic range of each measurement?</li></ul><p>This is the difference between an agent blindly running analysis and an agent making informed decisions.</p><h3>3. Visual Representations</h3><p>Humans are visual. Agents are learning to be. Those PNG plots aren’t just for us — they’re anchors for understanding distributions, spotting anomalies, and validating expectations.</p><p>When an agent can “see” that DNA intensity distributions are bimodal, it can infer that there’s likely a biological signal (maybe cell cycle phase) worth investigating.</p><h3>4. Rich README Context</h3><p>With Quilt, we can include README.md files in our data packages that explain:</p><ul><li>What this experiment was</li><li>What the columns mean</li><li>Known data quality issues</li><li>How to interpret the results</li></ul><p>Agents can read these READMEs and <strong>gain context</strong> while scientists ask questions. This is huge.</p><h3>The Quilt Plugin: Making Data Management Actually Pleasant</h3><p>Running the pipeline with Quilt was surprisingly smooth. The Nextflow Quilt plugin handles the heavy lifting of pushing outputs to S3 and creating the data package.</p><h3>The Bigger Picture: Data Enrichment as Science</h3><p>We spend so much time building pipelines to generate data. But then we dump that data in S3 with minimal context, and wonder why it’s hard to reuse. Specifically, pipeline engineers are known for handing results of a run off to scientists and skipping off into the sunset. With AI tools and context enrichment, we can begin to allow scientists to ask real questions of their data.</p><p>Adding metadata. Generating statistics. Creating visualizations. Writing READMEs. These aren’t “nice to haves” — they’re <strong>essential for making data usable</strong> at scale.</p><p>And with AI agents on the horizon, this becomes even more critical. An agent can’t ask clarifying questions about your data. It can’t intuit what you meant. It needs:</p><ul><li>Explicit schema information</li><li>Statistical summaries</li><li>Visual representations</li><li>Contextual documentation</li></ul><p>If we’re serious about making biology data-driven, we need to invest in making our data <strong>discoverable, understandable, and enriched</strong>.</p><h3>What’s Next</h3><p>The PR is submitted and under review. Once it’s merged, the nf-core/cellpainting pipeline will automatically generate these QC reports for every run.</p><p>I’m also planning to extend the Quilt integration to include:</p><ul><li>Automatic README generation with run metadata</li><li>Experiment metadata enrichment with papers, github repos, and any other resource I can get my hands on</li><li>Cross-plate QC aggregation</li><li>Outlier detection reports</li><li>Links back to the Nextflow execution report</li></ul><p>And I’m thinking about building a “data enrichment” module that can be plugged into any nf-core pipeline to automatically:</p><ol><li>Generate schema documentation</li><li>Calculate summary statistics</li><li>Create standard visualizations</li><li>Build a Quilt package with README</li></ol><h3>Key Takeaways</h3><p>✅ <strong>Contribute to open source.</strong> It’s good for your skills, good for your soul, and good for the community.</p><p>✅ <strong>Be open to new tools.</strong> I learned about marimo because a maintainer suggested it. Now I’ll use it in future projects.</p><p>✅ <strong>Automate your QC.</strong> Don’t manually check data quality. Build reporting into your pipeline.</p><p>✅ <strong>Make outputs machine-readable.</strong> JSON, parquet, and structured formats beat plain text every time.</p><p>✅ <strong>Use Quilt (or similar) for data cataloging.</strong> A web UI for browsing data packages is a game-changer.</p><p>✅ <strong>Enrich your data.</strong> Add metadata, statistics, visualizations, and documentation. Your future self (and future AI agents) will thank you.</p><p><em>If you’re interested in the nf-core/cellpainting pipeline, check out the </em><a href="https://nf-co.re/cellpainting/dev/"><em>official docs</em></a><em>. If you want to learn more about Quilt, visit </em><a href="https://quiltdata.com/"><em>quiltdata.com</em></a><em>. And if you haven’t tried </em><a href="https://marimo.io/"><em>marimo</em></a><em>, it’s worth checking out for any data analysis workflow.</em></p><p><em>If you’re thinking about attending a Nextflow hackathon: do it. You’ll learn new tools, meet great people, and contribute something useful. What’s not to love?</em></p><p><em>Next up: How to build AI agents that actually understand your bioinformatics data (hint: it starts with better data enrichment).</em></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=27a76a939d23" width="1" height="1" alt=""><hr><p><a href="https://medium.com/data-science-collective/nextflow-hackathon-2026-adding-qc-reports-to-nf-core-cellpainting-and-why-data-visualization-27a76a939d23">Nextflow Hackathon 2026: Adding QC Reports to nf-core/cellpainting (and Why Data Visualization…</a> was originally published in <a href="https://medium.com/data-science-collective">Data Science Collective</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Speeding Up UMI-tools with Cython: A 4.6x Performance Win]]></title>
            <link>https://medium.com/@jilliandrowe/speeding-up-umi-tools-with-cython-a-4-6x-performance-win-fe08891a6ef5?source=rss-87b832c70819------2</link>
            <guid isPermaLink="false">https://medium.com/p/fe08891a6ef5</guid>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[bioinformatics]]></category>
            <dc:creator><![CDATA[Jillian Rowe]]></dc:creator>
            <pubDate>Mon, 27 Apr 2026 00:00:46 GMT</pubDate>
            <atom:updated>2026-04-27T20:53:04.040Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1000/0*IMzoPSA0bJzgifIT.png" /></figure><h3>Overview</h3><p>You know that feeling when you’re running a bioinformatics pipeline and you watch the progress bar crawl along at a glacial pace? You grab a coffee, check your email, maybe start a second analysis on another dataset… and 90 seconds later your first job is <em>still</em> processing the same FASTQ file.</p><p>That was me with UMI-tools whitelist. Don&#39;t get me wrong—UMI-tools is a fantastic, battle-tested tool for single-cell RNA-seq analysis, and I love using <em>other people&#39;s code</em>. But when you&#39;re iterating on analysis parameters or processing dozens of samples, those 85-second runs start to add up. My patience is a very finite resource!</p><p>So I did what any reasonable person would do: I rewrote the hot paths in Cython and got a <strong>4.6x speedup</strong> (85.9s → 18.8s) without changing a single line of the actual algorithm. Same inputs, same outputs, just adding in cython to our project gets us a considerable speedup.</p><p>In this post, I’ll walk you through how I identified what to optimize, give you a quick Cython primer, and show you the specific changes that made the difference. By the end, you’ll know exactly how to apply these techniques to your own Python bottlenecks.</p><h3>Why UMI-tools? Why whitelist?</h3><p>If you work with single-cell RNA-seq data, you’ve probably used UMI-tools at some point. The whitelist command is particularly critical—it takes millions of raw cell barcodes from your FASTQ file and figures out which ones are &quot;real&quot; cells versus background noise or sequencing errors.</p><p>(More importantly, its the job I was hired to do.)</p><p>For a typical 10x Genomics dataset:</p><ul><li><strong>Input:</strong> 7.2 million reads with 194,000 unique barcodes</li><li><strong>Task:</strong> Identify the top 100 true cells, correct sequencing errors</li><li><strong>Output:</strong> A whitelist of valid barcodes with error-correction mappings</li></ul><p>This is a <em>perfect</em> candidate for optimization because:</p><ol><li>It’s pure computation (no I/O waiting)</li><li>It runs on every single-cell dataset (high-impact)</li><li>The algorithm is sound — we just need it to run faster</li></ol><h3>Finding the Hot Paths: Profile Before You Optimize</h3><p>Before you start optimizing, you need to know <em>what</em> to optimize. The cardinal rule: <strong>never guess, always profile</strong>.</p><p>I threw cProfile at the UMI-tools code and got this breakdown:</p><pre># Rough profiling breakdown (using cProfile + pstats)<br>Total runtime: 85.9 secondsTop functions by cumulative time:<br><br>1. FASTQ parsing + barcode counting: 76.0s (88%)<br>2. Error correction mapping: 8.6s (10%)<br>3. Knee detection: 1.3s (2%)<br>4. Everything else: ~0.1s (&lt;1%)</pre><p>Three functions accounted for 99% of the runtime. These are your hot paths — the only things worth optimizing.</p><p><strong>Key insight:</strong> The FASTQ parsing loop runs 7.2 million times. The Hamming distance function gets called ~194 million times inside the error-correction BK-tree. Even tiny per-call overhead adds up when you’re doing millions of iterations.</p><p>This is <em>exactly</em> the kind of code that benefits from Cython: tight loops with predictable types, called millions of times.</p><h3>What to Look For When Optimizing Code</h3><p>Here’s my mental checklist when I’m hunting for optimization targets:</p><h3>1. Tight Loops with Many Iterations</h3><pre># ❌ Python overhead on 7 million iterations<br>for read in fastq_file:<br>    cell_barcode = extract_barcode(read)  # function call overhead<br>    counter[cell_barcode] += 1            # dict lookup</pre><p>Every function call, every generator yield, every dictionary operation has Python overhead. When you’re doing it millions of times, that overhead becomes your bottleneck.</p><h3>2. String Operations in Hot Paths</h3><pre># ❌ Called 194 million times in the BK-tree<br>def hamming_distance(first, second):<br>    return sum([not a == b for a, b in zip(first, second)])</pre><p>This innocent-looking function allocates a list, creates a zip iterator, and does a comprehension — all in pure Python. Cython can replace this with a tight C loop that compares raw bytes.</p><h3>3. Repeated NumPy Allocations</h3><pre># ❌ Inside a convergence loop (10-15 iterations)<br>for iteration in range(max_iterations):<br>    distances = np.vstack(...)      # allocate<br>    repeated = np.repmat(...)        # allocate<br>    result = np.outer(...)           # allocate</pre><p>NumPy is fast for vectorized operations, but allocating new arrays repeatedly? That’s slow. Cython memoryviews let you iterate over arrays without creating intermediates.</p><h3>4. Type-Ambiguous Code</h3><p>Python’s dynamic typing is convenient, but it means the interpreter has to figure out types at runtime:</p><pre># Python has to check types on every iteration<br>for i in range(len(barcodes)):<br>    if barcodes[i] == target:  # type checks<br>        count += 1              # type checks</pre><p>If you know i is always an integer and count is always an integer, why make Python check every time?</p><p>(<strong>NOTE</strong> — this is one of the reasons why Cython is so much faster. You do need to add some type annotations, but once you’re there you get a considerable speed up.)</p><h3>Cython Primer: The Bare Minimum You Need to Know</h3><p>Cython sits in the sweet spot between “I want to write Python” and “I need C performance.” You write Python-like code, but with type annotations that let Cython compile it to C.</p><p>I’m a big believer in the magic of Cython. I can get all of my nice Python-y code. I love typer and the rest of the python ecosystem is so convenient. Most of the code that comes to me is written in Python, and I wouldn’t want to have to rewrite every line in C++. Typically I’ll benchmark the code, find a slow spot, and rewrite that function in Cython. Often I don’t even need to do much. Just the compilation step gets me the speedup I need.</p><h3>The Three Function Types</h3><p>Depending on your needs you write Cython in several ways. The easiest is to use the Python implementation with def, but if you have a function that lives in a deep dark cave, you can write it in pure Cython using cdef. Since we&#39;re not translating from Python &lt;-&gt; C we get a considerable speed up.</p><p>Most often I use cpdef, which is the mixing of the two.</p><p><em>Side Note</em>: Py_ssize_t is basically the int type. Its just a special int type the compiler translates to sane instructions so we don&#39;t have to care.</p><h4>1. def — Python-accessible functions</h4><pre>def hamming_distance(str first, str second):<br>    &quot;&quot;&quot;Can be called from Python code.&quot;&quot;&quot;<br>    # ... implementation</pre><p>Use def when:</p><ul><li>You need to call this function from regular Python code</li><li>It’s part of your public API</li><li>You’re okay with a tiny bit of Python/C conversion overhead</li></ul><h4>2. cdef — Internal C functions</h4><pre>cdef int _fast_count(char* sequence, int length):<br>    &quot;&quot;&quot;Pure C function, not accessible from Python.&quot;&quot;&quot;<br>    # ... implementation</pre><p>Use cdef when:</p><ul><li>This function is only called from other Cython code</li><li>You want maximum speed (no Python wrapper)</li><li>You don’t need to call it from regular Python</li></ul><h4>3. cpdef — Both worlds</h4><pre>cpdef int count_bases(str sequence):<br>    &quot;&quot;&quot;Callable from Python, but uses C internally.&quot;&quot;&quot;<br>    # ... implementation</pre><p>Use cpdef when:</p><ul><li>You want both Python accessibility AND C speed</li><li>Cython will create a C version for internal calls and a Python wrapper for external calls</li></ul><h3>Type Annotations: Where the Magic Happens</h3><pre># Without types (pure Python speed)<br>def slow_hamming(first, second):<br>    dist = 0<br>    for i in range(len(first)):<br>        if first[i] != second[i]:<br>            dist += 1<br>    return dist<br><br># With types (C speed)<br>def fast_hamming(str first, str second):<br>    cdef Py_ssize_t dist = 0<br>    cdef Py_ssize_t i<br>    cdef Py_ssize_t n = len(first)<br>    <br>    for i in range(n):<br>        if first[i] != second[i]:<br>            dist += 1<br>    return dist</pre><p><strong>The difference:</strong></p><ul><li>slow_hamming: Python checks types on every iteration</li><li>fast_hamming: Compiles to a tight C loop with no type checks</li></ul><p>Common type declarations:</p><ul><li>cdef int x — C integer</li><li>cdef Py_ssize_t i — Python-sized integer (for indices)</li><li>cdef str s — Python string (still a Python object, but declared)</li><li>cdef double[:] — Memory view of a double array (no copies!)</li></ul><h3>Compiler Directives: Safety Off, Speed On</h3><pre># cython: language_level=3, boundscheck=False, wraparound=False</pre><ul><li>boundscheck=False: Don&#39;t check if array indices are in bounds (assume you got it right)</li><li>wraparound=False: Disable Python-style negative indexing (list[-1])</li></ul><p>These are <em>dangerous</em> if you have bugs in your indexing, but they eliminate runtime checks. Use them only after your code is working and tested.</p><h3>The Three Optimizations</h3><p>Let me walk you through the three hot-path functions I rewrote and exactly what made them faster.</p><h3>1. FASTQ Parsing + Barcode Counting (4.7x faster)</h3><p><strong>What it does:</strong> Read 7.2 million FASTQ records, extract the 16-character cell barcode from each read, count occurrences.</p><p><strong>Original Python approach:</strong></p><pre># Three separate steps with overhead between each<br>def analyze_fastq_original(filepath, pattern):<br>    counter = Counter()<br>    <br>    # Step 1: Generator yielding FASTQ records<br>    for read in fastqIterate(filepath):<br>        # Step 2: Extract barcode via method call<br>        barcode = getBarcodes(read, pattern)<br>        # Step 3: Update counter<br>        counter[barcode] += 1<br>    <br>    return counter</pre><p><strong>Problems:</strong></p><ul><li>Generator overhead (fastqIterate yields through multiple layers)</li><li>Method call overhead (getBarcodes dispatched every read)</li><li>Dictionary operations in pure Python</li></ul><p><strong>Cython optimization:</strong></p><pre># _fastq_parser.pyx<br>def fastq_parse_and_count(object infile, str pattern, str method=&quot;reads&quot;, <br>                          Py_ssize_t subset_reads=100_000_000):<br>    cdef Py_ssize_t cell_len = 0<br>    cdef Py_ssize_t umi_start = 0<br>    cdef Py_ssize_t pattern_len = len(pattern)<br>    <br>    # Pre-compute barcode length from pattern<br>    cdef Py_ssize_t i<br>    for i in range(pattern_len):<br>        if pattern[i] == &#39;C&#39;:<br>            cell_len += 1<br>    <br>    umi_start = cell_len<br>    <br>    cdef Py_ssize_t n_reads = 0<br>    cdef Py_ssize_t n_valid = 0<br>    <br>    cell_barcode_counts = collections.Counter()<br>    <br>    cdef str line1, line2, line3, line4<br>    cdef str seq, cell<br>    <br>    # Fused loop: read + extract + count in one pass<br>    while True:<br>        line1 = infile.readline()<br>        if not line1:<br>            break<br>        line2 = infile.readline()<br>        line3 = infile.readline()<br>        line4 = infile.readline()<br>        <br>        n_reads += 1<br>        <br>        # Direct string slicing with typed indices<br>        seq = line2.strip()<br>        if len(seq) &lt; pattern_len:<br>            continue<br>            <br>        cell = seq[:cell_len]<br>        cell_barcode_counts[cell] += 1<br>        n_valid += 1<br>        <br>        if n_valid &gt;= subset_reads:<br>            break<br>    <br>    return cell_barcode_counts, n_reads, n_valid</pre><p><strong>Why it’s faster:</strong></p><ul><li><strong>Fused loop:</strong> Read → extract → count happens in one pass</li><li><strong>Typed indices:</strong> cdef Py_ssize_t i compiles to a C integer</li><li><strong>Direct slicing:</strong> seq[:cell_len] becomes pointer arithmetic</li><li><strong>No generator overhead:</strong> Direct readline() calls</li></ul><p><strong>Result:</strong> 76.0s → 16.2s (4.7x faster)</p><h3>2. Hamming Distance (3.3x faster in context)</h3><p><strong>What it does:</strong> Compare two barcode strings and count mismatches. Called ~194 million times inside the BK-tree during error correction.</p><p><strong>Original Python:</strong></p><pre>def hamming_distance(first, second):<br>    &quot;&quot;&quot;Pure Python with list comprehension.&quot;&quot;&quot;<br>    return sum([not a == b for a, b in zip(first, second)])</pre><p><strong>Problems:</strong></p><ul><li>zip() creates an iterator</li><li>List comprehension allocates memory</li><li>sum() iterates over the list</li><li>All in Python with type checks</li></ul><p><strong>Cython optimization:</strong></p><pre># _hamming.pyx<br>def hamming_distance(str first, str second):<br>    cdef Py_ssize_t n = len(first)<br>    if n != &lt;Py_ssize_t&gt;len(second):<br>        return float(&#39;inf&#39;)<br>    <br>    cdef Py_ssize_t dist = 0<br>    cdef Py_ssize_t i<br>    <br>    # Tight C loop, no allocations<br>    for i in range(n):<br>        if first[i] != second[i]:<br>            dist += 1<br>    <br>    return dist</pre><p><strong>Why it’s faster:</strong></p><ul><li><strong>No allocations:</strong> Just a counter and a loop</li><li><strong>Typed loop:</strong> i is a C integer, no Python overhead</li><li><strong>Direct character comparison:</strong> Compiles to char comparison</li><li><strong>Early exit:</strong> Returns inf immediately for length mismatch</li></ul><p><strong>Result:</strong> Called 194 million times, each call ~3x faster → 8.6s total → 2.6s total (3.3x faster)</p><h3>3. Knee Detection (13x faster)</h3><p><strong>What it does:</strong> Find the “elbow” in a curve by computing perpendicular distances. Involves iterative refinement with matrix operations.</p><p><strong>Original Python:</strong></p><pre>def getKneeDistance(values, x, y):<br>    # Build matrices on every iteration<br>    xy_data = np.vstack([x, y]).T<br>    first_point = xy_data[0]<br>    line_vec = xy_data[-1] - first_point<br>    line_vec_norm = line_vec / np.linalg.norm(line_vec)<br>    <br>    vec_from_first = xy_data - np.repmat(first_point, xy_data.shape[0], 1)<br>    scalar_product = np.sum(vec_from_first * np.repmat(line_vec_norm, xy_data.shape[0], 1), axis=1)<br>    <br>    # ... more matrix operations</pre><p><strong>Problems:</strong></p><ul><li>np.vstack allocates a new array</li><li>np.repmat creates repeated copies of data</li><li>np.outer creates another temporary matrix</li><li>All of this happens 10–15 times in the convergence loop</li></ul><p><strong>Cython optimization:</strong></p><pre># _knee.pyx<br>def knee_distance(double[:] x, double[:] y):<br>    cdef Py_ssize_t n = x.shape[0]<br>    cdef double x0 = x[0]<br>    cdef double y0 = y[0]<br>    cdef double x1 = x[n-1]<br>    cdef double y1 = y[n-1]<br>    <br>    # Pre-compute line parameters<br>    cdef double dx = x1 - x0<br>    cdef double dy = y1 - y0<br>    cdef double line_length = (dx*dx + dy*dy)**0.5<br>    <br>    cdef double[:] distances = np.empty(n, dtype=np.float64)<br>    cdef Py_ssize_t i<br>    <br>    # Single pass over data<br>    for i in range(n):<br>        # Perpendicular distance formula<br>        distances[i] = abs((dy * (x[i] - x0) - dx * (y[i] - y0)) / line_length)<br>    <br>    return np.asarray(distances)</pre><p><strong>Why it’s faster:</strong></p><ul><li><strong>Memory views:</strong> double[:] is a typed view, no copies</li><li><strong>Pre-computed constants:</strong> Line parameters computed once</li><li><strong>Single loop:</strong> One pass instead of multiple matrix operations</li><li><strong>No temporary arrays:</strong> Direct computation into result array</li></ul><p><strong>Result:</strong> 1.3s → 0.1s (13x faster)</p><h3>The Results: 4.6x Faster, Same Output</h3><p>Here’s the final breakdown:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/502/1*biw7PqtEcA-4QXDfeSIDGg.png" /></figure><p><strong>Memory usage:</strong> Basically unchanged (~30 MB)</p><p><strong>Output verification:</strong> All 21 test cases pass — byte-for-byte identical output to the original.</p><h3>Running the Optimized Version</h3><p>The full source code is available on GitHub: <a href="https://github.com/dabble-of-devops-bioanalyze/umitools-cython">dabble-of-devops-bioanalyze/umitools-cython</a></p><p>Installation is dead simple:</p><pre>git clone https://github.com/dabble-of-devops-bioanalyze/umitools-cython.git<br>cd umitools-cython<br>pip install -e .</pre><p>Then run it just like UMI-tools:</p><pre># Cython-accelerated version<br>umitools-cy whitelist \<br>    --stdin data/hgmm_100_R1.fastq.gz \<br>    --bc-pattern CCCCCCCCCCCCCCCCNNNNNNNNNN \<br>    --set-cell-number 100</pre><p><strong>Output:</strong> 18.8 seconds</p><p>Compare to the original:</p><pre># Original UMI-tools<br>umi_tools whitelist \<br>    --stdin data/hgmm_100_R1.fastq.gz \<br>    --bc-pattern=CCCCCCCCCCCCCCCCNNNNNNNNNN \<br>    --set-cell-number=100 \<br>    --log2stderr &gt; whitelist.txt</pre><p><strong>Output:</strong> 85.9 seconds</p><p>Same inputs, same outputs, 4.6x faster. No downsides.</p><h3>Lessons Learned: When Cython Wins</h3><p>After this project, I’ve got a pretty clear mental model for when Cython is the right tool:</p><h3>✅ Use Cython When:</h3><ol><li><strong>You have tight loops</strong> — Anything called thousands or millions of times</li><li><strong>Types are predictable</strong> — You know i is always an int, s is always a string</li><li><strong>You’re doing string operations</strong> — Character-level operations in hot paths</li><li><strong>You want Python compatibility</strong> — Drop-in replacements for existing code</li><li><strong>NumPy is allocating too much</strong> — Intermediate arrays that could be views</li></ol><h3>❌ Skip Cython When:</h3><ol><li><strong>It’s I/O-bound</strong> — Network calls, disk reads, database queries (Cython won’t help)</li><li><strong>It’s already using C libraries</strong> — If you’re calling NumPy/Pandas/Polars/PyArrow, those are already in C</li><li><strong>Types are highly dynamic</strong> — If you can’t declare types, Cython can’t optimize</li><li><strong>It’s rarely called</strong> — One-time setup code doesn’t benefit from optimization</li></ol><h3>Alternative Approaches (And Why I Didn’t Use Them)</h3><p><strong>NumPy vectorization:</strong> Limited here — we’re parsing text, not doing linear algebra. PyArrow would help for batch operations, but the hot path is still parsing.</p><p><strong>PyPy JIT:</strong> Incompatible with pybktree, which has C extensions. Would give ~1.5-2x at best.</p><p><strong>Rust + PyO3:</strong> Probably 5–10x faster, but requires:</p><ul><li>Learning Rust (high barrier)</li><li>Complex build system</li><li>FFI layer maintenance</li></ul><p><strong>Raw C extension:</strong> Same speed as Cython, but way more painful to write and maintain.</p><p><strong>Cython hits the sweet spot:</strong> Python-like syntax, compiles to C, works with existing Python packages, 4.6x speedup. That’s a win, and I’m happy to move on with my day.</p><h3>Key Takeaways</h3><ul><li><strong>Profile first, optimize second</strong> — Don’t guess what’s slow, measure it</li><li><strong>Hot paths matter most</strong> — 3 functions = 99% of runtime</li><li><strong>Cython is practical</strong> — Python syntax, C speed, minimal learning curve</li><li><strong>Type annotations unlock performance</strong> — cdef variables compile to C types</li><li><strong>Tight loops benefit most</strong> — Millions of iterations = huge gains</li><li><strong>Correctness is non-negotiable</strong> — Test everything, verify outputs match</li></ul><p>This project took me from “UMI-tools is slow” to “UMI-tools is 4.6x faster” in a few hours of focused work. Same algorithm, same outputs, just way less waiting around.</p><p>If you’ve got a Python bottleneck in your bioinformatics pipeline, Cython is probably the right tool for the job. You don’t need to become a C expert — you just need to find your slowest functions and add some type annotations.</p><p>Then watch those benchmarks fly!</p><h3>References</h3><ul><li><a href="https://github.com/dabble-of-devops-bioanalyze/umitools-cython">umitools-cython GitHub Repository</a> — Full source code and benchmarks</li><li><a href="https://cython.readthedocs.io/">Cython Documentation</a></li><li><a href="https://github.com/CGATOxford/UMI-tools">UMI-tools GitHub</a></li><li><a href="https://github.com/Jetsetter/pybktree">pybktree: BK-tree implementation</a></li><li><a href="https://docs.python.org/3/library/profile.html">Profiling Python code with cProfile</a></li></ul><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=fe08891a6ef5" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Bioinformatics Pipeline Engineering: Do This Not That: Split Tasks]]></title>
            <link>https://medium.com/data-science-collective/bioinformatics-pipeline-engineering-do-this-not-that-split-tasks-c66c3f4bb5b0?source=rss-87b832c70819------2</link>
            <guid isPermaLink="false">https://medium.com/p/c66c3f4bb5b0</guid>
            <category><![CDATA[drug-discovery]]></category>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[nextflow]]></category>
            <category><![CDATA[bioinformatics]]></category>
            <dc:creator><![CDATA[Jillian Rowe]]></dc:creator>
            <pubDate>Thu, 23 Apr 2026 00:00:21 GMT</pubDate>
            <atom:updated>2026-04-27T20:44:54.937Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1000/0*kXfjEf4MNelYMJ6w.png" /></figure><h3>Overview</h3><p>In the <a href="https://www.dabbleofdevops.com/blog/do-this-not-that-large-files/">last installment</a> of this series, we talked about how to handle large files without bringing down the wrath of your local sys admin. Today we’re tackling how to think about breaking up tasks.</p><p>You’ve probably written it. I’ve definitely written it. It looks perfectly reasonable when you first start:</p><pre>for sample in samples:<br>    run_qc(sample)<br>    align(sample)<br>    call_variants(sample)</pre><p>(I am aware that it is unlikely you’d write this exact code. Suspend your disbelief and walk with me. We’re going somewhere with this.)</p><p>Simple. Readable. Gets the job done.</p><p>Until it doesn’t!</p><p>Maybe you’re now running a 200GB alignment step on the same node where you’re doing QC, or you can’t checkpoint a failed run, or your HPC scheduler has no idea how to allocate resources for a step that sometimes needs 4GB and sometimes needs 400GB. Maybe your local admin is upset because nodes are underutilized and she’s got annoyed scientists waiting for a machine. (Day in the life!)</p><p>In this post we’ll look at <strong>splitting monolithic tasks into discrete CLI commands</strong>, and then show how naturally those map to a Nextflow pipeline.</p><h3>Don’t: Lump Everything Into a Single Script Loop</h3><h3>The Problem</h3><p>Here’s a pattern that feels intuitive but causes real pain at scale:</p><pre># ❌ DON&#39;T DO THIS<br>def run_pipeline(samples):<br>    for sample in samples:<br>        task1(sample)   # QC: uses ~4 GB RAM<br>        task2(sample)   # Alignment: uses ~200 GB RAM<br>        task3(sample)   # Variant calling: uses ~32 GB RAM</pre><p>When every task is tangled together in a loop, you have no control. The whole pipeline runs on a single machine, allocating resources for the <em>worst-case</em> step across <em>every</em> step. You can’t restart from the middle if something fails. Your HPC scheduler can’t optimize placement. And three months from now, when a colleague asks “what does task2 actually do?&quot;, the answer lives somewhere inside a 400-line script.</p><h3>Why This Hurts in Practice</h3><p>Consider a real-world scenario: you have 50 samples, QC takes 4 GB, alignment takes 200 GB, and variant calling takes 32 GB.</p><p>(Of course you would just use an existing pipeline for this workflow. Pretend we have some custom barcodes or something.)</p><p>With the monolithic approach:</p><ul><li>Every step runs on a node provisioned for the <strong>largest</strong> step (200 GB)</li><li>You’re paying for 200 GB of RAM during QC, when you only need 4 GB</li><li>A <em>single failure</em> at sample 49 means restarting everything from scratch</li><li>Adding a new step means modifying the loop and hoping you don’t break the others</li><li>There’s no obvious place to document what each step does or what it expects</li></ul><p>This is exactly the kind of thing that quietly costs teams thousands of dollars in cloud compute every month.</p><h3>Do: Split Tasks into Discrete CLI Commands</h3><h3>Step 1 — Give Each Task Its Own Typer Command</h3><p><a href="https://typer.tiangolo.com/">Typer</a> is a fantastic library for building CLIs from Python functions. The key insight is that a CLI command <em>is</em> documentation: the function signature, types, and help strings tell you exactly what goes in and what comes out.</p><p>Here’s what the refactored version looks like:</p><pre># ✅ DO THIS<br>import typer<br>from pathlib import Path<br><br>app = typer.Typer()<br><br>@app.command()<br>def qc(<br>    input_fastq: Path = typer.Argument(..., help=&quot;Path to input FASTQ file&quot;),<br>    output_dir: Path = typer.Argument(..., help=&quot;Directory to write QC results&quot;),<br>    min_quality: int = typer.Option(20, help=&quot;Minimum base quality score&quot;),<br>):<br>    &quot;&quot;&quot;<br>    Run quality control on a FASTQ file.<br>    Expected resource usage: ~4 GB RAM, 2 CPUs.<br>    &quot;&quot;&quot;<br>    output_dir.mkdir(parents=True, exist_ok=True)<br>    # ... QC logic here ...<br>    typer.echo(f&quot;QC complete. Results written to {output_dir}&quot;)<br><br>@app.command()<br>def align(<br>    input_fastq: Path = typer.Argument(..., help=&quot;Path to QC-passed FASTQ file&quot;),<br>    reference: Path = typer.Argument(..., help=&quot;Path to reference genome&quot;),<br>    output_bam: Path = typer.Argument(..., help=&quot;Path to output BAM file&quot;),<br>    threads: int = typer.Option(16, help=&quot;Number of alignment threads&quot;),<br>):<br>    &quot;&quot;&quot;<br>    Align reads to a reference genome using BWA-MEM2.<br>    Expected resource usage: ~200 GB RAM, 16 CPUs.<br>    &quot;&quot;&quot;<br>    output_bam.parent.mkdir(parents=True, exist_ok=True)<br>    # ... alignment logic here ...<br>    typer.echo(f&quot;Alignment complete. BAM written to {output_bam}&quot;)<br><br>@app.command()<br>def call_variants(<br>    input_bam: Path = typer.Argument(..., help=&quot;Path to aligned BAM file&quot;),<br>    reference: Path = typer.Argument(..., help=&quot;Path to reference genome&quot;),<br>    output_vcf: Path = typer.Argument(..., help=&quot;Path to output VCF file&quot;),<br>    min_depth: int = typer.Option(10, help=&quot;Minimum read depth to call a variant&quot;),<br>):<br>    &quot;&quot;&quot;<br>    Call variants from an aligned BAM file using GATK HaplotypeCaller.<br>    Expected resource usage: ~32 GB RAM, 4 CPUs.<br>    &quot;&quot;&quot;<br>    output_vcf.parent.mkdir(parents=True, exist_ok=True)<br>    # ... variant calling logic here ...<br>    typer.echo(f&quot;Variant calling complete. VCF written to {output_vcf}&quot;)<br><br>if __name__ == &quot;__main__&quot;:<br>    app()</pre><p>Now each task is invocable on its own:</p><pre># Run just QC<br>python pipeline.py qc sample1.fastq.gz ./results/qc/<br><br># Run just alignment<br>python pipeline.py align results/qc/sample1.fastq.gz ref/hg38.fa results/bam/sample1.bam<br><br># Run just variant calling<br>python pipeline.py call-variants results/bam/sample1.bam ref/hg38.fa results/vcf/sample1.vcf.gz</pre><p>And you get auto-generated help for free:</p><pre>$ python pipeline.py align --help<br><br>Usage: pipeline.py align [OPTIONS] INPUT_FASTQ REFERENCE OUTPUT_BAM<br><br>Align reads to a reference genome using BWA-MEM2.<br> Expected resource usage: ~200 GB RAM, 16 CPUs.<br>Arguments:<br>  INPUT_FASTQ   Path to QC-passed FASTQ file [required]<br>  REFERENCE     Path to reference genome [required]<br>  OUTPUT_BAM    Path to output BAM file [required]<br>Options:<br>  --threads     INTEGER  Number of alignment threads [default: 16]<br>  --help                 Show this message and exit.</pre><p>That’s documentation that stays in sync with the code, always.</p><h3>Step 2 — Wrap Commands in a Container</h3><p>Once your tasks are discrete CLI commands, containerizing them is straightforward. Each step can have its own Docker image with exactly the dependencies it needs — no more shipping a 15 GB image because alignment and QC happen to live in the same script.</p><pre># Dockerfile.align<br>FROM ubuntu:22.04<br><br>RUN apt-get update &amp;&amp; apt-get install -y bwa-mem2 samtools python3-pip<br>RUN pip install typer<br>COPY pipeline.py /opt/pipeline.py<br>ENTRYPOINT [&quot;python&quot;, &quot;/opt/pipeline.py&quot;, &quot;align&quot;]</pre><p>Build and run:</p><pre>docker build -f Dockerfile.align -t my-pipeline/align:latest .<br><br># or singularity run!<br>docker run --rm \<br>  -v /data:/data \<br>  my-pipeline/align:latest \<br>  /data/sample1.fastq.gz /data/hg38.fa /data/results/sample1.bam</pre><h3>Step 3 — Drop Them Into Nextflow</h3><p>Now that each task is a standalone command with well-defined inputs and outputs, translating to Nextflow is almost mechanical. Each CLI command becomes a Nextflow process. Once we have our code in a workflow manager we get all kinds of sanity checks! We can write checks, resume, look at memory consumption, etc.</p><h3>The Nextflow Module</h3><pre>// modules/qc.nf<br>process QC {<br>    tag &quot;$sample_id&quot;<br>    label &#39;process_low&#39;        // 4 GB RAM, 2 CPUs<br>    container &#39;my-pipeline/qc:latest&#39;<br><br>    input:<br>    tuple val(sample_id), path(fastq)<br>    output:<br>    tuple val(sample_id), path(&quot;${sample_id}_qc/&quot;), emit: qc_results<br>    script:<br>    &quot;&quot;&quot;<br>    python pipeline.py qc \\<br>        ${fastq} \\<br>        ${sample_id}_qc/<br>    &quot;&quot;&quot;<br>}<br>// modules/align.nf<br>process ALIGN {<br>    tag &quot;$sample_id&quot;<br>    label &#39;process_high_memory&#39;  // 200 GB RAM, 16 CPUs<br>    container &#39;my-pipeline/align:latest&#39;<br>    input:<br>    tuple val(sample_id), path(fastq)<br>    path reference<br>    output:<br>    tuple val(sample_id), path(&quot;${sample_id}.bam&quot;), emit: bam<br>    script:<br>    &quot;&quot;&quot;<br>    python pipeline.py align \\<br>        ${fastq} \\<br>        ${reference} \\<br>        ${sample_id}.bam \\<br>        --threads ${task.cpus}<br>    &quot;&quot;&quot;<br>}<br>// modules/call_variants.nf<br>process CALL_VARIANTS {<br>    tag &quot;$sample_id&quot;<br>    label &#39;process_medium&#39;   // 32 GB RAM, 4 CPUs<br>    container &#39;my-pipeline/call-variants:latest&#39;<br>    input:<br>    tuple val(sample_id), path(bam)<br>    path reference<br>    output:<br>    tuple val(sample_id), path(&quot;${sample_id}.vcf.gz&quot;), emit: vcf<br>    script:<br>    &quot;&quot;&quot;<br>    python pipeline.py call-variants \\<br>        ${bam} \\<br>        ${reference} \\<br>        ${sample_id}.vcf.gz<br>    &quot;&quot;&quot;<br>}</pre><h3>The Workflow</h3><pre>// main.nf<br>include { QC }            from &#39;./modules/qc&#39;<br>include { ALIGN }         from &#39;./modules/align&#39;<br>include { CALL_VARIANTS } from &#39;./modules/call_variants&#39;<br><br>workflow {<br>    samples_ch = Channel<br>        .fromPath(params.samplesheet)<br>        .splitCsv(header: true)<br>        .map { row -&gt; tuple(row.sample_id, file(row.fastq)) }<br>    reference_ch = file(params.reference)<br>    QC(samples_ch)<br>    ALIGN(QC.out.qc_results, reference_ch)<br>    CALL_VARIANTS(ALIGN.out.bam, reference_ch)<br>}</pre><p>And the resource labels map to concrete allocations in nextflow.config:</p><pre>// nextflow.config<br>process {<br>    withLabel: &#39;process_low&#39; {<br>        cpus   = 2<br>        memory = &#39;4 GB&#39;<br>    }<br>    withLabel: &#39;process_medium&#39; {<br>        cpus   = 4<br>        memory = &#39;32 GB&#39;<br>    }<br>    withLabel: &#39;process_high_memory&#39; {<br>        cpus   = 16<br>        memory = &#39;200 GB&#39;<br>    }<br>}</pre><p>Now Nextflow knows exactly how much memory each step needs, and your scheduler — whether that’s AWS Batch, SLURM, or Google Cloud Life Sciences — can place each process on an appropriately sized node.</p><p>Once you have your pipeline in Nextflow you no longer have to worry about orchestrating your pipeline. Nextflow will take over on assigning resources whether you’re working on a traditional HPC, AWS Batch, or locally. You’ll automatically get failover and retries. Each steps success/failure, time, and resource management will be logged automatically.</p><h3>Why This Matters: The Three Benefits</h3><h3>1. Organization</h3><p>A Typer command is self-documenting by design. The function name becomes the command name. Type annotations become argument validation. Docstrings become --help output. When you (or a new team member) comes back to this code in six months, every step has an obvious entry point, clear inputs and outputs, and inline documentation. No more hunting through a 500-line script to figure out what step 3 actually does.</p><h3>2. Resource Consumption</h3><p>This is where splitting tasks pays real dividends. When QC needs 4 GB and alignment needs 200 GB, running them in the same process means every sample occupies a 200 GB node for the entire pipeline — even during QC. With discrete Nextflow processes, each step runs on the smallest node that can handle it:</p><p>(<em>These numbers are sort of made up. Please check with your cloud provider for exact costs.</em>)</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/427/1*bUhnhSyTBCvZ8POY_UPUXg.png" /></figure><p>Running monolithically at the highest requirement for 50 samples means paying for 200 GB throughout. Splitting tasks means you only pay for what each step actually needs. At scale, that difference is not small.</p><h3>3. Pipeline Integration</h3><p>The moment each task is a discrete CLI command, it becomes a composable unit. Nextflow (or Snakemake, or WDL) can:</p><ul><li><strong>Parallelize</strong> across samples automatically — no manual multiprocessing</li><li><strong>Checkpoint</strong> and resume from any step without rerunning the whole pipeline</li><li><strong>Retry</strong> failed steps with different resource allocations</li><li><strong>Cache</strong> completed steps so re-runs skip work that’s already done</li><li><strong>Scale</strong> from your laptop to 1,000-node HPC clusters without changing a line of Python</li></ul><p>The CLI is the interface. The pipeline orchestrator is the caller. They don’t need to know anything about each other.</p><h3>Best Practices Checklist</h3><ul><li><strong>✅ One command per logical step</strong>: If you find yourself putting two typer.Options that only apply to half the function, you probably have two commands</li><li><strong>✅ Document resource expectations in docstrings</strong>: Future you will thank you</li><li><strong>✅ Use </strong><strong>Path types</strong>: Typer validates existence for you; no more FileNotFoundError buried in the middle of a long run</li><li><strong>✅ Match CLI arguments to Nextflow inputs</strong>: Keep the interface consistent so the Nextflow script is a thin wrapper, not a translation layer</li><li><strong>✅ Test each command independently</strong>: If a step can’t run standalone, it can’t run reliably in a pipeline</li></ul><h3>Key Takeaway</h3><p>The monolithic loop is a natural starting point, but it creates a ceiling. Splitting your pipeline into discrete CLI commands — each with defined inputs, outputs, and resource requirements — removes that ceiling entirely.</p><p><strong>Split your tasks. Document your interfaces. Let the scheduler worry about the rest.</strong></p><p><em>Next in the “Do This Not That” series: Batched vectorized operations over multiprocessing</em></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=c66c3f4bb5b0" width="1" height="1" alt=""><hr><p><a href="https://medium.com/data-science-collective/bioinformatics-pipeline-engineering-do-this-not-that-split-tasks-c66c3f4bb5b0">Bioinformatics Pipeline Engineering: Do This Not That: Split Tasks</a> was originally published in <a href="https://medium.com/data-science-collective">Data Science Collective</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Bioinformatics Pipeline Engineering: Do This Not That: Large Files]]></title>
            <link>https://medium.com/data-science-collective/bioinformatics-pipeline-engineering-do-this-not-that-large-files-9d903cbb0c65?source=rss-87b832c70819------2</link>
            <guid isPermaLink="false">https://medium.com/p/9d903cbb0c65</guid>
            <category><![CDATA[bioinformatics]]></category>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[ai]]></category>
            <dc:creator><![CDATA[Jillian Rowe]]></dc:creator>
            <pubDate>Tue, 21 Apr 2026 00:00:55 GMT</pubDate>
            <atom:updated>2026-04-22T16:06:41.720Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*1HV8omBlfqpqcjcJ-C6_Eg.png" /></figure><h3>Overview</h3><p>If you’ve worked with bioinformatics data, you’ve likely encountered the dreaded experience: your Python script chugging along, consuming memory at an alarming rate, and then — <em>crash</em>. Out of memory. Your computer becomes unresponsive, and you’re left frustrated wondering why your beautiful analysis script decided to obliterate your system. Maybe you’ve even taken down an entire HPC!</p><p>Here’s the thing: working with large bioinformatics files — FASTQ sequences, whole genome VCF files, BAM alignments, protein structure files — requires a fundamentally different approach than analyzing a CSV with a few thousand rows. These files often contain millions or billions of records, and the naive approach of “just load it all into memory” is often a recipe for a headache.</p><p>In this “Do This Not That” series, we’re going to explore practical patterns for optimizing bioinformatics software. Today, we’re tackling <strong>file handling at scale</strong>.</p><h3>Don’t: Load Everything Into Memory</h3><h3>The Problem</h3><p>The worst approach — and unfortunately the most common among newcomers — is to load an entire large file into memory:</p><pre># ❌ DON&#39;T DO THIS<br>def analyze_fastq(filepath):<br>    &quot;&quot;&quot;Load entire FASTQ file into memory. Catastrophic for large files.&quot;&quot;&quot;<br>    sequences = []<br>    with open(filepath, &#39;r&#39;) as f:<br>        lines = f.readlines()  # Load ALL lines into memory<br>    <br>    # Parse fastq into records<br>    for i in range(0, len(lines), 4):<br>        header = lines[i].strip()<br>        seq = lines[i+1].strip()<br>        plus = lines[i+2].strip()<br>        qual = lines[i+3].strip()<br>        sequences.append({<br>            &#39;header&#39;: header,<br>            &#39;sequence&#39;: seq,<br>            &#39;quality&#39;: qual<br>        })<br>    <br>    return sequences<br><br># This will crash with a 50GB FASTQ file<br>results = analyze_fastq(&quot;huge_genomic_dataset.fastq&quot;)</pre><p><strong>Why this fails:</strong></p><ul><li><strong>Memory explosion</strong>: A 100GB FASTQ file needs 100GB+ of RAM just to store the raw text, then additional memory for parsing into data structures</li><li><strong>Python object overhead</strong>: Each string and dictionary adds metadata overhead (Python objects are not memory-efficient)</li><li><strong>No streaming</strong>: You’re waiting for the entire file to load before processing even one record</li><li><strong>Swap thrashing</strong>: When RAM is exceeded, your OS swaps to disk, slowing everything to a crawl</li></ul><h3>Real-World Impact</h3><p>On a machine with 32GB of RAM, trying to load a 200GB human genome VCF file will:</p><ol><li>Use up all available RAM</li><li>Trigger swap, using disk as virtual memory (100x slower)</li><li>Freeze your system</li><li>Eventually crash with an out-of-memory error</li></ol><p>You’ve now lost all time and resources with nothing to show for it. You may be getting annoyed emails from your local sys admin.</p><h3>Do: Stream or Chunk Your Data</h3><h3>Approach 1: Streaming (Process Line-by-Line)</h3><p>The most memory-efficient approach is to process data line-by-line or record-by-record without storing everything:</p><pre># ✅ DO THIS - Streaming approach<br>def analyze_fastq_stream(filepath):<br>    &quot;&quot;&quot;Process FASTQ records one at a time.&quot;&quot;&quot;<br>    def fastq_reader(file_obj):<br>        &quot;&quot;&quot;Generator: yield one FASTQ record at a time.&quot;&quot;&quot;<br>        while True:<br>            header = file_obj.readline().strip()<br>            if not header:<br>                break<br>            <br>            seq = file_obj.readline().strip()<br>            plus = file_obj.readline().strip()<br>            qual = file_obj.readline().strip()<br>            <br>            yield [header, seq, qual]   <br>    # Process records as they&#39;re read (memory-constant)<br>    stats = {<br>        &#39;total_records&#39;: 0,<br>        &#39;avg_length&#39;: 0,<br>        &#39;total_bases&#39;: 0,<br>    }<br>    <br>    with open(filepath, &#39;r&#39;) as f:<br>        for record in fastq_reader(f):<br>            # Process each record individually<br>            stats[&#39;total_records&#39;] += 1<br>            stats[&#39;total_bases&#39;] += len(record[1])<br>    <br>    stats[&#39;avg_length&#39;] = stats[&#39;total_bases&#39;] / max(stats[&#39;total_records&#39;], 1)<br>    return stats<br><br># This handles terabyte-scale files with constant memory<br>results = analyze_fastq_stream(&quot;huge_genomic_dataset.fastq&quot;)</pre><p><strong>Advantages:</strong></p><ul><li><strong>Constant memory</strong>: Only one record in memory at any time</li><li><strong>Immediate processing</strong>: Start analyzing data before the file finishes loading</li><li><strong>Infinite scalability</strong>: Handle files larger than your system’s RAM</li><li><strong>Graceful degradation</strong>: Works whether your file is 1MB or 1TB</li></ul><h3>Approach 2: Chunked Reading (Process in Batches)</h3><p>For operations that benefit from batch processing (like machine learning inference), read in managed chunks.</p><p>Typically, when I do this, its because I want to interim results to parquet files. We get one parquet file per chunk, which can then be spread out across a scheduler.</p><pre>import pyarrow as pa<br>import pyarrow.parquet as pq<br><br># ✅ DO THIS - Chunked approach<br>def analyze_fastq_chunked(filepath, chunk_size=10000):<br>    &quot;&quot;&quot;Process FASTQ records in fixed-size batches.&quot;&quot;&quot;<br>    def batch_fastq_reader(file_obj, batch_size):<br>        &quot;&quot;&quot;Generator: yield batches of FASTQ records.&quot;&quot;&quot;<br>        seqs = []<br>        headers = []<br>        quals = []<br>        while True:<br>            header = file_obj.readline().strip()<br>            if not header:<br>                if batch:<br>                    yield batch<br>                break<br>            <br>            seq = file_obj.readline().strip()<br>            plus = file_obj.readline().strip()<br>            qual = file_obj.readline().strip()<br>            <br>            # batch.append([header, seq, qual])<br>            headers.append(header)<br>            seqs.append(seq)<br>            quals.append(qual)<br>            <br>            if len(batch) &gt;= batch_size:<br>                yield headers, seqs, quals<br>                headers = []<br>                seqs = []<br>                quals = []<br>    <br>    # Process in batches (bounded memory usage)<br>    <br>    batch_num = 0<br>    with open(filepath, &#39;r&#39;) as f:<br>        for batch in batch_fastq_reader(f, chunk_size):<br>            # Process this batch (e.g., run ML inference)<br>            headers, seqs, quals = process_sequences_batch(batch)<br>            table = pa.Table.from_pydict({&#39;header&#39;: headers, &#39;sequence&#39;, : seqs, &#39;quality&#39;: quals})<br>            # save additional processing for later<br>            pq.write_table(table, f&#39;fastq-records-{batch_num}.parquet&#39;)<br>            print(f&quot;Processed batch of {len(batch)} records&quot;)<br>            batch_num += 1<br>    <br>    return all_results</pre><p>Once you have your data in an arrow/pandas table, you can replace many bioinformatics string mantching type functions. Running code against all elements in the column of a table is nearly always faster than running code against all items in a list. (We’ll be talking about this next!)</p><p><strong>Advantages:</strong></p><ul><li><strong>Bounded memory</strong>: Memory usage stays constant regardless of file size</li><li><strong>Batch efficiency</strong>: Some operations (ML, vectorized computing) are faster in batches</li><li><strong>Progress tracking</strong>: You can report progress as batches complete</li><li><strong>Parallelizable</strong>: Each batch can be processed independently</li><li><strong>Simple</strong>: Keep your code as simple as possible. When running code millions of times even small delays really add up!</li></ul><h3>Approach 3: Using Libraries Built for Large Files</h3><p>This one is a bit fuzzy. I’m <em>all for</em> using code that I didn’t write. Ok? All about it.</p><p>But my dreams just don’t always match my reality of needing my code to run as quickly as possible. AWS credits don’t grow on trees!</p><p>Bioinformatics libraries are built to parse bioinformatics data. This can be a good thing. Some libraries, such as pysam, are optimized and fast.</p><p>Other libraries, such as BioPython, less so. Sometimes you need to do what you need to do, but most often you’ll be better off rolling your own, or better yet using a library such as pyarrow.</p><pre># ✅ SOMETIMES DO THIS - Use specialized libraries<br>from Bio import SeqIO  # BioPython<br><br># BioPython: Built-in streaming parser<br>def analyze_with_biopython(fastq_file):<br>    &quot;&quot;&quot;BioPython handles streaming automatically.&quot;&quot;&quot;<br>    stats = {&#39;count&#39;: 0, &#39;gc_content&#39;: 0}<br>    <br>    for record in SeqIO.parse(fastq_file, &quot;fastq&quot;):<br>        # Process each record without loading file into memory<br>        stats[&#39;count&#39;] += 1<br>        stats[&#39;gc_content&#39;] += record.seq.count(&#39;G&#39;) + record.seq.count(&#39;C&#39;)<br>    <br>    return stats</pre><p>For even faster batch operations on accumulated sequences, use PyArrow’s vectorized string functions:</p><pre># ✅ BEST FOR BATCH PROCESSING: Accumulate and use PyArrow compute functions<br>import pyarrow as pa<br>import pyarrow.compute as pc<br><br>def analyze_with_pyarrow(fastq_file):<br>    &quot;&quot;&quot;Accumulate sequences in batches, use vectorized string counting.&quot;&quot;&quot;<br>    sequences = []<br>    <br>    with pysam.FastqFile(fastq_file) as fq:<br>        for record in fq:<br>            sequences.append(record.sequence)<br>            <br>            # Process in batches of 10,000<br>            if len(sequences) &gt;= 10000:<br>                # Use PyArrow&#39;s vectorized count_substring for speed<br>                seq_array = pa.array(sequences)<br>                total_gc = int(pc.count_substring(seq_array, pattern=&quot;G&quot;)) + int(pc.count_substring(seq_array, pattern=&quot;C&quot;))<br>                <br>                print(f&quot;Batch GC content: {total_gc}&quot;)<br>                sequences = []<br>    <br>    # Handle remaining sequences<br>    if sequences:<br>        seq_array = pa.array(sequences)<br>        total_gc = int(pc.count_substring(seq_array, pattern=&quot;G&quot;)) + int(pc.count_substring(seq_array, pattern=&quot;C&quot;))<br>        print(f&quot;Final batch GC content: {total_gc}&quot;)</pre><p><a href="https://pysam.readthedocs.io/en/stable/api.html#api">PySam</a> is nearly always going to be fastest for file I/O. It accepts most major bioinformatics file format types.</p><pre># ✅ SOMETIMES DO THIS - Use specialized libraries<br><br># PysSAM: Efficient BAM file iteration<br>import pysam  # SAM/BAM handling<br>def analyze_bam_stream(bam_file):<br>    &quot;&quot;&quot;PysSAM streams BAM records efficiently.&quot;&quot;&quot;<br>    stats = {&#39;total_reads&#39;: 0, &#39;mapped_reads&#39;: 0}<br>    <br>    with pysam.AlignmentFile(bam_file, &quot;rb&quot;) as bam:<br>        for read in bam:<br>            stats[&#39;total_reads&#39;] += 1<br>            if not read.is_unmapped:<br>                stats[&#39;mapped_reads&#39;] += 1<br>    <br>    return stats</pre><p><strong>Why this matters:</strong></p><ul><li><strong>Battle-tested</strong>: These libraries are optimized for bioinformatics use cases</li><li><strong>Zero-copy operations</strong>: Many use C/C++ backends that don’t allocate excess memory</li><li><strong>Standard patterns</strong>: Your colleagues will recognize the code</li><li><strong>Feature-rich</strong>: Built-in support for filtering, transforming, etc.</li></ul><h3>Compressed Files: Even More Important</h3><p>Large bioinformatics files are almost always compressed (gzip, bgzip, zstd). Reading compressed files requires special attention:</p><pre># ❌ DON&#39;T: Decompress entire file first<br>import gzip<br>import shutil<br><br># This creates a huge temporary uncompressed file<br>with gzip.open(&#39;huge_file.fastq.gz&#39;, &#39;rb&#39;) as f_in:<br>    with open(&#39;huge_file.fastq&#39;, &#39;wb&#39;) as f_out:<br>        shutil.copyfileobj(f_in, f_out)  # Now you have 200GB on disk<br><br># ✅ DO: Use smartopen for transparent decompression<br>from smart_open import open as smart_open<br><br>def stream_compressed_fastq(compressed_file):<br>    &quot;&quot;&quot;Read compressed file without decompressing to disk.<br>    <br>    smartopen automatically detects compression format (gzip, bzip2, zstd, etc.)<br>    and handles decompression transparently.<br>    &quot;&quot;&quot;<br>    with smart_open(compressed_file, &#39;r&#39;) as f:<br>        for i, line in enumerate(f):<br>            # Process line by line, decompression happens on-the-fly<br>            if i % 4 == 0:  # Header line<br>                print(f&quot;Processing: {line.strip()}&quot;)<br>            if i &gt;= 15:  # Just demo first 4 records<br>                break<br><br># smartopen works with any compression format automatically<br>def process_any_format(filepath):<br>    &quot;&quot;&quot;Works with .gz, .bz2, .xz, .zst - just works!&quot;&quot;&quot;<br>    with smart_open(filepath, &#39;r&#39;) as f:<br>        for record in f:<br>            # Compression is transparent to your code<br>            yield record.strip()</pre><p><strong>Memory savings:</strong></p><ul><li><strong>Format-agnostic</strong>: Automatically detects gzip, bzip2, xz, zstd, and more</li><li><strong>Streaming decompression</strong>: Only decompress what you’re actively reading</li><li><strong>No intermediate files</strong>: Saves disk space equal to uncompressed file size</li><li><strong>No surprises</strong>: Same API regardless of compression type</li></ul><h3>Performance Comparison</h3><p>Let’s quantify the difference with concrete numbers:</p><pre>import time<br>import psutil<br>import os<br><br>def monitor_memory(func, *args, **kwargs):<br>    &quot;&quot;&quot;Decorator to measure peak memory usage.&quot;&quot;&quot;<br>    process = psutil.Process(os.getpid())<br>    mem_before = process.memory_info().rss / 1024 / 1024  # MB<br>    <br>    start = time.time()<br>    result = func(*args, **kwargs)<br>    elapsed = time.time() - start<br>    <br>    mem_after = process.memory_info().rss / 1024 / 1024  # MB<br>    peak_mem = mem_after - mem_before<br>    <br>    print(f&quot;Function: {func.__name__}&quot;)<br>    print(f&quot;  Time: {elapsed:.2f}s&quot;)<br>    print(f&quot;  Peak Memory Delta: {peak_mem:.1f}MB&quot;)<br>    <br>    return result<br># Hypothetical results with 1GB FASTQ file:<br># # load_into_memory: Time: 45s, Peak Memory: 1200MB ❌<br># stream_approach: Time: 32s, Peak Memory: 2MB ✅<br># chunked_approach: Time: 38s, Peak Memory: 150MB ✅</pre><p>For time tracking use <a href="https://funcy.readthedocs.io/en/stable/debug.html">funcy</a>.</p><pre>from funcy import tap, print_durations<br>import time<br><br>def stream_approach():<br>    &quot;&quot;&quot;Simulated streaming approach.&quot;&quot;&quot;<br>    time.sleep(2)<br>    return {&quot;records&quot;: 1000000}<br><br>def chunked_approach():<br>    &quot;&quot;&quot;Simulated chunked approach.&quot;&quot;&quot;<br>    time.sleep(2.3)<br>    return {&quot;records&quot;: 1000000}<br><br>@print_durations<br>def timed_stream():<br>    return stream_approach()<br><br>@print_durations<br>def timed_chunked():<br>    return chunked_approach()<br><br># Run and get automatic timing output<br>result1 = timed_stream()  # Prints: &lt;function timed_stream at 0x...&gt; took 2.00s<br>result2 = timed_chunked()  # Prints: &lt;function timed_chunked at 0x...&gt; took 2.30s</pre><h3>Best Practices Checklist</h3><ul><li><strong>✅ Always stream</strong>: Unless you have a very specific reason to load everything, use streaming or chunked approaches. You never know when your file will 200x in size!</li><li><strong>✅ Use proven libraries</strong>: BioPython, pySAM, pandas with chunksize, pyarrow with streaming</li><li><strong>✅ Monitor memory</strong>: Use psutil or memory_profiler to verify your approach</li><li><strong>✅ Profile your code</strong>: Some operations you think are expensive are actually fast, and vice versa</li><li><strong>✅ Test with real data</strong>: Your 1MB test file behaves very differently than a 100GB production file</li><li><strong>✅ Plan for growth</strong>: Design your system assuming files will get <em>out of hand</em>.</li></ul><h3>Common Patterns by File Type</h3><h3>FASTQ/FASTA</h3><pre>import pysam<br><br>with pysam.FastqFile(&quot;file.fastq&quot;) as fq:<br>    for record in fq:<br>        # record.name: read name<br>        # record.sequence: DNA sequence<br>        # record.query_qualities: quality scores<br>        print(f&quot;{record.name}: {record.sequence}&quot;)</pre><h3>BAM/SAM</h3><pre>import pysam<br>with pysam.AlignmentFile(&quot;file.bam&quot;, &quot;rb&quot;) as f:<br>    for read in f:<br>        # Process read</pre><h3>VCF</h3><pre>import vcf<br>reader = vcf.Reader(filename=&quot;file.vcf.gz&quot;)<br>for record in reader:<br>    # Process variant</pre><h3>CSV/Tabular</h3><pre>from pyarrow import csv<br>df = csv.read_csv(&quot;large_file.csv&quot;)<br>    # Process chunk</pre><h3>References</h3><ul><li><a href="https://pysam.readthedocs.io/en/stable/api.html#fastq-files">pySAM FASTQ Files Documentation</a></li><li><a href="https://github.com/RaRe-Technologies/smart_open">smart_open: Transparent file handling</a></li><li><a href="https://biopython.org/wiki/SeqIO">BioPython SeqIO</a></li><li><a href="https://pyvcf.readthedocs.io/">PyVCF: Variant Call Format handling</a></li></ul><h3>Key Takeaway</h3><p>Large file handling isn’t about fancy algorithms — it’s about respecting memory constraints and processing data efficiently. The pattern is always the same:</p><p><strong>Don’t load everything. Stream or chunk instead.</strong></p><p>This simple principle will make your bioinformatics software scale from laptop to HPC cluster without modification. Your future self (and your system administrator) will thank you.</p><p><em>Next in the “Do This Not That” series: Parallelization patterns for bioinformatics workflows.</em></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=9d903cbb0c65" width="1" height="1" alt=""><hr><p><a href="https://medium.com/data-science-collective/bioinformatics-pipeline-engineering-do-this-not-that-large-files-9d903cbb0c65">Bioinformatics Pipeline Engineering: Do This Not That: Large Files</a> was originally published in <a href="https://medium.com/data-science-collective">Data Science Collective</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[ML/AIOps for Drug Discovery with Boltz2 and MLflow]]></title>
            <link>https://medium.com/data-science-collective/ml-aiops-for-drug-discovery-with-boltz2-and-mlflow-f98bdc1ee414?source=rss-87b832c70819------2</link>
            <guid isPermaLink="false">https://medium.com/p/f98bdc1ee414</guid>
            <category><![CDATA[boltz2]]></category>
            <category><![CDATA[ai]]></category>
            <category><![CDATA[mlops]]></category>
            <category><![CDATA[drug-discovery]]></category>
            <category><![CDATA[mlflow]]></category>
            <dc:creator><![CDATA[Jillian Rowe]]></dc:creator>
            <pubDate>Wed, 15 Apr 2026 00:00:36 GMT</pubDate>
            <atom:updated>2026-04-22T20:39:35.340Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*ImDGYvfjS7LNnhTq.png" /></figure><h3>Overview</h3><p>If you’ve ever worked with scientific software you know that it doesn’t play well with the other children on the playground. Some of my favorite work related activities are getting rid of code and finding existing code, that I didn’t write, to use. MLOps for drug discovery has been an especially challenging arena.</p><p>Modern drug discovery relies on complex ML models to virtually screen millions of structures Computational structure prediction is meant to identify promising compounds and understand binding mechanisms.</p><p>This, of course, assumes we can:</p><ul><li>Inventory our models</li><li>Run our inference pipelines</li><li>Track our results</li><li>Track additional assets (graphs, plots, images, interim files.)</li></ul><p>Much easier said than done!</p><p>I’ve seen MLOps for drug discovery handled all kinds of ways, but the combination of <a href="https://mlflow.org/?ref=dabbleofdevops.com">MLFlow</a> has to be my favorite. MLFlow can be run locally. For testing it’s backed by a SQLite DB, but can also handle tracking multiple users over multiple experiments. Its an application that I don’t have to manage! It also provides a UI for viewing results.</p><p><a href="https://www.boltzai.info/?ref=dabbleofdevops.com">Boltz2</a>, the latest protein-ligand structure prediction model, combined with <a href="https://mlflow.org/docs/latest/ml/model/models-from-code/?ref=dabbleofdevops.com">MLflow’s models-from-code</a> framework, creates a powerful MLOps pipeline for tracking, versioning, and deploying AI-driven drug discovery workflows at scale.</p><h3>Why Models-From-Code for Drug Discovery?</h3><p>Scientific software is often in it’s own category. Custom programs that don’t play well with others. As much as I would like to coopt other people’s code, it’s just not always possible. Today though, it was possible!</p><p><strong>MLflow’s models-from-code</strong> stores your prediction pipeline as readable Python scripts instead of binary artifacts, enabling:</p><ul><li><strong>Reproducibility</strong>: Exact code used for each prediction is versioned and auditable</li><li><strong>Transparency</strong>: Audit model behavior without unpickling binaries. Version models, metrics, and any other artifacts we want</li><li><strong>Collaboration</strong>: Share prediction workflows across research teams without environment conflicts</li><li><strong>Deployment</strong>: Load any historical prediction configuration and reuse it with new compounds</li></ul><h3>Environment Setup</h3><p>Start by creating a conda environment with all necessary dependencies:</p><pre>name: boltz-mlflow<br>channels:<br>  - conda-forge<br>  - defaults<br>dependencies:<br>  - python=3.12<br>  - pip<br>  - ipython<br>  - pip:<br>    - mlflow<br>    - typer==0.23.1<br>    - click<br>    - boltz[cuda12] @ git+https://github.com/jwohlwend/boltz.git@v2.2.1<br>  # boltz dependencies<br>    - &quot;torch&gt;=2.2&quot;<br>    - &quot;numpy&gt;=1.26,&lt;2.0&quot;<br>    - &quot;hydra-core==1.3.2&quot;<br>    - &quot;pytorch-lightning==2.5.0&quot;<br>    - &quot;rdkit&gt;=2024.3.2&quot;<br>    - &quot;dm-tree==0.1.8&quot;<br>    - &quot;requests==2.32.3&quot;<br>    - &quot;pandas&gt;=2.2.2&quot;<br>    - &quot;types-requests&quot;<br>    - &quot;einops==0.8.0&quot;<br>    - &quot;einx==0.3.0&quot;<br>    - &quot;fairscale==0.4.13&quot;<br>    - &quot;mashumaro==3.14&quot;<br>    - &quot;modelcif==1.2&quot;<br>    - &quot;wandb==0.18.7&quot;<br>    - &quot;click==8.1.7&quot;<br>    - &quot;pyyaml==6.0.2&quot;<br>    - &quot;biopython==1.84&quot;<br>    - &quot;scipy==1.13.1&quot;<br>    - &quot;numba==0.61.0&quot;<br>    - &quot;gemmi==0.6.5&quot;<br>    - &quot;scikit-learn==1.6.1&quot;<br>    - &quot;chembl_structure_pipeline==1.2.2&quot;<br>    # cuda deps<br>    - &quot;cuequivariance_ops_cu12&gt;=0.5.0&quot;<br>    - &quot;cuequivariance_ops_torch_cu12&gt;=0.5.0&quot;<br>    - &quot;cuequivariance_torch&gt;=0.5.0&quot;</pre><p>Create and activate the environment:</p><pre>conda env create -f conda_env.yaml <br>conda activate boltz-mlflow</pre><h3>Implementation</h3><h3>1. MLflow Model Wrapper</h3><p>The Boltz2Model class wraps Boltz2 predictions in a format that MLflow understands. This enables seamless tracking, versioning, and deployment:</p><pre>import os<br>from pathlib import Path<br>from typing import Any, Dict, Optional<br>import mlflow<br>from mlflow.pyfunc import PythonModel<br>from mlflow.models import set_model<br><br><br>class Boltz2Model(PythonModel):<br>    &quot;&quot;&quot;MLflow wrapper for Boltz2 predictions.&quot;&quot;&quot;<br><br>    def load_context(self, context):<br>        &quot;&quot;&quot;Load model artifacts.&quot;&quot;&quot;<br>        import torch<br>        from boltz.model.models.boltz2 import Boltz2<br><br>        checkpoint_path = context.artifacts.get(&quot;checkpoint&quot;)<br>        if checkpoint_path:<br>            self.checkpoint = torch.load(checkpoint_path, map_location=&quot;cpu&quot;)<br>        else:<br>            self.checkpoint = None<br><br>    def predict(self, context, model_input: Dict[str, Any], params: Optional[Dict[str, Any]] = None) -&gt; Dict[str, Any]:<br>        &quot;&quot;&quot;Run Boltz2 prediction.&quot;&quot;&quot;<br>        from boltz.main import predict as boltz_predict<br><br>        input_path = model_input[&quot;input_path&quot;]<br>        out_dir = model_input.get(&quot;out_dir&quot;, &quot;./predictions&quot;)<br><br>        params = params or {}<br>        boltz_predict(<br>            data=input_path,<br>            out_dir=out_dir,<br>            cache=params.get(&quot;cache&quot;, &quot;~/.boltz&quot;),<br>            checkpoint=params.get(&quot;checkpoint&quot;),<br>            affinity_checkpoint=params.get(&quot;affinity_checkpoint&quot;),<br>            recycling_steps=params.get(&quot;recycling_steps&quot;, 3),<br>            sampling_steps=params.get(&quot;sampling_steps&quot;, 200),<br>            diffusion_samples=params.get(&quot;diffusion_samples&quot;, 1),<br>            use_msa_server=params.get(&quot;use_msa_server&quot;, False),<br>            model=params.get(&quot;model&quot;, &quot;boltz2&quot;),<br>        )<br><br>        return {&quot;output_dir&quot;: out_dir, &quot;status&quot;: &quot;completed&quot;}<br><br><br>set_model(Boltz2Model())</pre><p>This wrapper:<br>- <strong>Loads context</strong> using Boltz to manage the model<br>- <strong>Accepts structured input</strong> (input YAML path, output directory)<br>- <strong>Supports parameter overrides</strong> for experimentation (sampling steps, recycling iterations, etc.)<br>- <strong>Returns structured output</strong> for downstream pipeline steps</p><h3>2. Experiment Tracking Script</h3><p>This script orchestrates the complete MLflow workflow:</p><pre>&quot;&quot;&quot;Track Boltz2 predictions with MLflow models-from-code.&quot;&quot;&quot;<br>import mlflow<br>from pathlib import Path<br>from typing import Dict, Any, Optional<br><br><br>def log_boltz2_run(<br>    input_yaml: str,<br>    experiment_name: str = &quot;Boltz2-Drug-Discovery&quot;,<br>    run_name: Optional[str] = None,<br>    tags: Optional[Dict[str, str]] = None,<br>    params: Optional[Dict[str, Any]] = None,<br>):<br>    &quot;&quot;&quot;Log a Boltz2 run to MLflow.&quot;&quot;&quot;<br>    mlflow.set_experiment(experiment_name)<br><br>    with mlflow.start_run(run_name=run_name, tags=tags) as run:<br>        # Log parameters<br>        default_params = {<br>            &quot;model&quot;: &quot;boltz2&quot;,<br>            &quot;recycling_steps&quot;: 3,<br>            &quot;sampling_steps&quot;: 200,<br>            &quot;diffusion_samples&quot;: 1,<br>            &quot;use_msa_server&quot;: False,<br>        }<br>        if params:<br>            default_params.update(params)<br><br>        mlflow.log_params(default_params)<br>        mlflow.log_param(&quot;input_file&quot;, input_yaml)<br><br>        # Log the model using models-from-code<br>        model_info = mlflow.pyfunc.log_model(<br>            python_model=&quot;boltz2_mlflow_model.py&quot;,<br>            artifact_path=&quot;model&quot;,<br>            input_example={&quot;input_path&quot;: input_yaml, &quot;out_dir&quot;: &quot;./predictions&quot;},<br>            code_paths=[&quot;./code/boltz/src&quot;],<br>        )<br><br>        # Log input file as artifact<br>        mlflow.log_artifact(input_yaml, &quot;inputs&quot;)<br><br>        return run.info.run_id, model_info</pre><h3>3. CLI Application</h3><p>For production workflows, a command-line interface provides easier integration with HPC schedulers. Most often virtual screening projects involve screening millions of compounds.</p><pre>#!/usr/bin/env python3<br>&quot;&quot;&quot;Boltz MLflow CLI - End-to-end experiment tracking and model registration.&quot;&quot;&quot;<br>from typing import Optional<br>import typer<br>import mlflow<br>from mlflow.exceptions import MlflowException<br><br>app = typer.Typer(help=&quot;Boltz structure prediction with MLflow tracking&quot;)<br><br><br>@app.command()<br>def predict(<br>    input_yaml: str = typer.Argument(..., help=&quot;Input YAML file for prediction&quot;),<br>    out_dir: str = typer.Option(&quot;./predictions&quot;, help=&quot;Output directory&quot;),<br>    experiment_name: str = typer.Option(&quot;boltz-predictions&quot;, help=&quot;MLflow experiment name&quot;),<br>    model_name: str = typer.Option(&quot;boltz-structure-predictor&quot;, help=&quot;Registered model name&quot;),<br>    register: bool = typer.Option(True, help=&quot;Register model after prediction&quot;),<br>    recycling_steps: int = typer.Option(3, help=&quot;Number of recycling steps&quot;),<br>    sampling_steps: int = typer.Option(200, help=&quot;Number of sampling steps&quot;),<br>    diffusion_samples: int = typer.Option(1, help=&quot;Number of diffusion samples&quot;),<br>):<br>    &quot;&quot;&quot;Run Boltz prediction with full MLflow tracking.&quot;&quot;&quot;<br>    from boltz.main import predict as boltz_predict<br>    from pathlib import Path<br>    import platform<br><br>    input_yaml = Path(input_yaml)<br>    out_dir = Path(out_dir)<br><br>    mlflow.set_experiment(experiment_name)<br><br>    with mlflow.start_run(run_name=f&quot;predict_{input_yaml.stem}&quot;) as run:<br>        # Log parameters<br>        params = {<br>            &quot;model&quot;: &quot;boltz2&quot;,<br>            &quot;recycling_steps&quot;: recycling_steps,<br>            &quot;sampling_steps&quot;: sampling_steps,<br>            &quot;diffusion_samples&quot;: diffusion_samples,<br>            &quot;accelerator&quot;: &quot;cpu&quot; if platform.system() == &quot;Darwin&quot; else &quot;gpu&quot;,<br>            &quot;input_file&quot;: str(input_yaml),<br>        }<br>        mlflow.log_params(params)<br>        mlflow.set_tag(&quot;task&quot;, &quot;inference&quot;)<br><br>        # Log input<br>        mlflow.log_artifact(str(input_yaml), &quot;inputs&quot;)<br><br>        # Run prediction<br>        typer.echo(f&quot;Running prediction on {input_yaml}...&quot;)<br>        boltz_predict.callback(<br>            data=str(input_yaml),<br>            out_dir=str(out_dir),<br>            cache=&quot;~/.boltz&quot;,<br>            recycling_steps=recycling_steps,<br>            sampling_steps=sampling_steps,<br>            diffusion_samples=diffusion_samples,<br>            use_msa_server=False,<br>            model=&quot;boltz2&quot;,<br>            accelerator=params[&quot;accelerator&quot;],<br>            devices=1,<br>            num_workers=2,<br>            override=True,<br>            seed=None,<br>            msa_server_url=&quot;https://api.colabfold.com&quot;,<br>            msa_pairing_strategy=&quot;greedy&quot;,<br>        )<br><br>        # Log outputs<br>        if out_dir.exists():<br>            mlflow.log_artifacts(str(out_dir), &quot;predictions&quot;)<br><br>            # Log metrics from JSON files<br>            for json_file in out_dir.rglob(&quot;*.json&quot;):<br>                try:<br>                    import json<br>                    with open(json_file) as f:<br>                        data = json.load(f)<br>                        for key, value in data.items():<br>                            if isinstance(value, (int, float)):<br>                                mlflow.log_metric(key, value)<br>                except Exception as e:<br>                    typer.echo(f&quot;Could not parse {json_file}: {e}&quot;)<br><br>        typer.echo(f&quot;✓ Run ID: {run.info.run_id}&quot;)<br>        typer.echo(f&quot;✓ Predictions: {out_dir}&quot;)<br><br><br>if __name__ == &quot;__main__&quot;:<br>    app()</pre><h3>MLflow Experiment Tracking</h3><p>Once your models and pipelines are defined, tracking experiments becomes straightforward:</p><pre># Track a structure prediction run<br>run_id, model_info = log_boltz2_run(<br>    input_yaml=&quot;./code/boltz/examples/ligand.yaml&quot;,<br>    run_name=&quot;ligand_binding_screen&quot;,<br>    tags={&quot;target&quot;: &quot;kinase&quot;, &quot;stage&quot;: &quot;hit_discovery&quot;},<br>    params={&quot;sampling_steps&quot;: 200, &quot;use_msa_server&quot;: True}<br>)<br><br># Load and reuse the tracked model<br>loaded_model = mlflow.pyfunc.load_model(model_info.model_uri)<br>result = loaded_model.predict(<br>    {&quot;input_path&quot;: &quot;new_ligand.yaml&quot;, &quot;out_dir&quot;: &quot;./new_predictions&quot;},<br>    params={&quot;recycling_steps&quot;: 5}<br>)</pre><h3>Visualizing Experiments</h3><p>MLflow’s web UI provides comprehensive experiment tracking and comparison. To access it remotely:</p><pre>mlflow ui \<br>    --host 0.0.0.0 \<br>    --port 5000 \<br>    --allowed-hosts &quot;*&quot; \<br>    --cors-allowed-origins &quot;*&quot;</pre><p>Then navigate to http://&lt;your-server&gt;:5000 to view runs, compare parameters and metrics:</p><p>The runs tab shows all tracked experiments with their associated parameters and outputs:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*KMghZkHVLsTKg0if.png" /></figure><p>The metrics section displays computed properties from predictions:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*gr4SFmdvnSoGxUll.png" /></figure><p>The artifacts section stores input files, output structures, and prediction confidence data:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*Hr-qr12-16dQ3dr7.png" /></figure><h3>Best Practices for Drug Discovery MLOps</h3><p>Just my opinions. Of which I have many!</p><h3>1. Experiment Organization</h3><p>Structure experiments by discovery stage:</p><ul><li><strong>Hit-Discovery</strong>: Initial screening with binary affinity predictions (affinity_probability_binary). Use lower sampling steps for throughput</li><li><strong>Hit-to-Lead</strong>: Optimization using continuous affinity predictions (affinity_pred_value). Increase sampling for accuracy.</li><li><strong>Lead-Optimization</strong>: Fine-tuning with increased sampling steps (400+) and multiple diffusion samples</li></ul><p>Tag all runs with compound ID and discovery stage for easy querying and filtering.</p><h3>2. Parameter Tracking</h3><p>In virtual screening we always log:</p><ul><li><strong>Sampling parameters</strong> (steps, samples, recycling iterations)</li><li><strong>MSA configuration</strong> (server usage, pairing strategy)</li><li><strong>Target information</strong> (protein, ligand, binding site)</li><li><strong>Computational resources</strong> (GPU type, memory used, runtime)</li><li><strong>Input file hashes</strong> for full traceability</li></ul><h3>3. Artifact Management</h3><p>Store:</p><ul><li>Input YAML files (enables full reproducibility)</li><li>Generated structures (PDB/mmCIF format)</li><li>Confidence metrics (pLDDT, PAE matrices)</li><li>Affinity predictions and probability scores</li><li>MSA data (if custom, for complex targets)</li></ul><h3>4. Security</h3><p>Never hardcode credentials in scripts. Use environment variables:</p><pre># Use environment variables<br>os.environ[&quot;BOLTZ_MSA_USERNAME&quot;] = &quot;user&quot;<br>os.environ[&quot;BOLTZ_MSA_PASSWORD&quot;] = &quot;pass&quot;</pre><p>Or load from secure credential managers in your HPC environment.</p><h3>5. Reproducibility</h3><p>Tag runs with:</p><ul><li>Compound IDs</li><li>Target names</li><li>Discovery stage</li><li>Date/batch information</li><li>User/team information</li></ul><p>This enables filtering, comparison, and reconstruction of any discovery campaign.</p><h3>Advanced: Batch Screening Campaigns</h3><p>For high-throughput screening across multiple compounds:</p><pre>def track_screening_campaign(ligand_dir: Path, target: str):<br>    &quot;&quot;&quot;Track multiple ligand predictions across a target.&quot;&quot;&quot;<br>    mlflow.set_experiment(f&quot;Screening-{target}&quot;)<br><br>    for yaml_file in ligand_dir.glob(&quot;*.yaml&quot;):<br>        with mlflow.start_run(run_name=yaml_file.stem):<br>            mlflow.log_params({<br>                &quot;target&quot;: target,<br>                &quot;ligand&quot;: yaml_file.stem,<br>                &quot;model&quot;: &quot;boltz2&quot;,<br>            })<br><br>            model_info = mlflow.pyfunc.log_model(<br>                python_model=&quot;boltz2_mlflow_model.py&quot;,<br>                artifact_path=&quot;model&quot;,<br>                input_example={&quot;input_path&quot;: str(yaml_file)},<br>                code_paths=[&quot;./code/boltz/src&quot;],<br>            )<br><br>            mlflow.log_artifact(str(yaml_file), &quot;inputs&quot;)<br><br>            # Load and run prediction<br>            loaded_model = mlflow.pyfunc.load_model(model_info.model_uri)<br>            out_dir = Path(&quot;./screening_results&quot;) / yaml_file.stem<br>            out_dir.mkdir(parents=True, exist_ok=True)<br><br>            result = loaded_model.predict(<br>                {&quot;input_path&quot;: str(yaml_file), &quot;out_dir&quot;: str(out_dir)}<br>            )<br><br>            mlflow.log_artifacts(str(out_dir), &quot;predictions&quot;)</pre><h3>Integration with HPC Schedulers</h3><p>Since we created a CLI submitting to clusters is trivial:</p><pre>#!/bin/bash<br>#SBATCH --job-name=boltz-screening<br>#SBATCH --nodes=1<br>#SBATCH --gpus-per-node=1<br>#SBATCH --time=04:00:00<br><br>source activate boltz-mlflow<br><br>python boltz_mlflow_cli.py predict \<br>    input_ligands.yaml \<br>    --out-dir ./predictions \<br>    --experiment-name &quot;HT-Screening-2026-04&quot; \<br>    --sampling-steps 200 \<br>    --recycling-steps 3</pre><p>In a normal screening campaign you would probably have the input_ligands.yaml read in as an array of files, with one ligand per HPC / Batch / Nextflow job.</p><p>As a side note I try and add CLIs to most of my programs. It’s a favor to future me. When I inevitably need to run this a few hundred million times a well organized CLI makes future me happy.</p><h3>Key Advantages</h3><ol><li><strong>Transparent Tracking</strong>: All prediction code visible in MLflow UI for full auditability</li><li><strong>Version Control</strong>: Track model code changes alongside predictions-critical for regulatory submissions</li><li><strong>Deployment Ready</strong>: Load any historical prediction configuration and apply to new compounds</li><li><strong>Reproducibility</strong>: Exact computational conditions captured for every prediction</li><li><strong>Scalability</strong>: Trivially extends to high-throughput screening campaigns</li></ol><h3>References</h3><ul><li><a href="https://doi.org/10.1101/2025.06.14.659707">Boltz2 Paper</a></li><li><a href="https://github.com/jwohlwend/boltz">Boltz2 GitHub</a></li><li><a href="https://mlflow.org/docs/latest/ml/model/models-from-code/">MLflow Models-from-Code Documentation</a></li><li><a href="https://mlflow.org/docs/latest/python_api/index.html">MLflow Python API Reference</a></li></ul><h3>Conclusion</h3><p>Combining Boltz2’s cutting-edge structure prediction capabilities with MLflow’s models-from-code framework creates a robust, auditable ML/AIOps pipeline for drug discovery. By systematically tracking experiments, parameters, and artifacts, you gain the reproducibility needed. All without having to orchestrate your own MLOps system!</p><p>The code-based approach to model versioning eliminates black-box artifacts, enables seamless collaboration across teams, and provides the transparency that research teams demand. Start small with individual prediction tracking, then scale to high-throughput screening campaigns.</p><p><em>Originally published at </em><a href="https://www.dabbleofdevops.com/blog/boltz2-mlflow-mlops/"><em>https://www.dabbleofdevops.com</em></a><em> on April 15, 2026.</em></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=f98bdc1ee414" width="1" height="1" alt=""><hr><p><a href="https://medium.com/data-science-collective/ml-aiops-for-drug-discovery-with-boltz2-and-mlflow-f98bdc1ee414">ML/AIOps for Drug Discovery with Boltz2 and MLflow</a> was originally published in <a href="https://medium.com/data-science-collective">Data Science Collective</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Building a Natural Language Interface for OpenTargets Data with AI on AWS]]></title>
            <link>https://medium.com/data-science-collective/building-a-natural-language-interface-for-opentargets-data-with-ai-on-aws-109c6eb934fa?source=rss-87b832c70819------2</link>
            <guid isPermaLink="false">https://medium.com/p/109c6eb934fa</guid>
            <category><![CDATA[aws]]></category>
            <category><![CDATA[bioinformatics]]></category>
            <category><![CDATA[ai]]></category>
            <category><![CDATA[bedrock]]></category>
            <dc:creator><![CDATA[Jillian Rowe]]></dc:creator>
            <pubDate>Tue, 09 Sep 2025 18:48:18 GMT</pubDate>
            <atom:updated>2025-09-10T22:09:43.610Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*T4YUF3dKi9hr9F8wN99aaQ.jpeg" /></figure><p>With the advent of AI, knowledge bases and MCP servers I’m less inclined to write code than I ever have been. This blog post details how I built a complete natural language interface for querying <a href="https://www.opentargets.org/">OpenTargets data</a> using <a href="https://docs.aws.amazon.com/amazonq/latest/qdeveloper-ug/what-is.html">Amazon Q Developer</a>, AWS infrastructure and the <a href="https://mcp.so/">Model Context Protocol (MCP)</a>.</p><p>As a quick disclaimer the Amazon Q Developer service does allow for deploying AWS resources on your behalf. You should absolutely check and make sure the CLI isn’t going wild.</p><p>If you’d like to see the complete codebase, it’s available on <a href="https://github.com/dabble-of-devops-bioanalyze/opentargets-ai-agent/tree/main">GitHub</a>.</p><h3>Background: The Challenge of Genomic Data Access</h3><p>OpenTargets is a comprehensive platform that integrates genetic, genomic, and chemical data to support systematic drug target identification and prioritization. With over 3.8 million gene-disease associations, 78,000 genes, 39,000 diseases, and 18,000 drug molecules. It’s a lot of data.</p><p>However, accessing this data traditionally requires:</p><ul><li>Deep knowledge of the OpenTargets data model</li><li>SQL expertise to write complex queries</li><li>Understanding of table relationships and schema structures</li><li>Manual exploration to discover relevant associations</li><li>SDKs or APIs to programmatically access data</li></ul><p>Before the advent of LLMs and AI agents, I would have had to write hundreds of lines of code to build a web interface or API for researchers to query the data. Of course, I’d never have quite the correct use cases or queries, and we’d have to have many, many go arounds before getting something useful deployed.</p><p><strong>The Vision</strong>: What if researchers could simply ask questions in natural language like:</p><ul><li>“What genes are associated with breast cancer?”</li><li>“Show me drugs targeting BRCA1 with high confidence scores”</li><li>“Find rare diseases with few known genetic associations”</li></ul><p>More importantly, what if I can use the AI tools to write the code that then calls the AI services?</p><p>The answer to this question is <em>kind of</em>. You can’t just throw a problem at an LLM and expect a full solution. The AI is more like a really smart intern. You need to have a trust but verify mentality.</p><h3>Amazon Q Developer Process Overview</h3><p>I used <a href="https://docs.aws.amazon.com/amazonq/latest/qdeveloper-ug/what-is.html">Amazon Q Developer</a> throughout the entire project.</p><ol><li>Map out the steps you’ll take. Ask the Q developer to write these to a TODO.md file, and ask if it&#39;s unclear on anything.</li><li>Ask the Q chat to complete <em>one step at a time</em>. For example, I asked it to write the infrastructure as code in Terraform first.</li><li>Once Q generates the code ask it to review and summarize it for you.</li><li>Think of tests for your use case. If you’re using Q chat to create S3 buckets, ask if it the S3 buckets exist. Ask the AI to generate tests for your use case.</li><li>Leverage existing software where you can. I used the PyAthena library as a client to Athena, and the MCP library to create the MCP server. I told Q chat to use these libraries as context.</li><li>Describe, describe, describe. The more context you give the better the results. If you’re working with data ask the Q chat to describe the data and the schema. Then write tests against that schema. Something, somewhere will fail. Describe the error to the Q chat and ask for a solution.</li><li>Keep your end goal in mind, and be flexible on how you get there. I wanted to interact with the OpenTargets data without having to write SQL. I ended up with a CLI tool that does exactly that, and an MCP server that can be used with any MCP compatible AI tool.</li></ol><p>Here is my TODO file. I used the Q chat to generate this, and then I asked it to complete one task at a time. When it completed a task, I asked it to review the code, summarize what it did, and then write tests.</p><pre>## Completed ✅<br>- [x] Upload complete Open targets dataset to S3<br>- [x] Set up AWS Glue database and tables for OpenTargets data<br>- [x] Create Athena workgroup and named queries<br>- [x] Build SQL agent for natural language to SQL conversion<br>- [x] Implement MCP server for integration with AI tools<br>- [x] Create comprehensive schema validation tests<br>- [x] Document actual table schemas and query patterns</pre><pre>## Current Architecture<br>- **Data Storage**: S3 buckets with OpenTargets parquet files<br>- **Query Engine**: Amazon Athena with Glue Data Catalog<br>- **AI Integration**: Bedrock Claude for natural language processing<br>- **Interface**: MCP server + CLI agent</pre><h3>AWS Infrastructure Architecture</h3><p>As always, there are a million ways to architect a solution. I went with this because I wanted to understand the process that goes into building an MCP server, and I wanted to keep costs low. We’re using S3 with Athena and Glue because it’s serverless. No complicated databases to manage, and we only pay for what we use.</p><p>Here’s a description of each of the pieces of the AWS architecture:</p><h3>Data Storage and Cataloging</h3><ul><li><strong>Amazon S3</strong>: Stores OpenTargets parquet files in parquet format.</li><li>Data bucket: Raw OpenTargets datasets organized by table type</li><li>Results bucket: Athena query results with 30-day lifecycle management</li><li>Server-side encryption and private access controls</li><li><strong>AWS Glue Data Catalog</strong>: Provides schema discovery and metadata management</li><li>12 tables representing different aspects of the OpenTargets data model</li><li>Automatic schema inference from parquet files</li><li>Centralized metadata repository for consistent querying</li></ul><h3>Query Processing</h3><ul><li><strong>Amazon Athena</strong>: Serverless SQL query engine for interactive analytics</li><li>Direct queries against parquet files in S3</li><li>Pay-per-query pricing model (~$5 per TB scanned)</li><li>Workgroup configuration for query management and cost control</li><li>Named queries for common use cases</li></ul><h3>AI Integration</h3><ul><li><strong>Amazon Bedrock with Claude 3.5 Sonnet</strong>: Natural language to SQL conversion</li><li>Advanced reasoning capabilities for complex query generation</li><li>Context-aware SQL generation based on schema understanding</li><li>Cost-effective token-based pricing (~$0.003 per 1K input tokens)</li></ul><h3>Cost Analysis</h3><p>2025 is expensive and we like serverless.</p><ul><li><strong>Per Query</strong>: 0.01−0.11 (primarily Athena data scanning)</li><li><strong>Monthly Usage</strong> (100 queries): ~1−11</li><li><strong>No infrastructure overhead</strong>: Serverless components scale to zero</li></ul><h3>Schema Testing and Validation</h3><p>Biological data is messy. Always. It’s just how that works. Some of the columns were JSON or arrays. It took the AI several tries to get the correct structure into Athena, but we got there.</p><h3>Results</h3><p>Our MCP server/AI agent exposes three main tools for interacting with the OpenTargets data:</p><p><strong>Available Tools:</strong></p><ul><li>query_opentargets: Full natural language query processing</li><li>execute_sql: Direct SQL execution for advanced users</li><li>explain_query: SQL generation without execution for debugging</li></ul><p>In order to test the system, I ran a series of natural language queries through the CLI tool. Here are some example queries, along with the generated SQL, and their results:</p><pre>============================================================<br>Natural Query: Find high-confidence associations for Alzheimer&#39;s disease<br>============================================================<br>Generated SQL:<br>SELECT <br>    d.name AS disease_name,<br>    t.approvedSymbol AS gene_symbol,<br>    t.approvedName AS gene_name,<br>    a.score AS association_score,<br>    a.evidenceCount AS evidence_count<br>FROM <br>    association_overall_direct a<br>JOIN <br>    disease d ON a.diseaseId = d.id<br>JOIN <br>    target t ON a.targetId = t.id<br>WHERE <br>    (d.name LIKE &#39;%Alzheimer%&#39; OR array_join(d.synonyms.hasExactSynonym, &#39; &#39;) LIKE &#39;%Alzheimer%&#39;)<br>    AND a.score &gt; 0.7<br>ORDER BY <br>    a.score DESC<br>LIMIT 100;<br>============================================================<br>✅ Results (5 rows):</pre><pre>Row 1:<br>  disease_name: Alzheimer disease 3<br>  gene_symbol: PSEN1<br>  gene_name: presenilin 1<br>  association_score: 0.8370820664829206<br>  evidence_count: 566</pre><pre>Row 2:<br>  disease_name: early-onset autosomal dominant Alzheimer disease<br>  gene_symbol: PSEN2<br>  gene_name: presenilin 2<br>  association_score: 0.7960670314352667<br>  evidence_count: 271</pre><pre>Row 3:<br>  disease_name: Alzheimer disease type 1<br>  gene_symbol: APP<br>  gene_name: amyloid beta precursor protein<br>  association_score: 0.7855977451858343<br>  evidence_count: 62</pre><pre>Row 4:<br>  disease_name: Alzheimer disease<br>  gene_symbol: APP<br>  gene_name: amyloid beta precursor protein<br>  association_score: 0.7817288493909762<br>  evidence_count: 22877</pre><pre>Row 5:<br>  disease_name: Alzheimer disease<br>  gene_symbol: ABCA7<br>  gene_name: ATP binding cassette subfamily A member 7<br>  association_score: 0.706722491576517<br>  evidence_count: 477<br>Query: What are the mechanisms of action for cancer drugs?<br>----------------------------------------</pre><pre>============================================================<br>Natural Query: Which diseases have the most genetic associations?<br>============================================================<br>Generated SQL:<br>SELECT <br>    d.name AS disease_name,<br>    COUNT(DISTINCT a.targetId) AS genetic_association_count<br>FROM <br>    association_overall_direct a<br>JOIN <br>    disease d ON a.diseaseId = d.id<br>WHERE <br>    a.score &gt; 0.5<br>GROUP BY <br>    d.name<br>ORDER BY <br>    genetic_association_count DESC<br>LIMIT 100;<br>============================================================<br>✅ Results (100 rows):</pre><pre>Row 1:<br>  disease_name: neurodegenerative disease<br>  genetic_association_count: 1382</pre><pre>Row 2:<br>  disease_name: genetic disorder<br>  genetic_association_count: 404</pre><pre>Row 3:<br>  disease_name: platelet count<br>  genetic_association_count: 401</pre><pre>Row 4:<br>  disease_name: glomerular filtration rate<br>  genetic_association_count: 379</pre><pre>============================================================<br>Natural Query: What genes are associated with breast cancer?<br>============================================================<br>Generated SQL:<br>SELECT DISTINCT t.approvedSymbol AS gene_symbol, t.approvedName AS gene_name, d.name AS disease_name, a.score AS association_score<br>FROM association_overall_direct a<br>JOIN target t ON a.targetId = t.id<br>JOIN disease d ON a.diseaseId = d.id<br>WHERE d.name LIKE &#39;%breast cancer%&#39;<br>   OR array_join(d.synonyms.hasExactSynonym, &#39; &#39;) LIKE &#39;%breast cancer%&#39;<br>   OR array_join(d.synonyms.hasRelatedSynonym, &#39; &#39;) LIKE &#39;%breast cancer%&#39;<br>ORDER BY a.score DESC<br>LIMIT 100;<br>============================================================<br>✅ Results (100 rows):</pre><pre>Row 1:<br>  gene_symbol: BRCA2<br>  gene_name: BRCA2 DNA repair associated<br>  disease_name: breast cancer<br>  association_score: 0.8548846942723206</pre><pre>Row 2:<br>  gene_symbol: BRCA1<br>  gene_name: BRCA1 DNA repair associated<br>  disease_name: breast cancer<br>  association_score: 0.838391491656388</pre><pre>Row 3:<br>  gene_symbol: AR<br>  gene_name: androgen receptor<br>  disease_name: partial androgen insensitivity syndrome<br>  association_score: 0.8229769855485846</pre><pre>Row 4:<br>  gene_symbol: CHEK2<br>  gene_name: checkpoint kinase 2<br>  disease_name: breast cancer<br>  association_score: 0.8155025936777466</pre><pre>Row 5:<br>  gene_symbol: CDH1<br>  gene_name: cadherin 1<br>  disease_name: CDH1-related diffuse gastric and lobular breast cancer syndrome<br>  association_score: 0.7950273350532806</pre><p>Here are some additional results from various queries:</p><h3>Example Results</h3><p>High-confidence associations discovered:</p><ul><li><strong>CFTR ↔ cystic fibrosis</strong>: 91.3% confidence</li><li><strong>F8 ↔ hemophilia A</strong>: 90.8% confidence</li><li><strong>MYH7 ↔ hypertrophic cardiomyopathy</strong>: 89.8% confidence</li></ul><h3>Data Source Analysis</h3><p>Evidence diversity across different sources:</p><ul><li><strong>UniProt variants</strong>: Highest average confidence (75.8%)</li><li><strong>Reactome pathways</strong>: Strong pathway evidence (73.6%)</li><li><strong>Genomics England</strong>: Clinical evidence (71.7%)</li></ul><h3>Lessons Learned</h3><p>Break the problem down into small pieces. Don’t try to do everything at once. Especially don’t let the AI try to do everything at once!</p><h3>Conclusion</h3><p>Building a natural language interface for genomic data demonstrates the power of combining modern cloud infrastructure with AI capabilities. Our solution provides:</p><ul><li><strong>Accessibility</strong>: Researchers can query complex genomic data without SQL expertise. No code required!</li><li><strong>Cost Efficiency</strong>: Serverless architecture scales with usage</li><li><strong>Reliability</strong>: Comprehensive testing ensures data integrity</li><li><strong>Extensibility</strong>: MCP protocol enables broad integration possibilities</li></ul><p>The project showcases how thoughtful architecture decisions, thorough testing, and modern AI capabilities can democratize access to complex scientific datasets. By removing technical barriers, we enable researchers to focus on discovery rather than data wrangling.</p><p><strong>Key Takeaways:</strong></p><ol><li><strong>Test Early and Thoroughly</strong>: Schema validation prevented major production issues</li><li><strong>Embrace Serverless</strong>: Pay-per-use models align perfectly with research workloads</li><li><strong>Standardize Interfaces</strong>: MCP protocol provides future-proof integration capabilities</li><li><strong>Document Everything</strong>: Comprehensive testing and documentation enable team collaboration</li></ol><p>The complete codebase, including infrastructure definitions and test suites, demonstrates an approach to building AI-powered data interfaces for scientific research.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=109c6eb934fa" width="1" height="1" alt=""><hr><p><a href="https://medium.com/data-science-collective/building-a-natural-language-interface-for-opentargets-data-with-ai-on-aws-109c6eb934fa">Building a Natural Language Interface for OpenTargets Data with AI on AWS</a> was originally published in <a href="https://medium.com/data-science-collective">Data Science Collective</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Using LLMs to Query PubMed Knowledge Bases for BioMedical Research]]></title>
            <link>https://medium.com/data-science/using-llms-to-query-pubmed-knowledge-bases-for-biomedical-research-bc00ead9e540?source=rss-87b832c70819------2</link>
            <guid isPermaLink="false">https://medium.com/p/bc00ead9e540</guid>
            <category><![CDATA[drug-discovery]]></category>
            <category><![CDATA[bioinformatics]]></category>
            <category><![CDATA[aws-bedrock]]></category>
            <category><![CDATA[ai]]></category>
            <category><![CDATA[llms-in-healthcare]]></category>
            <dc:creator><![CDATA[Jillian Rowe]]></dc:creator>
            <pubDate>Wed, 24 Jul 2024 20:26:30 GMT</pubDate>
            <atom:updated>2024-07-24T20:26:30.122Z</atom:updated>
            <content:encoded><![CDATA[<h4>AI for fun and profit!</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*FK9P8wahsg9O3L3s" /><figcaption>Photo by <a href="https://unsplash.com/@itfeelslikefilm?utm_source=medium&amp;utm_medium=referral">🇸🇮 Janko Ferlič</a> on <a href="https://unsplash.com/?utm_source=medium&amp;utm_medium=referral">Unsplash</a></figcaption></figure><h3>Part 1: Creating the Knowledge base</h3><p>In this article, we’ll explore how to leverage large language models (LLMs) to search and scientific papers from <a href="https://www.ncbi.nlm.nih.gov/pmc/tools/openftlist/">PubMed Open Access</a> Subset, a free resource for accessing biomedical and life sciences literature. We’ll use Retrieval-Augmented Generation, RAG, to search our digital library.</p><p>AWS Bedrock will act as our AI backend, PostgreSQL as the vector database for storing embeddings, and the LangChain library in Python will ingest papers and query the knowledge base.</p><p>If you only care about the results generated by querying the knowledge base, skip down to the end.</p><p>The specific use case we’ll be focusing on is querying papers related to Rheumatoid Arthritis, a chronic inflammatory disorder affecting joints. We’ll use the query ((rheumatoid arthritis) AND gene) AND cell to retrieve around 10,000 relevant papers from PubMed and then sample that down to approximately 5,000 papers for our knowledge base.</p><h3>Disclaimer</h3><p>Not all research articles or sources have licensing that allows for ingesting with AI!</p><p>I’m not including all the source code because the AI libraries change so frequently and because there are oodles of ways to configure a knowledge base backend, but I have included some helper functions so you can follow along.</p><h3>PGVector: Storing Embeddings in a Vector Database</h3><p>To make it easier for the LLM to process and understand the textual data from the research papers, we’ll convert the text into numerical embeddings, which are dense vector representations of the text. These embeddings will be stored in a PostgreSQL database using the PGVector library. This step essentially simplifies the text data into a format that the LLM can more easily work with.</p><p>I’m running a local postgresql database, which is fine for my datasets. Hosting AWS Bedrock Knowledgebases can get expensive, and I’m not trying to run up my AWS bill this month. It’s summer, and I have kids camp to pay for!</p><h3>AWS Bedrock: The AI Backend</h3><p>AWS Bedrock is a managed service provided by Amazon Web Services (AWS), allowing you to easily deploy and operate large language models. In our setup, Bedrock will host the LLM that we’ll use to query and retrieve relevant information from our knowledge base of research papers.</p><h3>LangChain: Loading and Querying the Knowledge Base</h3><p>LangChain is a Python library that simplifies building applications with large language models. We’ll use <a href="https://www.langchain.com/">LangChain</a> to load our research papers and their associated embeddings into a knowledge base and then query this knowledge base using the LLM hosted on AWS Bedrock.</p><h3>Using PubGet for Data Acquisition</h3><p>While this setup can work with research papers from any source, we’re using PubMed because it’s a convenient source for acquiring a large volume of papers based on specific search queries. We’ll use the <a href="https://github.com/neuroquery/pubget">PubGet</a> tool to retrieve the initial set of 10,000 papers matching our query on Rheumatoid Arthritis, genes, and cells. Behind the scenes pubget fetches articles from the PubMed FTP service.</p><pre>pubget run -q &quot;((rheumatoid arthritis) AND gene) AND cell&quot; \<br> pubget_data</pre><p>This will get us articles in xml format.</p><h3>Structuring and Organizing the Dataset</h3><p>Beyond the technical aspects, this article will focus on how to structure and organize your dataset of research papers effectively.</p><ol><li><strong>Dataset</strong>: Managing your datasets on a global level using collections.</li><li><strong>Metadata Management</strong>: Handling and incorporating metadata associated with the papers, such as author information, publication dates, and keywords.</li></ol><p>You’ll want to think about this upfront. When using LangChain, you query datasets based on their collections. Each collection has a name and a unique identifier.</p><p>When you load your data, whether it’s pdf papers, xml downloads, markdown files, codebases, powerpoint slides, text documents, etc, you can attach additional metadata. You can later use this metadata to filter your results. The metadata is an open dictionary, and you can add tags, source, phenotype, or anything you think may be relevant.</p><h3>Loading and Querying the Knowledge Base</h3><p>The article will also cover best practices for loading your preprocessed and structured dataset into the knowledge base and provide examples of how to query the knowledge base effectively using the LLM hosted on AWS Bedrock.</p><p>By the end of this article, you should have a solid understanding of how to leverage LLMs to search and retrieve relevant information from a large corpus of research papers, as well as strategies for structuring and organizing your dataset to optimize the performance and accuracy of your knowledge base.</p><pre>import boto3<br>import pprint<br>import os<br>import boto3<br>import json<br>import hashlib<br>import funcy<br>import glob<br>from typing import Dict, Any, TypedDict, List<br>from langchain.llms.bedrock import Bedrock<br>from langchain.retrievers.bedrock import AmazonKnowledgeBasesRetriever<br>from langchain_core.documents import Document<br>from langchain_aws import ChatBedrock<br>from langchain_community.embeddings import BedrockEmbeddings # to create embeddings for the documents.<br>from langchain_experimental.text_splitter import SemanticChunker # to split documents into smaller chunks.<br>from langchain_text_splitters import CharacterTextSplitter<br>from langchain_postgres import PGVector<br>from pydantic import BaseModel, Field<br>from langchain_community.document_loaders import (<br>    WebBaseLoader,<br>    TextLoader,<br>    PyPDFLoader,<br>    CSVLoader,<br>    Docx2txtLoader,<br>    UnstructuredEPubLoader,<br>    UnstructuredMarkdownLoader,<br>    UnstructuredXMLLoader,<br>    UnstructuredRSTLoader,<br>    UnstructuredExcelLoader,<br>    DataFrameLoader,<br>)<br>import psycopg<br>import uuid</pre><h3>Set up your database connection</h3><p>I’m running a local <a href="https://supabase.com/">Supabase</a> postgresql database running using their docker-compose setup. In a production setup, I&#39;d recommend using a real database, like AWS AuroraDB or Supabase running someplace besides your laptop. Also, change your password to something besides password.</p><p>I didn’t notice any difference in performance for smaller datasets between an AWS-hosted knowledgebase and my laptop, but your mileage may vary.</p><pre>connection = f&quot;postgresql+psycopg://{user}:{password}@{host}:{port}/{database}&quot; <br># Establish the connection to the database<br>conn = psycopg.connect(<br>    conninfo = f&quot;postgresql://{user}:{password}@{host}:{port}/{database}&quot;<br>)<br># Create a cursor to run queries<br>cur = conn.cursor()</pre><h3>Insert AWS BedRock Embeddings into the table using Langchain</h3><p>We’re using AWS Bedrock as our AI Knowledgebase. Most of the companies I work with have some kind of proprietary data, and Bedrock has a guarantee that your data will remain private. You could use any of the AI backends here.</p><pre>os.environ[&#39;AWS_DEFAULT_REGION&#39;] = &#39;us-east-1&#39;<br>bedrock_client              = boto3.client(&quot;bedrock-runtime&quot;) <br>bedrock_embeddings          = BedrockEmbeddings(model_id=&quot;amazon.titan-embed-text-v1&quot;,client=bedrock_client)<br>bedrock_embeddings_image    = BedrockEmbeddings(model_id=&quot;amazon.titan-embed-image-v1&quot;,client=bedrock_client)<br>llm                         = ChatBedrock(model_id=&quot;anthropic.claude-3-sonnet-20240229-v1:0&quot;, client=bedrock_client)</pre><pre># function to create vector store<br># make sure to update this if you change collections!<br>def create_vectorstore(embeddings,collection_name,conn):                       <br>    vectorstore = PGVector(<br>        embeddings=embeddings,<br>        collection_name=collection_name,<br>        connection=conn,<br>        use_jsonb=True,<br>    )<br>    return vectorstore</pre><pre>def load_and_split_pdf_semantic(file_path, embeddings):<br>    loader = PyPDFLoader(file_path)<br>    pages = loader.load_and_split()<br>    return pages</pre><pre>def load_xml(file_path, embeddings):<br>    loader = UnstructuredXMLLoader(<br>        file_path,<br>    )<br>    docs = loader.load_and_split()<br>    return docs</pre><pre>def insert_embeddings(files, bedrock_embeddings, vectorstore):<br>    logging.info(f&quot;Inserting {len(files)}&quot;)<br>    x = 1<br>    y = len(files)<br>    for file_path in files:<br>        logging.info(f&quot;Splitting {file_path} {x}/{y}&quot;)<br>        docs = []<br>        if &#39;.pdf&#39; in file_path:<br>            try:<br>                with funcy.print_durations(&#39;process pdf&#39;):                <br>                    docs = load_and_split_pdf_semantic(file_path, bedrock_embeddings)<br>            except Exception as e:<br>                logging.warning(f&quot;Error loading docs&quot;)<br>        if &#39;.xml&#39; in file_path:<br>            try:<br>                with funcy.print_durations(&#39;process xml&#39;):                <br>                    docs = load_xml(file_path, bedrock_embeddings)<br>            except Exception as e:<br>                logging.warning(e)<br>                logging.warning(f&quot;Error loading docs&quot;)<br>        filtered_docs = []<br>        for d in docs:<br>            if len(d.page_content):<br>                filtered_docs.append(d)<br>        # Add documents to the vectorstore<br>        ids = []<br>        for d in filtered_docs:<br>            ids.append(<br>                hashlib.sha256(d.page_content.encode()).hexdigest()<br>            )<br>            <br>        if len(filtered_docs):<br>            texts = [ i.page_content for i in filtered_docs] <br>            # metadata is a dictionary. You can add to it!<br>            metadatas = [ i.metadata for i in filtered_docs]<br>            #logging.info(f&quot;Adding N: {len(filtered_docs)}&quot;)<br>            try:<br>                with funcy.print_durations(&#39;load psql&#39;):<br>                    vectorstore.add_texts(texts=texts, metadatas = metadatas, ids=ids)<br>            except Exception as e:<br>                logging.warning(e)<br>                logging.warning(f&quot;Error {x - 1}/{y}&quot;)<br>            #logging.info(f&quot;Complete {x}/{y}&quot;)<br>        x = x + 1</pre><pre>collection_name_text = &quot;MY_COLLECTION&quot; #pubmed, smiles, etc<br>vectorstore = create_vectorstore(bedrock_embeddings,collection_name_text,connection)</pre><h3>Load and process Pubmed XML Papers</h3><p>Most of our data was fetched using the pubget tool, and the articles are in XML format. We&#39;ll use the LangChain XML Loader to process, split and load the embeddings.</p><pre>files = glob.glob(&quot;/home/jovyan/data/pubget_ra/pubget_data/*/articles/*/*/article.xml&quot;)</pre><pre>#I ran this previously<br>insert_embeddings(files[0:2], bedrock_embeddings, vectorstore)</pre><h3>Load and Process Pubmed PDF Papers</h3><p>PDFs are easier to read, and I grabbed some for doing QA against the knowledge base.</p><pre>files = glob.glob(&quot;/home/jovyan/data/pubget_ra/papers/*pdf&quot;)<br>insert_embeddings(files[0:2], bedrock_embeddings, vectorstor</pre><h3>Part 2 — Query the Knowledge base</h3><p>Now that we have our knowledge base setup we can use Retrieval Augmented Generation, RAG methods, to use the LLMs to run queries.</p><p>Our queries are:</p><ul><li>Tell me about T cell–derived cytokines in relation to rheumatoid arthritis and provide citations and article titles</li><li>Tell me about single-cell research in rheumatoid arthritis.</li><li>Tell me about protein-protein associations in rheumatoid arthritis.</li><li>Tell me about the findings of GWAS studies in rheumatoid arthritis.</li></ul><pre>import hashlib<br>import logging<br>import os<br>from typing import Optional, List, Dict, Any<br>import glob<br>import boto3<br>from toolz.itertoolz import partition_all<br>import json<br>import funcy<br>import psycopg<br>from IPython.display import Markdown, display<br>from langchain.chains import create_retrieval_chain<br>from langchain.chains.combine_documents import create_stuff_documents_chain<br>from langchain.prompts import PromptTemplate<br>from langchain.retrievers.bedrock import (<br>    AmazonKnowledgeBasesRetriever,<br>    RetrievalConfig,<br>    VectorSearchConfig,<br>)<br>from aws_bedrock_utilities.models.base import BedrockBase, RAGResults<br>from aws_bedrock_utilities.models.pgvector_knowledgebase import BedrockPGWrapper<br>from sklearn.model_selection import train_test_split<br>import numpy as np<br>import pandas as pd<br>from pprint import pprint<br>import time<br>import logging<br>from rich.logging import RichHandler</pre><p>I don’t list it here, but I’ll always do some QA against my knowlegebase. Choose an article, parse out the summary or findings, and ask the LLM about it. You should get your article back.</p><h3>Structuring Your Queries</h3><p>You’ll need to first have the collection name you’re querying along with your queries.</p><p>I always recommend running a few QA queries. Ask the obvious questions in several different ways.</p><p>You’ll also want to adjust the MAX_DOCS_RETURNED based on your time constraints and how many articles are in your knowledgebase. The LLM will search until it hits that maximum, and then stops. You&#39;ll need to increase that number for an exhaustive search.</p><pre># Make sure to keep the collection name consistent!<br>COLLECTION_NAME   = &quot;MY_COLLECTION&quot;<br>MAX_DOCS_RETURNED = 50</pre><pre>p = BedrockPGWrapper(collection_name=COLLECTION_NAME)                              credentials.py:1147</pre><pre>#model = &quot;anthropic.claude-3-sonnet-20240229-v1:0&quot;<br>model = &quot;anthropic.claude-3-haiku-20240307-v1:0&quot;</pre><pre>model = &quot;anthropic.claude-3-haiku-20240307-v1:0&quot;<br>queries = [<br>    &quot;Tell me about T cell–derived cytokines in relation to rheumatoid arthritis and provide citations and article titles&quot;,<br>    &quot;Tell me about single-cell research in rheumatoid arthritis.&quot;,<br>    &quot;Tell me about protein-protein associations in rheumatoid arthritis.&quot;,<br>    &quot;Tell me about the findings of GWAS studies in rheumatoid arthritis.&quot;,<br>    <br>]<br>ai_responses = []<br>for query in queries:<br>    answer = p.run_kb_chat(query=query, collection_name= COLLECTION_NAME, model_id=model, search_kwargs={&#39;k&#39;: MAX_DOCS_RETURNED, &#39;fetch_k&#39;: 50000 })<br>    ai_responses.append(answer)<br>    time.sleep(1)</pre><pre>for answer in ai_responses:<br>    t = Markdown(f&quot;&quot;&quot;<br>### Query <br>{answer[&#39;query&#39;]}### Response<br>{answer[&#39;result&#39;]}<br> <br>    &quot;&quot;&quot;)<br>    display(t)</pre><h3>Part 3: Results!</h3><p>We’ve built our knowledge base, run some queries, and now we’re ready to look at the results the LLM generated for us.</p><p>Each result is a dictionary with the original query, the response, and the relevant snippets of the source document.</p><blockquote>Tell me about T cell–derived cytokines in relation to rheumatoid arthritis and provide citations and article titles</blockquote><blockquote>T cell-derived cytokines play a key role in the pathogenesis of rheumatoid arthritis (RA). Some key findings include:</blockquote><blockquote>Increased levels of IL-17, a cytokine produced by Th17 cells, have been found in the synovial fluid of RA patients. IL-17 can stimulate fibroblast-like synoviocytes (FLS) and macrophages to produce inflammatory mediators like VEGF, IL-1, IL-6, TNF-α, and prostaglandin E2, and promote osteoclast formation, contributing to joint inflammation and destruction (Honorati et al. 2006, Schurgers et al. 2011).</blockquote><blockquote>Th1 cells, which produce IFN-γ, are also implicated in RA pathogenesis. IFN-γ can induce macrophages to polarize towards a pro-inflammatory M1 phenotype (Schurgers et al. 2011, Kebir et al. 2009, Boniface et al. 2010).</blockquote><blockquote>CD161+ Th17 cells, which can produce both IL-17 and IFN-γ, are enriched in the synovium of RA patients and may contribute to the inflammatory environment (Afzali et al. 2013, Bovenschen et al. 2011, Koenen et al. 2008, Pesenacker et al. 2013).</blockquote><blockquote>Regulatory T cells (Tregs), which normally suppress inflammation, show impaired function in RA, potentially contributing to the dysregulated immune response (Moradi et al. 2014, Samson et al. 2012, Zhang et al. 2018, Walter et al. 2013, Wang et al. 2018, Morita et al. 2016).</blockquote><blockquote>In summary, the imbalance between pro-inflammatory T cell subsets (Th1, Th17) and anti-inflammatory Tregs is a hallmark of RA pathogenesis, with cytokines like IL-17 and IFN-γ playing central roles in driving joint inflammation and destruction.</blockquote><blockquote>Tell me about single-cell research in rheumatoid arthritis.</blockquote><blockquote>Single-cell research has provided important insights into the pathogenesis of rheumatoid arthritis (RA):</blockquote><blockquote>Single-cell RNA sequencing (scRNA-seq) studies have identified distinct cell states and subpopulations within the RA synovium, including pathogenic T cell subsets like T peripheral helper (Tph) cells and cytotoxic CD8+ T cells.</blockquote><blockquote>Analyses of the T cell receptor (TCR) repertoire in the RA synovium have revealed clonal expansion of CD4+ and CD8+ T cell populations, suggesting antigen-driven responses.</blockquote><blockquote>scRNA-seq has also characterized expanded populations of activated B cells, plasmablasts, and plasma cells in the RA synovium that demonstrate substantial clonal relationships.</blockquote><blockquote>Receptor-ligand analyses from scRNA-seq data have predicted key cell-cell interactions, such as between Tph cells and B cells, that may drive synovial inflammation.</blockquote><blockquote>Overall, single-cell studies have uncovered the cellular and molecular heterogeneity within the RA synovium, identifying specific immune cell subsets and pathways that could serve as targets for more personalized therapeutic approaches.</blockquote><blockquote>Tell me about protein-protein associations in rheumatoid arthritis.</blockquote><blockquote>Based on the information provided in the context, some key protein-protein associations in rheumatoid arthritis (RA) include:</blockquote><blockquote>Rheumatoid factor (RF) and anti-citrullinated protein antibodies (ACPAs):</blockquote><blockquote>RF is found in about 80% of patients in the pre-articular phase of RA.</blockquote><blockquote>ACPAs are highly specific for RA and can be detected years before the onset of clinical symptoms.</blockquote><blockquote>Peptidylarginine deiminase (PAD) enzymes and anti-PAD antibodies:</blockquote><blockquote>Anti-PAD2 antibodies are associated with a moderate disease course, while anti-PAD4 antibodies are linked to more severe and rapidly progressive RA.</blockquote><blockquote>Anti-PAD3/4 antibodies may signal the development of RA-associated interstitial lung disease.</blockquote><blockquote>Anti-carbamylated protein (anti-CarP) antibodies:</blockquote><blockquote>Anti-CarP antibodies are present in 25–50% of RA patients, independent of RF or ACPA positivity.</blockquote><blockquote>Anti-CarP antibodies are associated with poor prognosis and increased morbidity, including RA-associated interstitial lung disease.</blockquote><blockquote>Malondialdehyde-acetaldehyde (MAA) adducts and anti-MAA antibodies:</blockquote><blockquote>Anti-MAA antibodies are associated with radiological progression in seronegative RA.</blockquote><blockquote>Protein-protein interactions in signaling pathways:</blockquote><blockquote>The JAK-STAT, MAPK, PI3K-AKT, and SYK signaling pathways are all implicated in the pathogenesis of RA and are potential targets for therapeutic intervention.</blockquote><blockquote>In summary, the context highlights several key protein-protein associations in RA, including autoantibodies (RF, ACPAs, anti-PAD, anti-CarP, anti-MAA) and signaling pathway components (JAK, STAT, MAPK, PI3K, SYK), which play important roles in the pathogenesis and progression of the disease.</blockquote><blockquote>Tell me about the findings of GWAS studies in rheumatoid arthritis.</blockquote><blockquote>Here are some key findings from GWAS studies in rheumatoid arthritis:</blockquote><blockquote>Genome-wide association studies (GWAS) have identified over 100 genetic risk loci associated with rheumatoid arthritis susceptibility.</blockquote><blockquote>The HLA-DRB1 gene is the strongest genetic risk factor, accounting for about 50% of the genetic component of rheumatoid arthritis. Specific HLA-DRB1 alleles containing the “shared epitope” sequence are strongly associated with increased RA risk.</blockquote><blockquote>Other notable genetic risk factors identified through GWAS include PTPN22, STAT4, CCR6, PADI4, CTLA4, and CD40. These genes are involved in immune regulation and inflammation pathways.</blockquote><blockquote>Genetic risk factors can differ between seropositive (ACPA-positive) and seronegative rheumatoid arthritis. For example, HLA-DRB1 alleles have a stronger association with seropositive RA.</blockquote><blockquote>GWAS have also identified genetic variants associated with disease severity and response to treatment in rheumatoid arthritis. For example, variants in the FCGR3A and PTPRC genes have been linked to response to anti-TNF therapy.</blockquote><blockquote>Overall, GWAS have provided important insights into the genetic architecture and pathogenesis of rheumatoid arthritis, which has implications for developing targeted therapies and personalized treatment approaches.</blockquote><h3>Investigate the Source Documents</h3><p>Querying the Knowledge base will return relevant snippets of the source documents. Sometimes the formatting returned by langchain can be a bit off, but you can always go back to the source.</p><pre>x = 0<br>y = 10<br>for answer in ai_responses:<br>    for s in answer[&#39;source_documents&#39;]:<br>        if x &lt;= y:<br>            print(s.metadata)<br>        else:<br>            break<br>        x = x + 1</pre><pre>{&#39;page&#39;: 6, &#39;source&#39;: &#39;/home/jovyan/data/pubget_ra/papers/fimmu-12-790122.pdf&#39;}<br>{&#39;source&#39;: &#39;/home/jovyan/data/pubget_ra/pubget_data/query_55c6003c0195b20fd4bdc411f67a8dcf/articles/d52/pmcid_11167034/article.xml&#39;}<br>{&#39;page&#39;: 7, &#39;source&#39;: &#39;/home/jovyan/data/pubget_ra/papers/fimmu-12-790122.pdf&#39;}<br>{&#39;page&#39;: 17, &#39;source&#39;: &#39;/home/jovyan/data/pubget_ra/papers/fimmu-12-790122.pdf&#39;}<br>{&#39;source&#39;: &#39;/home/jovyan/data/pubget_ra/pubget_data/query_55c6003c0195b20fd4bdc411f67a8dcf/articles/657/pmcid_11151399/article.xml&#39;}<br>{&#39;page&#39;: 3, &#39;source&#39;: &#39;/home/jovyan/data/pubget_ra/papers/41392_2023_Article_1331.pdf&#39;}<br>{&#39;page&#39;: 4, &#39;source&#39;: &#39;/home/jovyan/data/pubget_ra/papers/41392_2023_Article_1331.pdf&#39;}<br>{&#39;page&#39;: 5, &#39;source&#39;: &#39;/home/jovyan/data/pubget_ra/papers/fimmu-12-790122.pdf&#39;}<br>{&#39;source&#39;: &#39;/home/jovyan/data/pubget_ra/pubget_data/query_55c6003c0195b20fd4bdc411f67a8dcf/articles/e02/pmcid_11219584/article.xml&#39;}<br>{&#39;page&#39;: 8, &#39;source&#39;: &#39;/home/jovyan/data/pubget_ra/papers/fimmu-12-790122.pdf&#39;}<br>{&#39;source&#39;: &#39;/home/jovyan/data/pubget_ra/pubget_data/query_55c6003c0195b20fd4bdc411f67a8dcf/articles/6fb/pmcid_11203675/article.xml&#39;}</pre><h3>Wrap Up</h3><p>There you have it! We created a knowledge base on the cheap, used AWS Bedrock to load the embeddings, and then used a Claude LLM to run our queries. Here we used PubMed papers, but we also could have used meeting notes, powerpoint slides, crawled websites, or in house databases.</p><p>If you have any questions, comments, or tutorial requests please don’t hesitate to reach out to me by email at jillian@dabbleofdevops.com</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=bc00ead9e540" width="1" height="1" alt=""><hr><p><a href="https://medium.com/data-science/using-llms-to-query-pubmed-knowledge-bases-for-biomedical-research-bc00ead9e540">Using LLMs to Query PubMed Knowledge Bases for BioMedical Research</a> was originally published in <a href="https://medium.com/data-science">TDS Archive</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Get a Fully Configured Apache Airflow Docker Dev Stack with Bitnami]]></title>
            <link>https://medium.com/data-science/get-a-fully-configured-apache-airflow-docker-dev-stack-with-bitnami-ed1671d63ea0?source=rss-87b832c70819------2</link>
            <guid isPermaLink="false">https://medium.com/p/ed1671d63ea0</guid>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[docker]]></category>
            <category><![CDATA[apache-airflow]]></category>
            <category><![CDATA[docker-compose]]></category>
            <category><![CDATA[python]]></category>
            <dc:creator><![CDATA[Jillian Rowe]]></dc:creator>
            <pubDate>Mon, 03 Aug 2020 14:48:12 GMT</pubDate>
            <atom:updated>2020-08-03T14:48:12.099Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*gR3JFYCpIj_OiNpR.jpg" /></figure><p>I’ve been using it for around 2 years now to build out custom workflow interfaces, like those used for Laboratory Information Management Systems (LIMs), Computer Vision pre and postprocessing pipelines, and to set and forget other genomics pipelines.</p><p>My favorite feature of Airflow is how completely agnostic it is to the work you are doing or where that work is taking place. It could take place locally, on a Docker image, on Kubernetes, on any number of AWS services, on an HPC system, etc. Using Airflow allows me to concentrate on the business logic of what I’m trying to accomplish without getting too bogged down in implementation details.</p><p>During that time I’ve adopted a set of systems that I use to quickly build out the main development stack with Docker and Docker Compose, using the <a href="https://github.com/bitnami/bitnami-docker-airflow">Bitnami Apache Airflow stack</a>. Generally, I either deploy the stack to production using either the same Docker compose stack if its a small enough instance that is isolated, or with Kubernetes when I need to interact with other services or file systems.</p><h3>Bitnami vs Roll Your Own</h3><p>I used to roll my own Airflow containers using Conda. I still use this approach for most of my other containers, including microservices that interact with my Airflow system, but configuring Airflow is a lot more than just installing packages. Also, even just installing those packages is a pain and I could rarely count on a rebuild actually working without some pain. Then, on top of the packages you need to configure database connections and a message queue.</p><p>In comes the <a href="https://github.com/bitnami/bitnami-docker-airflow">Bitnami Apache Airflow docker compose stack</a> for dev and <a href="https://bitnami.com/stack/apache-airflow/helm">Bitnami Apache Airflow Helm Chart</a> for prod!</p><p><a href="https://bitnami.com/">Bitnami</a>, in their own words:</p><blockquote><em>Bitnami makes it easy to get your favorite open source software up and running on any platform, including your laptop, Kubernetes and all the major clouds. In addition to popular community offerings, Bitnami, now part of VMware, provides IT organizations with an enterprise offering that is secure, compliant, continuously maintained and customizable to your organizational policies. </em><a href="https://bitnami.com/"><em>https://bitnami.com/</em></a></blockquote><p>Bitnami stacks (usually) work completely the same from their Docker Compose stacks to their Helm charts. This means I can test and develop locally using my compose stack, build out new images, versions, packages, etc, and then deploy to Kubernetes. The configuration, environmental variables, and everything else acts the same. It would be a fairly large undertaking to do all this from scratch, so I use Bitnami.</p><p>They have plenty of enterprise offerings, but everything included here is open source and there is no paywall involved.</p><p>And no, I am not affiliated with Bitnami, although I have kids that eat a lot and don’t have any particular ethical aversions to selling out. ;-) I’ve just found their offerings to be <em>excellent</em>.</p><h3>Project Structure</h3><p>I like to have my projects organized so that I can run tree and have a general idea of what&#39;s happening.</p><p>Apache Airflow has 3 main components, the application, the worker, and the scheduler. Each of these has it’s own Docker image to separate out the services. Additionally, there is a database and an message queue, but we won’t be doing any customization to these.</p><pre>.<br>└── docker<br>    └── bitnami-apache-airflow-1.10.10<br>        ├── airflow<br>        │   └── Dockerfile<br>        ├── airflow-scheduler<br>        │   └── Dockerfile<br>        ├── airflow-worker<br>        │   └── Dockerfile<br>        ├── dags<br>        │   └── tutorial.py<br>        ├── docker-compose.yml</pre><p>So what we have here is a directory called bitnami-apache-airflow-1.10.10. Which brings us to a very important point! Pin your versions! It will save you so, so much pain and frustration!</p><p>Then we have one Dockerfile per Airflow piece.</p><p>Create this directory structure with:</p><pre>mkdir -p docker/bitnami-apache-airflow-1.10.10/{airflow,airflow-scheduler,airflow-worker,dags}</pre><h3>The Docker Compose File</h3><p>This is my preference for the docker-compose.yml file. I made a few changes for my own preferences, mostly that I <em>pin versions</em>, build my own Docker images, I have volume mounts for the dags, plugins, and database backups along with adding in the docker socket so I can run DockerOperators from within my stack.</p><p>You can always go and grab the original docker-compose <a href="https://github.com/bitnami/bitnami-docker-airflow/blob/master/docker-compose.yml">here</a>.</p><pre>version: &#39;2&#39;<br><br>services:<br>  postgresql:<br>    image: &#39;docker.io/bitnami/postgresql:10-debian-10&#39;<br>    volumes:<br>      - &#39;postgresql_data:/bitnami/postgresql&#39;<br>    environment:<br>      - POSTGRESQL_DATABASE=bitnami_airflow<br>      - POSTGRESQL_USERNAME=bn_airflow<br>      - POSTGRESQL_PASSWORD=bitnami1<br>      - ALLOW_EMPTY_PASSWORD=yes<br>  redis:<br>    image: docker.io/bitnami/redis:5.0-debian-10<br>    volumes:<br>      - &#39;redis_data:/bitnami&#39;<br>    environment:<br>      - ALLOW_EMPTY_PASSWORD=yes<br>  airflow-scheduler:<br>#    image: docker.io/bitnami/airflow-scheduler:1-debian-10<br>    build:<br>      context: airflow-scheduler<br>    environment:<br>      - AIRFLOW_DATABASE_NAME=bitnami_airflow<br>      - AIRFLOW_DATABASE_USERNAME=bn_airflow<br>      - AIRFLOW_DATABASE_PASSWORD=bitnami1<br>      - AIRFLOW_EXECUTOR=CeleryExecutor<br>      # If you&#39;d like to load the example DAGs change this to yes!<br>      - AIRFLOW_LOAD_EXAMPLES=no<br>      # only works with 1.10.11<br>      #- AIRFLOW__WEBSERVER__RELOAD_ON_PLUGIN_CHANGE=true<br>      #- AIRFLOW__CORE__DAGS_ARE_PAUSED_AT_CREATION=False<br>    volumes:<br>      - airflow_scheduler_data:/bitnami<br>      - ./plugins:/opt/bitnami/airflow/plugins<br>      - ./dags:/opt/bitnami/airflow/dags<br>      - ./db_backups:/opt/bitnami/airflow/db_backups<br>      - /var/run/docker.sock:/var/run/docker.sock<br>  airflow-worker:<br>#    image: docker.io/bitnami/airflow-worker:1-debian-10<br>    build:<br>      context: airflow-worker<br>    environment:<br>      - AIRFLOW_DATABASE_NAME=bitnami_airflow<br>      - AIRFLOW_DATABASE_USERNAME=bn_airflow<br>      - AIRFLOW_DATABASE_PASSWORD=bitnami1<br>      - AIRFLOW_EXECUTOR=CeleryExecutor<br>      - AIRFLOW_LOAD_EXAMPLES=no<br>      # only works with 1.10.11<br>      #- AIRFLOW__WEBSERVER__RELOAD_ON_PLUGIN_CHANGE=true<br>      #- AIRFLOW__CORE__DAGS_ARE_PAUSED_AT_CREATION=False<br>    volumes:<br>      - airflow_worker_data:/bitnami<br>      - ./plugins:/opt/bitnami/airflow/plugins<br>      - ./dags:/opt/bitnami/airflow/dags<br>      - ./db_backups:/opt/bitnami/airflow/db_backups<br>      - /var/run/docker.sock:/var/run/docker.sock<br>  airflow:<br>#    image: docker.io/bitnami/airflow:1-debian-10<br>    build:<br>      # You can also specify the build context<br>      # as cwd and point to a different Dockerfile<br>      context: .<br>      dockerfile: airflow/Dockerfile<br>    environment:<br>      - AIRFLOW_DATABASE_NAME=bitnami_airflow<br>      - AIRFLOW_DATABASE_USERNAME=bn_airflow<br>      - AIRFLOW_DATABASE_PASSWORD=bitnami1<br>      - AIRFLOW_EXECUTOR=CeleryExecutor<br>      - AIRFLOW_LOAD_EXAMPLES=no<br>      # only works with 1.10.11<br>      #- AIRFLOW__WEBSERVER__RELOAD_ON_PLUGIN_CHANGE=True<br>      #- AIRFLOW__CORE__DAGS_ARE_PAUSED_AT_CREATION=False<br>    ports:<br>      - &#39;8080:8080&#39;<br>    volumes:<br>      - airflow_data:/bitnami<br>      - ./dags:/opt/bitnami/airflow/dags<br>      - ./plugins:/opt/bitnami/airflow/plugins<br>      - ./db_backups:/opt/bitnami/airflow/db_backups<br>      - /var/run/docker.sock:/var/run/docker.sock<br>volumes:<br>  airflow_scheduler_data:<br>    driver: local<br>  airflow_worker_data:<br>    driver: local<br>  airflow_data:<br>    driver: local<br>  postgresql_data:<br>    driver: local<br>  redis_data:<br>    driver: local</pre><h3>Pin your versions</h3><p>The version of Apache Airflow used here is 1.10.10. The 1.10.11 has some cool updates I would like to incorporate, so I will keep an eye on it!</p><p>You can always keep up with the latest Apache Airflow versions by checking out the <a href="https://airflow.apache.org/docs/stable/changelog.html">changelog</a> on the main site.</p><p>We are using Bitnami, which has bots that automatically build and update their images as new releases come along.</p><p>While this approach is great for bots, I <em>highly do not recommend</em> just hoping that the latest version will be backwards compatible and work with your setup.</p><p>Instead, pin a version, and when a new version comes along test it out in your dev stack. At the time of writing the most recent version is 1.10.11, but it doesn&#39;t quite work out of the box, so we are using 1.10.10.</p><h3>Bitnami Apache Airflow Docker Tags</h3><p>Generally speaking, a docker tag corresponds to the application version. Sometimes there are other variants as well, such as base OS. Here we can just go with the application version.</p><p><a href="https://hub.docker.com/r/bitnami/airflow-scheduler/tags">Bitnami Apache Airflow Scheduler Image Tags</a></p><p><a href="https://hub.docker.com/r/bitnami/airflow-worker/tags">Bitnami Apache Airflow Worker Image Tags</a></p><p><a href="https://hub.docker.com/r/bitnami/airflow/tags">Bitnami Apache Airflow Web Image Tags</a></p><h3>Build Custom Images</h3><p>In our docker-compose we have placeholders in order to build custom images.</p><p>We’ll just create a minimal Docker file for now. Later I’ll show you how to customize your docker container with extra system or python packages.</p><h3>Airflow Application</h3><pre>echo &quot;FROM docker.io/bitnami/airflow:1.10.10&quot; &gt; docker/bitnami-apache-airflow-1.10.10/airflow/Dockerfile</pre><p>Will give you this airflow application docker file.</p><pre>FROM docker.io/bitnami/airflow:1.10.10</pre><h3>Airflow Scheduler</h3><pre>echo &quot;FROM docker.io/bitnami/airflow-scheduler:1.10.10&quot; &gt; docker/bitnami-apache-airflow-1.10.10/airflow-scheduler/Dockerfile</pre><p>Will give you this airflow scheduler docker file.</p><pre>FROM docker.io/bitnami/airflow-scheduler:1.10.10</pre><h3>Airflow Worker</h3><pre>echo &quot;FROM docker.io/bitnami/airflow-worker:1.10.10&quot; &gt; docker/bitnami-apache-airflow-1.10.10/airflow-worker/Dockerfile</pre><p>Will give you this airflow worker docker file.</p><pre>FROM docker.io/bitnami/airflow-worker:1.10.10</pre><h3>Bring Up The Stack</h3><p>Grab the docker-compose file above and let&#39;s get rolling!</p><pre>cd docker/bitnami-apache-airflow-1.10.10 <br>docker-compose up</pre><p>If this is your first time running the command this will take some time. Docker will fetch any images it doesn’t already have, and build all the airflow-* images.</p><h3>Navigate to the UI</h3><p>Once everything is up and running navigate to the UI at <a href="http://localhost:8080.">http://localhost:8080.</a></p><p>Unless you changed the configuration, your default username/password is user/bitnami.</p><p>Login to check out your Airflow web UI!</p><h3>Add in a Custom DAG</h3><p>Here’s a DAG that I grabbed from the <a href="https://airflow.apache.org/docs/stable/tutorial.html">Apache Airflow Tutorial.</a> I’ve only included it here for the sake of completeness.</p><pre>from datetime import timedelta<br><br># The DAG object; we&#39;ll need this to instantiate a DAG<br>from airflow import DAG<br># Operators; we need this to operate!<br>from airflow.operators.bash_operator import BashOperator<br>from airflow.utils.dates import days_ago<br># These args will get passed on to each operator<br># You can override them on a per-task basis during operator initialization<br>default_args = {<br>    &#39;owner&#39;: &#39;airflow&#39;,<br>    &#39;depends_on_past&#39;: False,<br>    &#39;start_date&#39;: days_ago(2),<br>    &#39;email&#39;: [&#39;airflow@example.com&#39;],<br>    &#39;email_on_failure&#39;: False,<br>    &#39;email_on_retry&#39;: False,<br>    &#39;retries&#39;: 1,<br>    &#39;retry_delay&#39;: timedelta(minutes=5),<br>    # &#39;queue&#39;: &#39;bash_queue&#39;,<br>    # &#39;pool&#39;: &#39;backfill&#39;,<br>    # &#39;priority_weight&#39;: 10,<br>    # &#39;end_date&#39;: datetime(2016, 1, 1),<br>    # &#39;wait_for_downstream&#39;: False,<br>    # &#39;dag&#39;: dag,<br>    # &#39;sla&#39;: timedelta(hours=2),<br>    # &#39;execution_timeout&#39;: timedelta(seconds=300),<br>    # &#39;on_failure_callback&#39;: some_function,<br>    # &#39;on_success_callback&#39;: some_other_function,<br>    # &#39;on_retry_callback&#39;: another_function,<br>    # &#39;sla_miss_callback&#39;: yet_another_function,<br>    # &#39;trigger_rule&#39;: &#39;all_success&#39;<br>}<br>dag = DAG(<br>    &#39;tutorial&#39;,<br>    default_args=default_args,<br>    description=&#39;A simple tutorial DAG&#39;,<br>    schedule_interval=timedelta(days=1),<br>)<br><br># t1, t2 and t3 are examples of tasks created by instantiating operators<br>t1 = BashOperator(<br>    task_id=&#39;print_date&#39;,<br>    bash_command=&#39;date&#39;,<br>    dag=dag,<br>)<br><br>t2 = BashOperator(<br>    task_id=&#39;sleep&#39;,<br>    depends_on_past=False,<br>    bash_command=&#39;sleep 5&#39;,<br>    retries=3,<br>    dag=dag,<br>)<br>dag.doc_md = __doc__<br><br>t1.doc_md = &quot;&quot;&quot;\<br>#### Task Documentation<br>You can document your task using the attributes `doc_md` (markdown),<br>`doc` (plain text), `doc_rst`, `doc_json`, `doc_yaml` which gets<br>rendered in the UI&#39;s Task Instance Details page.<br>![img](http://montcs.bloomu.edu/~bobmon/Semesters/2012-01/491/import%20soul.png)<br>&quot;&quot;&quot;<br>templated_command = &quot;&quot;&quot;<br>{% for i in range(5) %}<br>    echo &quot;{{ ds }}&quot;<br>    echo &quot;{{ macros.ds_add(ds, 7)}}&quot;<br>    echo &quot;{{ params.my_param }}&quot;<br>{% endfor %}<br>&quot;&quot;&quot;<br><br>t3 = BashOperator(<br>    task_id=&#39;templated&#39;,<br>    depends_on_past=False,<br>    bash_command=templated_command,<br>    params={&#39;my_param&#39;: &#39;Parameter I passed in&#39;},<br>    dag=dag,<br>)<br><br>t1 &gt;&gt; [t2, t3]</pre><p>Anyways, grab this file and put it in your code/bitnami-apache-airflow-1.10.10/dags folder. The name of the file itself doesn&#39;t matter. The DAG name will be whatever you set in the file.</p><p>Airflow will restart itself automatically, and if you refresh the UI you should see your new tutorial DAG listed.</p><h3>Build Custom Airflow Docker Containers</h3><p>If you’d like to add additonal system or python packages you can do so.</p><pre># code/bitnami-apache-airflow-1.10.10/airflow/Dockerfile<br>FROM docker.io/bitnami/airflow:1.10.10<br># From here - https://github.com/bitnami/bitnami-docker-airflow/blob/master/1/debian-10/Dockerfile<br><br>USER root<br><br>RUN apt-get update &amp;&amp; apt-get upgrade -y &amp;&amp; \<br>    apt-get install -y vim &amp;&amp; \<br>    rm -r /var/lib/apt/lists /var/cache/apt/archives<br><br>RUN bash -c &quot;source /opt/bitnami/airflow/venv/bin/activate &amp;&amp; \<br>    pip install flask-restful &amp;&amp; \<br>    deactivate&quot;</pre><p>To be clear, I don’t especially endorse this approach anymore, except that I like to add flask-restful for creating custom REST API plugins.</p><p>I like to treat Apache Airflow the way I treat web applications. I’ve been <em>burned too many times</em>, so now my web apps take care of routing and rendering views, and absolutely nothing else.</p><p>Airflow is about the same, except it handles the business logic of my workflows and absolutely <em>nothing else</em>. If I have some crazy pandas/tensorflow/opencv/whatever stuff I need to do I’ll build that into a separate microservice and not touch my main business logic. I like to think of Airflow as the spider that sits in the web.</p><p>Still, I’m paranoid enough that I like to build my own images so I can then push them to my own docker repo.</p><h3>Wrap Up and Where to go from here</h3><p>Now that you have your foundation its time to build out your data science workflows! Add some custom DAGs, create some custom plugins, and generally <em>build stuff</em>.</p><p>If you’d like to request a tutorial please feel free to reach out to me at jillian@dabbleofdevops.com or on <a href="https://twitter.com/jillianerowe">twitter.</a></p><h3>Cheat Sheet</h3><p>Here are some hopefully helpful commands and resources.</p><h3>Log into your Apache Airflow Instance</h3><p>The default username and password is user and bitnami.</p><h3>Docker Compose Commands</h3><p>Build</p><pre>cd code/bitnami-apache-airflow-1.10.10/ <br>docker-compose build</pre><p>Bring up your stack! Running docker-compose up makes all your logs come up on STDERR/STDOUT.</p><pre>cd code/bitnami-apache-airflow-1.10.10/ <br>docker-compose build &amp;&amp; docker-compose up</pre><p>If you’d like to run it in the background instead use -d.</p><pre>cd code/bitnami-apache-airflow-1.10.10/ <br>docker-compose build &amp;&amp; docker-compose up -d</pre><h3>Bitnami Apache Airflow Configuration</h3><p>You can further customize your Airflow instance using environmental variables that you pass into the docker-compose file. Check out the <a href="https://github.com/bitnami/bitnami-docker-airflow/blob/master/README.md">README</a> for details.</p><h3>Load DAG files</h3><p>Custom DAG files can be mounted to /opt/bitnami/airflow/dags or copied during the Docker build phase.</p><h3>Specifying Environment variables using Docker Compose</h3><pre>version: &#39;2&#39;<br><br>services:<br>  airflow:<br>    image: bitnami/airflow:latest<br>    environment:<br>      - AIRFLOW_FERNET_KEY=46BKJoQYlPPOexq0OhDZnIlNepKFf87WFwLbfzqDDho=<br>      - AIRFLOW_EXECUTOR=CeleryExecutor<br>      - AIRFLOW_DATABASE_NAME=bitnami_airflow<br>      - AIRFLOW_DATABASE_USERNAME=bn_airflow<br>      - AIRFLOW_DATABASE_PASSWORD=bitnami1<br>      - AIRFLOW_PASSWORD=bitnami123<br>      - AIRFLOW_USERNAME=user<br>      - AIRFLOW_EMAIL=user@example.com</pre><h3>Clean up after Docker</h3><p>Docker can take up a lot of room on your filesystem.</p><p>If you’d like to clean up just the Airflow stack then:</p><pre>cd code/docker/bitnami-apache-airflow-1.10.10 <br>docker-compose stop <br>docker-compose rm -f -v</pre><p>Running docker-compose rm -f forcibly removes all the containers, and the -v also removes all data volumes.</p><h3>Remove all docker images everywhere</h3><p>This will stop all running containers and remove them.</p><pre>docker container stop $(docker container ls -aq) <br>docker system prune -f -a</pre><p>This will remove all containers AND data volumes</p><pre>docker system prune -f -a --volumes</pre><p><em>Originally published at </em><a href="https://www.dabbleofdevops.com/blog/get-a-fully-configured-apache-airflow-docker-dev-stack-with-bitnami"><em>https://www.dabbleofdevops.com</em></a><em>.</em></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=ed1671d63ea0" width="1" height="1" alt=""><hr><p><a href="https://medium.com/data-science/get-a-fully-configured-apache-airflow-docker-dev-stack-with-bitnami-ed1671d63ea0">Get a Fully Configured Apache Airflow Docker Dev Stack with Bitnami</a> was originally published in <a href="https://medium.com/data-science">TDS Archive</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Deploy Dash with Helm on Kubernetes with AWS EKS]]></title>
            <link>https://levelup.gitconnected.com/deploy-dash-with-helm-on-kubernetes-with-aws-eks-cabe035c0565?source=rss-87b832c70819------2</link>
            <guid isPermaLink="false">https://medium.com/p/cabe035c0565</guid>
            <category><![CDATA[python]]></category>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[dash]]></category>
            <category><![CDATA[data-visualization]]></category>
            <category><![CDATA[kubernetes]]></category>
            <dc:creator><![CDATA[Jillian Rowe]]></dc:creator>
            <pubDate>Wed, 15 Jul 2020 22:23:30 GMT</pubDate>
            <atom:updated>2020-07-15T22:23:30.517Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*CD4tCMsQ5F2UcduR.jpg" /></figure><p>Today we are going to be talking about Deploying a Dash App on Kubernetes with a Helm Chart using the AWS Managed Kubernetes Service <a href="https://aws.amazon.com/eks/">EKS</a>.</p><p>For this post, I’m going to assume that you have an EKS cluster up and running because I want to focus more on the strategy behind a real-time data visualization platform. If you don’t, please check out my <a href="https://www.dabbleofdevops.com/blog/deploy-rshiny-with-kubernetes-using-aws-eks-and-terraform">Deploy RShiny with Kubernetes using AWS EKS and Terraform</a>.</p><p>Dash is a data visualization platform written in Python.</p><blockquote><em>Dash is the most downloaded, trusted framework for building ML &amp; data science web apps.</em></blockquote><blockquote><em>Dash empowers teams to build data science and ML apps that put the power of Python, R, and Julia in the hands of business users. Full stack apps that would typically require a front-end, backend, and dev ops team can now be built and deployed in hours by data scientists with Dash. </em><a href="https://plotly.com/dash/"><em>https://plotly.com/dash/</em></a></blockquote><p>If you’d like to know what the Dash people say about Dash on Kubernetes you can read all about that <a href="https://plotly.com/dash/kubernetes/">here</a>.</p><p>Pretty much though, Dash is stateless, meaning you can throw it behind a load balancer and DOMINATE. Then, because Kubernetes is taking over the world, can load balance, and can autoscale applications along with the cluster itself being elastic, Dash + Kubernetes is a pretty perfect match!</p><h3>Data Visualization Infrastructure</h3><h3>Historical View</h3><p>I’ve been studying Dash as a part of my ongoing fascination with real-time data visualization of large scale genomic data sets. I suppose other datasets too, but I like biology so here we are.</p><p>I think we are reaching new capabilities data visualization capabilities with all the shiny new infrastructure out there. When I started off in Bioinformatics all the best data viz apps were desktop applications written in Java. If you’d like to see an example go check out <a href="http://software.broadinstitute.org/software/igv/">Integrate Genome Browser — IGV</a>.</p><p>There are definite downsides to desktop applications. They just don’t scale for the ever-increasing amount and resolution of data. With this approach, there is a limit to the amount of data you can visualize in real-time.</p><p>Then there’s the approach to set up pipelines that output a bunch of jpegs or PDFs and maybe wrapping them into some HTML report. This works well enough, assuming your pipeline is working as expected but doesn’t allow you to update parameters or generally peruse your data in real-time.</p><h3>Fancy New Data Viz Apps</h3><p>We’re no longer limited to a desktop app to get the level of interactivity we’re all looking for when interacting with data. Frameworks such as <a href="https://shiny.rstudio.com/">RShiny</a> and <a href="https://plotly.com/dash/">Dash</a>, and because they do such a nice job of integrating server-side computing with front end widgetizing (is that a word?) you can really think of how to scale your applications.</p><h3>Compute Infrastructure and Scaling</h3><p>Now, with all sorts of neat cloud computing, we can have <strong>automatically scaling clusters</strong>. You can say things like “hey, Kubernetes, when any one of my pods is above 50% CPU throw some other pods on there”.</p><p>Python and R both have very nice data science ecosystems. You can scale your Python apps with Dask and/or Spark, and your R apps with Spark by completely separating out your web stuff with Dash/Shiny from your heavy-duty computation with Dask/Spark.</p><p>If you’d like to read more about how I feel about Dask go check out my blog post <a href="https://www.dabbleofdevops.com/blog/deploy-and-scale-your-dask-cluster-with-kubernetes">Deploy and Scale your Dask Cluster with Kubernetes</a>. (As a side note you can install the Dask chart as is alongside your Dash app and then you can use both!)</p><p>Then, of course, you can throw some networked file storage up in Kubernetes cluster, because seriously no matter how fancy my life gets I’m still using bash, ssh, and networked file storage!</p><h3>Let’s Build!</h3><p>We’re using this <a href="https://github.com/plotly/dash-sample-apps/tree/master/apps/dash-cytoscape-lda">app</a>. I did not write this app, so many thanks to the Plotly Sample apps! ;-)</p><p>I’ve already built it into a <a href="https://hub.docker.com/repository/docker/jerowe/dash-sample-app-dash-cytoscape-lda">docker container</a> and this post is going to focus on the Helm Chart and Kubernetes Deployment.</p><h3>Customize the Bitnami/NGINX Helm Chart</h3><p>I base most of my Helm Charts off of the <a href="https://github.com/bitnami/charts/tree/master/bitnami/nginx/templates">NGINX Bitnami Helm chart</a>. Today is no exception. I will be changing a few things though!</p><p>Then, for the sake of brevity, I’m going to leave out all the metrics-server and load from git niceness. I actually like these and include them in my production deployments, but its kind of overkill for just throwing a Dash app up on AWS EKS. If you’re working towards a production instance I’d recommend going through the bitnami/nginx helm chart and seeing what all they have there, particularly for metrics along with replacing the default Ingress with the NGINX Ingress.</p><p>We’re not going to be modifying any files today. Instead, we will abuse the power of the --set flag in the helm CLI.</p><h3>A quick note about SSL</h3><p>Getting into SSL is a bit beyond the scope of this tutorial, but here are two resources to get you started.</p><p>The first is a <a href="https://www.digitalocean.com/community/tutorials/how-to-set-up-an-nginx-ingress-with-cert-manager-on-digitalocean-kubernetes">Digital Ocean tutorial</a> on securing your application with the NGINX Ingress. I recommend giving this article a thorough read as this will give you a very good conceptual understanding of setting up https. In order to do this, you will need to switch the default Ingress with the NGINX Ingress.</p><p>The second is an article by <a href="https://docs.bitnami.com/tutorials/secure-kubernetes-services-with-ingress-tls-letsencrypt/">Bitnami</a>. It is a very clear tutorial on using helm charts to get up and running with HTTPS, and I think it does an excellent job of walking you through the steps as simply as possible.</p><p>If you don’t care about understanding the ins and outs of https with Kubernetes just go with the Bitnami tutorial. ;-)</p><h3>Install the Helm Chart</h3><p>Let’s install the Helm Chart. First off, we’ll start with the simplest configuration in order to test that nothing too strange is happening.</p><pre># Install the bitnami/nginx helm chart with a RELEASE_NAME dash<br>helm repo add bitnami https://charts.bitnami.com/bitnami<br>helm repo update<br>helm upgrade --install dash bitnami/nginx \<br>    --set image.repository=&quot;jerowe/dash-sample-app-dash-cytoscape-lda&quot; \<br>    --set image.tag=&quot;1.0&quot;  \<br>    --set containerPort=8050</pre><h3>Get the Dash Service</h3><p>Alrighty. So you can either pay attention to what the notes say, get the notes, or use the Kubernetes CLI (kubectl)</p><h3>Use Helm</h3><p>This assumes that the information you need is actually in the Helm notes.</p><pre>helm get notes dash</pre><p>This should spit out:</p><pre>export SERVICE_IP=$(kubectl get svc --namespace default dash-nginx --template &quot;{{ range (index .status.loadBalancer.ingress 0) }}{{.}}{{ end }}&quot;)</pre><pre>echo &quot;NGINX URL: http://$SERVICE_IP/&quot;</pre><p>If you’re on AWS you’ll see something similar to this:</p><pre>NAME         TYPE           CLUSTER-IP      EXTERNAL-IP                                                              PORT(S)                      AGE<br>dash-nginx   LoadBalancer   172.20.48.151   acf3379ed0fcb4790afc8036310259dc-994191965.us-east-1.elb.amazonaws.com   80:31019/TCP,443:32684/TCP   18m<br>kubernetes   ClusterIP      172.20.0.1      &lt;none&gt;</pre><h3>Use kubectl to get the SVC</h3><p>You can also just run</p><pre>kubectl get svc | grep dash</pre><p>Or you can use the JSON output to programmatically grab your SVC. This is handy if you want to update it with a CI/CD service.</p><pre>export EXPOSED_URL=$(kubectl get svc --namespace default dash-nginx-ingress-controller -o json | jq -r &#39;.status.loadBalancer.ingress[]?.hostname&#39;)</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*c6iCALJCfuyUmLL_.jpeg" /></figure><h3>Check out Dash on AWS!</h3><p>Grab that URL and check out our Dash App!</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*tPeb_zhgJQC_AW4Z.jpeg" /></figure><h3>SCALE</h3><p>Now, this is where things get fun! We can scale the application, dynamically or manually. The helm chart is already setup to work as a Load Balancer, so we can have 1 dash app or as many dash apps as we have compute power for. The Load Balancer will take care of serving all these under a single URL in a way that’s completely hidden from the end user.</p><h3>Manually Scale the Number of Dash Apps Running</h3><p>First, we’re going to manually scale the number of Dash Apps running by increasing the number of replicas. This is pretty standard in web land.</p><pre>helm upgrade --install dash bitnami/nginx \<br>    --set image.repository=&quot;jerowe/dash-sample-app-dash-cytoscape-lda&quot; \<br>    --set image.tag=&quot;1.0&quot;  \<br>    --set containerPort=8050 \<br>    --set replicaCount=3</pre><p>Now when you run kubectl get pods you should see 3 dash-nginx-* pods.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*1ipjuVeaji9-yjUH.jpeg" /></figure><p>Then, when you run kubectl get svc you&#39;ll see that there is still one LoadBalancer service for the Dash App.</p><p>That’s the most straight forward way to statically scale your app. Let’s bring it back down, because next we’re going to dynamically scale it!</p><pre>helm upgrade --install dash bitnami/nginx \<br>    --set image.repository=&quot;jerowe/dash-sample-app-dash-cytoscape-lda&quot; \<br>    --set image.tag=&quot;1.0&quot;  \<br>    --set containerPort=8050 \<br>    --set replicaCount=1</pre><h3>Dynamically Scale your Dask App with a Horizontal Pod AutoScaler</h3><p>The <a href="https://kubernetes.io/docs/tasks/run-application/horizontal-pod-autoscale">Kubernetes Horizontal Pod Autoscaler</a> allows you dynamically scale your application based on the CPU or Memory load on the pod. Pretty much you set a rule, say that once a rule reaches some percent of total CPU to increase the number of pods.</p><p>This is kind of tricky for people, so I’m going to show the code from the Nginx helm chart.</p><p>Here it is in the values.yaml.</p><pre># https://github.com/bitnami/charts/blob/master/bitnami/nginx/values.yaml#L497<br>## Autoscaling parameters<br>##<br>autoscaling:<br>  enabled: false<br>  # minReplicas: 1<br>  # maxReplicas: 10<br>  # targetCPU: 50<br>  # targetMemory: 50</pre><p>And here it is in the templates/hpa.yaml.</p><pre># https://github.com/bitnami/charts/blob/master/bitnami/nginx/templates/hpa.yaml<br>{{- if .Values.autoscaling.enabled }}<br>apiVersion: autoscaling/v2beta1<br>kind: HorizontalPodAutoscaler<br>metadata:<br>  name: {{ template &quot;nginx.fullname&quot; . }}<br>  labels: {{- include &quot;nginx.labels&quot; . | nindent 4 }}<br>spec:<br>  scaleTargetRef:<br>    apiVersion: apps/v1<br>    kind: Deployment<br>    name: {{ template &quot;nginx.fullname&quot; . }}<br>  minReplicas: {{ .Values.autoscaling.minReplicas }}<br>  maxReplicas: {{ .Values.autoscaling.maxReplicas }}<br>  metrics:<br>    {{- if .Values.autoscaling.targetCPU }}<br>    - type: Resource<br>      resource:<br>        name: cpu<br>        targetAverageUtilization: {{ .Values.autoscaling.targetCPU }}<br>    {{- end }}<br>    {{- if .Values.autoscaling.targetMemory }}<br>    - type: Resource<br>      resource:<br>        name: memory<br>        targetAverageUtilization: {{ .Values.autoscaling.targetMemory  }}<br>    {{- end }}<br>{{- end }}</pre><p>You set your minimum number of replicas, your maximum number of replicas, and the amount of Memory/CPU you’re targeting.</p><p>Now, this is a blog post and not particularly indicative of the real world. I’m going to show you some values so this is interesting, but really when you’re doing your own autoscaling you’ll have to play around with this.</p><p>First, we’ll have to install the metrics server.</p><pre>helm repo add stable https://kubernetes-charts.storage.googleapis.com<br>helm repo update<br>helm install metrics stable/metrics-server</pre><p>If you get an error here about metrics already taken you probably have the metrics chart installed already. It’s installed by default on a lot of platforms or may have been required by another chart.</p><p>In order to get this to work you need to set <a href="https://kubernetes.io/docs/concepts/configuration/manage-resources-containers/#meaning-of-memory">limits on the resources of the deployment itself</a> so Kubernetes has a baseline for scaling.</p><p>I’m using a t2.medium instance for my worker groups, which has 2 CPUs, 24 CPU credits/hour, and 4GB of memory. You should be fine if you&#39;re on a larger instance, but if you&#39;re on a smaller you may need to play with the resource values.</p><pre>helm upgrade --install dash bitnami/nginx \<br>    --set image.repository=&quot;jerowe/dash-sample-app-dash-cytoscape-lda&quot; \<br>    --set image.tag=&quot;1.0&quot;  \<br>    --set autoscaling.enabled=true \<br>    --set autoscaling.minReplicas=2 \<br>    --set autoscaling.maxReplicas=3 \<br>    --set autoscaling.targetCPU=1 \<br>    --set resources.limits.cpu=&quot;200m&quot; \<br>    --set resources.limits.memory=&quot;200m&quot; \<br>    --set resources.requests.cpu=&quot;200m&quot; \<br>    --set resources.requests.memory=&quot;200m&quot; \<br>    --set containerPort=8050</pre><p>We should see, at a minimum, 2 pods (or 1 pod up with one scaling up).</p><pre>kubectl get pods</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*CN042DbgXIQSxy23.jpeg" /></figure><p>Then we can describe our HPA.</p><pre>kubectl get hpa kubectl describe hpa dash-nginx</pre><p>I set the resource requirements very low, so grab your SVC and just keep refreshing the page. You should see the pods start scaling up.</p><pre>kubectl get svc |grep dash</pre><h3>Figure out your Autoscaling Values with htop</h3><p>Now, what you should do is to install Prometheus/Grafana and use the metrics server to keep track of what’s happening on your cluster.</p><p>But, sometimes you just don’t want to be bothered you can always just exec into a Kubernetes pod and run htop.</p><pre>kubectl get pods |grep dash </pre><pre># grab the POD_NAME - something like this dash-nginx-75c5c8649-rbdj7 </pre><pre>kubectl exec -it POD_NAME bash</pre><p>Then, depending on when you read this you may need to install htop.</p><pre>apt-get install -y htop<br>htop</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*ezw3r9lLeCFqQ8F-.jpeg" /></figure><h3>Wrap Up</h3><p>That’s it! I hope you see how you can gradually build up your data science and visualization infrastructure, piece by piece, in order to do real-time data visualization of large data sets!</p><p>If you have any questions, comments, or tutorials requests please reach out to me at <a href="/service/mailto:jillian@dabbleofdevops.com,">jillian@dabbleofdevops.com, </a>or leave a comment below. ;-)</p><h3>Helpful Commands</h3><p>Here are some helpful commands for navigating your Dash deployment on Kubernetes.</p><h3>Helm Helpful Commands</h3><pre># List the helm releases<br>helm list<br># Add the Bitnami Repo<br>helm repo add bitnami https://charts.bitnami.com/bitnami<br>helm repo update<br># Install a helm release from a helm chart (the nice way to integrate with CI/CD)<br>helm upgrade --install RELEASE_NAME bitnami/nginx<br># Install a helm chart from a local filesystem<br>helm upgrade --install RELEASE_NAME ./path/to/folder/with/values.yaml<br># Get the notes<br>helm get notes RELEASE_NAME</pre><h3>Kubectl Helpful Commands</h3><pre># Get a list of all running pods<br>kubectl get pods <br># Describe the pod. This is very useful for troubleshooting! <br>kubectl describe pod PODNAME<br># Drop into a shell in your pod. It&#39;s like docker run.<br>kubectl exec -it PODNAME bash<br># Get the logs from stdout on your pod<br>kubectl logs PODNAME <br># Get all the services and urls<br>kubectl get svc</pre><p><em>Originally published at </em><a href="https://www.dabbleofdevops.com/blog/deploy-dash-with-helm-on-kubernetes-with-aws-eks"><em>https://www.dabbleofdevops.com</em></a><em>.</em></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=cabe035c0565" width="1" height="1" alt=""><hr><p><a href="https://levelup.gitconnected.com/deploy-dash-with-helm-on-kubernetes-with-aws-eks-cabe035c0565">Deploy Dash with Helm on Kubernetes with AWS EKS</a> was originally published in <a href="https://levelup.gitconnected.com/">Level Up Coding</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[RShiny Authentication with Polished on AWS Kubernetes]]></title>
            <link>https://levelup.gitconnected.com/rshiny-authentication-with-polished-on-aws-kubernetes-358baff9a61c?source=rss-87b832c70819------2</link>
            <guid isPermaLink="false">https://medium.com/p/358baff9a61c</guid>
            <category><![CDATA[r-shiny]]></category>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[kubernetes]]></category>
            <category><![CDATA[aws]]></category>
            <dc:creator><![CDATA[Jillian Rowe]]></dc:creator>
            <pubDate>Tue, 07 Jul 2020 13:21:48 GMT</pubDate>
            <atom:updated>2020-07-07T13:21:48.728Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*RBYDUXeizAdy02TP.jpg" /></figure><p>If you’re looking for a hassle free way to add authentication to your RShiny Apps you should check out <a href="http://polished.tech/">polished.tech</a>. In their own words:</p><blockquote><em>Polished is an R package that adds authentication, user management, and other goodies to your Shiny apps. Polished.tech is the quickest and easiest way to use polished.</em></blockquote><blockquote><em>Polished.tech provides a hosted API and dashboard so you don’t need to worry about setting up and maintaining the polished API. With polished.tech you can stop worrying about boilerplate infrastructure and get back to building your core Shiny application logic.</em></blockquote><p>Polished.tech is a hosted solution for adding authentication to your RShiny Apps. It is completely free for the first 10 users, which gives you plenty of freedom to play around with it.</p><p>In this post we’ll go over how to:</p><ul><li>Quickly and easily get started with the polished.tech hosted solution and grab your API credentials</li><li>Test out your API credentials with a pre built docker container</li><li>Deploy an AWS Kubernetes (EKS) Cluster</li><li>Securely add your API credentials to an EKS cluster</li><li>Deploy a helm chart with a sample RShiny app using polished.tech authentication!</li></ul><p>So let’s get started!</p><h3>Prerequisites</h3><p>In order to complete the AWS portion of the tutorial you’ll need:</p><h3>Docker</h3><p>You will need Docker to run the Quickstart guide.</p><p>I always create a Docker image with all the CLI tools I need and keep a copy of any credentials directly in the project folder. You don’t need to especially know how to use Docker, as we will just be treating the container as a terminal.</p><p>If you prefer not to use Docker you will need to install all the CLI tools shown in the Dockerfile and run the shiny app locally.</p><h3>AWS IAM User</h3><p>If you want to deploy to AWS you’ll need an AWS IAM User, and that user’s API keys. For more information see the <a href="https://docs.aws.amazon.com/cli/latest/userguide/cli-configure-quickstart.html#cli-configure-quickstart-creds">AWS docs</a>.</p><p>This same AWS IAM user must have permission to create an EKS cluster. You can use <a href="https://docs.aws.amazon.com/eks/latest/userguide/service_IAM_role.html">this role</a>, or a user with administrator privileges.</p><h3>Quick Start</h3><p>If you already have your Polished API Credentials you can start here.</p><pre>docker run -it \<br> --rm \<br>-p 3838:3838 \<br> -e POLISHED_APP_NAME=&quot;my_first_shiny_app&quot; \<br> -e POLISHED_API_KEY=&quot;XXXXXXXXXXXXXX&quot; \<br> jerowe/polished-tech:latest</pre><p>Add your own credentials to the POLISHED_APP_NAME and the POLISHED_API_KEY.</p><p>Then open up a browser at localhost:3838 and login to your application using your polished.tech email and password combo.</p><h3>Sign up and Get your Credentials</h3><p>All these instructions are <a href="https://polished.tech/docs/get-started">in the docs</a>, and only here for the sake of completeness.</p><h3>Sign up</h3><p>Head over to the <a href="https://dashboard.polished.tech/">dashboard URL</a> and register. From there, log in, and you’ll see your account dashboard.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*q0pOxykE8gjnC8f79kwZ6A.png" /></figure><p>If you use Google or Microsoft you will be prompted to log in with those accounts. If you use email you will be prompted to create a password.</p><h3>Optional — Social Login vs Email &amp; Password</h3><p>If you want to use the Social Login and deploy on AWS you will need to setup Firebase with <a href="https://polished.tech/docs/firebase-setup">your external domain</a>. You only need to complete this step if you are deploying to a host other than localhost and want to use a social login provider such as Google. We’ll go into more detail when we deploy to AWS further down.</p><h3>Get your API Secret Key</h3><p>Head on over to the <a href="https://dashboard.polished.tech/">Dashboard</a>, on the left-hand menu, click <em>Accounts</em> to get your Secret Key.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*UVc8OyIolKG9PMCF.jpg" /></figure><p>Right it down somewhere! From here on out we will refer to it as the POLISHED_API_KEY.</p><h3>Create an App</h3><p>You must create an app for this whole scheme to work.</p><p>On the left-hand menu click <em>Shiny Apps</em>.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*4xFEmjszaCUUHIx0.png" /></figure><p>Write it down somewhere! From here on out we will refer to it as the POLISHED_APP_NAME.</p><h3>Associate the App to your User</h3><p>(And potentially other users!)</p><p>Once you have an app, make sure to associate it with your user. This one got me the first time around because I don’t follow instructions well. ;-) If you skip this step you won’t be able to log in.</p><p>On your dashboard in the left-hand menu go to <em>Users</em>, and associate the app to the user(s).</p><p>You can either edit the user to allow access to the app or edit the app to give access to the user. Both accomplish the same goal.</p><p><strong>From the Users Menu</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*ceu_XBG54muf0t9g.jpg" /></figure><p><strong>From the App Menu</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*VAizJkVgWEKVCHhR.jpg" /></figure><h3>Test Your Credentials</h3><p>If you just want to test your credentials run the docker run command.</p><pre>docker run -it \<br> --rm \<br> -p 3838:3838 \<br> -e POLISHED_APP_NAME=&quot;my_first_shiny_app&quot; \<br> -e POLISHED_API_KEY=&quot;XXXXXXXXXXXXXX&quot; \<br> jerowe/polished-tech:latest</pre><p>With, of course, the credentials that <em>I know you wrote down earlier</em>. ;-)</p><p>Open your browser at localhost:3838 and login with your user email and password. You should see a screen that has your user UID and session info. Success!</p><h3>Let’s Deploy Our App!</h3><p>For the next part of the tutorial, we will discuss getting your app out there into the wild! I chose Kubernetes for this because it works absolutely everywhere, and the hosted AWS solution is <em>nice</em>. Also, deploying to Kubernetes allows you to easily encrypt your API keys, database credentials, etc.</p><h3>Grab the code</h3><p>If you would like to follow along clone the Github repo.</p><pre>git clone <a href="https://github.com/jerowe/rshiny-with-polished-tech-eks">https://github.com/jerowe/rshiny-with-polished-tech-eks</a></pre><h3>Project Structure</h3><p>There are two directories in the GitHub repo, auto-deployment, and rshiny-app-polished-auth.</p><p>The auto-deployment has a (slightly modified) <a href="https://github.com/terraform-aws-modules/terraform-aws-eks">Terraform EKS</a> recipe to first deploy our EKS cluster, install our helm chart, and do some networking magic.</p><p>If you’ve never used Terraform before don’t worry. It’s really like a Makefile with a nicer syntax. A <a href="https://helm.sh/docs/topics/charts/">helm chart</a> is the Kubernetes preferred method of packaging and defining applications.</p><p>Getting deep into the Helm chart would make this post <em>way</em> too long, but I go into more <a href="https://www.dabbleofdevops.com/blog/deploy-rshiny-on-kubernetes-with-a-helm-chart">in this post.</a></p><p>There is also a Dockerfile in the root of the project system. This Dockerfile has everything you need to both deploy the EKS cluster and install the helm chart. It installs the aws-cli, terraform, helm, and kubectl cli tools. You could install these things on your local computer, but that would be painful. ;-)</p><pre>├── Dockerfile<br>├── README.md<br>├── auto-deployment<br>│   ├── eks<br>│   │   ├── aws.tf<br>│   │   ├── helm.tf<br>│   │   ├── helm_charts<br>│   │   │   ├── rshiny-eks<br>│   │   │   │   ├── Chart.yaml<br>│   │   │   │   ├── charts<br>│   │   │   │   ├── templates<br>│   │   │   │   │   ├── NOTES.txt<br>│   │   │   │   │   ├── _helpers.tpl<br>│   │   │   │   │   ├── deployment.yaml<br>│   │   │   │   │   ├── ingress.yaml<br>│   │   │   │   │   ├── service.yaml<br>│   │   │   │   │   ├── serviceaccount.yaml<br>│   │   │   │   │   └── tests<br>│   │   │   │   │       └── test-connection.yaml<br>│   │   │   │   └── values.yaml<br>│   │   ├── main.tf<br>│   │   ├── outputs.tf<br>│   │   └── variables.tf<br>│   └── terraform-state<br>│       ├── main.tf<br>├── build_and_run_polished_app.sh<br>└── rshiny-app-polished-auth<br>    ├── Dockerfile<br>    ├── app.R<br>    └── shiny-server.sh</pre><h3>The Docker Image</h3><p>I built a docker image based on the <a href="https://hub.docker.com/r/rocker/shiny">Rocker/Shiny</a> image.</p><p>The important things to note here are that I deleted the default index.html and other sample apps, and added in my own app.R.</p><h4>Environmental Variables</h4><p>I added a startup CMD to add the POLISHED_* variables to /home/shiny/.Renviron. The shiny-server runs a clean process where it does not <a href="https://community.rstudio.com/t/passing-environment-variable-to-an-app-in-shiny-server/13355">pick up on any outside environmental variables</a>.</p><pre># rshiny-app-polished-auth/Dockerfile<br>FROM rocker/shiny:4.0.0<br><br># Install everything needed for polished<br><br>RUN apt-get update &amp;&amp; apt-get install -y \<br>    libssl-dev<br><br><br>RUN R -e &quot;install.packages(&#39;remotes&#39;)&quot;<br>RUN R -e &quot;remotes::install_github(&#39;tychobra/polished&#39;)&quot;<br>RUN rm -rf /srv/shiny-server/*<br>COPY app.R /srv/shiny-server/app.R<br><br>COPY shiny-server.sh /usr/bin/shiny-server.sh</pre><p>The shiny-server process runs cleanly, meaning we have to load in our POLISHED_*environmental variables from a ~/.Renviron file. If you need to add any other variables make sure you do that here and rebuild the docker image!</p><pre># rshiny-app-polished-auth/shiny-server.sh<br>#!/bin/sh<br><br># Make sure that we add our Polished to our /home/shiny/.Renviron<br>env | grep POLISHED &gt; /home/shiny/.Renviron<br>chown shiny:shiny /home/shiny/.Renviron<br><br># Make sure the directory for individual app logs exists<br>mkdir -p /var/log/shiny-server<br>chown shiny.shiny /var/log/shiny-server<br><br>if [ &quot;$APPLICATION_LOGS_TO_STDOUT&quot; != &quot;false&quot; ];<br>then<br>    # push the &quot;real&quot; application logs to stdout with xtail in detached mode<br>    exec xtail /var/log/shiny-server/ &amp;<br>fi<br><br># start shiny server<br>exec shiny-server 2&gt;&amp;1</pre><h3>Update the AWS Region</h3><p>The AWS region is set as us-east-1. If you need to change it take a look at auto-deployment/eks/variables.tf and auto-deployment/terraform-state/main.tf.</p><h3>Optional — Firebase Setup for Social Login</h3><p>If you want to use social login (Google, Microsoft, etc) you will need to get your own Firebase account and credentials. If you’re using email and password you can skip this step!</p><p>First, we’ll set up the Firebase Project, and then we’ll register an app to that project.</p><h3>FireBase Project Setup</h3><p>Head on over to the <a href="https://console.firebase.google.com/u/0/?pli=1">FireBase console</a> to set up a project. It’s pretty simple and the wizard will walk you through the steps. From there go to _Develop -&gt; Authentication -&gt; Sign-in Method. _Then enable Email + Password along with whichever Social Provider you’d like.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*OMUihLVo7F5u5Z0a.jpg" /></figure><p>Make sure your social logins are green and enabled!</p><h3>Setup your FireBase App</h3><p>Within your project, you’ll need to register an app and grab the credentials. In your Firebase project click on the settings icon, then on <em>Add App.</em></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*cIKoIamA28s5SgQy.jpg" /></figure><p>Once you’ve registered your app you’ll get your app credentials.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*NlIZWI1gu_bAwMYm.jpg" /></figure><p>You’ll add these to your terraform recipe in the next step.</p><p>Add your Polished Credentials to the Terraform Recipe</p><p>Before you deploy your EKS cluster, make sure that you add in your Polished credentials.</p><pre># auto-deployment/eks/variables.tf<br># CHANGE THESE!<br><br>...<br><br># Then make sure to add your user to the app!<br>variable &quot;POLISHED_APP_NAME&quot; {<br>  default = &quot;my_first_shiny_app&quot;<br>}<br><br># Grab this from the polished.tech Dashboard -&gt; Account -&gt; Secret<br>variable &quot;POLISHED_API_KEY&quot; {<br>  default = &quot;XXXXXXXXXXXXXXXXXX&quot;<br>}<br><br># Firebase credentials<br># Only add these if you&#39;ve setup your own firebase credentials<br>variable &quot;POLISHED_FIREBASE_API_KEY&quot; {<br>  default = &quot;&quot;<br>}</pre><p>If you’re following along with the Firebase instructions make sure you add your Firebase credentials too. If you aren’t just leave those as is.</p><h3>Kubernetes Secrets</h3><p>You see that we add in our POLISHED_* variables as terraform variables, which are then added to our Kubernetes cluster as <a href="https://kubernetes.io/docs/concepts/configuration/secret/">Secrets</a>.</p><p>Secrets get a little tricky because it feels like you have to add them to a million places, and it’s easy to get them out of sync. I’d suggest picking ONE, and only ONE way to reference your variables. Don’t get cute with it! ;-)</p><p>Your Kubernetes Secrets (generally) appear in 3 places:</p><ul><li>In your Terraform Recipe (as secrets)</li><li>In your helm chart’s environmental variables in either values.yaml</li><li>Or in the templates/deployment.yaml.</li></ul><p>The second just really depends on the way that your chart is organized.</p><p>Secrets can be tricky, so I’m showing them in a few different ways.</p><h3>Kubernetes Secrets — Code View</h3><p>Here are our secrets in the terraform deployment:</p><pre># auto-deployment/eks/main.tf<br>locals {<br>  secrets = [<br>    {<br>      name = &quot;polished&quot;<br>      data = {<br>        app_name = var.POLISHED_APP_NAME<br>        api_key = var.POLISHED_API_KEY<br>      }<br>    },<br>  ]<br>}<br><br>resource &quot;kubernetes_secret&quot; &quot;main&quot; {<br>  depends_on = [<br>    module.eks,<br>  ]<br>  count = length(local.secrets)<br><br>  metadata {<br>    name = local.secrets[count.index].name<br>    labels = {<br>      Project = local.cluster_name<br>      Owner = &quot;terraform&quot;<br>    }<br><br>  }<br><br>  data = local.secrets[count.index].data<br><br>  type = &quot;Opaque&quot;<br>}</pre><p>The Firebase secrets are there too, just not shown for the sake of not making this an encyclopedia. Now we read them into our RShiny App Deployment, telling Helm that they are Kubernetes secrets.</p><pre># auto-deployment/helm_charts/rshiny-eks/values.yaml<br>env:<br>  - name: POLISHED_API_KEY<br>    valueFrom:<br>      secretKeyRef:<br>        # In terraform this is locals.secrets.[0].polish.data.api_key<br>        name: polished<br>        key: api_key<br>  - name: POLISHED_APP_NAME<br>    valueFrom:<br>      secretKeyRef:<br>        # In terraform this is locals.secrets.[0].polish.data.app_name<br>        name: polished<br>        key: app_name</pre><p>Then, finally, in our actual deployment we add in the env keys from our values.yaml.</p><pre>containers:<br>   - name: {{ .Chart.Name }}<br>     securityContext:<br>       {{- toYaml .Values.securityContext | nindent 12 }}<br>     image: &quot;{{ .Values.image.repository }}:{{ .Values.image.tag }}&quot;<br>     imagePullPolicy: {{ .Values.image.pullPolicy }}<br>     # Env vars are read in HERE<br>     env:<br>       - name: TAG<br>         value: {{ .Values.image.tag }}<br>       {{- toYaml .Values.env | nindent 12 }}</pre><h4>Kubernetes Secrets — Image View</h4><p>Here’s a side by side view of the Secrets declared in our:</p><ul><li>Terraform Variables — auto-deployment/eks/variables.tf</li><li>Kubernetes Secrets — auto-deployment/eks/main.tf</li><li>In the Helm chart — auto-deployment/eks/helm_charts/rshiny-eks/values.yaml( and in the containers[0].env of auto-deployment/eks/helm_charts/rshiny-eks/templates/deployment.yaml)</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*vpVXrtvVEud31VkA.jpg" /></figure><h3>Deploy to an AWS Kubernetes Cluster (EKS)</h3><p>Now that we’ve covered all the basics let’s finally deploy our cluster!</p><h3>Build the Terraform Docker Image</h3><p>I like having everything I need to deploy a particular project in a Dockerfile. I have one of these for nearly every project I work on, because no one wants to keep track of deployments.</p><p>Most of my projects look like:</p><pre>├── .aws<br>├── Dockerfile<br>├── README.md<br>├── auto-deployment</pre><p>With the .aws directory having the AWS credentials for the project, the Dockerfile having the cli tools needed for deployment, and then the actual auto-deploy. If you don&#39;t already have your AWS CLI configured you can read more about it on the <a href="https://docs.aws.amazon.com/cli/latest/userguide/cli-configure-quickstart.html#cli-configure-quickstart-creds">AWS docs</a>.</p><pre># From the project directory<br>docker build -t eks-k8 .<br># If your aws credentials are someplace else be sure to change this!<br># The default AWS cli credentials location is home<br># If you keep your .aws credentials in the project folder use this command<br>#docker run -it -v &quot;$(pwd):/project&quot; -v &quot;$(pwd)/.aws:/root/.aws&quot; eks-k8 bash<br># If your AWS credentials are in ~/.aws use this one<br>docker run -it \<br>    -v &quot;$(pwd):/project&quot; -v &quot;${HOME}/.aws:/root/.aws&quot; eks-k8 bash</pre><p>This will create an image with everything you need, and then drop you into the shell. From there treat your image like any other terminal.</p><h3>Initialize The Terraform State</h3><p>Remember how I said earlier Terraform is sort of like a Makefile? It keeps track of what operations it’s performed, resources it’s created or destroyed, as a part of its <a href="https://www.terraform.io/docs/state/">state</a>. It can either keep track of those locally or in an S3 bucket or other types of storage. In our case, we’ll keep track of the state in an S3 Bucket.</p><p>Now, S3 bucket names must be globally unique. This means that you and I can’t have the same S3 bucket name! I use a random number generator but you can also just change the name to something unique in_ auto-deployment/terraform-state/main.tf_</p><pre># From within the eks-k8 docker image<br>cd /project/auto-deployment/terraform-state<br>terraform init; terraform refresh; terraform apply -auto-approve</pre><h3>Deploy the Cluster!</h3><p><strong>Finally!</strong></p><p>In one fell swoop, we will deploy our cluster, configure our various CLIs, install our helm chart, and get our external URL!</p><pre># From within the eks-k8 docker image<br>cd /project/auto-deployment/eks<br>terraform init; terraform refresh; terraform apply -auto-approve</pre><p>Occasionally, stuff times out. You may have to run this command a few times. Grab a snack and watch it go!</p><p>By the end, you should see something like this -</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*Z2mV0P9BZO3XkZih.png" /></figure><h3>Check out our RShiny App</h3><p>If you were paying attention you will see lots and lots of output from the terraform command.</p><p>For the impatient, what we really want is to run this command to get the external URL.</p><pre>export EXPOSED_URL=$(kubectl get svc --namespace default rshiny-nginx-ingress-controller -o json | jq -r &#39;.status.loadBalancer.ingress[]?.hostname&#39;) <br>echo $EXPOSED_URL</pre><h3>Optional — Add External Domain to Firebase</h3><p>If you’ve been following along with setting up your own Firebase account then you will need to grab that external URL and add it to the Firebase Console. <em>Go to Develop -&gt; Authentication -&gt; Sign in Methods -&gt; Authorized Domains.</em></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*khPfj5lN-Gi9RhJW.jpg" /></figure><h3>Success!</h3><p>Grab the $EXTERNAL_URL and open up a browser. You should see the polished.tech login page!</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*0gmPuu-T8e9iRcBB.png" /></figure><p>Then, when you login, you’ll see the session data!</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*1yYTkmcHyBwBURkV.jpg" /></figure><h3>Helpful Commands</h3><p>Here are some helpful commands for navigating your RShiny deployment on Kubernetes.</p><pre># Get a list of all running pods<br>kubectl get pods <br># Describe the pod. This is very useful for troubleshooting! <br>kubectl describe pod PODNAME<br># Drop into a shell in your pod. It&#39;s like docker run.<br>kubectl exec -it PODNAME bash<br># Get the logs from stdout on your pod<br>kubectl logs PODNAME <br># Get all the services and urls<br>kubectl get svc</pre><h3>A quick note about SSL</h3><p>Getting into SSL is a bit beyond the scope of this tutorial, but here are two resources to get you started.</p><p>The first is a <a href="https://www.digitalocean.com/community/tutorials/how-to-set-up-an-nginx-ingress-with-cert-manager-on-digitalocean-kubernetes">Digital Ocean tutorial</a> on securing your application with the NGINX Ingress. I recommend giving this article a thorough read as this will give you a very good conceptual understanding of setting up https.</p><p>The second is an article by <a href="https://docs.bitnami.com/tutorials/secure-kubernetes-services-with-ingress-tls-letsencrypt/">Bitnami</a>. It is a very clear tutorial on using helm charts to get up and running with HTTPS, and I think it does an excellent job of walking you through the steps as simply as possible.</p><p>If you don’t care about understanding the ins and outs of https with Kubernetes just go with the Bitnami tutorial. ;-)</p><h3>Wrap Up</h3><p>That’s it! Hopefully, you see how easy it is to get up and running with secure authentication using the polished.tech hosted service!</p><p>If you have any questions or comments please reach out to me directly at <a href="/service/mailto:jillian@dabbleofdevops.com">jillian@dabbleofdevops.com</a>. If you have any questions about the polished.tech authentication service <a href="http://polished.tech/">check them out on their website</a>.</p><h3>Resources</h3><p><a href="https://polished.tech/docs/get-started">Polished Tech RShiny Authentication</a></p><p><a href="https://helm.sh/docs/topics/charts/">Helm Charts</a></p><p><a href="https://github.com/terraform-aws-modules/terraform-aws-eks">Terraform EKS Recipes</a></p><p><a href="https://hub.docker.com/r/rocker/shiny">Rocker Shiny Docker Image</a></p><p><a href="https://aws.amazon.com/eks/">EKS — AWS Managed Kubernetes Clusters</a></p><p><a href="https://docs.bitnami.com/tutorials/secure-kubernetes-services-with-ingress-tls-letsencrypt/">Add SSL Tutorial</a> <a href="https://aws.amazon.com/about-aws/global-infrastructure/regions_az/#:~:text=AWS%20maintains%20multiple%20geographic%20Regions,Africa%2C%20and%20the%20Middle%20East.">AWS — Regions</a></p><p><a href="https://docs.aws.amazon.com/cli/latest/userguide/cli-configure-quickstart.html#cli-configure-quickstart-creds">AWS — Configure the CLI</a> <a href="https://docs.aws.amazon.com/eks/latest/userguide/service_IAM_role.html">AWS — EKS IAM Policy</a></p><p><em>Originally published at </em><a href="https://www.dabbleofdevops.com/blog/rshiny-authentication-with-polished-on-aws-kubernetes"><em>https://www.dabbleofdevops.com</em></a><em>.</em></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=358baff9a61c" width="1" height="1" alt=""><hr><p><a href="https://levelup.gitconnected.com/rshiny-authentication-with-polished-on-aws-kubernetes-358baff9a61c">RShiny Authentication with Polished on AWS Kubernetes</a> was originally published in <a href="https://levelup.gitconnected.com/">Level Up Coding</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
    </channel>
</rss>