<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Tulshiram Kadam on Medium]]></title>
        <description><![CDATA[Stories by Tulshiram Kadam on Medium]]></description>
        <link>https://medium.com/@tulshiram.kadam?source=rss-d67062a61bce------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/1*B0QFMNphsVUVDU-mqhMZYQ.png</url>
            <title>Stories by Tulshiram Kadam on Medium</title>
            <link>https://medium.com/@tulshiram.kadam?source=rss-d67062a61bce------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Fri, 20 Nov 2026 15:09:00 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@tulshiram.kadam/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[Running Bioinformatics Pipelines in Production: A Strategic Guide for Decision-Makers]]></title>
            <link>https://blog.nonstopio.com/running-bioinformatics-pipelines-in-production-a-strategic-guide-for-decision-makers-4dce18963a83?source=rss-d67062a61bce------2</link>
            <guid isPermaLink="false">https://medium.com/p/4dce18963a83</guid>
            <category><![CDATA[aws-healthomics]]></category>
            <category><![CDATA[kubernetes]]></category>
            <category><![CDATA[devops]]></category>
            <category><![CDATA[bioinformatics]]></category>
            <category><![CDATA[batch-processing]]></category>
            <dc:creator><![CDATA[Tulshiram Kadam]]></dc:creator>
            <pubDate>Tue, 13 Jan 2026 10:39:28 GMT</pubDate>
            <atom:updated>2026-01-14T10:18:09.025Z</atom:updated>
            <content:encoded><![CDATA[<h4>HealthOmics, Batch, or Kubernetes? Stop debating and start shipping your genomics workflows.</h4><h4><em>How to choose the right infrastructure for your Nextflow workflows based on business requirements, not just technical features.</em></h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*988CaqX3MHsHU4bXQpqF7w.png" /></figure><p>The global bioinformatics market generated <strong>$8,614.29 million in 2019</strong> and is projected to reach <strong>$24,731.61 million by 2027</strong>, growing at a <strong>CAGR of 13.4%</strong> from 2020 to 2027 (<a href="https://www.alliedmarketresearch.com/bioinformatics-market">source</a>).</p><p>Yet despite this growth, most organisations stumble on a far more basic problem. Not how to design algorithms. Not how to analyse data. But how to <strong>run their computational workflows reliably in production</strong>?</p><p>This gap between analysis and execution is where most bioinformatics efforts quietly struggle, and where real business impact is either created or lost.</p><p>Let us start by understanding <strong>why this deployment problem exists</strong>, especially in the era of cloud computing. The root of the issue often lies in how we define workflows and pipelines in bioinformatics.</p><h3>The Real Problem: Workflows vs Pipelines</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*EnToecMokF7LMj91ZMhpjA.png" /><figcaption>Workflows vs Pipelines</figcaption></figure><p><strong>A workflow is the computational logic.</strong> It’s the executable definition of what runs, in what order, and how data flows between steps. This is what you write in Nextflow, WDL, Snakemake, or CWL. It bundles tools into processes, connects them with data channels, and defines the DAG for execution. A workflow answers one question clearly: <em>what computation happens?</em></p><p><strong>A pipeline is the full operational system</strong> that turns raw data into a usable result, reliably and repeatedly, under real-world constraints. It includes the workflow, but also everything around it: reference databases, storage architecture, compute infrastructure, security controls, access management, monitoring, cost controls, retries, audits, and operational ownership. A pipeline answers a different question: <em>can we trust this to run in production?</em></p><p>The relationship is simple but critical. <strong>The workflow is a major component of the pipeline, but it is not the pipeline by itself.</strong></p><h3>Why This Decision Matters Now</h3><p>Four converging trends are forcing bioinformatics teams to professionalize their infrastructure:</p><p>1. Explosive Data Growth</p><p>2. Regulatory Pressure</p><p>3. Competitive Advantage</p><p>4. Talent Gap</p><h3>Understanding Your Requirements</h3><p>Before choosing an approach, assess two key factors:</p><p><strong>1. Scale</strong></p><ul><li>Small: Under 1,000 samples/month</li><li>Medium: 1,000–10,000 samples/month</li><li>Large: Over 10,000 samples/month</li></ul><p><strong>2. Team Capability</strong></p><ul><li>Basic: Bioinformaticians with minimal cloud experience</li><li>Intermediate: Some cloud knowledge</li><li>Advanced: DevOps team with Kubernetes expertise</li></ul><p>These two factors determine which option fits best.</p><h3>Three Approaches to Production Pipelines:</h3><h4>Option 1: AWS HealthOmics</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1000/1*C16H1TgbwXdNqxywFLnuVQ.png" /></figure><p><a href="https://aws.amazon.com/healthomics/">AWS HealthOmics</a> is a purpose-built, fully managed service for genomics and multiomics workloads. It provides storage, analytics, and workflow execution specifically designed for healthcare and life sciences.</p><h4>Perfect For:</h4><ol><li>Clinical diagnostics labs</li><li>Healthcare organizations handling patient data</li><li>Pharma companies with FDA submission requirements</li><li>Organizations prioritizing compliance over flexibility</li></ol><p><strong>Managed By:</strong> AWS (fully managed service)</p><h4><strong>Team Requirements:</strong></h4><ul><li>Need someone who understands bioinformatics and basic AWS concepts</li><li>No DevOps expertise required</li><li>Can be run by the bioinformatics team directly</li></ul><h4>How Fast Can You Launch:</h4><ul><li>Setup: Quick (mostly account configuration)</li><li>First production run: Very fast</li><li>Ongoing management: Minimal (few hours per week)</li></ul><h4>The Big Promise<strong>:</strong></h4><p>HealthOmics removes infrastructure complexity entirely. You focus on workflows, AWS handles everything else: compute provisioning, scaling, security, compliance controls, and monitoring.</p><h4>The Money Talk<strong>:</strong></h4><ul><li><strong>Most expensive per sample</strong> (premium for managed service)</li><li><strong>Lowest operational cost</strong> (minimal staff time required)</li><li><strong>Fastest time to value</strong> (no infrastructure setup delays)</li></ul><p>At small to medium scale (under 5,000 samples/month), the total cost is competitive because you save on personnel. At a very large scale, per-sample costs become the dominant factor.</p><h4>The Good Stuff<strong>:</strong></h4><ul><li><strong>Compliance Built-In:</strong> HIPAA-ready architecture, audit logging, encryption by default</li><li><strong>Zero Infrastructure Management:</strong> AWS handles everything</li><li><strong>Healthcare-Specific Optimizations:</strong> Purpose-built for genomics</li><li><strong>Fastest Time to Production:</strong> Days, not weeks</li><li><strong>Predictable Costs:</strong> Clear pricing, no surprise bills</li><li><strong>Integrated Analytics:</strong> Built-in variant analysis and annotation tools</li></ul><h4><strong>Critical HIPAA Compliance Note:</strong></h4><p>While AWS HealthOmics is designed for healthcare workloads, <strong>using cloud storage for PHI (Protected Health Information) requires specific HIPAA compliance measures</strong></p><h4>The Tradeoffs<strong>:</strong></h4><ul><li><strong>Vendor Lock-In:</strong> Tightly coupled to the AWS HealthOmics ecosystem</li><li><strong>Limited Flexibility:</strong> Cannot customize the infrastructure deeply</li><li><strong>Higher Per-Sample Cost at Scale:</strong> Premium pricing for managed service</li><li><strong>Regional Availability:</strong> Not available in all AWS regions yet</li><li><strong>Newer Service:</strong> Smaller community, fewer third-party integrations</li></ul><h4><strong>Strategic Considerations:</strong></h4><ul><li>Migrating away requires workflow rewrites</li><li>Cost per sample decreases at high volume but remains higher than self-managed options</li><li>Faster time to market can justify premium pricing</li></ul><p><a href="https://medium.com/nonstopio/how-to-deploy-production-grade-bioinformatics-pipelines-using-aws-healthomics-b7899b19b34f">Read the AWS HealthOmics implementation guide</a></p><h3>Option 2: Batch Services with Nextflow</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*U4DruNdvhH0syoLzB9A0CA.png" /><figcaption>Batch Architecture</figcaption></figure><p>Batch computing services (<a href="https://aws.amazon.com/batch/">AWS Batch</a>, <a href="https://azure.microsoft.com/en-us/products/batch">Azure Batch</a>, <a href="https://docs.cloud.google.com/batch/docs/get-started">Google Cloud Batch</a>) are managed services that run batch jobs at any scale. Combined with Nextflow, they enable production-grade workflow execution without requiring infrastructure management.</p><h4>Perfect For:</h4><ol><li>Research labs and academic institutions</li><li>Biotech startups with limited IT resources</li><li>Organizations prioritizing simplicity and cost efficiency</li><li>Teams are comfortable with bioinformatics, but have minimal DevOps experience</li></ol><p><strong>Managed By:</strong> Partially managed (cloud provider handles compute scaling, you manage workflow orchestration)</p><h4><strong>Team Requirements:</strong></h4><ul><li>Need someone who understands bioinformatics and understands cloud</li><li>Cloud knowledge is required</li><li>No dedicated DevOps team required</li></ul><h4>How Fast Can You Launch<strong>:</strong></h4><ul><li>Setup: Quick</li><li>First production run: Fast</li><li>Ongoing management: Moderate (several hours per week)</li></ul><h4>The Big Promise:</h4><p>Batch services provide the simplest path to production-grade pipelines. No server management, automatic scaling, pay only for compute time used. You get reliability without complexity.</p><h4>The Money Talk<strong>:</strong></h4><ul><li><strong>Lowest infrastructure cost</strong> (especially with spot instances)</li><li><strong>Moderate operational cost</strong> (some cloud management required)</li><li><strong>Best cost efficiency at small to medium scale</strong></li></ul><p>Using spot instances reduces compute costs by up to 90%. At medium scale (100–5,000 samples/month), this is typically the most cost-effective option overall.</p><h4>The Good Stuff<strong>:</strong></h4><ul><li><strong>Lowest Entry Cost:</strong> Minimal upfront investment</li><li><strong>Simple Architecture:</strong> Fewer moving parts than Kubernetes</li><li><strong>No Server Management:</strong> Cloud provider handles provisioning</li><li><strong>Spot Instance Support:</strong> Up to 90% savings on compute</li><li><strong>Quick Setup:</strong> Production-ready in days</li><li><strong>Flexible Scaling:</strong> Handles 10 samples or 100,000 automatically</li><li><strong>Low Operational Overhead:</strong> Minimal ongoing maintenance</li></ul><h4><strong>Important for Clinical/PHI Data:</strong></h4><p>If handling Protected Health Information, <strong>batch services require manual HIPAA compliance configuration</strong></p><h4>The Tradeoffs<strong>:</strong></h4><ul><li><strong>Limited Customization:</strong> Cannot deeply tune infrastructure</li><li><strong>Spot Instance Interruptions:</strong> Occasional job restarts (mitigated by Nextflow resume)</li><li><strong>Smaller Community:</strong> Less ecosystem than Kubernetes</li><li><strong>Cloud-Specific:</strong> Skills are less transferable between AWS Batch, Azure Batch, and GCP Batch</li><li><strong>HIPAA Compliance Manual:</strong> No built-in compliance controls</li></ul><h4>Strategic Considerations:</h4><ul><li>Small to medium teams with limited cloud expertise</li><li>Cost-sensitive organizations</li><li>Standard Nextflow workflows without deep customization needs</li><li>Want to start simple and scale gradually</li></ul><p><a href="https://medium.com/nonstopio/running-nextflow-on-aws-batch-a-practical-guide-c7033cb8594c">Read the AWS Batch implementation guide</a></p><h3>Option 3: Nextflow on Kubernetes</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*4y2hEF5rRvOFt5hsFpAl5w.png" /><figcaption>Kubernetes Architecture</figcaption></figure><p>Kubernetes is a container orchestration platform that provides maximum control over infrastructure. Running Nextflow on Kubernetes (Amazon EKS, Azure AKS, Google GKE) provides the flexibility to customise every aspect.</p><h4>Perfect For<strong>:</strong></h4><ol><li>Enterprise organizations with DevOps teams</li><li>Companies running multiple complex workflows</li><li>Teams requiring deep infrastructure customization</li><li>Organizations with existing Kubernetes expertise</li></ol><p><strong>Managed By:</strong> Partially managed (cloud provider manages Kubernetes control plane, you manage most other aspects)</p><h4><strong>Team Requirements:</strong></h4><ul><li>Need dedicated DevOps engineers with Kubernetes expertise</li><li>People who understand Kubernetes, cloud networking, and security</li><li>Not suitable for teams without Kubernetes experience</li></ul><h4>How Fast Can You Launch<strong>:</strong></h4><ul><li>Setup: Longer (cluster setup, networking, RBAC, monitoring)</li><li>First production run: Significant time investment</li><li>Ongoing management: Substantial (requires dedicated attention)</li></ul><h4>The Big Promise<strong>:</strong></h4><p>Kubernetes provides maximum flexibility and control. If you have complex requirements, need advanced features, or already have Kubernetes expertise, this gives you the power to build exactly what you need.</p><h4>The Money Talk<strong>:</strong></h4><ul><li><strong>Infrastructure costs are competitive with Batch</strong> (at large scale, potentially lower)</li><li><strong>Highest operational cost</strong> (requires a dedicated DevOps team)</li><li><strong>Highest fixed costs</strong> (team required regardless of volume)</li></ul><p>The infrastructure is cost-efficient, but you need personnel to manage it. At small to medium scale, the team cost outweighs infrastructure savings. At very large scale (10,000+ samples/month), infrastructure optimization makes this cost-competitive overall.</p><h4>The Good Stuff<strong>:</strong></h4><ul><li><strong>Maximum Flexibility:</strong> Complete control over infrastructure</li><li><strong>Multi-Cloud Portability:</strong> Kubernetes skills work across EKS, AKS, GKE</li><li><strong>Advanced Features:</strong> Custom networking, service mesh, complex orchestration</li><li><strong>Huge Ecosystem:</strong> Extensive community, mature tooling</li><li><strong>Enterprise Features:</strong> Sophisticated RBAC, audit logging, compliance controls</li><li><strong>Multiple Workflows:</strong> Easily run many pipelines concurrently</li><li><strong>Deep Optimization Possible:</strong> Can tune for maximum efficiency at scale</li></ul><h4><strong>HIPAA Compliance:</strong></h4><p>Kubernetes provides the <strong>most control</strong> over compliance but also <strong>the most responsibility.</strong></p><h4>The Tradeoffs<strong>:</strong></h4><ul><li><strong>High Operational Cost:</strong> Requires a dedicated DevOps team</li><li><strong>Complex Setup:</strong> Weeks to production instead of days</li><li><strong>Steep Learning Curve:</strong> Team needs Kubernetes expertise</li><li><strong>Ongoing Maintenance:</strong> Security patches, version upgrades, monitoring</li><li><strong>Over-Engineering Risk:</strong> Often overkill for simpler use cases</li><li><strong>Higher Fixed Costs:</strong> Team required regardless of processing volume</li><li><strong>Troubleshooting Complexity:</strong> More layers to debug when issues occur</li><li><strong>HIPAA Compliance Complexity:</strong> Complete manual configuration required</li></ul><h4>Strategic Considerations:</h4><ul><li>Enterprise organizations with existing DevOps/SRE teams</li><li>Already running Kubernetes for other services</li><li>Multiple complex workflows with interdependencies</li><li>Teams are already experts in Kubernetes</li><li>Regulatory requirements needing deep infrastructure control</li><li>Very large scale, where infrastructure optimization becomes critical</li></ul><p><a href="https://medium.com/nonstopio/how-we-built-scalable-bioinformatics-pipelines-on-aws-eks-using-nextflow-bff42df37883">Read the Amazon EKS implementation guide</a></p><h3>Side-by-Side Comparison</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/787/1*KsAqOhMrJlVSjmS2jnRkQw.png" /></figure><h3>Decision Framework</h3><p><strong>Choose based on your situation:</strong></p><p><strong>HealthOmics</strong> → Clinical diagnostics, need HIPAA compliance, limited IT team.<br> <strong>Batch Services</strong> → Research labs, cost-sensitive, small to medium scale.<br> <strong>Kubernetes</strong> → Enterprise with a DevOps team needs advanced customization, large scale.</p><h3>Conclusion</h3><p>The bioinformatics infrastructure market is growing at 13% annually, reaching $18.7 billion by 2027. Organizations that choose infrastructure wisely gain a competitive advantage through faster results, lower costs, and better reliability. Those who choose poorly waste money on over-engineered solutions or face scaling problems later.</p><blockquote>Here to make the community stronger by sharing our knowledge. Follow <a href="https://medium.com/@tulshiram.kadam"><strong>me</strong></a>, <a href="https://medium.com/@tabishaliansari"><strong>Tabish </strong></a>and <a href="https://medium.com/nonstopio"><strong>my team</strong></a> to stay updated on the latest and greatest in the web &amp; mobile tech world.</blockquote><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=4dce18963a83" width="1" height="1" alt=""><hr><p><a href="https://blog.nonstopio.com/running-bioinformatics-pipelines-in-production-a-strategic-guide-for-decision-makers-4dce18963a83">Running Bioinformatics Pipelines in Production: A Strategic Guide for Decision-Makers</a> was originally published in <a href="https://blog.nonstopio.com/">nonstopio</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Optimizing Computational Workflows: Running Nextflow on AWS Batch]]></title>
            <link>https://blog.nonstopio.com/running-nextflow-on-aws-batch-a-practical-guide-c7033cb8594c?source=rss-d67062a61bce------2</link>
            <guid isPermaLink="false">https://medium.com/p/c7033cb8594c</guid>
            <category><![CDATA[genomics]]></category>
            <category><![CDATA[devops]]></category>
            <category><![CDATA[aws-batch]]></category>
            <category><![CDATA[nextflow]]></category>
            <category><![CDATA[bioinformatics]]></category>
            <dc:creator><![CDATA[Tulshiram Kadam]]></dc:creator>
            <pubDate>Thu, 08 Jan 2026 13:17:25 GMT</pubDate>
            <atom:updated>2026-01-12T10:35:28.472Z</atom:updated>
            <content:encoded><![CDATA[<p>Run production-grade Nextflow pipelines using AWS Batch with minimal setup. No complexity. No orchestration headaches. Just working on pipelines.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*hyBepaSYe6dTHewu2xX3eQ.png" /></figure><h3>What is Nextflow?</h3><p>Before we dive into the “why AWS Batch” question, let’s talk about what <a href="https://www.nextflow.io/">Nextflow</a> actually is.</p><p>Nextflow is a workflow orchestration engine designed for data-intensive computational pipelines. Think of it as a sophisticated task manager that can:</p><ul><li><strong>Chain together multiple steps</strong>: Take raw data, run it through analysis tools, and generate reports in a defined sequence</li><li><strong>Handle dependencies automatically</strong>: Task B won’t run until Task A finishes successfully</li><li><strong>Parallelize work</strong>: If you have 100 samples to analyze, Nextflow can process them all at once (resource permitting)</li><li><strong>Resume from failures</strong>: If your pipeline crashes at step 5 out of 10, you can restart from step 5 instead of starting over</li></ul><p>Originally built for genomics (where a single analysis might have 20+ steps and run for days), Nextflow is now popular in data science, ML pipelines, and any field dealing with complex data processing.</p><p>The key feature? Nextflow utilises containers (such as Docker) for each task, ensuring your analysis is reproducible; it’ll run the same way on a laptop as it does on a cloud cluster.</p><h3>Why AWS Batch for Nextflow?</h3><p><a href="https://aws.amazon.com/batch">AWS Batch</a> is a managed service that runs batch computing workloads. You define what to compute, AWS handles everything else: provisioning servers, queuing jobs, scaling up and down, and cleanup.</p><p>Here’s why it pairs perfectly with Nextflow:</p><ul><li>Hands-off infrastructure: No servers to patch, no clusters to maintain</li><li>Smart scaling: Starts instances when jobs arrive, stops them when idle</li><li>Flexible pricing: Choose on-demand for or spot for 90% cost savings</li><li>Built-in monitoring: CloudWatch tracks everything automatically</li><li>Simple architecture: Fewer moving parts means fewer things to break</li></ul><p>Compare this to running your own compute cluster or managing complex infrastructure. AWS Batch removes all that operational burden.</p><h3>How it Works:</h3><p>EC2 Instance or Local Machine(Nextflow) <br>↓ <br>Submits jobs to the Batch Queue <br>↓ <br>Batch spins up compute instances(Managed by AWS Batch)<br>↓ <br>Runs containers (FASTQC, MultiQC)<br>↓ <br>Saves results to S3 <br>↓ <br>Shuts down compute instances</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*sHHyXZ2R20JTkXdLvhulrg.avif" /><figcaption>Nextflow working on AWS Batch</figcaption></figure><h3>What We’re Building</h3><ol><li>S3 bucket (stores input data, results, and temp files)</li><li>Batch compute environment (the worker pool)</li><li>Batch job queue (routes work to available compute)</li><li>One EC2 instance (runs the Nextflow coordinator)</li><li>The rest will be managed by AWS Batch</li></ol><h3>Phase 1: Building Our Infrastructure</h3><h4>Step 1: Create S3 Bucket</h4><p><strong>1. Open S3 Console:</strong></p><ul><li>AWS Console → S3 → Create bucket</li></ul><p><strong>2. Configure:</strong></p><ul><li>Bucket name: nextflow-demo-yourname (must be globally unique)</li><li>Region: select your nearest region</li><li>Keep defaults</li><li>Click <strong>Create bucket</strong></li></ul><p><strong>3. Create Folders:</strong></p><p>Inside the bucket:</p><ul><li>Click <strong>Create folder</strong> → Name: inputs → Create</li><li>Create folder: results</li><li>Create folder: work</li></ul><p><strong>4. Upload Data:</strong></p><ul><li>Open inputs/ folder</li><li>Click <strong>Upload</strong></li><li>Add your FASTQ file</li><li>Click <strong>Upload</strong></li></ul><h4>Step 2: Create IAM Roles</h4><p>We need two roles: one for the EC2 instance, one for the Batch compute environment.</p><h4>Role 1: EC2 Instance Role</h4><p><strong>1. Open IAM Console:</strong></p><ul><li>AWS Console → IAM → Roles → Create role</li></ul><p><strong>2. Select Trusted Entity:</strong></p><ul><li>AWS service → EC2</li><li>Click <strong>Next</strong></li></ul><p><strong>3. Attach Policies:</strong></p><p>Add these three policies:</p><ul><li>AmazonS3FullAccess</li><li>AWSBatchFullAccess</li><li>AmazonECS_FullAccess</li></ul><p><strong>Note</strong>: To restrict this to the exact bucket, you can create a custom policy and attach it instead of the AmazonS3FullAccess policy.</p><p>Click <strong>Next</strong></p><p><strong>4. Name and Create:</strong></p><ul><li>Role name: NextflowEC2Role</li><li>Click <strong>Create role</strong></li></ul><h4>Role 2: Batch Compute Environment Role</h4><p><strong>1. Create New Role:</strong></p><ul><li>Click <strong>Create role</strong></li></ul><p><strong>2. Select Trusted Entity:</strong></p><ul><li>AWS service → EC2</li><li>Click <strong>Next</strong></li></ul><p><strong>3. Attach Policy:</strong></p><ul><li>AmazonEC2ContainerServiceforEC2Role</li><li>AmazonS3FullAccess</li></ul><p><strong>Note</strong>: To restrict this to the exact bucket, you can create a custom policy and attach it instead of the AmazonS3FullAccess policy.</p><p>Click <strong>Next</strong></p><p><strong>4. Name and Create:</strong></p><ul><li>Role name: BatchComputeInstanceRole</li><li>Click <strong>Create role</strong></li></ul><h4>Step 3: Create AWS Batch Compute Environment</h4><p><strong>1. Open Batch Console:</strong></p><ul><li>AWS Console → Batch → Compute environments → Create</li><li>Select Amazon EC2</li></ul><p><strong>2. Orchestration Type:</strong></p><ul><li>Select <strong>Managed</strong></li></ul><p><strong>3. Name:</strong></p><ul><li>Compute environment name: nextflow-compute</li></ul><p><strong>4. Service Role:</strong></p><ul><li>Select <strong>Create a new service role</strong></li><li>AWS creates this automatically</li></ul><p><strong>5. Instance Configuration:</strong></p><ul><li>Instance role: Select BatchComputeInstanceRole</li><li>Click <strong>Next</strong></li></ul><p><strong>6. Capacity:</strong></p><p>PS: You can modify this as per your requirements</p><ul><li>Provisioning model: <strong>On-Demand or </strong>select <strong>Spot</strong></li><li>Minimum vCPUs: <strong>0</strong></li><li>Desired vCPUs: <strong>0</strong></li><li>Maximum vCPUs: <strong>16</strong></li><li>Instance types: select <strong>default_x86_64</strong></li><li>Allocation Strategy: keep it <strong>default</strong>, or you can select as per your need</li><li>Click<strong> Next</strong></li></ul><p><strong>7. Network:</strong></p><ul><li>Use the default VPC</li><li>Select all available subnets</li><li>Security groups: Default</li></ul><p><strong>8. Create:</strong></p><ul><li>Click <strong>Create compute environment</strong></li><li>Wait for status: <strong>VALID</strong></li></ul><h4>Step 4: Create Job Queue</h4><p><strong>1. Navigate:</strong></p><ul><li>Batch → Job queues → Create</li></ul><p><strong>2. Configure:</strong></p><ul><li>Select Orchestration type: <strong>EC2</strong></li><li>Job queue name: nextflow-queue</li><li>Priority: <strong>1</strong></li><li>Connected compute environments: Select nextflow-compute</li></ul><p><strong>3. Create:</strong></p><ul><li>Click <strong>Create job queue</strong></li></ul><h3>Step 5: Launch EC2 Instance</h3><p><strong>1. Open EC2 Console:</strong></p><ul><li>AWS Console → EC2 → Launch instance</li></ul><p><strong>2. Basic Configuration:</strong></p><ul><li>Name: nextflow-launcher</li><li>AMI: <strong>Amazon Linux</strong></li><li>Instance type: <strong>t3.small</strong></li><li>Key pair: Select existing or create new</li></ul><p><strong>3. Network Settings:</strong></p><ul><li>VPC: Default</li><li>Auto-assign public IP: <strong>Enable</strong></li><li>Security group: Create new or use existing</li><li>Allow SSH (port 22) from your IP</li></ul><p><strong>4. Advanced Details:</strong></p><ul><li>Scroll down to <strong>IAM instance profile</strong></li><li>Select: NextflowEC2Role</li></ul><p><strong>5. Launch:</strong></p><ul><li>Click <strong>Launch instance</strong></li><li>Wait for status: <strong>Running</strong></li></ul><h3>Step 6: Install Nextflow</h3><p><strong>1. Connect to the Instance</strong></p><p>If required, the pipeline can also be run from a local system after installing and configuring the AWS CLI. However, we’ve set up a dedicated EC2 instance for this purpose, and we’ll be using that going forward.</p><p><strong>2. Install Nextflow:</strong></p><pre>sudo dnf install java -y<br>curl -s https://get.nextflow.io | bash<br>sudo mv nextflow /usr/local/bin/<br>sudo chmod +x /usr/local/bin/nextflow<br>nextflow -version</pre><h3>Phase 2: The Nextflow Pipeline</h3><p>For this example, we’re running a bioinformatics quality control workflow on genomic sequencing data(<a href="https://drive.google.com/file/d/1YAmlDtCp3HU0HPIYtzqcTqISKWlqvwNk/view?usp=sharing">Download</a>). Specifically:</p><p><strong>What we’re doing:</strong></p><ul><li><strong>FASTQC</strong>: Analyzes our raw sequencing data (FASTQ file) and generates quality control reports. It checks things like read quality scores, sequence duplication levels, GC content, and potential contamination. Each dataset gets its own HTML report with pretty graphs.</li><li><strong>MultiQC</strong>: Takes all those individual FASTQC reports and aggregates them into a single, beautiful summary report. Instead of opening 10 different HTML files, you get a comprehensive view of your data quality in one place.</li></ul><p>Think of it this way: FASTQC is your detailed inspector, checking each piece individually, and MultiQC is the manager who summarises everything into a single executive report.</p><p><strong>main.nf-</strong> This is your pipeline definition:</p><pre>#!/usr/bin/env nextflow<br>nextflow.enable.dsl=2<br><br>params {<br>    reads  = &#39;s3://your-bucket-name/inputs/sample_data.fastq&#39;<br>    outdir = &#39;s3://your-bucket-name/results&#39;<br>}<br><br>process FASTQC {<br>    container &#39;biocontainers/fastqc:v0.11.9_cv8&#39;<br>    publishDir &quot;${params.outdir}/fastqc&quot;, mode: &#39;copy&#39;<br>    <br>    input:<br>    path reads<br>    <br>    output:<br>    tuple path(&quot;*.html&quot;), path(&quot;*_fastqc.zip&quot;)<br>    <br>    script:<br>    &quot;&quot;&quot;<br>    echo &quot;Running FASTQC on ${reads}&quot;<br>    fastqc ${reads}<br>    echo &quot;FASTQC completed for ${reads}&quot;<br>    &quot;&quot;&quot;<br>}<br><br>process MULTIQC {<br>    container &#39;multiqc/multiqc:dev&#39;<br>    publishDir &quot;${params.outdir}/multiqc&quot;, mode: &#39;copy&#39;<br>    <br>    input:<br>    path fastqc_reports<br>    <br>    output:<br>    path &quot;multiqc_report.html&quot;<br>    path &quot;multiqc_data&quot;<br>    <br>    script:<br>    &quot;&quot;&quot;<br>    echo &quot;Aggregating FASTQC reports with MultiQC&quot;<br>    multiqc .<br>    echo &quot;MultiQC report generated successfully&quot;<br>    &quot;&quot;&quot;<br>}<br><br>workflow {<br>    reads_ch = Channel.fromPath(params.reads)<br>    fastqc_reports_ch = FASTQC(reads_ch)<br>    MULTIQC(fastqc_reports_ch.collect())<br>}</pre><p><strong>nextflow.config- </strong>This tells Nextflow how to run on batch:</p><pre>plugins {<br>    id &#39;nf-wave&#39;<br>    id &#39;nf-amazon&#39;<br>}<br><br>wave {<br>    enabled = true<br>}<br><br>fusion {<br>    enabled = true<br><br>}<br><br>process {<br>    executor = &#39;awsbatch&#39;<br>    queue    = &#39;&lt;job-queue-name&gt;&#39;<br>    <br>    cpus   = 1<br>    memory = &#39;2 GB&#39;<br><br>    // Error strategy: retry on transient AWS Batch issues<br>    errorStrategy = { task.exitStatus in [130, 137, 143, 151] ? &#39;retry&#39; : &#39;finish&#39; }<br>    maxRetries = 3<br><br>    withName: FASTQC {<br>        container = &#39;biocontainers/fastqc:v0.11.9_cv8&#39;<br>    }<br><br>    withName: MULTIQC {<br>        container = &#39;multiqc/multiqc:v1.14&#39;<br>    }<br>}<br><br>aws {<br>    region = &#39;us-east-1&#39;<br>    batch {<br>        volumes = &#39;/fusion&#39;<br>    }<br>}<br><br>workDir = &#39;s3://&lt;your-bucket-name&gt;/work&#39;</pre><p><strong>Important:</strong> Replace &lt;yourbucket-name&gt; and &lt;job-queue-name&gt;with your actual name in both files.<br>Push this to your GitHub repo.</p><h4>Run the Pipeline</h4><pre>nextflow run https://github.com/username/repo-name.git</pre><p>Output will look like:</p><pre>N E X T F L O W  ~  version 24.04.4<br>Pulling username/repo-name ...<br>downloaded from https://github.com/username/repo-name.git<br>Downloading plugin nf-wave@1.16.1<br>Downloading plugin nf-amazon@3.4.2<br>Downloading plugin nf-tower@1.17.3<br>Launching `https://github.com/username/repo-name` [awesome_lichterman] DSL2 - revision: e4b5f72abf [main]<br>executor &gt;  awsbatch (2)<br>[a1/b2c3] process &gt; FASTQC   [100%] 1 of 1 ✔<br>[e5/f6g7] process &gt; MULTIQC  [100%] 1 of 1 ✔<br>Completed at: 08-Jan-2025 10:30:15<br>Duration    : 3m 45s<br>CPU hours   : 0.4<br>Succeeded   : 2</pre><p>What happens:</p><ol><li>Nextflow submits jobs to AWS Batch</li><li>Batch automatically spins up EC2 instances</li><li>Runs FASTQC and MultiQC in containers</li><li>Saves results to S3</li><li>Shuts down instances when done</li></ol><h3>Conclusion</h3><p>You’ve just built a production-ready Nextflow pipeline on AWS Batch. With minimal setup (S3, two IAM roles, Batch environment, and one EC2 instance), you now have a system that:</p><ul><li>Automatically scales compute based on workload</li><li>Handles job queuing and execution without manual intervention</li><li>Saves results directly to S3</li><li>Costs only pennies per run</li></ul><p>The beauty of this setup is its simplicity. No complex networking. No multiple services to maintain. Just Nextflow coordinating jobs and AWS Batch handling the compute.</p><p><strong>What makes this powerful:</strong></p><p>AWS Batch removes the operational burden. You don’t patch servers, manage clusters, or worry about capacity planning. Submit your pipeline, and AWS takes care of the rest.</p><p>Nextflow handles the complexity of your workflow. Dependencies, retries, parallelization, and resume capability are all built in. Your job is defining what to analyze, not how to manage the infrastructure.</p><p><strong>Real-world impact:</strong></p><p>This same setup scales from analyzing a single sample to processing thousands. The pipeline we built today (FASTQC + MultiQC) can easily extend to variant calling, RNA-seq analysis, or any multi-step computational workflow.</p><p>Teams using this approach report:</p><ul><li>90% reduction in infrastructure management time</li><li>80–90% cost reduction on spot instance</li><li>Faster iterations on analysis pipelines</li><li>Reproducible results across different environments</li></ul><p>Happy Hosting!</p><blockquote><em>Here to make the community stronger by sharing our knowledge. Follow </em><a href="https://medium.com/@tulshiram.kadam"><strong><em>me</em></strong></a><strong><em> </em></strong><em>and </em><a href="https://medium.com/nonstopio"><strong><em>my team</em></strong></a><em> to stay updated on the latest and greatest in the web &amp; mobile tech world.</em></blockquote><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=c7033cb8594c" width="1" height="1" alt=""><hr><p><a href="https://blog.nonstopio.com/running-nextflow-on-aws-batch-a-practical-guide-c7033cb8594c">Optimizing Computational Workflows: Running Nextflow on AWS Batch</a> was originally published in <a href="https://blog.nonstopio.com/">nonstopio</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[How We Built Scalable Bioinformatics Pipelines on AWS EKS Using Nextflow]]></title>
            <link>https://blog.nonstopio.com/how-we-built-scalable-bioinformatics-pipelines-on-aws-eks-using-nextflow-bff42df37883?source=rss-d67062a61bce------2</link>
            <guid isPermaLink="false">https://medium.com/p/bff42df37883</guid>
            <category><![CDATA[aws-eks]]></category>
            <category><![CDATA[devops]]></category>
            <category><![CDATA[nextflow]]></category>
            <category><![CDATA[aws]]></category>
            <category><![CDATA[bioinformatics]]></category>
            <dc:creator><![CDATA[Tulshiram Kadam]]></dc:creator>
            <pubDate>Thu, 08 Jan 2026 06:30:41 GMT</pubDate>
            <atom:updated>2026-01-08T06:30:41.331Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*3cMSVfgXWDaNVrCXxEtQCg.png" /></figure><p>If you’ve ever tried running Nextflow workflows in the cloud, you know the struggle is real. Local machines? Too slow. Manual scaling? A nightmare. That’s why I decided to dive headfirst into running Nextflow on Amazon EKS (Elastic Kubernetes Service).</p><p>In this guide, I’ll walk you through exactly how I set up a Nextflow environment on EKS, complete with spot instances for cost savings and secure S3 access using Pod Identity.</p><h3>What is Nextflow?</h3><p>Before we dive into the “why EKS” question, let’s talk about what <a href="https://www.nextflow.io/">Nextflow</a> actually is.</p><p>Nextflow is a workflow orchestration engine designed for data-intensive computational pipelines. Think of it as a sophisticated task manager that can:</p><ul><li><strong>Chain together multiple steps</strong>: Take raw data, run it through analysis tools, and generate reports in a defined sequence</li><li><strong>Handle dependencies automatically</strong>: Task B won’t run until Task A finishes successfully</li><li><strong>Parallelize work</strong>: If you have 100 samples to analyze, Nextflow can process them all at once (resource permitting)</li><li><strong>Resume from failures</strong>: If your pipeline crashes at step 5 out of 10, you can restart from step 5 instead of starting over</li></ul><p>It was originally built for bioinformatics (genomics workflows can have dozens of steps and take days to run), but it’s now used in machine learning, data science, and anywhere you have complex, multi-step computational workflows.</p><p>The key feature? Nextflow uses <strong>containers</strong> (like Docker) for each task, which means your analysis is reproducible’ll run the same way on my laptop as it does on your cloud cluster.</p><h4>Why EKS for Nextflow?</h4><p>Before we jump in, let’s talk about <em>why</em> this setup makes sense.</p><p>Nextflow is fantastic for orchestrating complex computational workflows, but it needs resources and lots of them. Running everything locally or on a single EC2 instance quickly becomes a bottleneck. Kubernetes, on the other hand, gives us:</p><ul><li><strong>Auto-scaling</strong>: Spin up pods as needed, shut them down when done</li><li><strong>Cost optimization</strong>: Use spot instances to cut costs by up to 90%</li><li><strong>Flexibility</strong>: Run multiple workflows concurrently without stepping on each other’s toes</li><li><strong>Reliability</strong>: Built-in retry logic and fault tolerance</li></ul><p>EKS takes all of this and makes it managed, which means less time wrestling with control planes and more time getting actual work done.</p><h3>The Game Plan</h3><p>Here’s what we’re building:</p><ol><li>An EKS cluster optimized for batch workloads</li><li>Secure S3 access using Pod Identity</li><li>Kubernetes RBAC so our Nextflow head pod can manage worker pods</li><li>GitHub integration using SSH keys for pipeline code management</li></ol><p>Let’s break it down phase by phase.</p><h3>Prerequisites: Getting Your Tools Ready</h3><p>Before we dive into building the cluster, we need to make sure our local environment is set up properly. You’ll need two essential tools: the AWS CLI and kubectl.</p><h3>Configure AWS CLI</h3><p>If you haven’t already, install and configure the AWS CLI. This is how you’ll interact with AWS services from your terminal.</p><p>Install the AWS CLI (if needed):</p><pre># For Linux<br>curl &quot;https://awscli.amazonaws.com/awscli-exe-linux-x86_64.zip&quot; -o &quot;awscliv2.zip&quot;<br>unzip awscliv2.zip<br>sudo ./aws/install</pre><pre># For macOS<br>brew install awscli</pre><p>Now configure it with your credentials:</p><pre>aws configure</pre><h3>You’ll be prompted for:</h3><ul><li><strong>AWS Access Key ID</strong>: Your IAM user access key</li><li><strong>AWS Secret Access Key</strong>: Your secret key</li><li><strong>Default region name</strong>: I’m using us-east-1 for this guide</li><li><strong>Default output format</strong>: json works great</li></ul><p><strong>Pro tip</strong>: If you’re working with multiple AWS accounts or profiles, use aws configure --profile &lt;profile-name&gt; to keep things organized.</p><h3>Install kubectl</h3><p>kubectl is your command-line tool for interacting with Kubernetes clusters. You absolutely need this.</p><p>For Linux:</p><pre>curl -LO &quot;https://dl.k8s.io/release/$(curl -L -s https://dl.k8s.io/release/stable.txt)/bin/linux/amd64/kubectl&quot;<br>sudo install -o root -g root -m 0755 kubectl /usr/local/bin/kubectl</pre><p>For macOS:</p><pre>brew install kubectl</pre><p>Verify the installation:</p><pre>kubectl version --client</pre><h3>Install eksctl</h3><p>While we’re at it, let’s install eksctl too. We&#39;ll need it for creating the cluster:</p><pre># For Linux<br>curl --silent --location &quot;https://github.com/weaveworks/eksctl/releases/latest/download/eksctl_$(uname -s)_amd64.tar.gz&quot; | tar xz -C /tmp<br>sudo mv /tmp/eksctl /usr/local/bin</pre><pre># For macOS<br>brew install eksctl</pre><p>Alright, tools ready!</p><h3>Phase 1: Building Our Infrastructure</h3><h3>Setting Up the EKS Cluster</h3><p>First, we need a cluster. I’m using eksctl because it handles a lot of the heavy lifting for us.</p><pre>eksctl create cluster \<br>  --name nextflow-eks-cluster \<br>  --region us-east-1 \<br>  --nodegroup-name spot-nodes \<br>  --instance-types t3.medium \<br>  --managed \<br>  --spot \<br>  --nodes 1 \<br>  --with-oidc \<br>  --vpc-nat-mode Disable</pre><p><strong>A quick note on that last flag</strong>: I disabled NAT Gateway here to keep costs down during testing. In production, you’ll want to enable it for better security and connectivity.</p><p>The --spot flag is doing the real magic here. It tells EKS to use spot instances, which are way cheaper than on-demand instances. Perfect for batch workloads where a little interruption risk is acceptable.</p><h3>Installing Pod Identity Agent</h3><p>Now here’s where things get interesting. Instead of creating IAM users and storing access keys, we’re using EKS Pod Identity. This lets our pods assume IAM roles directly.</p><pre>eksctl create addon \<br>  --cluster nextflow-eks-cluster \<br>  --name eks-pod-identity-agent \<br>  --region us-east-1</pre><p>This add-on runs as a DaemonSet on your cluster and handles all the credential magic behind the scenes.</p><h3>S3 Bucket Setup</h3><p>Head over to the S3 console and create a bucket. Inside it, create a folder called inputs . This is where we&#39;ll store our input data.</p><p>For this example, I’m using a FASTQ file (bioinformatics folks will know what this is). You can grab a sample dataset from public genomics repositories if you want to follow along.</p><h3>IAM Permissions</h3><p>We need to create a role that allows our Nextflow pods to:</p><ul><li>Access S3 buckets</li><li>Create and manage other pods in the cluster</li></ul><pre>eksctl create podidentityassociation \<br>  --cluster nextflow-eks-cluster \<br>  --namespace nextflow \<br>  --serviceaccount nextflow-sa \<br>  --role-name nextflow-pod-role \<br>  --permission-policy-arn arn:aws:iam::aws:policy/AmazonS3FullAccess \<br>  --region us-east-1</pre><p>You can also do this through the AWS console.</p><h3>Phase 2: Kubernetes Configuration</h3><p>Now that our infrastructure is ready, let’s configure Kubernetes to handle our Nextflow workloads.</p><h3>Creating the Namespace and Service Account</h3><p>Namespaces keep things organized. Let’s create one specifically for Nextflow:</p><pre>kubectl create namespace nextflow<br>kubectl create serviceaccount nextflow-sa --namespace nextflow</pre><h3>RBAC: Giving Permissions</h3><p>RBAC (Role-Based Access Control) is how we tell Kubernetes what our Nextflow head pod is allowed to do. We need it to create worker pods, check their status, and clean them up when done.</p><p>Save this as rbac.yaml:</p><pre># Define what the role can do<br>apiVersion: rbac.authorization.k8s.io/v1<br>kind: Role<br>metadata:<br>  name: nextflow-role<br>  namespace: nextflow<br>rules:<br>  - apiGroups: [&quot;&quot;]<br>    resources: [&quot;pods&quot;, &quot;pods/log&quot;, &quot;pods/status&quot;]<br>    verbs: [&quot;get&quot;, &quot;list&quot;, &quot;watch&quot;, &quot;create&quot;, &quot;delete&quot;]<br>  - apiGroups: [&quot;batch&quot;]<br>    resources: [&quot;jobs&quot;]<br>    verbs: [&quot;get&quot;, &quot;list&quot;, &quot;watch&quot;, &quot;create&quot;, &quot;delete&quot;]<br><br>---<br># Bind the role to our service account<br>apiVersion: rbac.authorization.k8s.io/v1<br>kind: RoleBinding<br>metadata:<br>  name: nextflow-rolebinding<br>  namespace: nextflow<br>subjects:<br>  - kind: ServiceAccount<br>    name: nextflow-sa<br>    namespace: nextflow<br>roleRef:<br>  kind: Role<br>  name: nextflow-role<br>  apiGroup: rbac.authorization.k8s.io</pre><p>Apply it:</p><pre>kubectl apply -f rbac.yaml</pre><h3>SSH Keys for GitHub Access</h3><p>Since we’re pulling our Nextflow pipeline from GitHub, we need authentication. SSH keys are the way to go.</p><p><strong>Spoiler alert</strong>: If your GitHub repo is public, you can skip this entire SSH key setup and use HTTPS URLs to clone. But for private repos, SSH keys are essential.</p><p><strong>On your local machine</strong>, generate a new key pair:</p><pre>ssh-keygen -t ed25519 -C &quot;nextflow-eks-pipeline&quot;</pre><p>When prompted, save it to a specific location, like ~/.ssh/nextflow_eks_key (or just hit Enter to use the default location). Don&#39;t set a passphrase; Kubernetes won&#39;t be able to use a password-protected key.</p><p>Now store the private key as a Kubernetes secret:</p><pre>kubectl create secret generic git-ssh-key \<br>  --from-file=ssh-privatekey=/path/to/your/private/key \<br>  -n nextflow</pre><h3>Phase 3: GitHub Configuration</h3><p>Take the public key you just generated and add it to your GitHub repository:</p><ol><li>Go to your repo’s <strong>Settings</strong> → <strong>Deploy keys</strong></li><li>Click <strong>Add deploy key</strong></li><li>Paste your public key</li><li>Give it a descriptive name like “EKS Nextflow Runner.”</li><li>Check “Allow write access” if your workflow needs to push results back (optional)</li></ol><h3>Phase 4: The Implementation</h3><p>This is where it all comes together. We need two things: our Nextflow pipeline code and a Kubernetes Job to run it.</p><h3>The Nextflow Pipeline</h3><p>For this example, we’re running a bioinformatics quality control workflow on genomic sequencing data. Specifically:</p><p><strong>What we’re doing:</strong></p><ul><li><strong>FASTQC</strong>: Analyzes our raw sequencing data (FASTQ file) and generates quality control reports. It checks things like read quality scores, sequence duplication levels, GC content, and potential contamination. Each dataset gets its own HTML report with pretty graphs.</li><li><strong>MultiQC</strong>: Takes all those individual FASTQC reports and aggregates them into a single, beautiful summary report. Instead of opening 10 different HTML files, you get a comprehensive view of your data quality in one place.</li></ul><p>Think of it this way: FASTQC is your detailed inspector, checking each piece individually, and MultiQC is the manager who summarises everything into a single executive report.</p><p><strong>main.nf-</strong> This is your pipeline definition:</p><pre>#!/usr/bin/env nextflow<br>nextflow.enable.dsl=2<br><br>params.reads = &quot;s3://&lt;your-bucket-name&gt;/inputs/dataset.fastq&quot;<br><br>process FASTQC {<br>    container &#39;biocontainers/fastqc:v0.11.9_cv8&#39;<br>    publishDir &quot;${params.outdir}/fastqc&quot;, mode: &#39;copy&#39;<br>    <br>    input:<br>    path reads<br>    <br>    output:<br>    tuple path(&quot;*.html&quot;), path(&quot;*_fastqc.zip&quot;)<br>    <br>    script:<br>    &quot;&quot;&quot;<br>    echo &quot;Running FASTQC on ${reads}&quot;<br>    fastqc ${reads}<br>    echo &quot;FASTQC completed for ${reads}&quot;<br>    &quot;&quot;&quot;<br>}<br><br>process MULTIQC {<br>    container &#39;multiqc/multiqc:dev&#39;<br>    publishDir &quot;${params.outdir}/multiqc&quot;, mode: &#39;copy&#39;<br>    <br>    input:<br>    path fastqc_reports<br>    <br>    output:<br>    path &quot;multiqc_report.html&quot;<br>    path &quot;multiqc_data&quot;<br>    <br>    script:<br>    &quot;&quot;&quot;<br>    echo &quot;Aggregating FASTQC reports with MultiQC&quot;<br>    multiqc .<br>    echo &quot;MultiQC report generated successfully&quot;<br>    &quot;&quot;&quot;<br>}<br><br>workflow {<br>    reads_ch = Channel.fromPath(params.reads)<br>    fastqc_reports_ch = FASTQC(reads_ch)<br>    MULTIQC(fastqc_reports_ch.collect())<br>}</pre><p><strong>Important notes</strong>:</p><ul><li>Update &lt;your-bucket-name&gt;.git to match your S3 Bucket</li></ul><p><strong>nextflow.config- </strong>This tells Nextflow how to run on Kubernetes:</p><pre>plugins {<br>    id &#39;nf-wave&#39;<br>    id &#39;nf-amazon&#39;<br>}<br><br>wave.enabled = true<br>fusion.enabled = true<br>fusion.exportStorageCredentials = true <br><br>process {<br>    executor = &#39;k8s&#39;<br>    cpus = 1<br>    memory = &#39;2 GB&#39;<br>    maxForks = 1<br><br>    withName: &#39;FASTQC&#39; {<br>        container = &#39;biocontainers/fastqc:v0.11.9_cv8&#39;<br>    }<br>    withName: &#39;MULTIQC&#39; {<br>        container = &#39;multiqc/multiqc:dev&#39;<br>    }<br>}<br><br>k8s {<br>    namespace = &#39;nextflow&#39;<br>    serviceAccount = &#39;nextflow-sa&#39;<br>}</pre><p>Push this to your GitHub repo.</p><h3>The Kubernetes Job</h3><p>Now for the piece that ties everything together. But first, let’s talk about <em>why</em> we’re using a Kubernetes Job instead of a Deployment or a plain Pod.</p><p><strong>Why a Job?</strong></p><p>Kubernetes gives us several ways to run workloads, and choosing the right one matters:</p><ul><li><strong>Pod</strong>: Runs once, but if it fails, it’s done. No retry logic, no cleanup guarantees. Not ideal for batch workloads.</li><li><strong>Deployment</strong>: Designed for long-running services that should stay up. If a pod dies, it gets restarted automatically. Great for web servers, terrible for batch jobs that need to run once and finish.</li><li><strong>Job</strong>: Perfect for batch workloads! It runs to completion, has built-in retry logic (controlled by backoffLimit), and can automatically clean itself up when done (via ttlSecondsAfterFinished).</li></ul><p>For Nextflow pipelines, we want:</p><ul><li>Run once and finish</li><li>Automatic cleanup after completion</li><li>Clear success/failure status</li><li>No accidental restarts</li></ul><p>That’s exactly what a Job gives us.</p><p>Save this as nextflow-job.yaml:</p><pre>apiVersion: batch/v1<br>kind: Job<br>metadata:<br>  namespace: nextflow<br>  generateName: &quot;nextflow-job-&quot;<br>spec:<br>  backoffLimit: 0<br>  ttlSecondsAfterFinished: 300<br>  template:<br>    spec:<br>      serviceAccountName: nextflow-sa<br>      restartPolicy: Never<br>      <br>      initContainers:<br>        - name: fetch-pipeline<br>          image: alpine/git:latest<br>          env:<br>            - name: BRANCH<br>              value: &quot;main&quot;  # Change to your branch name<br>          command:<br>            - sh<br>            - -c<br>            - |<br>              set -e<br>              mkdir -p /root/.ssh<br>              cp /var/ssh/ssh-privatekey /root/.ssh/id_rsa<br>              chmod 600 /root/.ssh/id_rsa<br>              ssh-keyscan github.com &gt;&gt; /root/.ssh/known_hosts<br>              git clone -b &quot;${BRANCH}&quot; git@github.com:&lt;your-username&gt;/&lt;your-repo&gt;.git /workspace<br>          volumeMounts:<br>            - name: workspace<br>              mountPath: /workspace<br>            - name: git-ssh-key<br>              mountPath: /var/ssh<br>              readOnly: true<br>      <br>      containers:<br>        - name: nextflow<br>          image: nextflow/nextflow:25.12.0-edge<br>          env:<br>            - name: NXF_PLUGINS_DEFAULT<br>              value: &quot;nf-amazon,nf-wave,nf-k8s&quot;<br>            - name: JOB_NAME<br>              value: &quot;fastqc-analysis&quot;<br>          command:<br>            - sh<br>            - -c<br>            - |<br>              set -e<br>              cd /workspace<br>              WORK_DIR=&quot;s3://&lt;your-bucket-name&gt;/work/${JOB_NAME}&quot;<br>              RESULTS_DIR=&quot;s3://&lt;your-bucket-name&gt;/results/${JOB_NAME}&quot;<br>              echo &quot;Work directory: $WORK_DIR&quot;<br>              echo &quot;Results directory: $RESULTS_DIR&quot;<br>              exec nextflow run main.nf \<br>                -c nextflow.config \<br>                --outdir $RESULTS_DIR \<br>                -work-dir &quot;$WORK_DIR&quot; \<br>                -resume<br>          volumeMounts:<br>            - name: workspace<br>              mountPath: /workspace<br><br>      volumes:<br>        - name: workspace<br>          emptyDir: {}<br>        - name: git-ssh-key<br>          secret:<br>            secretName: git-ssh-key<br>            defaultMode: 0400</pre><p><strong>Important notes</strong>:</p><ul><li>Update &lt;your-username&gt;/&lt;your-repo&gt;.git to match your GitHub repo</li><li>Update &lt;your-bucket-name&gt; to match your S3 bucket</li><li>The generateName field means each run creates a unique job</li></ul><h3>Running the Pipeline</h3><p>Here’s the fun part. Launch your pipeline:</p><pre>kubectl create -f nextflow-job.yaml</pre><p>Note: We use kubectl create instead of kubectl apply because generateName creates a new job each time.</p><h3>Monitoring Your Workflow</h3><p>Watch the job spin up:</p><pre>kubectl get jobs -n nextflow</pre><p>Check the logs in real-time:</p><pre>kubectl logs job/&lt;job-name&gt; -n nextflow -f</pre><p>You’ll see Nextflow do its thing, spawning worker pods, running tasks, and saving results to S3.</p><h3>Checking Your Results</h3><p>Once everything completes, head to S3</p><p>You should see your FASTQC HTML reports and MultiQC summary. Beautiful!</p><h3>Cleanup</h3><p>The ttlSecondsAfterFinished: 300 Setting in our Job spec means Kubernetes automatically deletes completed jobs after 5 minutes. This keeps your cluster clean.</p><h3>Conclusion</h3><p>Running Nextflow on EKS gives you the best of both worlds: the power and flexibility of Nextflow with the scalability and reliability of Kubernetes. Sure, there’s a learning curve, but once you have this foundation in place, you can run virtually any computational workflow efficiently and cost-effectively.</p><p>If you found this helpful, consider giving it a clap or sharing it with your DevOps and bioinformatics friends. And if you run into issues or have questions, drop a comment below. I’m always happy to help troubleshoot.</p><p>Happy Hosting!</p><blockquote><em>Here to make the community stronger by sharing our knowledge. Follow </em><a href="https://medium.com/@tulshiram.kadam"><strong><em>me</em></strong></a><strong><em> </em></strong><em>and </em><a href="https://medium.com/nonstopio"><strong><em>my team</em></strong></a><em> to stay updated on the latest and greatest in the web &amp; mobile tech world.</em></blockquote><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=bff42df37883" width="1" height="1" alt=""><hr><p><a href="https://blog.nonstopio.com/how-we-built-scalable-bioinformatics-pipelines-on-aws-eks-using-nextflow-bff42df37883">How We Built Scalable Bioinformatics Pipelines on AWS EKS Using Nextflow</a> was originally published in <a href="https://blog.nonstopio.com/">nonstopio</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[The Ultimate Guide to Configuring a Free Site-to-Site VPN Server and Remote Access on AWS EC2 Using…]]></title>
            <link>https://blog.nonstopio.com/setting-up-site-to-site-vpn-and-remote-access-with-libreswan-and-openvpn-on-aws-ec2-5825be36137b?source=rss-d67062a61bce------2</link>
            <guid isPermaLink="false">https://medium.com/p/5825be36137b</guid>
            <category><![CDATA[libreswanvpn]]></category>
            <category><![CDATA[site-to-site-vpn]]></category>
            <category><![CDATA[vpn-configuration]]></category>
            <category><![CDATA[devops]]></category>
            <category><![CDATA[openvpn-setup]]></category>
            <dc:creator><![CDATA[Tulshiram Kadam]]></dc:creator>
            <pubDate>Thu, 07 Aug 2025 11:29:08 GMT</pubDate>
            <atom:updated>2025-08-13T12:32:18.362Z</atom:updated>
            <content:encoded><![CDATA[<h3>The Ultimate Guide to Configuring a Free Site-to-Site VPN Server and Remote Access on AWS EC2 Using Libreswan and OpenVPN</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*-PZihwacZeHyhL3UGsSHXg.jpeg" /></figure><p>With cloud infrastructure powering modern businesses, securing communication between services across VPCs or granting remote access to internal environments has become essential. Managed VPN services often come with high costs and limited flexibility. Fortunately, open-source solutions like <strong>Libreswan</strong> and <strong>OpenVPN</strong> offer production-grade security without locking you into expensive platforms.</p><p>This guide walks through setting up two critical VPN components using Ubuntu EC2 instances on AWS:</p><p>A. <strong>site-to-site IPsec VPN</strong> using <strong>Libreswan</strong>.</p><p>B. <strong>remote-access SSL VPN</strong> using <strong>OpenVPN</strong>.</p><p>By the end, you’ll have a functioning private tunnel between two AWS VPCs and a secure access point for clients to connect remotely.</p><h3>1. Infra Setup</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*XHlKwvTGfUftrypjegHk9g.jpeg" /></figure><h4>Instance Configuration</h4><ul><li><strong>Region:</strong> Select the AWS Region as per your preferred location</li><li><strong>OS:</strong> Select the Linux distribution as per your requirement</li><li><strong>Instance Type:</strong> Select the Instance type as per your requirement</li><li><strong>Network:</strong> create a new one with an auto-assigned public IP</li><li><strong>Security Group:</strong> Allowed Ports<strong> &gt;&gt;</strong> UDP <strong>500</strong> and <strong>4500</strong> for IPsec (LibreSwan), UDP <strong>1194</strong> for OpenVPN (Open to all)</li><li><strong>Storage:</strong> 8GB gp3 (Encrypted with default EBS key)</li></ul><h3>2. Installing and Configuring Libreswan (Site-to-Site VPN)</h3><p>Libreswan is an open-source IPsec-based VPN solution for secure tunnel communication.</p><h4>Step 1: Install Libreswan</h4><p>SSH into each EC2 instance and run</p><pre>sudo apt update<br>sudo apt install libreswan -y</pre><h4>Step 2: Enable IP Forwarding</h4><p>This is required for routing packets between VPCs:</p><pre>echo &quot;net.ipv4.ip_forward=1&quot; | sudo tee -a /etc/sysctl.conf<br>sudo sysctl -p</pre><h4>Step 3: Configure Libreswan</h4><p>Edit the following files on the EC2 instance.</p><p><strong>On the EC2 (Local VPC):<br></strong>File: /etc/ipsec.conf</p><pre>config setup<br>    protostack=netkey<br>    dumpdir=/run/pluto/<br>    nat_traversal=yes<br>    virtual_private=%v4:10.0.0.0/8,%v4:&lt;LOCAL_VPC_CIDR&gt;<br>    oe=off<br>    logfile=/var/log/pluto.log<br><br>conn to-remote  #Connection Name<br>authby=secret<br>    auto=start<br>    ikev2=insist<br>    type=tunnel<br><br>    left=&lt;LOCAL_PRIVATE_IP&gt;<br>    leftid=&lt;LOCAL_PUBLIC_IP&gt;<br>    leftsubnet=&lt;LOCAL_VPC_CIDR&gt;<br><br>    right=&lt;REMOTE_PUBLIC_IP&gt;<br>    rightid=&lt;REMOTE_PUBLIC_IP&gt;<br>    rightsubnet=&lt;REMOTE_VPC_CIDRS&gt; #You can add multiple CIDR<br><br>    ike=aes256-sha2_256;modp2048<br>    esp=aes256-sha2_256<br>    dpddelay=30<br>    dpdtimeout=120<br>    dpdaction=restart</pre><p>File: /etc/ipsec.secrets</p><pre>&lt;LOCAL_PUBLIC_IP&gt; &lt;REMOTE_PUBLIC_IP&gt; : PSK &quot;YourStrongSharedSecretHere&quot;</pre><h4>Step 5: NAT and Routing</h4><p>Enable NAT to allow private traffic routing and install iptables-persistent to make our iptables persistent:</p><pre>sudo iptables -t nat -A POSTROUTING -s &lt;YOUR_VPC_CIDR&gt; -o eth0 -j MASQUERADE<br>sudo apt install iptables-persistent<br>sudo netfilter-persistent save</pre><h4>Step 6: Configure Firewall for EC2 IP (Tunnel Access)</h4><p>We will be using the Palo Alto firewall</p><p>In your Palo Alto configuration, add EC2 IP to the Tunnel Configuration:</p><ul><li>Go to <strong>Network &gt; IPSec Tunnels</strong></li><li>Select or create your IPSec tunnel</li><li>Under <strong>IKE Gateway</strong>, ensure the <strong>Peer IP Address</strong> is set to the EC2 IP address</li></ul><p>This allows your EC2 to initiate and receive traffic over the tunnel.</p><h4>Step 7: Start the Connection</h4><pre>sudo systemctl enable ipsec<br>sudo systemctl restart ipsec<br>sudo ipsec auto --up to-remote   #Connection Name</pre><p>You can check the status with:</p><pre>sudo ipsec status</pre><h3>4. Installing and Configuring OpenVPN (Remote Access VPN)</h3><p>OpenVPN will enable external clients to connect to the internal AWS infrastructure securely.</p><h4>Step 1: Install OpenVPN and Easy-RSA</h4><p>To avoid the complexity of manual certificate generation, we’ll use a popular OpenVPN installation script by <a href="https://github.com/angristan/openvpn-install">Angristan</a></p><p>First, get the script and make it executable:</p><pre>curl -O https://raw.githubusercontent.com/angristan/openvpn-install/master/openvpn-install.sh<br>chmod +x openvpn-install.sh</pre><p>Then run it:</p><pre>./openvpn-install.sh</pre><p>You need to run the script as root and have the TUN module enabled.</p><p>The first time you run it, you’ll have to follow the assistant and answer a few questions to set up your VPN server.</p><p>When OpenVPN is installed, you can run the script again, and you will get the choice to:</p><ul><li><strong>Add a client</strong></li><li><strong>Remove a client</strong></li><li><strong>Uninstall OpenVPN</strong></li></ul><h4>Step 2: Configure the OpenVPN Server</h4><p>Create a config file:</p><pre>sudo nano /etc/openvpn/server/server.conf</pre><p>Paste the following and adjust:</p><pre>port 1194<br>proto udp<br>dev tun<br><br>ca ca.crt<br>cert myservername.crt<br>key myservername.key<br>dh dh.pem<br><br>server 10.8.0.0 255.255.255.0<br>push &quot;route &lt;YOUR_EC2_SUBNET_CIDR&gt;&quot;<br>push &quot;dhcp-option DNS 1.1.1.1&quot;<br>push &quot;dhcp-option DNS 8.8.8.8&quot;<br><br>keepalive 10 120<br>persist-key<br>persist-tun<br><br>cipher AES-256-CBC<br>auth SHA256<br>user nobody<br>group nogroup<br><br>status /var/log/openvpn-status.log<br>log-append /var/log/openvpn.log<br>verb 3<br>explicit-exit-notify 1</pre><p>Edit /etc/ufw/before.rules and add:</p><pre>*nat<br>:POSTROUTING ACCEPT [0:0]<br>-A POSTROUTING -s 10.8.0.0/24 -o eth0 -j MASQUERADE<br>COMMIT</pre><p>Set forwarding policy in /etc/default/ufw:</p><pre>DEFAULT_FORWARD_POLICY=&quot;ACCEPT</pre><p>Then reload:</p><pre>sudo ufw disable &amp;&amp; sudo ufw enable</pre><h4>Step 6: Start OpenVPN Server</h4><pre>sudo systemctl start openvpn@server<br>sudo systemctl enable openvpn@server</pre><p>Check status:</p><pre>sudo systemctl status openvpn@server</pre><h4>Step 7: Add a User</h4><p>To add more OpenVPN users:</p><pre>./openvpn-install.sh</pre><ul><li>Choose Add a new user</li><li>The script will generate a new .ovpn config file per user</li></ul><p>Your client .ovpn file will be saved in your home directory. Use this file to connect from any OpenVPN client (like the OpenVPN GUI or mobile app).</p><h4>Conclusion</h4><p>In this guide, you configured both <strong>Site-to-Site VPN with Libreswan</strong> and <strong>Remote Access VPN with OpenVPN</strong> on a single AWS EC2 instance. The Libreswan setup connects your EC2 to an on-premises Palo Alto firewall using IPsec, while OpenVPN (installed via a simple script) enables secure remote access for users.</p><p>This setup offers a cost-effective and efficient way to support both infrastructure-level and user-level VPN access within a single server — ideal for hybrid or small-scale cloud deployments.</p><p>Happy Hosting!</p><blockquote>Here to make the community stronger by sharing our knowledge. Follow <a href="https://medium.com/@tulshiram.kadam"><strong>me</strong></a><strong> </strong>and <a href="https://medium.com/nonstopio"><strong>my team</strong></a> to stay updated on the latest and greatest in the web &amp; mobile tech world.</blockquote><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=5825be36137b" width="1" height="1" alt=""><hr><p><a href="https://blog.nonstopio.com/setting-up-site-to-site-vpn-and-remote-access-with-libreswan-and-openvpn-on-aws-ec2-5825be36137b">The Ultimate Guide to Configuring a Free Site-to-Site VPN Server and Remote Access on AWS EC2 Using…</a> was originally published in <a href="https://blog.nonstopio.com/">nonstopio</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[How to Host a WordPress Website on AWS EC2 (Ubuntu 24.04) — Step-by-Step Guide]]></title>
            <link>https://blog.nonstopio.com/how-to-host-a-wordpress-website-on-aws-ec2-ubuntu-24-04-step-by-step-guide-381510b47952?source=rss-d67062a61bce------2</link>
            <guid isPermaLink="false">https://medium.com/p/381510b47952</guid>
            <dc:creator><![CDATA[Tulshiram Kadam]]></dc:creator>
            <pubDate>Tue, 25 Feb 2025 13:31:45 GMT</pubDate>
            <atom:updated>2025-02-26T06:09:49.219Z</atom:updated>
            <content:encoded><![CDATA[<h3>How to Host a WordPress Website on AWS EC2 (Ubuntu 24.04) — Step-by-Step Guide</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*-t5q1adW4Glu81sifPBY0Q.png" /></figure><p>WordPress is a powerful and versatile content management system (CMS) that enables users to build and manage websites efficiently. Hosting WordPress on a cloud instance such as AWS EC2 or a local VirtualBox VM ensures flexibility, scalability, and control over your website. This guide walks you through setting up WordPress on a fresh VM, configuring a LAMP stack (Linux, Apache, MySQL, PHP), and ensuring the configurations for a fully functional WordPress website.</p><h3>1. Create a Virtual Machine</h3><p>create a VM on AWS EC2, VirtualBox, or VMware as per your requirements.</p><p>Update the server packages to the latest version</p><pre>sudo apt update &amp;&amp; sudo apt upgrade -y</pre><p>Set hostname as per your naming conventions</p><pre>hostnamectl set-hostname ns-site1-vm</pre><h3>2. Install Apache Web Server</h3><p>Apache is the web server that will serve your WordPress site. We need to install and enable it to run automatically when the system starts.</p><p>Install the latest version of the Apache web server</p><pre>sudo apt install apache2</pre><p>Enable and start the Apache Server</p><pre>sudo systemctl enable apache2<br>sudo systemctl start apache2</pre><h3>3. Install PHP and Required Plugins</h3><p>PHP is the scripting language required for WordPress to function. We also install necessary extensions that WordPress relies on.</p><p>To install the latest version of PHP and its extensions on Ubuntu, follow these steps:</p><p>Add the Repository (For Latest PHP Versions)</p><pre>sudo apt install software-properties-common ca-certificates lsb-release apt-transport-https -y<br>sudo add-apt-repository ppa:ondrej/php -y<br>sudo apt update</pre><p>Install PHP Version 8.0 for better compatibility with WordPress.<br>You can refer to the <a href="https://make.wordpress.org/core/handbook/references/php-compatibility-and-wordpress-versions/">documentation</a></p><pre>sudo apt install php8.0 php8.0-cli php8.0-common php8.0-mysql php8.0-curl php8.0-gd php8.0-intl \<br>php8.0-mbstring php8.0-soap php8.0-xml php8.0-xmlrpc php8.0-zip libapache2-mod-php8.0 -y</pre><p>Verify Installation</p><pre>php -v</pre><p>This should display the installed PHP version.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*aepeQG-Tl6j5cX7AUqkN1A.png" /></figure><h3>4. Install and Configure MySQL</h3><p>MySQL is the database management system that stores all of WordPress&#39;s data, including posts, users, and settings.</p><p>Install MySQL server:</p><pre>sudo apt install mysql-server</pre><p>Enable and start the MYSQL server</p><pre>sudo systemctl enable mysql.service<br>sudo systemctl start mysql.service</pre><p>Do MySQL secure installation</p><pre>mysql_secure_installation</pre><p>Create a new user with restricted permissions for security</p><p>To access your MySQL prompt, use the below command</p><pre>mysql -u root -p</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*VWd5-s5tLShJybhtTm5-oA.png" /></figure><p>First, we need to create a database that will be restricted to the specific user only</p><pre>CREATE DATABASE site1_ns_db;</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*dHwsZctGf4GEf5J5LZOmtA.png" /></figure><p>Create a user</p><pre>CREATE USER &#39;site1_ns_user&#39;@&#39;localhost&#39; IDENTIFIED WITH mysql_native_password BY &#39;@MySecurePass123&#39;;</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*8qCQxygsHnuGSWZsmbxMog.png" /></figure><p>When creating a new user with CREATE USER, you provide a username followed by an @ symbol and the hostname from which the user will connect.<br>Change &lt;password&gt; with your desired password.</p><p>Grant Privileges to the User for the database we created</p><pre>GRANT ALL PRIVILEGES ON site1_ns_db.* TO &#39;site1_ns_user&#39;@&#39;localhost&#39;;</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*_QP2oUqw9BO2djYEVPk7Ig.png" /></figure><p>Apply Changes</p><pre>FLUSH PRIVILEGES;</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Kooa-_0CINj_UiemqIH9ig.png" /></figure><p>Verify the Privileges</p><pre>SHOW GRANTS FOR &#39;site1_ns_user&#39;@&#39;localhost&#39;;</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*L4wuZnHcuwVf2Tm4aqU4xg.png" /></figure><h3>5. Install and Configure WordPress</h3><p>We are deploying WordPress, a content management system (CMS). This step involves downloading and configuring WordPress to work with our web server and database.</p><p>You can create a separate directory for WordPress if needed. Recommended installation path: /var/www/html/yoursite</p><pre>mkdir -p /var/www/html/ns.site1.com</pre><p>Set the correct ownership and permissions</p><pre>sudo chown -R www-data:www-data /var/www/html/ns.site1.com/public_html<br>sudo chmod -R 755 /var/www/html/ns.site1.com/public_html</pre><p>Download WordPress</p><pre>cd /var/www/html/ns.site1.com<br>wget https://wordpress.org/latest.tar.gz</pre><p>Unzip the package</p><pre>tar -xvf latest.tar.gz</pre><p>Change the name of the WordPress directory to public_html</p><pre>mv wordpress public_html</pre><p>Update the WordPress configuration file</p><pre>cd /var/www/html/ns.site1.com/public_html<br>nano wp-config.php</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*EW1Bz1ulC5t_lZZsKeNgog.png" /></figure><p>Modify the database settings in wp-config.php to match your MySQL setup.</p><h3>6. Configure Apache Virtual Host</h3><p>A virtual host is required to tell Apache how to serve your WordPress site. This configuration ensures that requests to your domain or IP address are properly handled.</p><p>Navigate to the Apache configuration directory:</p><pre>cd /etc/apache2/sites-enabled/</pre><p>First, delete the default configuration files that are already there:</p><pre>sudo rm /etc/apache2/sites-enabled/*</pre><p>To store the log files for Apache, we’ll be creating a separate directory and files</p><pre>mkdir -p /var/log/apache2/site1.ns.com/<br>sudo touch /var/log/apache2/site1.ns.com/error.log<br>sudo touch /var/log/apache2/site1.ns.com/access.log</pre><p>Create a new configuration file</p><pre>sudo vi ns-site1.com.conf</pre><p>Refer to the following configuration</p><pre>&lt;VirtualHost *:80&gt;<br>   ServerAdmin webmaster@localhost<br>   DocumentRoot /var/log/apache2/site1.ns.com/public_html<br>   ErrorLog /var/log/apache2/site1.ns.com/error.log<br>   CustomLog /var/log/apache2/site1.ns.com/access.log combined<br>   &lt;Directory &lt;/var/log/apache2/site1.ns.com/public_html&gt;<br>       Options Indexes FollowSymLinks<br>       AllowOverride All<br>       Require all granted<br>   &lt;/Directory&gt;<br>&lt;/VirtualHost&gt;</pre><p>To enable the config, use the soft link</p><pre>ln -s /etc/apache2/sites-available/ns-site1.com.conf /etc/apache2/sites-enabled/ns-site1.com.conf<br>sudo a2ensite example</pre><p>Check that the configuration is correct</p><pre>apache2ctl -t</pre><p>Enable necessary Apache modules</p><pre>sudo a2enmod rewrite<br>sudo a2enmod proxy<br>sudo a2enmod proxy_http</pre><p>Set up Apache server timezone to Asia/Kolkata</p><pre>sudo timedatectl set-timezone Asia/Kolkata<br>timedatectl</pre><p>Configure Apache to Use the System Time zone</p><pre>SetEnv TZ Asia/Kolkata</pre><p>Restart Apache</p><pre>sudo systemctl restart apache2</pre><h3>8. Additional Configuration</h3><p>Access the WordPress setup page from a web browser</p><p>Additionally, you can set up a local domain to access<br>The host file is where you can map domain names to IP addresses. This file exists on both Linux and Windows systems.</p><p>Edit the host file</p><pre>sudo nano /etc/hosts</pre><p>Add a line that maps your local domain to your server’s IP</p><pre>&lt;VM ip&gt;   ns-site1.com</pre><p>Modify the Apache configuration file</p><pre>&lt;VirtualHost *:80&gt;<br>   ServerAdmin webmaster@localhost<br>   DocumentRoot /var/log/apache2/ns-site1.com/public_html<br>    ServerName ns-site1.com #Add ServerName<br>   ErrorLog /var/log/apache2/ns-site1.com/error.log<br>   CustomLog /var/log/apache2/ns-site1.com/access.log combined<br>   &lt;Directory &lt;Wordpress installation path&gt;&gt;<br>       Options Indexes FollowSymLinks<br>       AllowOverride All<br>       Require all granted<br>   &lt;/Directory&gt;<br>&lt;/VirtualHost&gt;</pre><p>Enable the Site and Restart Apache</p><pre>sudo a2ensite ns-site1.com.conf<br>sudo systemctl reload apache2</pre><p>Try to access it by using the DNS you mapped to your server’s IP</p><h3>Step 5: Install SSL using Certbot</h3><p>If you are using any DNS, you can secure your site by installing SSL using certbot</p><ol><li>Install Certbot</li></ol><pre>sudo apt install certbot python3-certbot-apache -y</pre><p>2. Generate an SSL certificate</p><pre>sudo certbot --apache -d ns-site1.com</pre><h3>Conclusion</h3><p>By following this guide, you have successfully hosted a WordPress website on an EC2 instance or a VirtualBox VM. Your LAMP stack is fully configured, and your WordPress installation is ready for use. You can now access the WordPress setup page from a web browser and proceed with site configuration.</p><p>This setup provides you with complete control over your WordPress hosting, allowing for scalability, customization, and enhanced security.</p><p>Happy Hosting!</p><blockquote><em>Here to make the community stronger by sharing our knowledge. </em><a href="https://medium.com/@tulshiram.kadam"><em>Follow</em></a><em> me and </em><a href="https://medium.com/nonstopio"><em>my team</em></a><em> to stay updated on the latest and greatest in the web &amp; mobile tech world.</em></blockquote><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=381510b47952" width="1" height="1" alt=""><hr><p><a href="https://blog.nonstopio.com/how-to-host-a-wordpress-website-on-aws-ec2-ubuntu-24-04-step-by-step-guide-381510b47952">How to Host a WordPress Website on AWS EC2 (Ubuntu 24.04) — Step-by-Step Guide</a> was originally published in <a href="https://blog.nonstopio.com/">nonstopio</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Hosting a Streamlit Python App on AWS EC2: A Step-by-Step Guide]]></title>
            <link>https://blog.nonstopio.com/hosting-a-streamlit-python-app-on-aws-ec2-a-step-by-step-guide-0148062b992d?source=rss-d67062a61bce------2</link>
            <guid isPermaLink="false">https://medium.com/p/0148062b992d</guid>
            <dc:creator><![CDATA[Tulshiram Kadam]]></dc:creator>
            <pubDate>Mon, 17 Feb 2025 19:01:44 GMT</pubDate>
            <atom:updated>2025-02-25T08:41:23.097Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*pzGJXogrxW5mI2msAFXvIA.png" /></figure><p>Deploying a <strong>Streamlit application</strong> in a production environment requires a secure and scalable infrastructure. In this guide, we will walk through <strong>setting up an EC2 instance</strong> on AWS, configuring it for a Python-based Streamlit app, and securing it with <strong>Nginx and SSL certificates</strong>.</p><p>We will also automate the application as a system service and ensure a smooth deployment by setting up a <strong>reverse proxy</strong> with Nginx. By the end of this guide, you will have a <strong>fully functional</strong> and <strong>secure</strong> deployment of your Streamlit application on AWS.</p><h3>Step 1: Infra Setup</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*kOi5DrOAL-A4g-ksT7RoTQ.jpeg" /><figcaption><strong>Infrastructure for the application</strong></figcaption></figure><p>For this demo, we are focusing on the core deployment process, skipping CI/CD, and monitoring setup. While these are essential for production environments, our goal here is to keep things simple and demonstrate the deployment workflow efficiently.</p><h4>Instance Configuration</h4><ul><li><strong>Region:</strong> Select the AWS Region as per your preferred location</li><li><strong>OS:</strong> Select the Linux distribution as per your requirement</li><li><strong>Instance Type:</strong> Select the Instance type as per your requirement</li><li><strong>Network:</strong> Default VPC or create a new one with auto-assigned public IP</li><li><strong>Security Group:</strong> Allowed Ports<strong> &gt;&gt;</strong> 22, 80, 443 (Open to all)</li><li><strong>Storage:</strong> 8GB gp3 (Encrypted with default EBS key)</li></ul><p>We are not configuring AWS Cloudwatch and CICD for this.</p><h3><strong>Step 2: Configure the server</strong></h3><p>Configuring a server for Python applications involves setting up a dedicated environment that ensures stability and security. This includes updating system packages, installing Python, and creating an isolated virtual environment to manage dependencies efficiently. Additionally, a new user is created to enhance security, with a designated directory assigned as their home. This structured setup ensures a clean workspace, minimizing conflicts and providing better control over application deployment.</p><ol><li>Update the list of available packages and their versions</li></ol><pre>sudo apt update <br>sudo apt upgrade -y</pre><p>2. Create a separate directory for application /webapps/demo</p><pre>Sudo -p mkdir /webapps/demo</pre><p>3. Create a separate user and set the home directory</p><pre>Sudo adduser &lt;usename&gt;<br>Sudo chown &lt;usename&gt;:&lt;usename&gt; /webapps/demo<br>Sudo usermod -d /webapps/demo -m &lt;usename&gt;</pre><p>4. Confirm Python is installed. If not install Python</p><pre>sudo apt install python3.11 python3.11-venv -y</pre><p>5. Clone the Application Repository</p><pre>git clone &lt;repo_link&gt;</pre><p>6. Create a Python Virtual Environment</p><pre>python3.11 -m venv venv<br>source venv/bin/activate</pre><h3>Step 3: Set Up Nginx as a Reverse Proxy</h3><p>Nginx reverse proxy functions as a middleware between clients and backend servers, intercepting incoming requests and directing them to the appropriate destination. It also handles returning responses from the server to the client. Acting as an abstraction layer, a reverse proxy enhances performance, security, and scalability for backend services.</p><ol><li>Install Nginx</li></ol><pre>sudo apt install nginx</pre><p>2. Configure Nginx as a proxy to the application port</p><pre>sudo nano /etc/nginx/conf.d/python_app</pre><ul><li>Sample configuration file for the nginx proxy</li></ul><pre>server {<br>    server_name example.com;<br>    location / {<br>        proxy_pass http://&lt;private_ip&gt;:8501;<br>        pproxy_set_header Host $host;<br>        proxy_set_header X-Real-IP $remote_addr;  <br>    }</pre><p>3. Restart Nginx</p><pre>sudo systemctl restart nginx</pre><h3>Step 4: Set Up the Application as a Service</h3><p>Setting up an application as a service ensures that it runs seamlessly in the background and automatically restarts after a server reboot. This involves defining the application as a system service, allowing it to start, stop, and restart as needed. By configuring the service to launch on boot, the application remains consistently available without manual intervention. This approach enhances reliability, ensures continuous operation, and provides better process management for deployed applications.</p><ol><li>Create a service file for the application</li></ol><pre>sudo nano /etc/systemd/system/python_app.service</pre><p>2. Sample configuration for the service file</p><pre>[Unit]<br>Description=Demo python app<br>After=network.target<br><br>[Service]<br>User=&lt;Usename&gt;<br>Group=&lt;Usename&gt;<br>WorkingDirectory=/webapps/demo/my_app/<br>ExecStart=/webapps/demo/venv/bin/streamlit run app.py<br>Restart=always<br><br>[Install]<br>WantedBy=multi-user.target</pre><p>3. Enable and Start the Service</p><pre>sudo systemctl daemon-reload<br>sudo systemctl enable python_app<br>sudo systemctl start Python_app<br>sudo systemctl status python_app</pre><h3>Step 5: Install SSL using Certbot</h3><p>Installing an SSL certificate using Certbot provides a free and automated solution for securing server communications. This process involves obtaining and configuring an SSL certificate, integrating it with the web server, and setting up automatic renewal. By leveraging Let’s Encrypt, Certbot ensures encrypted data transmission at no cost, enhancing security and trust for users while eliminating the need for manual certificate management.</p><ol><li>Install Certbot</li></ol><pre>sudo apt install certbot python3-certbot-nginx -y</pre><p>2. Generate an SSL certificate</p><pre>sudo certbot --nginx -d example.com</pre><p>Following these steps, you successfully deployed a <strong>Streamlit application</strong> on an AWS EC2 instance with <strong>Nginx acting as a reverse proxy</strong>. The application now runs as a <strong>systemd service</strong>, ensuring it automatically restarts when needed. Additionally, an SSL certificate has been configured using <strong>Certbot</strong>, making your application secure and accessible over HTTPS.</p><p>Happy Hosting!</p><blockquote>Here to make the community stronger by sharing our knowledge. <a href="https://medium.com/@tulshiram.kadam">Follow</a> me and <a href="https://medium.com/nonstopio">my team</a> to stay updated on the latest and greatest in the web &amp; mobile tech world.</blockquote><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=0148062b992d" width="1" height="1" alt=""><hr><p><a href="https://blog.nonstopio.com/hosting-a-streamlit-python-app-on-aws-ec2-a-step-by-step-guide-0148062b992d">Hosting a Streamlit Python App on AWS EC2: A Step-by-Step Guide</a> was originally published in <a href="https://blog.nonstopio.com/">nonstopio</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
    </channel>
</rss>