<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Danish Siddiqui on Medium]]></title>
        <description><![CDATA[Stories by Danish Siddiqui on Medium]]></description>
        <link>https://medium.com/@danishsidd203?source=rss-7d9b62c475b8------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/0*5OSzUY2WaQcFzpGQ</url>
            <title>Stories by Danish Siddiqui on Medium</title>
            <link>https://medium.com/@danishsidd203?source=rss-7d9b62c475b8------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Mon, 16 Nov 2026 20:58:31 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@danishsidd203/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[Understanding Ansible: Infrastructure Automation Made Simple]]></title>
            <link>https://medium.com/@danishsidd203/understanding-ansible-infrastructure-automation-made-simple-b94c5991d2b9?source=rss-7d9b62c475b8------2</link>
            <guid isPermaLink="false">https://medium.com/p/b94c5991d2b9</guid>
            <category><![CDATA[devops]]></category>
            <category><![CDATA[infrastructure-as-code]]></category>
            <category><![CDATA[automation]]></category>
            <category><![CDATA[ansible-playbook]]></category>
            <category><![CDATA[ansible]]></category>
            <dc:creator><![CDATA[Danish Siddiqui]]></dc:creator>
            <pubDate>Mon, 18 Aug 2025 17:18:04 GMT</pubDate>
            <atom:updated>2025-08-18T17:18:04.349Z</atom:updated>
            <content:encoded><![CDATA[<p>Ansible is an open-source automation tool that manages IT infrastructure, application deployment, and configuration management using simple YAML playbooks. It operates agentlessly over SSH, allowing you to automate repetitive tasks across multiple servers without installing additional software on target machines.</p><figure><img alt="Ansible" src="https://cdn-images-1.medium.com/max/1024/1*ytuQ2YtKmZ2q3QKh2YX1YQ.jpeg" /></figure><h3>Installing Ansible on local machine</h3><pre>sudo apt update<br>sudo apt install software-properties-common<br>sudo add-apt-repository --yes --update ppa:ansible/ansible<br>sudo apt install ansible</pre><p>ansible uses a declarative approach to configure systems by writing explicit yaml configuration files.</p><h3>Ansible Architecture</h3><p>Ansible follows a simple push-based architecture where a control node manages multiple target hosts without requiring agents on remote systems.</p><figure><img alt="Ansible Architecture" src="https://cdn-images-1.medium.com/max/819/1*vGacH4MY21uyL2FTx2bAdQ.png" /></figure><h4>Core Components</h4><p><strong>Control Node</strong>: The machine where Ansible is installed and playbooks are executed from. It connects to managed nodes via SSH and pushes configurations.</p><p><strong>Managed Nodes</strong>: Target servers that Ansible configures. No Ansible software needs to be installed on these hosts — they just need SSH access and Python.</p><p><strong>Inventory</strong>: File or directory containing information about managed hosts, including IP addresses, connection details, and host groupings.</p><p><strong>Modules</strong>: Python scripts that perform specific tasks (install packages, manage services, copy files, etc.). Modules are pushed to managed nodes, executed, then removed.</p><p><strong>Playbooks</strong>: YAML files containing ordered lists of tasks to execute on managed hosts. They define the desired state of your infrastructure.</p><h4>How It Works</h4><ul><li>Control node reads inventory and playbook</li><li>Establishes SSH connections to managed nodes</li><li>Transfers and executes modules on target hosts</li><li>Collects results and provides feedback</li><li>No persistent agents or daemons required on managed nodes</li><li>Uses existing SSH infrastructure for secure communication</li></ul><p>This agentless design makes Ansible lightweight and easy to deploy compared to other automation tools.</p><h4>File Structure</h4><ul><li><strong>Inventory file</strong>: Contains machine IP addresses and SSH information for remote access</li><li><strong>Playbook (main.yaml)</strong>: Contains declarative steps for Ansible to execute</li><li><strong>ansible.cfg</strong>: Configuration file for Ansible settings (optional)</li></ul><h3>Some Essential Ansible commands</h3><pre># Run playbook<br>ansible-playbook playbook.yml<br><br># Run with specific inventory<br>ansible-playbook -i inventory.yml playbook.yml<br><br># Testing connectivity<br>ansible all -m ping                              # Ping all hosts<br>ansible webservers -m ping                       # Ping specific group<br><br># Inventory management<br>ansible-inventory --list                         # List all hosts<br>ansible-inventory --list --limit webservers      # List hosts in specific group<br><br># Dry run (check mode)<br>ansible-playbook playbook.yml --check<br><br># Verbose output<br>ansible-playbook playbook.yml -v                 # -v, -vv, -vvv for more detail</pre><h3>Ansible Handlers</h3><p>Ansible handlers are special tasks that run only when triggered by other tasks, typically used for actions like restarting services after configuration changes. They ensure operations happen only when necessary, not on every playbook run.</p><h4>Handler Execution Flow:</h4><ol><li>Tasks run in order</li><li>If a task changes something, it “notifies” the handler</li><li>All tasks complete first</li><li>Handlers run at the end, only once each</li><li>Handlers run in the order they’re defined</li></ol><p>This makes Ansible playbooks more efficient and prevents unnecessary service disruptions!</p><h3>Variables</h3><p>Variables in Ansible are placeholders that store values like server names, configurations, or passwords. They make playbooks reusable and flexible by allowing you to change values without modifying the playbook code. Variables enable dynamic configuration management across different environments and hosts.</p><h4>Variable Precedence in Ansible (Low to High Priority)</h4><pre># Role defaults<br># roles/webserver/defaults/main.yml<br>apache_port: 80<br><br># Inventory group vars<br># group_vars/webservers.yml<br>apache_port: 8080<br><br># Inventory host vars<br># host_vars/web1.yml<br>apache_port: 8081<br><br># Playbook vars<br># playbook.yml<br>vars:<br>  apache_port: 8082<br><br># Role vars<br># roles/webserver/vars/main.yml<br>apache_port: 8083<br><br># Block vars<br># Within playbook block<br>- block:<br>    vars:<br>      apache_port: 8084<br><br># Task vars<br># Within task<br>- name: Configure Apache<br>  vars:<br>    apache_port: 8085<br><br># set_fact/register<br>- set_fact:<br>    apache_port: 8086<br><br># Extra vars (highest priority)<br>ansible-playbook playbook.yml -e &quot;apache_port=8087&quot;</pre><h4>Command Line Variable Assignment</h4><pre># Single variable<br>ansible-playbook playbook.yml -e &quot;env=production&quot;<br><br># Multiple variables<br>ansible-playbook playbook.yml -e &quot;env=prod debug=true&quot;<br><br># JSON format<br>ansible-playbook playbook.yml -e &#39;{&quot;env&quot;:&quot;prod&quot;,&quot;port&quot;:8080}&#39;<br><br># From file<br>ansible-playbook playbook.yml -e &quot;@vars.yml&quot;<br><br># YAML format<br>ansible-playbook playbook.yml -e &quot;env: production&quot;</pre><h4>Environment Variables:</h4><p>Shell variables that are available to commands, scripts, and processes running on the target system.</p><h3>Difference between Variables and Environment Variables</h3><ul><li><strong>Ansible Variables</strong> are internal to Ansible — used for templating, conditionals, and passing data between tasks within playbooks.</li><li><strong>Environment Variables</strong> are external — they’re actual shell environment variables that your commands and scripts can access using $VARIABLE_NAME.</li></ul><p>Use Ansible variables for Ansible logic and configuration, use environment variables when you need to pass values to shell commands, scripts, or applications running on the target system.</p><pre>---<br>- name: Quick Comparison<br>  hosts: all<br>  vars:<br>    app_name: &quot;myapp&quot;           # Ansible variable<br>    config_path: &quot;/etc/myapp&quot;   # Ansible variable<br>    <br>  environment:<br>    APP_NAME: &quot;{{ app_name }}&quot;  # Environment variable (uses Ansible var)<br>    CONFIG_PATH: &quot;/etc/myapp&quot;   # Environment variable<br>    <br>  tasks:<br>    # Ansible variable usage<br>    - name: Create directory using Ansible variable<br>      file:<br>        path: &quot;{{ config_path }}&quot;  # ✅ Ansible variable in module<br>        state: directory<br>        <br>    # Environment variable usage  <br>    - name: Run script that needs environment variables<br>      shell: |<br>        echo &quot;App: $APP_NAME&quot;           # ✅ Environment variable in shell<br>        mkdir -p $CONFIG_PATH/logs      # ✅ Environment variable in shell<br>      register: output</pre><h3>Conditionals</h3><p><strong>Conditionals</strong> in Ansible allow you to control task execution based on specific conditions — like running tasks only on certain OS types, when variables have specific values, or when previous tasks succeed/fail.</p><h4>Key Components</h4><ul><li><strong>when:</strong> - Primary conditional statement</li><li><strong>ansible_facts</strong> - System information (OS, hardware, network, etc.)</li><li><strong>failed, </strong><strong>succeeded, </strong><strong>changed</strong> - Task result states</li><li><strong>is defined, </strong><strong>is not defined</strong> - Variable existence checks</li><li><strong>Logical operators</strong>: and, or, not</li></ul><h3>Templating in Ansible</h3><p><strong>Templating</strong> in Ansible allows you to create dynamic configuration files by inserting variables, conditionals, and loops into template files. It enables you to generate customized files for different hosts or environments from a single template source.</p><h4>Jinja2 Templating</h4><p><strong>Jinja2</strong> is the templating engine used by Ansible — it’s a Python-based template language that uses {{ }} for variables, {% %} for logic (loops, conditionals), and {# #} for comments. It&#39;s used to make templates dynamic and reusable instead of having static configuration files.</p><h3>Ansible Modules</h3><p>Modules are reusable units of code that Ansible uses to perform specific tasks. They’re the building blocks of Ansible playbooks.</p><h3>Ansible Roles</h3><p>Roles are a way to organize playbooks and reuse code. They provide a structured way to group tasks, variables, files, and templates.</p><pre># Role Structure<br>roles/<br>└── webserver/<br>    ├── tasks/<br>    │   └── main.yml<br>    ├── handlers/<br>    │   └── main.yml<br>    ├── templates/<br>    ├── files/<br>    ├── vars/<br>    │   └── main.yml<br>    ├── defaults/<br>    │   └── main.yml<br>    └── meta/<br>        └── main.yml<br>        <br>        <br> # Using Roles<br>- name: Configure web servers<br>  hosts: webservers<br>  roles:<br>    - webserver<br>    - database<br>    - { role: nginx, nginx_port: 8080 }</pre><h3>Deployment Strategies</h3><p>Method of rolling out applications/configurations across multiple servers — can be all-at-once, rolling updates, or canary deployments to minimize downtime and risk.</p><h4>Blue-Green Strategy</h4><p>Maintain two identical production environments (blue=current, green=new) — deploy to green, test it, then switch traffic from blue to green for zero-downtime deployments.</p><h4>Common Ansible Deployment Strategies</h4><ul><li><strong>Rolling Deployment</strong>: Update servers one by one or in small batches to maintain service availability during updates.</li><li><strong>All-at-Once</strong>: Deploy to all servers simultaneously — fastest but causes downtime if issues occur.</li><li><strong>Canary Deployment</strong>: Deploy to a small subset of servers first, monitor for issues, then gradually roll out to remaining servers.</li></ul><p>The --limit flag restricts playbook execution to specific hosts or groups from your inventory.</p><pre>ansible-playbook -i inventory.yaml playbook.yaml --limit blue<br># Runs only on blue group<br><br>ansible-playbook -i inventory.yaml playbook.yaml --limit green  <br># Runs only on green group</pre><h3>Error Handling in Ansible</h3><p><strong>Error Handling</strong> in Ansible allows you to control what happens when tasks fail — you can ignore errors, retry failed tasks, or define custom failure conditions instead of stopping the entire playbook.</p><h4>Key Error Handling Methods</h4><p><strong>ignore_errors</strong>: Continue playbook execution even if task fails <strong>failed_when</strong>: Define custom failure conditions</p><p><strong>rescue/always</strong>: Handle errors with try-catch-like blocks <strong>retry/until</strong>: Retry tasks until they succeed or max attempts reached</p><h3>Ansible Vault</h3><p><strong>Ansible Vault</strong> is a built-in encryption feature that allows you to encrypt sensitive data like passwords, API keys, and certificates in your playbooks and variable files, keeping them secure while stored in version control.</p><pre># File operations<br>ansible-vault create secrets.yaml                    # Create new encrypted file<br>ansible-vault encrypt vars.yaml                      # Encrypt existing file<br>ansible-vault decrypt secrets.yaml                   # Decrypt file (makes readable)<br>ansible-vault view secrets.yaml                      # View encrypted file without decrypting<br>ansible-vault edit secrets.yaml                      # Edit encrypted file<br>ansible-vault rekey secrets.yaml                     # Change vault password<br><br># String Operations<br>ansible-vault encrypt_string &#39;mysecret&#39;             # Encrypt single string<br>ansible-vault encrypt_string &#39;mysecret&#39; --name &#39;db_pass&#39;  # Encrypt string with variable name<br><br># Running Playbooks with Vault<br>ansible-playbook playbook.yml -e vault.yaml --ask-vault-pass      # Prompt for vault password<br>ansible-playbook playbook.yml --ask-vault-pass      # Prompt for vault password<br>ansible-playbook playbook.yml --vault-password-file ~/.vault_pass  # Use password file<br>ansible-playbook playbook.yml --vault-id dev@~/.vault_dev  # Use labeled vault<br><br># Password File Management<br>echo &#39;mypassword&#39; &gt; ~/.vault_pass                   # Create password file<br>chmod 600 ~/.vault_pass                             # Secure password file permissions<br><br># Multiple Vault IDs<br>ansible-vault create secrets.yml --vault-id prod@prompt     # Create with specific vault ID<br>ansible-vault view secrets.yml --vault-id prod@~/.vault_prod  # View with specific vault ID<br>ansible-playbook site.yml --vault-id dev@~/.vault_dev --vault-id prod@~/.vault_prod  # Multiple vaults</pre><h3>Best Practices</h3><h4>Playbook Organization</h4><ul><li>Use meaningful names for tasks and playbooks</li><li>Group related tasks in roles</li><li>Keep playbooks idempotent (safe to run multiple times)</li><li>Use tags for selective execution</li></ul><h4>Security</h4><ul><li>Always use Ansible Vault for sensitive data</li><li>Use SSH keys instead of passwords</li><li>Limit inventory file permissions</li><li>Don’t commit unencrypted secrets to version control</li></ul><h4>Performance</h4><ul><li>Use gather_facts: no when facts aren&#39;t needed</li><li>Limit host connections with serial or throttle</li><li>Use async for long-running tasks</li><li>Cache fact gathering when possible</li></ul><h4>Variables and Inventory</h4><ul><li>Use group_vars and host_vars for organization</li><li>Follow consistent naming conventions</li><li>Document variable purposes</li><li>Use defaults in roles</li></ul><h3>Conclusion</h3><p>Ansible is a powerful automation tool that simplifies infrastructure management through its agentless architecture and declarative YAML syntax. The key to mastering Ansible is understanding its core concepts: playbooks for orchestration, variables for flexibility, templates for dynamic configuration, and roles for code organization.</p><p>Start with simple tasks and gradually build more complex automation workflows. Remember that Ansible’s strength lies in its idempotent nature — your playbooks should be safe to run multiple times. Use handlers for efficient service management, leverage conditionals and loops for dynamic execution, and always secure sensitive data with Ansible Vault.</p><p>Practice with different modules, experiment with deployment strategies, and don’t forget to implement proper error handling. With these fundamentals in place, you’ll be able to automate complex infrastructure tasks and maintain consistent, reliable deployments across your environment.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=b94c5991d2b9" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Terraform Basics: A Beginner’s Guide to Infrastructure as Code]]></title>
            <link>https://medium.com/@danishsidd203/terraform-basics-a-beginners-guide-to-infrastructure-as-code-4ea613d10578?source=rss-7d9b62c475b8------2</link>
            <guid isPermaLink="false">https://medium.com/p/4ea613d10578</guid>
            <category><![CDATA[aws]]></category>
            <category><![CDATA[infrastructure-as-code]]></category>
            <category><![CDATA[devops]]></category>
            <category><![CDATA[iaac]]></category>
            <category><![CDATA[terraform]]></category>
            <dc:creator><![CDATA[Danish Siddiqui]]></dc:creator>
            <pubDate>Sat, 28 Jun 2025 20:44:20 GMT</pubDate>
            <atom:updated>2025-06-28T20:45:21.203Z</atom:updated>
            <content:encoded><![CDATA[<h3><strong>Terraform</strong></h3><p>Terraform is an open-source Infrastructure as Code (IaC) tool developed by HashiCorp that allows you to build, change, and version infrastructure safely and efficiently. Unlike traditional methods of manually configuring servers and services, Terraform uses declarative configuration files to describe your desired infrastructure state.</p><figure><img alt="Terraform" src="https://cdn-images-1.medium.com/max/1024/1*VIh2kTIcZnJLbBhOclQCkw.png" /></figure><h3><strong>Key Characteristics of Terraform:</strong></h3><p><strong>Cloud Agnostic</strong>: Terraform supports multiple cloud providers including AWS, Azure, Google Cloud, and many others. It can manage any service that has an API, making it incredibly versatile.</p><p><strong>Declarative Approach</strong>: You describe what you want your infrastructure to look like, and Terraform figures out how to make it happen.</p><p><strong>State Management</strong>: Terraform maintains a state file that tracks the current state of your infrastructure, enabling it to make intelligent decisions about what changes need to be applied.</p><p><strong>Resource Graph</strong>: Terraform builds a dependency graph of your resources and parallelizes operations whenever possible for efficient execution.</p><figure><img alt="terraform basics" src="https://cdn-images-1.medium.com/max/1024/1*jF9sz03eWhT4OMTqv6PQZg.png" /></figure><h3><strong>Terraform vs Configuration Management Tools</strong></h3><p>t’s important to understand that Terraform serves a different purpose than configuration management tools:</p><ul><li><strong>Terraform</strong>: Focuses on <strong>provisioning</strong> infrastructure (creating servers, networks, databases, etc.)</li><li><strong>Ansible/Chef/Puppet</strong>: Focus on <strong>configuration management</strong> (installing software, configuring applications, managing services)</li></ul><p>These tools complement each other perfectly — use Terraform to provision your infrastructure and configuration management tools to configure the software running on that infrastructure.</p><h3>Setup and Installation:</h3><ol><li>Download terraform using choco install terraform</li><li>Create a user group on AWS console for terraform, then create a user and add it into the new group with the required permissions</li><li>Install AWS CLI</li><li>Configure the AWS using aws configure</li><li>Verify installation: terraform version</li></ol><h4>Some Essential Terraform Commands:</h4><pre># Initialize a Terraform working directory<br>terraform init<br><br># Create an execution plan<br>terraform plan<br><br># Apply the changes required to reach the desired state<br>terraform apply<br><br># Destroy the infrastructure<br>terraform destroy<br><br># Login to Terraform Cloud<br>terraform login<br><br># State management commands<br>terraform state list                    # List all resources in state<br>terraform state show [resource_name]   # Show detailed resource info<br>terraform show                         # Display all resource details<br><br># Formatting and validation<br>terraform fmt                          # Format configuration files<br>terraform validate                     # Validate configuration syntax<br><br># Plan Review<br>terraform plan -out=tfplan             # Save plan to file<br>terraform apply tfplan                 # Apply saved plan</pre><h3>Terraform Workflow:</h3><p>The typical Terraform workflow follows these stages:</p><ol><li><strong>Write</strong>: Define your infrastructure in configuration files</li><li><strong>Plan</strong>: Review the execution plan to understand what changes will be made</li><li><strong>Apply</strong>: Execute the plan to create, update, or delete resources</li><li><strong>Manage</strong>: Use state management and other commands to maintain your infrastructure</li></ol><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*-4megwpkDmrqynCYY8hJ5A.png" /></figure><h3>Understanding Terraform State</h3><p>The state file is arguably the most critical component of Terraform. It serves as a source of truth for your infrastructure and enables Terraform to:</p><ul><li>Track resource metadata</li><li>Improve performance by caching resource attributes</li><li>Enable collaboration through remote state storage</li><li>Provide dependency mapping</li></ul><figure><img alt="Terraform state and backend" src="https://cdn-images-1.medium.com/max/1024/1*RP-e1ZpfZsc9HvhVhmaYQw.png" /></figure><h3>Types of Backends:</h3><p>Terraform supports two main types of backends for state storage:</p><h4>1. Local Backend</h4><p>Best for individual developers or when collaboration isn’t required. The state file is stored locally on your machine.</p><h4>2. Remote Backend</h4><p>Essential for team collaboration and production environments. Common options include:</p><p><strong>S3 Backend with DynamoDB State Locking</strong>:</p><pre><br>terraform {<br>  backend &quot;s3&quot; {<br>    bucket         = &quot;terraform-backend-bucket&quot;<br>    key            = &quot;terraform/terraform.tfstate&quot;<br>    region         = &quot;us-west-2&quot;<br>    encrypt        = true<br>    dynamodb_table = &quot;terraform-lock&quot;<br>  }<br>}</pre><p><strong>Terraform Cloud Backend:</strong></p><pre><br>terraform {<br>  backend &quot;remote&quot; {<br>    organization = &quot;your-organization&quot;<br>    <br>    workspaces {<br>      name = &quot;production-infrastructure&quot;<br>    }<br>  }<br>}</pre><h3><strong>Variables and Outputs: Making Configurations Dynamic</strong></h3><p>Variables make your Terraform configurations reusable and flexible:</p><pre>variable &quot;instance_type&quot; {<br>  description = &quot;EC2 instance type&quot;<br>  type        = string<br>  default     = &quot;t2.micro&quot;<br>  <br>  validation {<br>    condition     = contains([&quot;t2.micro&quot;, &quot;t2.small&quot;, &quot;t2.medium&quot;], var.instance_type)<br>    error_message = &quot;Invalid instance type. Must be one of: t2.micro, t2.small, t2.medium.&quot;<br>  }<br>}<br><br># Usage in resources<br>resource &quot;aws_instance&quot; &quot;web&quot; {<br>  instance_type = var.instance_type<br>  # ... other configuration<br>}</pre><h4><strong>Variable Types:</strong></h4><p>Terraform supports various data types:</p><pre>Primitive Types -&gt; string, number, bool<br>Collection Types -&gt; list, map, set<br>Complex Types -&gt; object, tuple</pre><h3>Variable Assignment Precedence (lowest to highest)</h3><p>Terraform follows a specific order when determining variable values (from lowest to highest priority):</p><ol><li>Manual entry during plan/apply</li><li>Default values in variable declaration</li><li>TF_VAR_&lt;NAME&gt; environment variables</li><li>terraform.tfvars file</li><li>.auto.tfvars files</li><li>Command line flags (var=&quot;key=value&quot;)</li></ol><h4><strong>Local Variables:</strong></h4><p>Local values help you avoid repetition and make your code more maintainable:</p><pre>locals {<br>  common_tags = {<br>    Environment = var.environment<br>    Project     = &quot;web-application&quot;<br>    ManagedBy   = &quot;terraform&quot;<br>  }<br>  <br>  service_name = &quot;${var.environment}-web-service&quot;<br>  service_port = 8080<br>}<br><br># Usage<br>resource &quot;aws_instance&quot; &quot;web&quot; {<br>  tags = local.common_tags<br>  # ... other configuration<br>}</pre><h4>Output variables:</h4><p>Outputs allow you to extract useful information from your Terraform configurations:</p><pre>output &quot;instance_public_ip&quot; {<br>  description = &quot;Public IP address of the web server&quot;<br>  value       = aws_instance.web.public_ip<br>}<br><br>output &quot;database_connection_string&quot; {<br>  description = &quot;Database connection string&quot;<br>  value       = &quot;postgresql://${aws_db_instance.main.username}:${var.db_password}@${aws_db_instance.main.endpoint}/${aws_db_instance.main.name}&quot;<br>  sensitive   = true<br>}</pre><p>Passing variables and variable files via CLI:</p><pre>terraform apply -var=&quot;db_pass=root&quot;<br>terraform apply -var-file=&quot;terraform.tfvars&quot;</pre><h3>Expressions and Functions</h3><p>Terraform provides a rich expression language for dynamic configurations:</p><h4>String</h4><pre>&quot;foo&quot; # literal string<br>&quot;foo ${var.bar}&quot; # template string</pre><h4>Conditionals</h4><pre># Ternary operator<br>instance_type = var.environment == &quot;production&quot; ? &quot;t3.large&quot; : &quot;t2.micro&quot;<br><br># More complex conditions<br>storage_encrypted = var.environment == &quot;production&quot; || var.environment == &quot;staging&quot; ? true : false</pre><h4>Other expression types:</h4><ul><li>For expressions</li><li>Splat expressions</li><li>Dynamic blocks</li><li>Type constraints</li><li>Version constraints</li></ul><h3>Useful Built-in Functions</h3><pre># String functions<br>upper(var.environment)                    # Convert to uppercase<br>lower(var.region)                        # Convert to lowercase<br>join(&quot;-&quot;, [&quot;web&quot;, &quot;server&quot;, var.env])   # Join strings with delimiter<br>split(&quot;,&quot;, var.availability_zones)      # Split string into list<br><br># Collection functions<br>length(var.subnet_ids)                   # Get length of list/map<br>concat(var.public_subnets, var.private_subnets)  # Combine lists<br>merge(local.common_tags, var.custom_tags)        # Merge maps<br><br># File functions<br>file(&quot;${path.module}/user-data.sh&quot;)     # Read file contents<br>templatefile(&quot;config.tpl&quot;, var.config_vars)  # Template file with variables<br><br># Encoding functions<br>base64encode(local.user_data)           # Base64 encode<br>jsonencode(local.policy_document)       # Convert to JSON</pre><h3>Meta-Arguments: Controlling Resource Behavior</h3><p>Meta-arguments provide special functionality for controlling how Terraform manages resources:</p><h4>depends_on</h4><p>Allows specifying dependencies which do not manifest directly through consumption of data from another resource. For example if the creation order of two resources matters, the latter can be specified to depend on the former.</p><h4>count</h4><p>Allows for creation of multiple of a particular resource or module. This is most useful if each instance configuration is nearly identical.</p><pre>resource &quot;aws_instance&quot; &quot;web&quot; {<br>  count         = 3<br>  instance_type = &quot;t2.micro&quot;<br>  <br>  tags = {<br>    Name = &quot;web-server-${count.index + 1}&quot;<br>  }<br>}</pre><h4>for_each</h4><p>Also allows for multiple of a particular resource or module but allows for more control across the instances by iterating over a list.</p><pre>resource &quot;aws_instance&quot; &quot;web&quot; {<br>  for_each = toset(var.availability_zones)<br>  <br>  availability_zone = each.key<br>  instance_type     = &quot;t2.micro&quot;<br>  <br>  tags = {<br>    Name = &quot;web-server-${each.key}&quot;<br>  }<br>}</pre><h4>lifecycle</h4><p>Lifecycle meta-arguments control how Terraform treats particular resources.</p><pre>resource &quot;aws_instance&quot; &quot;web&quot; {<br>  # ... configuration<br>  <br>  lifecycle {<br>    create_before_destroy = true<br>    prevent_destroy       = true<br>    ignore_changes       = [tags]<br>  }<br>}</pre><h3>Provisioners: Executing Actions</h3><p>A provisioner allows you to perform some action either locally or on a remote machine. Theres a number of different provisioners like:</p><pre>- file<br>- local-exec<br>- remote-exec<br>- vendor<br>	- chef<br>	- puppet</pre><p><strong>Important Note</strong>: Provisioners should be used as a last resort. It’s generally better to use cloud-init, user data, or configuration management tools.</p><h3>Modules: Organizing and Reusing Code</h3><p>Modules are containers for multiple resources that are used together. A module consists of a collection of .tf files kept together in a directory.</p><p>Modules are the main way to package and reuse resource configurations with Terraform.</p><pre>module &quot;web_app&quot; {<br>  source = &quot;../web-app-module&quot;<br>  <br>  # Input variables<br>  bucket_name = &quot;bucket-name&quot;<br>  domain      = &quot;abc.com&quot;<br>  db_name     = &quot;mydb&quot;<br>  db_user     = &quot;danish&quot;<br>  db_pass     = var.db_pass<br>}</pre><h4>Types Of Modules:</h4><p>Terraform has two types of modules:</p><ul><li><strong>Root Module</strong> which contains all .tf files in your main working directory where you run commands</li><li><strong>Child Modules</strong> which are separate, reusable modules referenced from other modules using the module block.</li></ul><p><strong>Module sources:</strong></p><p>Modules can be sourced from various locations including local filesystem paths, the official Terraform Registry with community modules, Git repositories like GitHub/Bitbucket, HTTP URLs for custom distribution, and cloud storage buckets like S3 or GCS for private organizational modules.</p><p><strong>What makes a good module?</strong></p><p>Good modules raise abstraction levels by hiding complex configurations, group related resources logically, provide sensible defaults while allowing customization, expose necessary input variables for flexibility, and return useful outputs to enable integration with other infrastructure components.</p><h3>Managing Multiple Environments:</h3><p>There are generally two main approaches to manage multiple environments:</p><h4>Workspaces:</h4><p>Use multiple named sections within a single backend:</p><pre># Create and switch to workspaces<br>terraform workspace new production<br>terraform workspace new staging<br>terraform workspace new development<br><br># List workspaces<br>terraform workspace list<br># Switch workspace<br>terraform workspace select production<br># Use workspace in configuration<br>resource &quot;aws_instance&quot; &quot;web&quot; {<br>  instance_type = terraform.workspace == &quot;production&quot; ? &quot;t3.large&quot; : &quot;t2.micro&quot;<br>  <br>  tags = {<br>    Environment = terraform.workspace<br>  }<br>}</pre><h4>Directory Structure</h4><p>Organize environments using separate directories:</p><pre>environments/<br>├── production/<br>│   ├── main.tf<br>│   ├── variables.tf<br>│   └── terraform.tfvars<br>├── staging/<br>│   ├── main.tf<br>│   ├── variables.tf<br>│   └── terraform.tfvars<br>└── development/<br>    ├── main.tf<br>    ├── variables.tf<br>    └── terraform.tfvars</pre><h4>Addressing Code Rot</h4><p>Common issues that can cause “code rot” in Terraform:</p><ol><li><strong>Out-of-band changes</strong>: Manual changes made directly in the cloud console</li><li><strong>Unpinned versions</strong>: Not specifying provider or module versions</li><li><strong>Deprecated dependencies</strong>: Using outdated resources or arguments</li><li><strong>Unapplied changes</strong>: Configuration drift between code and actual infrastructure</li></ol><p><strong>Best Practices to Prevent Code Rot</strong>:</p><ul><li>Use version constraints for providers and modules</li><li>Regularly run terraform plan to detect drift</li><li>Implement CI/CD pipelines for infrastructure changes</li><li>Use policy as code tools like Sentinel or OPA</li><li>Regular infrastructure audits and updates</li></ul><h3>Conclusion</h3><p>Terraform stands as the industry-leading Infrastructure as Code solution because it combines cloud-agnostic flexibility with declarative simplicity, allowing teams to manage complex infrastructure through version-controlled code rather than manual processes. Its powerful state management, extensive provider ecosystem, and modular architecture make it the optimal choice for organizations seeking to achieve infrastructure consistency, reduce human error, and scale their operations efficiently.</p><p>By adopting Terraform, you’re not just automating infrastructure provisioning — you’re embracing a methodology that brings software development best practices to infrastructure management, resulting in more reliable, auditable, and collaborative infrastructure operations that can evolve with your business needs.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=4ea613d10578" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Mastering Kubernetes Monitoring with Prometheus and Grafana]]></title>
            <link>https://medium.com/@danishsidd203/mastering-kubernetes-monitoring-with-prometheus-and-grafana-3f1d6d2a5896?source=rss-7d9b62c475b8------2</link>
            <guid isPermaLink="false">https://medium.com/p/3f1d6d2a5896</guid>
            <category><![CDATA[prometheus]]></category>
            <category><![CDATA[grafana-prometheus]]></category>
            <category><![CDATA[kubernetes]]></category>
            <category><![CDATA[monitoring]]></category>
            <category><![CDATA[grafana]]></category>
            <dc:creator><![CDATA[Danish Siddiqui]]></dc:creator>
            <pubDate>Fri, 23 Aug 2024 16:51:25 GMT</pubDate>
            <atom:updated>2024-08-23T16:51:25.871Z</atom:updated>
            <content:encoded><![CDATA[<p>Monitoring and logging in Kubernetes are essential practices for observing, tracking, and analyzing the performance, health, and behavior of both the Kubernetes cluster and the applications running on it.</p><h4>Key Components:</h4><ol><li><strong>Monitoring:</strong> Collects and analyzes metrics from cluster components and applications.</li><li><strong>Logging:</strong> Captures and stores event data and messages from various sources within the cluster.</li></ol><h4>Primary Functions:</h4><ul><li>Ensure cluster and application health</li><li>Facilitate troubleshooting and debugging</li><li>Enable performance optimization</li><li>Support capacity planning and scaling decisions</li><li>Enhance security and compliance efforts</li></ul><h4>Tools:</h4><ul><li><strong>Monitoring:</strong> Prometheus, Grafana, Nagios, Zabbix, PRTG.</li><li><strong>Logging:</strong> Grafana Loki, ELK Stack (Elasticsearch, Logstash, Kibana), EFK Stack (Elasticsearch, FluentBit, Kibana) Splunk, Jaeger, Zipkin, New Relic, Dynatrace, Datadog.</li></ul><figure><img alt="Prometheus Monitoring architecture" src="https://cdn-images-1.medium.com/max/1024/1*aMeHr4qXohVx4fMpm5vnvA.png" /><figcaption>Monitoring Architecture Example</figcaption></figure><h3>Prometheus</h3><p>Prometheus has become the mainstream monitoring tool of choice in container and micro service world, designed to monitor highly dynamic environments like Docker Swarm and Kubernetes. It is used for monitoring and gaining metrics, not only in containerized setups but also in traditional non-container infrastructures with bare servers and direct application deployments.</p><p>Prometheus constantly monitors all services, alerting when crashes or issues occur on any server, and even identifies problems before they occur. For optimal performance, it’s recommended to install Prometheus instances directly on the Kubernetes cluster, deploying them as close to the targets as possible and leveraging the existing Kubernetes infrastructure.</p><h4>Monitoring Targets</h4><p>Prometheus can monitor both, the applications running on Kubernetes and the Kubernetes cluster itself. It tracks various cluster components, including:</p><ul><li>Kubernetes cluster components (api-server, kube-scheduler)</li><li>Kubelets for container metrics</li><li>kube-state-metrics for cluster-level metrics</li><li>Node-exporter for host-related metrics (CPU, memory, network)</li></ul><p>For comprehensive monitoring, a node_exporter should run on every host, ideally implemented as a Kubernetes DaemonSet to ensure coverage across all cluster nodes.</p><h4>Why Use Prometheus?</h4><p>Prometheus is crucial for modern DevOps as it helps manage increasingly complex infrastructures with multiple servers running containerized applications and interconnected processes. Maintaining such setups smoothly and without downtime is challenging. Without insight into hardware and application levels, identifying issues when services or applications crash or misbehave, causing other failures, becomes difficult and time-consuming. Prometheus provides the necessary visibility to quickly identify and resolve problems.</p><h4>Prometheus Architecture</h4><p>At its core it has the main component called Prometheus Server that does the actual monitoring work. It is made up of three parts</p><ul><li><strong>Time-series database: </strong>stores all the metrics data</li><li><strong>Data-Retrieval worker: </strong>responsible for pulling those metrics from application, services, servers, etc.</li><li><strong>Server Api:</strong> accepts queries for that stored data. It accepts <strong>PromQL </strong>queries. It can be accessed using a web interface through Prometheus Web UI, or any other monitoring tool like <strong>Grafana</strong>.</li></ul><p>Prometheus just a require a scraping endpoint, it is usually /metrics. Multiple Prometheus instances can pull metrics data.</p><h3>Installation:</h3><h4>Prometheus Operator:</h4><ul><li>The Kube-Prometheus-stack chart makes use of the Prometheus Operator.</li><li>The Prometheus Operator is a Kubernetes Operator, which is an application-specific controller that extends the Kubernetes API to create, configure, and manage instances of complex applications, such as Prometheus.</li><li>The Prometheus Operator has several custom resources to aid the deployment and management of a Prometheus instance, such as AlertManager, PrometheusRule, ServiceMonitor, and PodMonitor.</li><li>The Prometheus Operator chart can be found at: <a href="https://github.com/prometheus-community/helm-charts/tree/main/charts/kube-prometheus-stack">https://github.com/prometheus-community/helm-charts/tree/main/charts/kube-prometheus-stack</a></li></ul><pre>helm repo add prometheus-community https://prometheus-community.github.io/helm-charts<br>helm repo update<br><br>helm install prometheus prometheus-community/kube-prometheus-stack<br><br># Open dashboard through port-forwarding<br>kubectl port-forward prometheus-prometheus-kube-prometheus-prometheus-0 9090<br><br># Open dashboard<br>kubectl expose service prometheus-server --type=NodePort --target-port=9090 --name=prometheus-server-ext</pre><figure><img alt="Prometheus Web UI" src="https://cdn-images-1.medium.com/max/1024/1*qmti8ieTvi-kIgNCjYJOkA.png" /></figure><h3>Grafana: Visualizing Your Metrics</h3><p>Grafana is an open-source platform for monitoring and observability that complements Prometheus perfectly. It allows users to query, visualize, and understand metrics from various data sources, including Prometheus, Elasticsearch, and InfluxDB. Grafana uses PromQL queries under the hood to fetch and visualize data, making it an ideal companion for Prometheus-based monitoring systems.</p><h4>Key features of Grafana:</h4><ol><li>Wide-ranging support for various data sources.</li><li>Customizable and dynamic dashboards with a variety of visualization options.</li><li>Advanced alerting system with multiple notification channels.</li><li>Annotation features for adding context to graphs.</li><li>A rich plugin ecosystem for expanded functionality.</li></ol><p>These features make Grafana a powerful tool for creating comprehensive, insightful visualizations of your monitoring data, enabling better understanding and management of your infrastructure and applications.</p><h4>Installation:</h4><pre>helm repo add grafana https://grafana.github.io/helm-charts<br>helm repo update<br>helm install grafana grafana/grafana<br><br># Open Grafana<br>kubectl expose service grafana --type=NodePort --target-port=3000 --name=grafana-ext<br><br># Get password for user: admin<br>kubectl get secret --namespace default grafana -o jsonpath=&quot;{.data.admin-password}&quot; | base64 --decode ; echo</pre><ul><li>Add Data source in your dashboard by choosing Prometheus and adding the url for Prometheus running on your cluster, then click “Save &amp; Test”</li><li>Add a new dashboard from open source by importing it using its id: some are: 3662, 8171, 11159</li></ul><figure><img alt="Grafana Dashboard" src="https://cdn-images-1.medium.com/max/1024/1*QUv76GoI52gvUWC_HJFtwA.png" /></figure><h3>Creating Custom Metrics In Your Application</h3><ul><li>Install Prometheus Client: npm i prom-client@11.5.3</li><li>Configure Prometheus client and create route for metrics</li></ul><pre>const register = new promClient.Registry();<br>register.setDefaultLabels({<br>  app: &quot;monitor-node-service&quot;,<br>});<br><br>const collectDefaultMetrics = promClient.collectDefaultMetrics;<br><br>collectDefaultMetrics({ register });<br><br>app.get(&quot;/metrics&quot;, async (req, res) =&gt; {<br>  res.setHeader(&quot;Content-Type&quot;, register.contentType);<br>  const metrics = await register.metrics();<br>  res.end(metrics);<br>});</pre><ul><li>Create custom metrics</li></ul><pre>const httpReqCounter = new promClient.Counter({<br>  name: &quot;http_requests_total&quot;,<br>  help: &quot;Total number of HTTP requests&quot;,<br>  labelNames: [&quot;method&quot;, &quot;route&quot;, &quot;status_code&quot;],<br>  registers: [register],<br>});<br><br>const requestDurationHistogram = new promClient.Histogram({<br>  name: &quot;http_request_duration_seconds&quot;,<br>  help: &quot;Duration of HTTP requests in seconds&quot;,<br>  labelNames: [&quot;method&quot;, &quot;route&quot;, &quot;status_code&quot;],<br>  buckets: [0.1, 0.5, 1, 5, 10], // Buckets for the histogram in seconds<br>  registers: [register],<br>});<br><br>const requestDurationSummary = new promClient.Summary({<br>  name: &quot;http_request_duration_summary_seconds&quot;,<br>  help: &quot;Summary of the duration of HTTP requests in seconds&quot;,<br>  labelNames: [&quot;method&quot;, &quot;route&quot;, &quot;status_code&quot;],<br>  percentiles: [0.5, 0.9, 0.99], // Define your percentiles here<br>  registers: [register],<br>});<br><br>const gauge = new promClient.Gauge({<br>  name: &quot;node_gauge_example&quot;,<br>  help: &quot;Example of a gauge tracking async task duration&quot;,<br>  labelNames: [&quot;method&quot;, &quot;status&quot;],<br>  registers: [register],<br>});</pre><ul><li>Create a middleware to track metrics.</li></ul><pre>app.use(<br>  responseTime((req, res, time) =&gt; {<br>    httpReqCounter.labels(req.method, req.url, res.statusCode).inc();<br>    requestDurationHistogram<br>      .labels(req.method, req.url, res.statusCode)<br>      .observe(time);<br>    requestDurationSummary<br>      .labels(req.method, req.url, res.statusCode)<br>      .observe(time);<br>  })<br>);</pre><h4>Results:</h4><figure><img alt="Grafana Custom Metrics Visualization" src="https://cdn-images-1.medium.com/max/1024/1*704fGwaxjCefSuX8ME0k5w.png" /></figure><h3>Conclusion</h3><p>In summary, monitoring and logging are key to keeping Kubernetes clusters and applications running smoothly. Prometheus and Grafana provide the tools you need to track metrics, spot issues early, and maintain a reliable and efficient system, helping you make informed decisions for scaling and optimization.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=3f1d6d2a5896" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Kubernetes Essentials: Understanding Ingress, Helm, and Data Persistence]]></title>
            <link>https://medium.com/@danishsidd203/kubernetes-essentials-understanding-ingress-helm-and-data-persistence-61e0f165897a?source=rss-7d9b62c475b8------2</link>
            <guid isPermaLink="false">https://medium.com/p/61e0f165897a</guid>
            <category><![CDATA[ingress]]></category>
            <category><![CDATA[helm]]></category>
            <category><![CDATA[data-persistence]]></category>
            <category><![CDATA[devops]]></category>
            <category><![CDATA[kubernetes]]></category>
            <dc:creator><![CDATA[Danish Siddiqui]]></dc:creator>
            <pubDate>Thu, 22 Aug 2024 12:23:02 GMT</pubDate>
            <atom:updated>2024-08-22T12:23:02.153Z</atom:updated>
            <content:encoded><![CDATA[<h3>Introduction</h3><p>To fully harness the power of Kubernetes, it’s essential to understand key components like Ingress, Helm, and data persistence mechanisms. This blog delves into these critical aspects of Kubernetes, explaining how they function and why they are indispensable in a modern cloud-native architecture.</p><h3>Ingress</h3><p>Ingress is a powerful resource in Kubernetes that manages external access to services within a cluster. It provides HTTP and HTTPS routing to services based on defined rules, allowing you to control how external users access your applications.</p><figure><img alt="Ingress In Kubernetes" src="https://cdn-images-1.medium.com/max/1024/1*QvbQTmFpgFi91g22z7599A.jpeg" /></figure><h4>Key Features of Ingress</h4><ul><li><strong>No Open IP Address or Port</strong>: Ingress allows you to expose your services without opening specific IP addresses or ports directly. Instead, it routes incoming requests to internal services based on defined rules.</li><li><strong>Traffic Routing</strong>: Incoming requests are forwarded to the appropriate internal service based on the specified rules, enabling flexible routing configurations.</li></ul><h4>Example Ingress Resource</h4><p>Here is an example of how to define an Ingress resource in Kubernetes:</p><pre>apiVersion: networking.k8s.io/v1<br>kind: Ingress<br>metadata:<br>  name: myapp-ingress<br>spec: <br>  rules:<br>  - host: myapp.com<br>    http:<br>      paths:<br>        - path: /<br>          pathType: Prefix<br>          backend: <br>            service:<br>              name: myapp-internal-service<br>              port:<br>                number: 8081</pre><p><strong>Corresponding Service Definition:</strong></p><p>To route traffic to the appropriate service, you need a corresponding Service resource. Here’s an example of a Service definition:</p><pre>apiVersion: v1<br>kind: Service<br>metadata:<br>  name: myapp-internal-service<br>spec:<br>  selector:<br>    app: myapp<br>  ports:<br>    - protocol: TCP<br>      port: 8081<br>      targetPort: 8081</pre><h4>Ingress Controller</h4><p>The implementation of Ingress is handled by an <strong>Ingress Controller.</strong> This component is responsible for managing the routing of external traffic to the appropriate services based on the Ingress rules defined. Key responsibilities of the Ingress Controller include:</p><ul><li><strong>Evaluating Rules</strong>: The Ingress Controller evaluates all defined Ingress rules to determine how to route incoming requests.</li><li><strong>Managing Redirection</strong>: It handles redirection and rewriting of URLs as specified in the Ingress rules, allowing for flexible traffic management.</li><li><strong>Entry Point to the Cluster</strong>: The Ingress Controller acts as the entry point for external traffic into the Kubernetes cluster, enabling secure and organized access to services.</li><li><strong>Popular Implementations</strong>: One of the most widely used Ingress Controllers is the <strong>Kubernetes Nginx Ingress Controller</strong>. It is easy to install and configure, providing robust features for managing Ingress resources.</li></ul><p>Ingress is a vital component in Kubernetes for managing external access to services. By using Ingress resources and an Ingress Controller, you can effectively route traffic to your applications while maintaining security and flexibility in your cluster architecture. This enhanced version provides clearer explanations, additional context, and maintains a structured format for better readability.</p><h3>Helm — Package Manager</h3><figure><img alt="Helm — Package Manager for Kubernetes" src="https://cdn-images-1.medium.com/max/728/1*QYpFDlGrTLrnOpOu6D3TLw.png" /></figure><h4>What is Helm?</h4><ul><li><strong>Helm</strong> is a package manager for Kubernetes that simplifies the deployment and management of applications. It allows users to package YAML files into reusable charts and distribute them through public and private repositories.</li><li>Helm acts as a <strong>templating engine</strong>, enabling users to define complex Kubernetes applications with customizable configurations.</li><li>It facilitates <strong>release management</strong>, allowing users to install, upgrade, and roll back applications easily.</li></ul><h4>What are Helm Charts ?</h4><ul><li><strong>Helm Charts</strong> are bundles of YAML files that define the resources needed to deploy an application on Kubernetes. Each chart contains all the necessary configuration files, including deployments, services, secrets, and config maps.</li><li>Users can create their own Helm charts using Helm, push them to a Helm repository, or download and use existing charts from public repositories.</li></ul><h4>when to use them ?</h4><p>Helm is particularly useful in complex Kubernetes environments where applications consist of multiple microservices and configurations. It simplifies the management of these applications by allowing users to:</p><ul><li>Maintain consistent deployments across different environments (e.g., development, staging, production).</li><li>Easily manage application versions and configurations.</li><li>Streamline CI/CD processes by integrating Helm into deployment pipelines.</li></ul><h4>What is Tiller? (Server)</h4><ul><li><strong>Tiller</strong> was the in-cluster server component of Helm in version 2, responsible for managing the interaction between the Helm client and the Kubernetes API. It evaluated Helm charts and deployed applications based on defined configurations.</li><li>Tiller was removed in <strong>Helm 3</strong> for security reasons, as it had extensive permissions within the Kubernetes cluster. The removal of Tiller simplified the architecture and improved security by allowing direct interaction with the Kubernetes API without an intermediary.</li></ul><h4>Directory Structure</h4><p>A typical Helm chart follows this directory structure:</p><pre>mychart/         # Folder name of the chart<br>	Chart.yaml     # Metadata about the chart (name, version, etc.)<br>	values.yaml    # Default values for the templates<br>	charts/        # Folder for chart dependencies<br>	templates/     # Directory containing the actual template files</pre><h4>Common Helm commands</h4><pre>helm install [chart_name]    # Installs a Helm chart and deploys the application.<br>helm upgrade [chart_name]    # Upgrades an existing release to a new version.<br>helm rollback [chart_name]   # Rolls back to a previous version of a release.</pre><p>Helm is an essential tool for managing Kubernetes applications, providing a streamlined approach to packaging, deploying, and maintaining applications. By using Helm charts, developers can automate complex deployments, manage application configurations more effectively, and enhance the overall efficiency of their Kubernetes workflows.</p><h3>Data Persistence In Kubernetes</h3><p>In Kubernetes, a <strong>volume</strong> is a directory accessible to containers within a pod. Volumes provide a way to persist data beyond the lifecycle of a container or pod. These volumes are essential for data persistence in Kubernetes.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/869/1*m4V9-7RMgYGHnAD0hudIMA.png" /></figure><h4>Need for volume:</h4><ul><li><strong>Stateful applications</strong>: Applications like databases require persistent storage to store and retrieve data. If a pod containing a database crashes, the data must be preserved.</li><li><strong>Shared data</strong>: Volumes enable sharing of data among containers within a pod or across pods. This is useful for scenarios where multiple containers need to access the same data.</li><li><strong>Data backup and restore</strong>: Volumes provide a mechanism to back up and restore data, ensuring data integrity and enabling disaster recovery.</li></ul><h4>Persistent Volume</h4><p><strong>Persistent Volumes (PVs)</strong> are cluster-level resources created using YAML files. They represent actual physical or cloud storage and are not namespaced, making them accessible to the entire cluster across all namespaces.</p><p>PVs can be provisioned statically by administrators or dynamically using <strong>Storage Classes</strong>.</p><h4>Persistent Volume Claim</h4><p><strong>Persistent Volume Claims (PVCs)</strong> are requests for storage resources made by applications. They specify the desired storage size, access mode, and other requirements. Applications need to claim a PV by creating a PVC in the same namespace.</p><p>PVCs are bound to PVs based on the requested storage size, access mode, and other constraints. If a matching PV is available, the PVC is bound to it. If no matching PV exists, the PVC remains unbound until a suitable PV becomes available or a Storage Class provisions a new PV dynamically.</p><h4>Storage Class</h4><p><strong>Storage Classes</strong> are used to provision PVs dynamically based on the requested storage requirements in a PVC. They abstract the underlying storage implementation, allowing administrators to define different storage options with varying performance characteristics, backup policies, and access modes.</p><p>When a PVC requests storage using a specific Storage Class, the Kubernetes control plane dynamically provisions a new PV that matches the PVC’s requirements. This simplifies storage management and enables self-service provisioning for application developers.</p><h3>Conclusion</h3><p>Kubernetes is a powerful platform, but to unlock its full potential, it’s crucial to master its core components like Ingress, Helm, and data persistence strategies. Ingress provides seamless and secure external access to services, Helm simplifies application deployment and management, and persistent volumes ensure data integrity and availability. By understanding and effectively utilizing these tools, you can build scalable and manageable applications. Whether you’re deploying micro-services, managing complex configurations, or ensuring data durability, these Kubernetes features form the backbone of a reliable and efficient system.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=61e0f165897a" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Kubernetes Essentials: Mastering Namespaces for Effective Resource Management]]></title>
            <link>https://medium.com/@danishsidd203/kubernetes-essentials-mastering-namespaces-for-effective-resource-management-f16d34299e81?source=rss-7d9b62c475b8------2</link>
            <guid isPermaLink="false">https://medium.com/p/f16d34299e81</guid>
            <category><![CDATA[kubernetes-cluster]]></category>
            <category><![CDATA[orchestration]]></category>
            <category><![CDATA[namespaces]]></category>
            <category><![CDATA[devops]]></category>
            <category><![CDATA[kubernetes]]></category>
            <dc:creator><![CDATA[Danish Siddiqui]]></dc:creator>
            <pubDate>Thu, 22 Aug 2024 12:11:42 GMT</pubDate>
            <atom:updated>2024-08-22T12:11:42.299Z</atom:updated>
            <content:encoded><![CDATA[<h3>Introduction:</h3><p>Kubernetes is a powerful container orchestration platform that offers a range of features to manage, deploy, and scale applications. Among these features, Namespaces play a crucial role in organizing resources. This blog will explore the key concepts of Namespaces, and why do we need it, providing a clear understanding of it’s importance and usage in Kubernetes environments.</p><figure><img alt="Kubernetes Namespaces" src="https://cdn-images-1.medium.com/max/1024/1*r8LDG603Oh2e8ZNyvEGShg.png" /></figure><h3>Namespaces: Organizing Resources in Kubernetes</h3><p>Namespaces are a powerful feature in Kubernetes that allow for the organization and management of resources within a cluster. They provide a way to create virtual clusters within a physical cluster, enabling better resource allocation, access control, and isolation among different teams and applications.</p><h4>Key Features of Namespaces</h4><ul><li><strong>Organize Resources</strong>: Namespaces help in organizing resources logically, making it easier to manage applications and services. For example, you can group resources related to databases, monitoring tools, the Elastic Stack, and Nginx Ingress under specific namespaces.</li><li><strong>Virtual Clusters</strong>: Namespaces act as virtual clusters inside a physical cluster, allowing multiple teams to work independently without interfering with each other’s resources.</li></ul><h4>Default Namespaces in Kubernetes</h4><p>Kubernetes comes with four default namespaces, each serving a specific purpose:</p><ol><li><strong>kube-system</strong></li></ol><ul><li>Contains system processes managed by Kubernetes.</li><li>It is advisable <strong>not to create or modify resources</strong> in this namespace unless necessary, as it can affect cluster stability.</li><li>Includes critical components such as the Kubernetes master processes and kubectl.</li></ul><p><strong>2. kube-public</strong></p><ul><li>Designed for publicly accessible data.</li><li>Contains a ConfigMap that holds cluster information, which can be accessed by all users, including those who are not authenticated.</li></ul><p><strong>3. kube-node-lease</strong></p><ul><li>Stores information about the heartbeats of nodes in the cluster.</li><li>Each node has an associated lease object in this namespace, which helps determine the availability of nodes.</li></ul><p><strong>4. default</strong></p><ul><li>This is the namespace where resources created without a specified namespace are located.</li><li>It serves as the default working area for users who do not define a namespace for their resources.</li></ul><h3>Need for Namespaces</h3><p>Namespaces are essential for several reasons:</p><ul><li><strong>Resource Grouping</strong>: They allow for logical grouping of resources, such as databases, monitoring tools, and application services, making management easier.</li><li><strong>Conflict Resolution</strong>: In environments with multiple teams working on the same application, namespaces help avoid resource conflicts by allowing each team to operate within its own isolated namespace.</li><li><strong>Resource Sharing</strong>: Namespaces facilitate resource sharing for staging and deployment environments, enabling teams to collaborate effectively.</li><li><strong>Access Control</strong>: Namespaces can enforce access controls and resource limits, ensuring that teams have the necessary permissions while maintaining security.</li><li><strong>Isolation</strong>: Each team can have its own isolated environment, reducing the risk of accidental interference with other teams’ resources.</li></ul><h3>Characteristics of Namespaces</h3><ul><li><strong>Isolation</strong>: Most resources in Kubernetes cannot be accessed across namespaces. Each namespace must define its own ConfigMap, Secrets, and other resources.</li><li><strong>Global Resources</strong>: Some components, such as Volumes and Nodes, exist outside of namespaces and are global within the cluster. They cannot be isolated within a namespace.</li></ul><figure><img alt="Characteristics of Namespaces in Kubernetes" src="https://cdn-images-1.medium.com/max/739/1*hYShsjvyupxMqIPWjZY5dA.png" /></figure><h3>Example of Creating a ConfigMap in a Namespace</h3><p>Here is an example of how to create a ConfigMap within a specific namespace:</p><pre>apiVersion: v1<br>kind: ConfigMap<br>metadata:<br>  name: mongo-config<br>  namespace: my-namespace<br>data: <br>  db_url: mongo-service.database  # The dot notation specifies the service&#39;s namespace</pre><h3>Switching Between Namespaces</h3><p>To switch between active namespaces in your Kubernetes environment, you can use the kubens command-line tool. This tool simplifies the process of managing and navigating through different namespaces, enhancing your productivity when working with Kubernetes.</p><h3>Conclusion:</h3><p>Namespaces in Kubernetes are essential for organizing and managing resources within a cluster. They provide logical separation, enable resource isolation, and allow multiple teams to work concurrently without interference. By leveraging namespaces, you can enhance security, streamline resource management, and efficiently manage complex Kubernetes environments, ensuring smooth and scalable operations across your applications.</p><p><strong>Happy Organizing and Isolating!</strong></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=f16d34299e81" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Kubernetes 101: A Deep Dive Into Container Orchestration]]></title>
            <link>https://medium.com/@danishsidd203/kubernetes-101-a-deep-dive-into-container-orchestration-5617fbde8e00?source=rss-7d9b62c475b8------2</link>
            <guid isPermaLink="false">https://medium.com/p/5617fbde8e00</guid>
            <category><![CDATA[devops-tool]]></category>
            <category><![CDATA[container-orchestration]]></category>
            <category><![CDATA[kubernetes-cluster]]></category>
            <category><![CDATA[kubernetes]]></category>
            <category><![CDATA[devops]]></category>
            <dc:creator><![CDATA[Danish Siddiqui]]></dc:creator>
            <pubDate>Mon, 19 Aug 2024 22:51:10 GMT</pubDate>
            <atom:updated>2024-08-19T22:51:10.986Z</atom:updated>
            <content:encoded><![CDATA[<h3>What is Kubernetes and Why Do We Need It?</h3><p>Kubernetes, often abbreviated as K8s, is a powerful platform designed to automate the deployment, scaling, and management of containerized applications. It is an open-source container orchestration tool developed by google which helps you manage containerized applications in different deployment environments.</p><p>Need for Kubernetes increased because of some of the following reasons:</p><ul><li>Increased usage of containers.</li><li>Trend from Monolithic architecture to Micro-services.</li><li>Demand for an efficient way to manage hundreds of containers.</li></ul><h4>Key Features of Kubernetes:</h4><ul><li>High availability or no downtime: Applications are always accessible to users.</li><li>Scalability and high performance: Ability to handle increased loads efficiently.</li><li>Disaster recovery: Automated backup and restore capabilities.</li></ul><figure><img alt="Kubernetes — a container orchestration tool." src="https://cdn-images-1.medium.com/max/760/1*TR7Acg5YWqIcRan10CVmmw.jpeg" /></figure><h3>Core Kubernetes Components:</h3><h4>1. Node</h4><ul><li>A worker machine in the Kubernetes cluster.</li><li>Can be a physical or virtual machine.</li><li>Runs pods and is managed by the control plane.</li></ul><h4>2. Pods</h4><ul><li>Smallest unit of Kubernetes.</li><li>Abstraction over container, providing a layer on top of a container.</li><li>Usually hosts one application per pod.</li><li>Each pod gets its own IP address.</li><li>Pods can die very easily, and a new one is created on its place with a new IP address, so IP address can be changed.</li><li>If a pod dies, a new one is created with a new IP address.</li><li>When a pod dies and restarts, it pulls the image from the remote repository and reruns the application, which may cause brief downtime.</li></ul><h4>3. Services</h4><ul><li>Provides a permanent IP address for accessing pods</li><li>Lifecycle of Pod and Service are not connected</li><li>Two types: <br> — External Service (e.g., your website)<br> — Internal Service (e.g., database)</li></ul><h4>4. Secret</h4><ul><li>Used to store sensitive data.</li><li>Base64 encoded.</li><li>Keeps confidential information separate from application code.</li></ul><h4>5. ConfigMap</h4><ul><li>Stores external configuration for your application.</li><li>Allows separation of configuration from application code.</li></ul><h4>6. Deployment</h4><ul><li>If one pod dies, our application will also go down, so to overcome this we use Deployments and ReplicaSets to create replicas of the same pod, so it will create a number of pods, if one goes down, other will handle the traffic.</li><li>Manages a set of identical pods.</li><li>Ensures high availability by creating replicas of pods.</li><li>Acts as a load balancer to manage high traffic with minimal latency.</li><li>In production environments, you typically work with Deployments rather than individual Pods.</li></ul><h4>7. StatefulSet</h4><ul><li>Used for stateful applications like databases.</li><li>Maintains a sticky identity for each pod.</li><li>Ensures stable storage and network identifiers across pod rescheduling.</li></ul><h4>8. Volumes</h4><ul><li>Provides persistent storage for pods.</li><li>Allows data to survive container restarts and pod rescheduling.</li><li>Various types available, including local storage and cloud-provider specific options.</li></ul><h4>9. Storage</h4><ul><li>If a database pod goes down, data stored within the container is lost.</li><li>Kubernetes doesn’t manage data persistence by default.</li><li>Persistent Volumes (PV) and Persistent Volume Claims (PVC) are used to handle persistent storage.</li><li>Storage Classes can be used to dynamically provision storage resources.</li></ul><h4>Additional Kubernetes concepts:</h4><ul><li>Namespace: Virtual cluster within a physical cluster, used for resource isolation.</li><li>Ingress: Manages external access to services within a cluster.</li><li>Helm: Package manager for Kubernetes, simplifying application deployment.</li><li>Horizontal Pod Autoscaler: Automatically scales the number of pods based on CPU utilization or custom metrics.</li></ul><h3>Kubernetes Architecture:</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*2iOqWSsQNUgBWI9xmNqbSg.jpeg" /></figure><h4>Worker Nodes:</h4><ul><li>Perform the actual workload</li><li>Key components:<br> — Kubelet: Agent that runs on each node, ensuring containers are running in a pod<br> — Container Runtime: Software responsible for running containers (e.g., Docker, containerd)<br> — Kube Proxy: Network proxy that maintains network rules on nodes</li><li>Cluster Structure:<br> — A Kubernetes cluster consists of multiple nodes<br> — Each node must have container runtime and kubelet services installed<br> — Communication between nodes is facilitated by Services, which act as load balancers<br> — Kube Proxy, installed on every node, is responsible for forwarding requests from services to pods and maintaining network rules</li></ul><h4>Master Node (Control Plane):</h4><p>The master node manages the cluster state and worker nodes. Four essential services run on every master node:</p><h4>1. API Server</h4><ul><li>Acts as the cluster gateway.</li><li>Handles all API operations.</li><li>Serves as a gatekeeper for authentication and authorization.</li><li>Validates and processes API requests.</li></ul><h4>2. Scheduler:</h4><ul><li>Receives requests from the API server.</li><li>Decides where to place pods based on resource requirements, policies, and constraints.</li><li>Considers factors such as resource availability, affinity/anti-affinity rules, and tolerations.</li></ul><h4>3. Controller Manager:</h4><ul><li>Monitors cluster state.</li><li>Detects and responds to changes, such as pod failures.</li><li>Attempts to maintain the desired cluster state.</li><li>Makes requests to the scheduler to reschedule failed pods.</li><li>Includes various controllers: Node Controller, Replication Controller, Endpoints Controller, etc.</li></ul><h4>4. etcd:</h4><ul><li>Distributed key-value store that serves as the cluster’s “brain”.</li><li>Stores all cluster configuration data and state.</li><li>Provides a reliable way to store and retrieve cluster information.</li><li>Does not store application data.</li><li>Crucial for communication between master processes and worker processes.</li><li>A cluster state information which is used for master proceses to communicate with the workprocess and vice versa.</li></ul><p>In production environments, a Kubernetes cluster usually consists of:</p><ul><li>Multiple master nodes for high availability and load distribution.</li><li>A load balancer in front of API servers to distribute client requests.</li><li>Multiple worker nodes to handle application workloads.</li></ul><p>This architecture ensures that Kubernetes can efficiently manage containerized applications at scale, providing features like self-healing, load balancing, and automated roll-outs and rollbacks.</p><h3>Kubernetes Networking:</h3><h4>Service networking:</h4><ul><li>Abstracts the way Pods are accessed within a Kubernetes cluster.</li><li>Provides a stable endpoint for a set of Pods, regardless of their actual IP addresses or locations.</li><li>Uses labels and selectors to group Pods under a Service.</li><li>Offers load balancing across multiple Pod instances.</li><li>Supports different types: ClusterIP, NodePort, LoadBalancer, and ExternalName.</li></ul><h4>Pod networking:</h4><ul><li>Each Pod gets its own IP address.</li><li>Containers within a Pod share the same network namespace.</li><li>Containers can communicate with each other using localhost.</li><li>Pods on the same node can communicate directly.</li><li>Pods on different nodes communicate through overlay network.</li></ul><h4>Network policies:</h4><ul><li>Define how groups of Pods are allowed to communicate.</li><li>Act as a firewall for controlling traffic flow between Pods.</li><li>Can specify both ingress and egress rules.</li></ul><h4>Ingress controllers:</h4><ul><li>Manages external access to services within a cluster.</li><li>Provides HTTP/HTTPS routing, SSL termination, and name-based virtual hosting.</li><li>Acts as a reverse proxy and load balancer.</li><li>Typically runs as pods within the cluster.</li><li>Popular implementations include NGINX Ingress Controller, HAProxy, and Traefik.</li></ul><h3>Common Kubectl commands:</h3><pre>kubectl get node<br>kubectl get pods<br>kubectl get pods -o wide # for more details<br>kubectl get secrets<br>kubectl get configmap<br>kubectl get services<br>kubectl get deployments<br>kubectl get replicaset<br>kubectl get all<br><br>kubectl describe pod [pod_name] # for complete information<br><br># To add file<br>kubectl apply -f [yaml_file_name]<br>kubectl create -f [yaml_file_name]<br><br>kubectl apply -f [directory]<br>kubectl delete -f [directory]<br><br># To remove files<br>kubectl delete pods --all<br>kubectl delete deployment [name]<br>kubectl delete service [name]<br>kubectl delete secret [name]<br>kubectl delete configmap [name]<br><br>kubectl logs [pod_name]<br><br>minikube ip<br>minikube service [service_name]</pre><h4>Pod Yaml File Example:</h4><pre>apiVersion: v1<br>kind: Pod<br>metadata:<br>  name: myapp-pod<br>  labels:<br>    app: myapp<br>spec:<br>  containers:<br>    - name: myapp<br>      image: repo-link<br>      ports:<br>        - containerPort: 8000</pre><h4>Service Yaml File Example:</h4><pre>apiVersion: v1<br>kind: Service<br>metadata:<br>  name: myapp-service<br>spec:<br>  selector:<br>    app: myapp<br>  type: LoadBalancer<br>  ports:<br>    - port: 7000<br>      targetPort: 8000</pre><h4>Secret Yaml File Example:</h4><pre>apiVersion: v1<br>kind: Secret<br>metadata:<br>  name: myapp-secret<br>type: Opaque<br>data:<br>  MONGO_URI: value-in-Base64<br>  PORT: value-in-Base64</pre><h4>ConfigMap Yaml File Example:</h4><pre>apiVersion: v1<br>kind: ConfigMap<br>metadata:<br>  name: myapp-config<br>data:<br>  mongo-url: mongo-service</pre><h4>Deployment Yaml File Example:</h4><pre>apiVersion: apps/v1<br>kind: Deployment<br>metadata:<br>  name: deployment-name<br>spec:<br>  replicas: 3<br>  selector:<br>    matchLabels:<br>      app: myapp<br>  template:<br>    metadata:<br>      labels:<br>        app: myapp<br>    spec:<br>      containers:<br>        - name: myapp-container<br>          image: prodanish203/myapp<br>          ports:<br>            - containerPort: 8000<br>          env:<br>            - name: MONGO_URI<br>              valueFrom:<br>                secretKeyRef:<br>                  name: myapp-secret<br>                  key: MONGO_URI<br>            - name: PORT<br>              valueFrom:<br>                secretKeyRef:<br>                  name: myapp-secret<br>                  key: PORT</pre><h3>Conclusion:</h3><p>Kubernetes has revolutionized the way we deploy and manage containerized applications. By providing a platform for container orchestration, it enables organizations to build scalable, resilient, and efficient infrastructure. Whether you’re just starting with containerization or looking to optimize your existing setup, understanding Kubernetes is crucial in today’s cloud-native world.</p><p>As you delve deeper into Kubernetes, remember that its ecosystem is vast and constantly evolving. Stay curious, keep learning, and don’t hesitate to experiment with different configurations to find what works best for your specific use cases.</p><p><strong>Happy orchestrating and keep on scaling!</strong></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=5617fbde8e00" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Mastering MERN Stack Deployment: A Comprehensive Guide to CI/CD with GitHub Actions]]></title>
            <link>https://medium.com/@danishsidd203/introduction-2e9266279624?source=rss-7d9b62c475b8------2</link>
            <guid isPermaLink="false">https://medium.com/p/2e9266279624</guid>
            <category><![CDATA[ci-cd-pipeline]]></category>
            <category><![CDATA[vps]]></category>
            <category><![CDATA[devops]]></category>
            <category><![CDATA[github-actions]]></category>
            <category><![CDATA[deployment]]></category>
            <dc:creator><![CDATA[Danish Siddiqui]]></dc:creator>
            <pubDate>Fri, 16 Aug 2024 20:13:22 GMT</pubDate>
            <atom:updated>2024-08-16T20:29:22.694Z</atom:updated>
            <content:encoded><![CDATA[<h3>Introduction</h3><p>Deploying a MERN stack application can be tricky. It often involves many steps and can be frustrating. But don’t worry! This guide will help you. I’ll show you how to put your MERN app on a Virtual Private Server (VPS). But we won’t stop there. We’ll also look how to make the whole process happen automatically using CI/CD pipelines with GitHub Actions.</p><p>CI/CD might sound complicated, but it’s actually quite helpful. Here’s what it does: Once you set it up, you just need to push your code to GitHub. Then, the CI/CD system takes over. It will automatically deploy your latest code into production. This means you don’t have to do the deployment yourself each time you make changes.</p><p>This guide is for everyone. If you’re new to MERN stack deployment, you’ll learn the basics. If you’re more experienced, you’ll see how to make your process smoother. By the end, you’ll know how to set up a strong, production-ready system that updates itself. This can save you time and make the whole process less stressful.</p><figure><img alt="Complete workflow of a CI/CD Pipeline" src="https://cdn-images-1.medium.com/max/1024/1*AGQDZReW0cbCQZZ-20olIw.png" /><figcaption>Complete workflow of a CI/CD Pipeline</figcaption></figure><h4>1. Setting Up Your VPS</h4><p>First, you’ll need to configure a VPS on a cloud provider. We’ll use Digital Ocean for this example, but the process is similar for other providers. Once your VPS is set up, you’ll need to SSH into it:</p><pre># Create an SSH key (if you haven&#39;t already)<br>ssh-keygen -t rsa -b 4096 -C &quot;github-actions-deploy&quot; -f ~/.ssh/github_actions_deploy -N &quot;&quot;<br><br># Login to your VPS<br>ssh -i ~/.ssh/PRIVATE_KEY_FILE_NAME USERNAME@SERVER_IP_ADDRESS</pre><h4>2. Preparing Your Server</h4><p>Update your system to ensure you have the latest packages:</p><pre>sudo apt update<br>sudo apt upgrade</pre><h4>3. Installing Node.js</h4><p>We’ll use NVM (Node Version Manager) to install Node.js, which allows for easy version management:</p><pre>curl -o- https://raw.githubusercontent.com/nvm-sh/nvm/v0.40.0/install.sh | bash<br><br># Install the latest LTS version of Node.js<br>nvm install --lts</pre><h4>4. Setting Up Git and GitHub CLI</h4><p>Install Git and the GitHub CLI for easier management of your repositories:</p><pre>apt install git<br>apt install gh<br><br># Login to GitHub<br>gh auth login</pre><h4>5. Installing PM2</h4><p>PM2 is a process manager for Node.js applications that helps you keep your app running in the background:</p><pre>npm i -g pm2<br><br># Some useful pm2 commands<br>pm2 start npm --name &quot;your-app-name&quot; -- start<br>pm2 list<br>pm2 logs your-app-name<br>pm2 delete all<br>pm2 save</pre><h4>6. Configuring Nginx</h4><p>Nginx will act as a reverse proxy for our Node.js application:</p><pre>apt install nginx<br><br># Start and enable Nginx<br>sudo systemctl start nginx<br>sudo systemctl enable nginx</pre><p>Create a new Nginx configuration file: sudo nano /etc/nginx/sites-available/your-app.conf</p><p>Add the following configuration:</p><pre>server {<br>    listen 80;<br>    server_name _;<br><br>    location / {<br>        proxy_pass http://localhost:3000;<br>        limit_req zone=mylimit burst=20 nodelay;<br>        try_files $uri $uri/ /index.html =404;<br>        proxy_set_header Host $host;<br>        proxy_set_header X-Real-IP $remote_addr;<br>        proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;<br>        proxy_set_header X-Forwarded-Proto $scheme;<br><br>        #Important For Cookies<br>        proxy_set_header X-Forwarded-Host $host;<br>        proxy_cookie_path / &quot;/; Secure; HttpOnly; SameSite=None&quot;;<br>    }<br><br>    location /api {<br>        proxy_pass http://localhost:8000;<br>        # ... (same settings as above)<br>    }<br>}</pre><h4>7. Implementing Rate Limiting</h4><p>To protect your server from abuse, add rate limiting to Nginx:</p><p>Edit the main Nginx configuration file: sudo nano /etc/nginx/nginx.conf</p><p>Add the following inside the http block:</p><pre>http {<br>    limit_req_zone $binary_remote_addr zone=mylimit:10m rate=2r/s;<br>    # ... (other settings)<br>}</pre><h4>8. Activating Your Nginx Configuration</h4><p>Create a symbolic link to enable your configuration:</p><pre>sudo ln -s /etc/nginx/sites-available/your-app.conf /etc/nginx/sites-enabled/<br>sudo rm /etc/nginx/sites-enabled/default</pre><p>Test and reload Nginx:</p><pre>sudo nginx -t<br>sudo systemctl reload nginx</pre><h4>9. Setting Up GitHub Actions for CI/CD</h4><p>Create a .github/workflows/deploy.yml file in your repository with the following content:</p><pre>name: Build And Deploy<br><br>on:<br>  push:<br>    branches:<br>      [ &quot;main&quot; ]<br><br>jobs:<br>    build-and-deploy:<br>        runs-on: ubuntu-latest<br>        steps:<br>            - uses: actions/checkout@v2<br><br>            - name: Use Node.js 22.2.0<br>              uses: actions/setup-node@v2<br>              with:<br>                node-version: &#39;22.2.0&#39;<br><br>            - name: Install Dependencies (Client)<br>              run: |<br>                cd client<br>                npm ci<br>            <br>            - name: Install Dependencies (Server)<br>              run: |<br>                cd server<br>                npm ci<br>            <br>            - name: Build Client<br>              run: |<br>                cd client<br>                npm run build<br><br>            - name: Create client .env file<br>              run: |<br>                cd client<br>                echo &quot;${{ secrets.CLIENT_ENV }}&quot; &gt; .env<br>          <br>            - name: Create server .env file<br>              run: |<br>                cd server<br>                echo &quot;${{ secrets.SERVER_ENV }}&quot; &gt; .env<br><br>            - name: Deploy to DigitalOcean<br>              env: <br>                SSH_PRIVATE_KEY: ${{ secrets.SSH_PRIVATE_KEY }}<br>                SSH_USERNAME: ${{ secrets.SSH_USERNAME }}<br>                SSH_HOST: ${{ secrets.SSH_HOST }}<br>              run: |<br>                mkdir -p ~/.ssh/<br>                echo &quot;$SSH_PRIVATE_KEY&quot; &gt; ~/.ssh/id_rsa<br>                chmod 600 ~/.ssh/id_rsa<br>                ssh-keyscan -H $SSH_HOST &gt;&gt; ~/.ssh/known_hosts<br>                ssh -i ~/.ssh/id_rsa $SSH_USERNAME@$SSH_HOST &#39;echo SSH connection successful&#39;<br>                rsync -avz --delete-after --quiet client/ $SSH_USERNAME@$SSH_HOST:~/MERN-CICD/client/<br>                rsync -avz --delete-after --quiet server/ $SSH_USERNAME@$SSH_HOST:~/MERN-CICD/server/<br>                ssh $SSH_USERNAME@$SSH_HOST &#39;<br>                    export NVM_DIR=&quot;$HOME/.nvm&quot; &amp;&amp;<br>                    [ -s &quot;$NVM_DIR/nvm.sh&quot; ] &amp;&amp; \. &quot;$NVM_DIR/nvm.sh&quot; &amp;&amp;<br>                    cd ~/MERN-CICD/client &amp;&amp;<br>                    npm install &amp;&amp;<br>                    npm run build &amp;&amp;<br>                    cd ../server &amp;&amp;<br>                    npm install &amp;&amp;<br>                    pm2 restart all<br>                &#39;</pre><p>This GitHub Actions workflow will automatically deploy your application whenever you push to the main branch. It installs dependencies, builds the client, creates necessary environment files, and deploys the application to your DigitalOcean server using SSH and rsync.</p><h3>Conclusion</h3><p>By following this guide, you’ve set up a robust MERN stack deployment pipeline. Your application is now running on a VPS, protected by Nginx with rate limiting, and automatically deploys whenever you push changes to your main branch. This setup provides a solid foundation for scaling your application and implementing further optimizations as needed.</p><p>Remember to secure your server by setting up a firewall, implementing HTTPS, and regularly updating your packages. Happy deploying!</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=2e9266279624" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Implementing AWS S3 and CloudFront for Efficient Storage and Content Delivery]]></title>
            <link>https://medium.com/@danishsidd203/implementing-aws-s3-and-cloudfront-for-efficient-storage-and-content-delivery-7e13d9472974?source=rss-7d9b62c475b8------2</link>
            <guid isPermaLink="false">https://medium.com/p/7e13d9472974</guid>
            <category><![CDATA[multer]]></category>
            <category><![CDATA[file-upload]]></category>
            <category><![CDATA[aws-cloudfront]]></category>
            <category><![CDATA[cdn]]></category>
            <category><![CDATA[aws-s3]]></category>
            <dc:creator><![CDATA[Danish Siddiqui]]></dc:creator>
            <pubDate>Fri, 09 Aug 2024 14:03:46 GMT</pubDate>
            <atom:updated>2024-08-09T14:03:46.769Z</atom:updated>
            <content:encoded><![CDATA[<p>Managing and storing files in a MERN stack application can be challenging. In this blog post, I’ll walk you through how I recently implemented Amazon S3 for file storage and CloudFront as a Content Delivery Network (CDN) in my MERN stack application, significantly improving its performance and scalability.</p><figure><img alt="Represents how users will access the S3 bucket content using the AWS CloudFront CDN" src="https://cdn-images-1.medium.com/max/1024/1*Fo52wgR_qDvgXqnk8w9fJg.png" /><figcaption>AWS S3 and Cloudfront Architecture</figcaption></figure><h3>Why AWS S3 ?</h3><p>Amazon S3 (Simple Storage Service) is an excellent choice for file storage in web applications. Its scalability allows it to handle any amount of data, from a few files to petabytes of information. Amazon guarantees an impressive 99.999999999% (11 9’s) durability for S3 objects, ensuring your data remains safe and accessible. S3 also offers robust security features, including encryption at rest and in transit. Its flexibility makes it perfect for storing various types of files, including user-uploaded content, static assets, and backups. Moreover, S3’s cost-effective pay-as-you-go model makes it economical for growing applications, as you only pay for what you use.</p><h3>Getting Started with S3:</h3><h4>Step 1: Create a Private S3 Bucket</h4><p>First, you’ll need to create a private S3 bucket in your AWS console. Make sure it’s set to private to ensure the security of your stored files.</p><h4>Step 2: Set Up User Access</h4><p>Create a user that can access the bucket. You’ll need to attach a policy to this user. Here’s an example policy that grants the necessary permissions:</p><pre>{<br>  &quot;Version&quot;: &quot;2012-10-17&quot;,<br>  &quot;Statement&quot;: [<br>    {<br>      &quot;Sid&quot;: &quot;VisualEditor0&quot;,<br>      &quot;Effect&quot;: &quot;Allow&quot;,<br>      &quot;Action&quot;: [<br>          &quot;s3:PutObject&quot;,<br>          &quot;s3:GetObject&quot;,<br>          &quot;s3:DeleteObject&quot;<br>        ],<br>        &quot;Resource&quot;: &quot;arn:aws:s3:::unique-bucket-name/*&quot;<br>    }<br>  ]<br>}</pre><p>Replace unique-bucket-name with your actual bucket name.</p><h4>Step 3: Create API Access and Secret Key</h4><p>After creating the user, generate an API Access Key and Secret Key. You’ll need these to authenticate your application with AWS.</p><h3>Implementing S3 in Node.js</h3><h4>Installing Required Packages</h4><p>First, install the necessary npm packages:</p><pre>npm i @aws-sdk/client-s3<br>npm i @aws-sdk/s3-request-presigner</pre><h4>Configuring the S3 Client</h4><p>In your Node.js application, set up the S3 client:</p><pre>const { S3Client } = require(&quot;@aws-sdk/client-s3&quot;);<br><br>const s3Client = new S3Client({<br>  region: bucket_region,<br>  credentials: {<br>    accessKeyId: bucket_access_key,<br>    secretAccessKey: bucket_secret_key,<br>  },<br>});</pre><h3>File Operations with S3</h3><h4>Storing Files (POST)</h4><p>Here’s how to upload a file to S3:</p><pre>const { PutObjectCommand } = require(&quot;@aws-sdk/client-s3&quot;);<br><br>const fileName = `uploads/${getRandomFileName()}-${req.file.originalname}`;<br><br>const command = new PutObjectCommand({<br>  Bucket: bucket_name,<br>  Key: fileName,<br>  Body: req.file.buffer,<br>  ContentType: req.file.mimetype,<br>});<br><br>const response = await s3Client.send(command);</pre><h4>Retrieving Files (GET)</h4><p>To get a signed URL for a file:</p><pre>const { GetObjectCommand } = require(&quot;@aws-sdk/client-s3&quot;);<br>const { getSignedUrl } = require(&quot;@aws-sdk/s3-request-presigner&quot;);<br><br>const command = new GetObjectCommand({<br>  Bucket: bucket_name,<br>  Key: object_key,<br>});<br>const url = await getSignedUrl(s3Client, command, { expiresIn: 3600 });</pre><h4>Deleting Files (DELETE)</h4><p>To delete a file from S3:</p><pre>const { DeleteObjectCommand } = require(&quot;@aws-sdk/client-s3&quot;);<br><br>const command = new DeleteObjectCommand({<br>  Bucket: bucket_name,<br>  Key: object_key,<br>});<br><br>await s3Client.send(command);</pre><h3>Why Amazon CloudFront?</h3><p>Amazon CloudFront is a global Content Delivery Network (CDN) that significantly enhances our application’s performance and user experience. By caching content at edge locations worldwide, CloudFront reduces latency and improves load times for users across the globe. It efficiently handles traffic spikes, enhances security with built-in SSL/TLS and DDoS protection, and can lower infrastructure costs by reducing the load on origin servers. CloudFront’s ability to customize content delivery makes it an ideal choice for optimizing our MERN stack application’s content distribution, ensuring fast, secure, and reliable access for users regardless of their location.</p><h3>Integrating CloudFront</h3><p>CloudFront acts as a global CDN, caching your content at edge locations worldwide for faster delivery.</p><h4>Configuring CloudFront in Node.js</h4><p>First, install the CloudFront client:</p><pre>npm i @aws-sdk/client-cloudfront</pre><p>Then, set it up in your application:</p><pre>const { CloudFrontClient } = require(&quot;@aws-sdk/client-cloudfront&quot;);<br><br>const cloudFront = new CloudFrontClient({<br>  region: bucket_region,<br>  credentials: {<br>    accessKeyId: bucket_access_key,<br>    secretAccessKey: bucket_secret_key,<br>  },<br>});</pre><h4>Updating URL Generation</h4><p>With CloudFront, your URL structure changes:</p><pre>const url = `${cloudfront_url}/${object_name}`;</pre><h4>Cache Invalidation</h4><p>After deleting an object from S3, you should invalidate the CloudFront cache:</p><pre>const { CreateInvalidationCommand } = require(&quot;@aws-sdk/client-cloudfront&quot;);<br><br>const invalidationCommand = new CreateInvalidationCommand({<br>  DistributionId: cloudfront_distribution_id,<br>  InvalidationBatch: {<br>    CallerReference: image_key,<br>    Paths: {<br>      Quantity: 1,<br>      Items: [`/${image_key}`],<br>    },<br>  },<br>});<br><br>await cloudFront.send(invalidationCommand);</pre><h3>Simpler Usage: Multer S3</h3><p>For a more easier approach to file uploads, you can use Multer S3:</p><pre>const multerS3 = require(&#39;multer-s3&#39;);<br><br>const upload = multer({<br>  storage: multerS3({<br>    s3: s3Client,<br>    bucket: bucket_name,<br>    metadata: function (req, file, cb) {<br>      cb(null, { fieldName: file.fieldname });<br>    },<br>    key: function (req, file, cb) {<br>      const filename = getRandomFileName();<br>      cb(null, `uploads/test/${filename}-`);<br>    },<br>  }),<br>});</pre><p>Using multerS3 for file uploads simplifies implementation and saves development time, but it comes with trade-offs. While it&#39;s great for simpler use cases, it limits control compared to manually handling uploads with the AWS SDK. Developers should balance convenience with the need for flexibility when choosing the best approach for their project.</p><h3>Conclusion</h3><p>Implementing AWS S3 and CloudFront in your MERN stack application can significantly improve your file storage capabilities and content delivery speed. By following this guide, you can set up a robust system for handling file uploads, storage, and delivery in your application.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=7e13d9472974" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Docker: Ending the “It Works on My Machine” Era]]></title>
            <link>https://medium.com/@danishsidd203/docker-ending-the-it-works-on-my-machine-era-b90a3f933ae9?source=rss-7d9b62c475b8------2</link>
            <guid isPermaLink="false">https://medium.com/p/b90a3f933ae9</guid>
            <category><![CDATA[docker]]></category>
            <category><![CDATA[dockerfiles]]></category>
            <category><![CDATA[docker-compose]]></category>
            <category><![CDATA[containers]]></category>
            <category><![CDATA[containerization]]></category>
            <dc:creator><![CDATA[Danish Siddiqui]]></dc:creator>
            <pubDate>Fri, 02 Aug 2024 21:53:16 GMT</pubDate>
            <atom:updated>2024-08-02T21:53:16.651Z</atom:updated>
            <content:encoded><![CDATA[<p>In the world of software development, we’ve all heard (or said) the infamous phrase, “But it works on my machine!” This common frustration highlights a persistent challenge: ensuring software runs consistently across different environments. That’s where Docker comes into play, by transforming how we create, distribute, and run applications.</p><figure><img alt="Docker | Shipping your machine" src="https://cdn-images-1.medium.com/max/659/1*A_2KNHazhddgSqHkrt0M3g.png" /></figure><p>Imagine Docker as an open-source universal packaging system for software. Instead of sending your colleague a list of instructions to set up their environment, you’re handing them a complete, self-contained package that includes your application and everything it needs to run correctly.</p><p>At its heart, Docker relies on containers — imagine them as compact, portable, self-contained units that can be run anywhere Docker is set up. These containers are like virtual packages that include everything your application needs to function, from the code itself to the necessary libraries and settings, all neatly bundled together.</p><p>The popularity of Docker stems from its ability to solve common development and operations challenges, such as the “it works on my machine” problem, and its seamless integration with cloud platforms and Dev-Ops practices.</p><h3>Benefits of Using Docker</h3><ol><li><strong>Consistency</strong>: Docker ensures that your application runs the same way in development, testing, and production environments.</li><li><strong>Isolation</strong>: Containers are isolated from each other and the host system, reducing conflicts between different components or applications.</li><li><strong>Portability</strong>: Docker containers can run on any system that supports Docker, regardless of the underlying operating system.</li><li><strong>Efficiency</strong>: Containers are lightweight and start quickly, allowing for rapid deployment and scaling.</li><li><strong>Version Control</strong>: Docker images can be versioned, making it easy to roll back to previous versions if needed.</li><li><strong>Micro-services Architecture</strong>: Docker facilitates the implementation of micro-services, allowing complex applications to be broken down into smaller, manageable components.</li></ol><h3>The core concepts of Docker can be summarized as:</h3><ol><li><strong>Image</strong>: A lightweight, standalone, and executable package that includes everything needed to run a piece of software.</li><li><strong>Container</strong>: A runtime instance of an image.</li></ol><h3>Getting Started with Docker</h3><p>To begin your Docker journey, follow these steps:</p><ol><li><strong>Install Docker</strong>: Visit the official Docker website and download Docker Desktop for your operating system.</li><li><strong>Understand Basic Concepts</strong>: Familiarize yourself with key Docker concepts like images, containers, and Dockerfiles.</li><li><strong>Learn Essential Commands</strong>: Start with basic Docker commands to build images, run containers, and manage your Docker environment.</li><li><strong>Create a Dockerfile</strong>: Learn to write Dockerfiles to define your application’s environment.</li><li><strong>Use Docker Compose</strong>: For multi-container applications, learn to use Docker Compose to define and run your application stack.</li></ol><p>Let’s dive into some essential Docker commands and concepts:</p><h4>Building a Docker Image</h4><p>To build a Docker image, you create a Dockerfile and use the docker build command:</p><pre>docker build -t [image_name] .</pre><h4>Running a Container</h4><p>To run a container from your image:</p><pre>docker run -p 5173:5173 [image_name]</pre><p>This command runs the container and maps port 5173 from the container to port 5173 on your host machine.</p><h4>Managing Containers</h4><p>Some useful commands for managing containers:</p><pre>docker ps  # List running containers<br>docker ps -a  # List all containers<br>docker stop [container_id/name]  # Stop a running container<br>docker rm [container_id/name] --force  # Remove a container</pre><h4>Working with Images</h4><p>To manage your Docker images:</p><pre>docker images  # List all images<br>docker rmi [image_id] --force  # Remove an image</pre><h4>Publishing Your Docker Image</h4><p>To share your Docker image with others:</p><pre>docker login<br>docker tag [image_name] docker_username/image_name<br>docker push docker_username/image_name</pre><h4>Using Docker Compose</h4><p>Docker Compose allows you to define and run multi-container Docker applications. Create a compose.yaml file to specify your services, then use:</p><pre>docker compose up --watch</pre><p>This command starts your services and watches for changes, automatically rebuilding when necessary.</p><h4>DockerFile Example To Containerize a Next.js Application:</h4><pre>ARG NODE_VERSION=22.2.0<br><br>FROM node:${NODE_VERSION}-alpine<br><br>WORKDIR /app<br><br>COPY package*.json ./<br><br>RUN npm install<br><br>COPY . .    <br><br>EXPOSE 3000<br><br>CMD npm run dev</pre><h4>Docker Compose Example To Containerize a Next.Js Application:</h4><pre>version: &#39;3.8&#39;<br><br>services:<br>  web:<br>    build:<br>      context: .<br>      dockerfile: Dockerfile<br>    ports:<br>      - &quot;3000:3000&quot;<br>    develop:<br>      watch:<br>        - path: ./package.json<br>          action: rebuild<br>        - path: ./next.config.js<br>          action: rebuild<br>        - path: ./package-lock.json<br>          action: rebuild<br>        - path: .<br>          target: /app<br>          action: sync<br><br>    environment:<br>      DB_URL: YOUR_MONGO_ATLAS_URL<br><br>volumes:<br>  tasked:</pre><h3>Advanced Docker Usage</h3><p>As you become more comfortable with Docker, you can explore advanced topics like:</p><ol><li>Multi-stage builds for optimizing your images</li><li>Docker networking for complex multi-container applications</li><li>Docker volumes for persistent data storage</li><li>Docker Swarm or Kubernetes for container orchestration</li><li>Continuous Integration/Continuous Deployment (CI/CD) pipelines with Docker</li><li>Docker Scout for enhanced container security</li></ol><h3>Conclusion:</h3><p>Docker has reshaped the software development landscape, offering a powerful way to build, deploy, and run applications. By ensuring consistency across different environments, boosting efficiency, and enabling modern architectural practices, Docker has become an essential tool for both developers and operations teams.</p><p>As you begin your Docker journey, remember that practice is key to mastering this technology. Start with simple applications and gradually tackle more complex projects as you grow comfortable with Docker’s ecosystem. With its vibrant community and extensive documentation, you’ll have plenty of resources to support your learning and help you fully maximize Docker’s potential in your projects.</p><p>Happy Dockerizing! ✨</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=b90a3f933ae9" width="1" height="1" alt="">]]></content:encoded>
        </item>
    </channel>
</rss>