<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Sanjeev Gautam on Medium]]></title>
        <description><![CDATA[Stories by Sanjeev Gautam on Medium]]></description>
        <link>https://medium.com/@me.sanjeev3d?source=rss-89634bf8d93a------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/1*DCoVfJwO4AFc40bm462pmg.jpeg</url>
            <title>Stories by Sanjeev Gautam on Medium</title>
            <link>https://medium.com/@me.sanjeev3d?source=rss-89634bf8d93a------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Thu, 19 Nov 2026 11:20:56 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@me.sanjeev3d/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[Hierarchy of AI Machine Learning and Deep Learning]]></title>
            <link>https://medium.com/@me.sanjeev3d/hierarchy-of-ai-machine-learning-and-deep-learning-d5ea1350fb6f?source=rss-89634bf8d93a------2</link>
            <guid isPermaLink="false">https://medium.com/p/d5ea1350fb6f</guid>
            <category><![CDATA[supervised-learning]]></category>
            <category><![CDATA[online-learning]]></category>
            <category><![CDATA[artificial-intelligence]]></category>
            <category><![CDATA[deep-learning]]></category>
            <category><![CDATA[machine-learning]]></category>
            <dc:creator><![CDATA[Sanjeev Gautam]]></dc:creator>
            <pubDate>Sat, 28 Feb 2026 10:30:07 GMT</pubDate>
            <atom:updated>2026-02-28T10:44:01.184Z</atom:updated>
            <content:encoded><![CDATA[<h3>Hierarchy of Machine Learning and Deep Learning</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*dBmm1T64xVpcuhOPvs_Tag.png" /></figure><p>The relationship between Artificial Intelligence, Machine Learning, and Deep Learning can be visualized as three concentric circles: <strong>the outermost circle is AI, the middle circle is ML (a subset of AI), and the innermost circle is DL (a subset of ML)</strong>.</p><p><strong>Artificial Intelligence (AI)</strong></p><ul><li><strong>Goal:</strong> The main objective of AI is to download or replicate human intelligence into machines.</li><li><strong>Current State:</strong> Human intelligence is highly complex, encompassing puzzle-solving, coding, creativity, imagination, and emotional intelligence. Because things like creativity and imagination are difficult to quantify or measure, <strong>current AI focuses on specific, narrow tasks</strong> rather than achieving human-like “General Intelligence”.</li><li><strong>Symbolic AI &amp; Expert Systems (Early AI):</strong> Emerging around the 1950s, early AI relied on <strong>Symbolic AI</strong>, which involved building massive knowledge bases governed by explicit logical rules. Developers built <strong>Expert Systems</strong> (like early chess-playing programs) by extracting knowledge from human experts and hard-coding it.</li><li><strong>The Flaw:</strong> Expert systems failed when faced with “fuzzy logic” problems, such as identifying a dog in an image. It is impossible to manually write if-else rules for every possible physical feature or variation of a dog.</li></ul><p><strong>Machine Learning (ML)</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*EP-I4rz3ntSK3Fqm5WasLQ.png" /></figure><p>ML is a branch of computer science that uses statistical techniques to find mathematical patterns in data.</p><ul><li><strong>The Paradigm Shift:</strong> The biggest difference between ML and Symbolic AI is that <strong>ML requires no explicit programming</strong>. Instead of giving a machine the data and the rules to get an output, you <strong>give the machine the data and the expected output, and it automatically generates the rules</strong>.</li><li><strong>Learning Process:</strong> If you want a system to identify a dog, you feed it thousands of photographs labeled as “dog” or “not dog.” The system automatically tries to find the underlying patterns (learning), eventually becoming able to classify new images.</li><li><strong>Recent Boom:</strong> While the concepts are decades old, ML has become revolutionary in recent years due to the massive availability of data and powerful hardware.</li></ul><p><strong>Deep Learning (DL)</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*T245aPzgs7P9fPlbc9fQpQ.png" /></figure><p>DL is not a completely different procedure from ML; the core steps (feeding data, training the algorithm, reducing errors, and predicting) remain exactly the same.</p><ul><li><strong>Inspiration:</strong> DL is inspired by biology and the human brain, utilizing mathematical units called <strong>perceptrons</strong> (artificial neurons). However, they are fundamentally mathematical models and do not perfectly replicate how a biological brain functions.</li></ul><p><strong>Why Deep Learning? (Key Differences from ML):</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*mkNByyO3EY0x3MP60GPXZg.png" /></figure><ol><li><strong>Automatic Feature Extraction:</strong> In traditional ML, humans must explicitly provide “features” for the model (e.g., telling it to look for a pointed snout or tail to identify a dog, or specific certifications to predict a candidate’s job placement). <strong>In DL, the system automatically extracts these features from raw data</strong>, which is essential when working with unstructured data where you don’t know what features to look for.</li><li><strong>Layered Learning:</strong> DL uses multiple layers of neurons. As you add layers, the system’s prediction power improves. For example, the first layer might detect edges, the second layer combines those edges, and the third layer identifies the final object or digit.</li><li><strong>Scalability with Data:</strong> Traditional ML models eventually plateau, meaning their accuracy stops improving even if you add more data. In contrast, <strong>Deep Learning models continuously improve in performance as you feed them more data</strong>.</li></ol><p><strong>When to use ML vs. Deep Learning?</strong></p><ul><li><strong>Use Deep Learning:</strong> When you are tackling complex tasks such as <strong>image classification, object detection, or text-related tasks</strong>, and you have access to a massive amount of data.</li><li><strong>Use Machine Learning:</strong> When you have a <strong>smaller dataset</strong>. In many real-world industries like banking and insurance, companies do not always have massive amounts of data; in these scenarios, traditional Machine Learning is highly useful and will provide better results than Deep Learning</li></ul><p><strong>Types of Machine Learning (Based on Supervision)</strong> Machine learning algorithms can be divided into four main categories based on the amount of external supervision they require to get trained.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*ggKVETeklRAJbMSAM1Ef7A.png" /></figure><p><strong>1. Supervised Machine Learning</strong></p><p>The algorithm learns from a dataset that contains both <strong>input</strong> and <strong>output</strong> data.</p><ul><li><strong>Goal:</strong> The system learns the mathematical relationship between the input and output so that it can accurately predict the output for future, unseen inputs.</li><li><strong>Example:</strong> A bank’s system is trained using millions of past transactions that are already labeled as either <strong>“Legitimate”</strong> or <strong>“Fraudulent.”</strong> The algorithm learns the mathematical patterns associated with each label — such as the transaction amount, geographic location, and time of day. When a new transaction occurs, the system compares it against these learned patterns to immediately flag it as safe or suspicious.</li></ul><p><strong>Sub-categories (Based on Data Type):</strong></p><ul><li><strong>Regression:</strong> Predicting the <strong>exact dollar amount</strong> of a user’s next monthly credit card bill.</li><li><strong>Classification:</strong> Deciding if a specific transaction is <strong>Fraud (Yes)</strong> or <strong>Not Fraud (No)</strong>.</li></ul><p><strong>2. Unsupervised Machine Learning</strong></p><p>The dataset contains only <strong>input</strong> data and no output data or labels. Because there are no provided answers to learn from, there is no direct prediction task.</p><ul><li><strong>Sub-categories:</strong></li><li><strong>Clustering:</strong> Grouping data points based on hidden similarities. <em>Example: Customer segmentation in e-commerce, where customers are divided into distinct groups based on purchasing behavior without prior labeling.</em></li><li><strong>Dimensionality Reduction:</strong> Reducing the number of input columns (features) while keeping the essential information intact. This prevents algorithms from slowing down, removes unnecessary columns (feature extraction), and allows high-dimensional data (like 784-dimension images) to be visualized in 2D or 3D graphs.</li><li><strong>Anomaly Detection:</strong> Identifying abnormal data points or outliers that deviate from the norm. <em>Examples: Detecting manufacturing defects, credit card fraud, or unusual loan approvals.</em></li><li><strong>Association Rule Learning:</strong> Finding hidden patterns or rules indicating how items are associated with each other. <em>Example: Arranging products in a supermarket based on items frequently bought together (e.g., Walmart placing beer next to baby diapers after discovering a correlation).</em></li></ul><p><strong>3. Semi-Supervised Machine Learning</strong></p><p>A combination approach used because manually labeling massive amounts of data is expensive and time-consuming.</p><ul><li><strong>How it works:</strong> It uses unsupervised learning to group similar data points, and then allows you to label just one or two points in a group to automatically label the rest.</li><li><strong>Example:</strong> Google Photos automatically clustering pictures of the same person together (unsupervised) and then asking you to provide the person’s name once to label the entire cluster (semi-supervised).</li></ul><p><strong>4. Reinforcement Learning</strong></p><p>The algorithm (called an “agent”) starts from scratch without any pre-existing dataset.</p><ul><li><strong>How it works:</strong> The agent interacts with its “environment” and learns through trial and error. It operates on a system of rewards and punishments based on a set “policy” (rulebook), constantly updating its policy to maximize positive rewards and minimize punishments.</li><li><strong>Examples:</strong> Self-driving cars learning to navigate roads, or AI playing complex games. DeepMind’s AlphaGo agent used reinforcement learning to defeat the world champion at the highly complex game of Go</li></ul><p><strong>Batch (Offline) Machine Learning vs. Online Machine Learning</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*i4do9r0hWeOxqTMTPDqupA.jpeg" /></figure><p><strong>Part 1: Batch Machine Learning (Offline Learning)</strong></p><p><strong>Definition &amp; Process</strong></p><ul><li><strong>Batch Learning:</strong> The conventional method where the machine learning algorithm is trained using the <strong>entire dataset at once</strong>.</li><li><strong>No Incremental Learning:</strong> The model does not learn step-by-step; it processes data in one massive batch.</li><li><strong>Offline Training:</strong> Because training on large datasets is computationally expensive, it is done offline on local systems. Once fully trained, the static model is deployed to the production server to make predictions.</li></ul><p><strong>The Core Problem: Static Models</strong></p><ul><li>Once deployed, the model is completely static and only knows the data it was originally trained on.</li><li>If business scenarios evolve (e.g., Netflix adding new movies, or spammers inventing new tactics), the model will become obsolete.</li><li><strong>The Workaround:</strong> Developers must constantly combine new data with the old data and <strong>retrain the model from scratch</strong> at set intervals (e.g., every 24 hours, weekly, or monthly) to keep it updated.</li></ul><p><strong>Major Disadvantages</strong></p><ul><li><strong>Data Overload:</strong> As datasets grow exponentially, existing hardware and tools may fail to process the massive chunks of data all at once.</li><li><strong>Hardware/Connectivity Limits:</strong> Batch learning fails in remote or low-connectivity environments (e.g., defense applications in Ladakh, space satellites, or moving trains) because you cannot easily fetch the model, retrain it, and push updates.</li><li><strong>Inability to Catch Sudden Trends:</strong> Because models update on a delay (e.g., 24 hours), they cannot adapt to sudden spikes in user interest caused by breaking news (e.g., sudden government announcements), making the recommendations irrelevant by the time the model finally updates.</li></ul><p><strong>Part 2: Online Machine Learning</strong></p><p><strong>Definition &amp; Process</strong></p><ul><li><strong>Online Learning:</strong> The model is trained <strong>incrementally</strong> by feeding it small chunks or “mini-batches” of data sequentially.</li><li><strong>Dynamic Learning:</strong> The training happens dynamically on the fly while the model is live on the production server. As new data comes in, the model continuously learns and improves its performance.</li></ul><p><strong>Real-World Examples</strong></p><ul><li><strong>Chatbots &amp; Voice Assistants:</strong> Systems like Google Now and Alexa continuously learn and improve based on user interactions.</li><li><strong>Smart Keyboards:</strong> Applications like SwiftKey adapt dynamically to a user’s typing style over time.</li><li><strong>YouTube Recommendations:</strong> The recommendation feed changes instantly based on the specific video you just clicked and watched, demonstrating real-time online learning.</li></ul><p><strong>When to Use Online Learning</strong></p><ul><li><strong>Concept Drift:</strong> Ideal for scenarios where the nature of the data or user behavior changes rapidly over time (e.g., e-commerce buying trends).</li><li><strong>Cost Efficiency &amp; Speed:</strong> Training in small, continuous batches is often faster and saves on massive computational costs compared to retraining huge offline datasets.</li></ul><p><strong>Implementation &amp; Tools</strong></p><ul><li><strong>Algorithms:</strong> Requires algorithms that support “partial fitting,” such as SGDRegressor in the Scikit-Learn library, which allows training to continue from where it left off.</li><li><strong>Dedicated Libraries:</strong> Developers use specialized libraries like <strong>River</strong> (created by merging two previous libraries) and <strong>Vowpal Wabbit</strong> to implement online machine learning effectively.</li></ul><p><strong>Important Concepts &amp; Risks</strong></p><ul><li><strong>Out-of-Core Learning:</strong> This is an offline technique that uses the <em>principles</em> of online learning. It is used when a dataset is too massive to fit into computer memory (e.g., 50GB data on an 8GB RAM machine). The data is broken into small chunks and fed sequentially to train the model.</li><li><strong>The Learning Rate:</strong> A crucial setting that dictates how fast the model learns new information versus how quickly it forgets old data. If set incorrectly, the model will misbehave.</li><li><strong>The Risk of Bad Data:</strong> Because the model updates dynamically on the server, if bad, faulty, or hacked data is fed into the system, the model will quickly become biased or corrupted.</li><li><strong>Mitigation:</strong> To prevent data corruption, developers must build strict anomaly detection systems and have fallback options to instantly take the system offline and revert to a previous safe state.</li></ul><p><strong>Summary Comparison</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*ZiIe-a32P251xkUAPioeVg.png" /></figure><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=d5ea1350fb6f" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[GCP:3-Cloud Load balancer]]></title>
            <link>https://medium.com/@me.sanjeev3d/gcp-3-cloud-load-balancer-be3daa3ae810?source=rss-89634bf8d93a------2</link>
            <guid isPermaLink="false">https://medium.com/p/be3daa3ae810</guid>
            <category><![CDATA[google-cloud]]></category>
            <category><![CDATA[gcp]]></category>
            <category><![CDATA[load-balancing]]></category>
            <category><![CDATA[gcp-load-balancer]]></category>
            <category><![CDATA[load-balancer]]></category>
            <dc:creator><![CDATA[Sanjeev Gautam]]></dc:creator>
            <pubDate>Thu, 25 Jul 2024 17:41:06 GMT</pubDate>
            <atom:updated>2024-07-25T17:49:39.429Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/664/1*idbPUgqPUcTwybJAum7WGQ.png" /></figure><p><strong>Exploring Cloud Load Balancing for VM Instances</strong></p><p><strong>What is Cloud Load Balancing?</strong></p><p>A cloud load balancer distributes user traffic across instances of an application within a single region or across multiple regions. Whether you have multiple VM instances in various zones or regions, a cloud load balancer can distribute traffic among them. This is a fully distributed, software-defined managed service. A reliable cloud platform ensures that this service is highly available and replicated as needed.</p><p><strong>Key Features of Cloud Load Balancing</strong></p><p><strong>Health Checks:</strong></p><p>A crucial feature of any load balancer is the ability to perform health checks. These checks allow the load balancer to determine if an instance is healthy. With health checks, cloud load balancing can route traffic only to healthy instances, helping to recover from failures quickly.</p><p><strong>Auto Scaling:</strong></p><p>Based on the incoming traffic, cloud load balancing can automatically scale. This ensures that your application can handle varying traffic loads without manual intervention.</p><p><strong>Single Anycast IP:</strong></p><p>Cloud load balancing provides a single Anycast IP, which can be used to receive traffic from multiple regions globally. This allows you to serve a global audience using one specific IP address.</p><p><strong>Internal Load Balancing:</strong></p><p>Cloud load balancing supports internal load balancing, enabling load distribution for applications deployed within a specific network.</p><p><strong>Why Use Cloud Load Balancing?</strong></p><p><strong>High Availability:</strong></p><p>Cloud load balancing ensures high availability. Even if instances go down and come up, the load balancer ensures that users are not affected by distributing traffic only among healthy instances.</p><p><strong>Autoscaling:</strong></p><p>Cloud load balancing supports autoscaling based on the number of requests. It works with backend services like compute engines or instance groups, which can dynamically scale up or down.</p><p><strong>Resiliency:</strong></p><p>With health checks in place, the load balancer can detect when an instance is down and redirect traffic to healthy instances. This ensures that your application remains resilient and available to users.</p><p><strong>Loosely Coupled Architecture:</strong></p><p>By using a cloud load balancer, you can create a loosely coupled architecture. This means that your system components are independent, improving the overall flexibility and robustness of your application.</p><p><strong>Multi-Zone and Multi-Region Support:</strong></p><p>One standout feature is the ability to distribute load across instances in multiple zones and regions. This not only enhances availability but also ensures that your application can handle traffic from different geographical locations effectively.</p><p><strong>Understanding Communication Protocols in Virtual Servers and Load Balancers</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/482/1*WiQHpREmSP61ACrygm9OIg.png" /></figure><p>In the realm of virtual servers and load balancers, communication between systems is crucial. This communication happens through various protocols like HTTP, HTTPS, TCP, TLS, and UDP. Understanding these protocols is essential for anyone working with cloud infrastructure. In this article, we’ll explore these protocols and their roles in system communication.</p><p><strong>Why Do We Need Protocols?</strong></p><p>Protocols are to computers what languages are to humans. Just as we use structured languages to communicate, computers use protocols to exchange information. These protocols operate at different layers of the network communication model, ensuring that data is transmitted accurately and securely.</p><p><strong>The Layered Approach to Communication</strong></p><p>Communication between systems occurs over multiple layers: the Network Layer, the Transport Layer, and the Application Layer. Each layer has a specific role in the process.</p><p><strong>Network Layer</strong></p><p>The Network Layer is responsible for transferring bits and bytes (zeros and ones) across the network. This layer handles the basic transfer of data but doesn’t guarantee delivery or order.</p><p><strong>Transport Layer</strong></p><p>The Transport Layer ensures that the data sent by one system is received correctly by another system. It deals with issues like data loss and ensures reliable transmission through protocols like TCP and TLS.</p><p><strong>Application Layer</strong></p><p>The Application Layer is where most applications operate. It uses higher-level protocols like HTTP and HTTPS to facilitate communication for web applications, emails, file transfers, and more.</p><p><strong>Key Protocols and Their Functions</strong></p><p><strong>Internet Protocol (IP):</strong> Operates at the Network Layer, responsible for routing data packets to their destinations. It’s unreliable on its own, which is why higher layers provide additional reliability.</p><p><strong>Transmission Control Protocol (TCP):</strong> Works at the Transport Layer, ensuring reliable data transmission by verifying that data packets arrive correctly and in order.</p><p><strong>Transport Layer Security (TLS):</strong> A secure version of TCP, encrypting data to protect it during transmission over the network.</p><p><strong>User Datagram Protocol (UDP):</strong> Also at the Transport Layer, UDP prioritizes performance over reliability. It’s used in applications like video streaming where occasional data loss is acceptable for faster transmission.</p><p><strong>Hypertext Transfer Protocol (HTTP):</strong> An Application Layer protocol used for web communications. It follows a stateless request-response model.</p><p><strong>Hypertext Transfer Protocol Secure (HTTPS):</strong> A secure version of HTTP that uses TLS to encrypt data, ensuring secure communication over the internet.</p><p><strong>Why Understanding Protocols Matters</strong></p><p>Protocols play a critical role in ensuring that applications run smoothly and securely. By understanding the layered approach and the specific functions of each protocol, you can design and manage systems that are both robust and efficient.</p><p><strong>High Performance Applications:</strong></p><p>For applications requiring high performance, such as gaming or live video streaming, UDP may be preferred due to its low-latency characteristics despite its lower reliability compared to TCP.</p><p><strong>Web Applications:</strong></p><p>Most web applications use HTTP/HTTPS, leveraging TCP/TLS for reliable and secure data transmission. This ensures that web pages load correctly and securely, providing a safe user experience.</p><p><strong>Creating Your First Load Balancer: A Step-by-Step Guide</strong></p><p><strong>Introduction</strong></p><p>Are you excited to create your first load balancer? Let’s dive in and set up a load balancer to distribute the load between all instances in your Managed Instance Group.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*MLFn2i5cD7kswk5_ns0Dxw.png" /></figure><p><strong>Step 1: Access Load Balancing</strong></p><p>First, navigate to the load balancing section in your cloud platform. Load balancing is part of network services, and it’s where you can create a load balancer to distribute incoming traffic among multiple VM instances.</p><p><strong>Step 2: Choose the Type of Load Balancer</strong></p><p>Click on “Create Load Balancer” to see the different types available:</p><p>• <strong>HTTPS Load Balancers:</strong> Ideal for secure web applications.<br>• <strong>TCP Load Balancers:</strong> Used for applications needing high performance, like gaming.<br>• <strong>UDP Load Balancers:</strong> Suitable for applications using the UDP protocol. Note that UDP load balancing is restricted to a single region.</p><p>You can create either an internet-facing or an internal load balancer, depending on whether external users on the internet or internal network users will use it.</p><p><strong>Step 3: Configure the Load Balancer</strong></p><p>Select the type of load balancer that suits your needs. For this guide, we’ll focus on creating an HTTPS load balancer.</p><p>1. <strong>Traffic Source:</strong> Choose “Traffic from Internet” to balance traffic from external users.<br>2. <strong>Load Balancer Version:</strong> Opt for the classic HTTPS load balancer for stability, as the new version is still in preview.</p><p><strong>Step 4: Backend Configuration</strong></p><p>Three main configurations are needed: backend, host and path rules, and front end.</p><p><strong>Backend</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*0FeD-oUmBU9Hqom7MoMsTA.png" /></figure><p>1. Select “Create a Backend Service.”<br>2. Name it (e.g., My Instance Group Backend Service).<br>3. Choose “Instance Group” as the backend type and select your managed instance group.<br>4. Set the protocol to HTTP and the port to 80.<br>5. Choose the balancing mode (default is utilization-based).</p><p>You can also add multiple backends if using a microservices architecture.</p><p><strong>Host and Path Rules</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*jxcJIntT8jVNRynoJN3BDw.png" /></figure><p>1. Configure how traffic will be redirected to your backend.<br>2. For a simple setup, leave the host and path empty to redirect all requests to the instance group.</p><p>Advanced configurations like URL redirects can be set up as needed.</p><p><strong>Step 5: Front End Configuration</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*zY_6Z71j1fIADMLvzGEzTg.png" /></figure><p>1. Set the protocol to HTTP and the port to 80.<br>2. Review the configuration to ensure everything is correct.</p><p><strong>Step 6: Finalize and Create<br></strong>1. Give your load balancer a name (e.g., My HTTP Load Balancer).<br>2. Click “Create.”</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*KpefcAz9UIHfuiF4CESSqw.png" /></figure><p>The creation process typically takes 8 to 10 minutes.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*naWxuQXIk8eql3V9m3Y0rQ.png" /></figure><p><strong>SSL/TLS Termination</strong></p><p>SSL termination (or SSL offloading) and TLS termination (or TLS offloading) are important concepts in load balancing. These processes involve handling the secure communication between clients and the load balancer.</p><p>• <strong>SSL Termination (Layer 7):</strong> When exposing an application on the internet, HTTPS is recommended for secure communication between the client and the load balancer. The load balancer handles the SSL certificate, ensuring secure communication. Between the load balancer and the VM instance, communication occurs over HTTP, as it traverses the Google internal network. This reduces the load on instances, as they do not need to handle HTTPS.</p><p>• <strong>TLS Termination (Layer 4):</strong> Similar to SSL termination, but used for layer 4 communications. The client and the load balancer communicate using TLS, while communication between the load balancer and the VM instance happens over TCP.</p><p>The advantage of SSL/TLS termination is reducing the load on instances, allowing them to handle requests directly without worrying about HTTPS or TLS.</p><p><strong>How to Choose the Right Load Balancer</strong></p><p>Choosing the right load balancer can significantly impact the efficiency and reliability of your network traffic management.</p><p><strong>Internal vs. External Traffic<br></strong>First, determine whether your traffic is internal or external. Internal traffic refers to traffic within a network, while external traffic comes from the internet.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*g9N1G_kbTEbi9QcPT0gZ_Q.png" /></figure><p><strong>Internal Traffic Load Balancing<br></strong>For internal traffic, the choice of load balancer depends on the type of traffic:</p><p>1. <strong>TCP or UDP Traffic</strong>: Use Internal TCP/UDP Load Balancing.<br>2. <strong>HTTP or HTTPS Traffic</strong>: Use Internal HTTPS Load Balancing.</p><p><strong>External Traffic Load Balancing</strong></p><p>For external traffic, the decision-making process is more complex. Let’s break it down:<br><strong>Type of Traffic</strong>: Determine whether the traffic is HTTP/HTTPS, TCP, UDP, or ICMP.</p><p><strong>HTTP/HTTPS Traffic</strong></p><p>If your external traffic is HTTP or HTTPS, consider the following:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*VqYyqnNOK8k-N2ejAqvQUA.png" /></figure><p>• <strong>New HTTPS Load Balancer</strong>: This provides advanced traffic management features. Choose between a global or regional load balancer:<br>• <strong>Regional HTTPS Load Balancer</strong>: Use this for traffic specific to a region.<br>• <strong>Global HTTPS Load Balancer</strong>: Use this for global traffic.</p><p>The new version offers advanced features such as request/response header modification, retrying failed requests, traffic splitting, and traffic metering.<br>• <strong>Classic HTTPS Load Balancer</strong>: This is the older version and can be used for global traffic as well.</p><p><strong>TCP Traffic</strong></p><p>For external TCP traffic, consider these options:<br>• <strong>SSL Proxy</strong>: Use this if you need SSL offloading.<br>• <strong>TCP Proxy</strong>: Use this for global load balancing or IPv6 load balancing. If you need to preserve client IPs, choose Network Load Balancing.</p><p><strong>UDP Traffic</strong></p><p>For external UDP traffic, the recommended option is External Network Load Balancing. This also applies to ESP or ICMP traffic.</p><p><strong>Features:</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*JpTPMvteqhix3Ai5tLHTEg.png" /></figure><p><strong>Key Differences Between Proxy and Pass-Through Load Balancers</strong></p><p><strong>Proxy Load Balancers</strong>:<br>• Transform requests before sending to the backend.<br>• Backend cannot see client details such as IP addresses.<br>• Suitable for scenarios requiring request modifications or SSL offloading.</p><p><strong>Pass-Through Load Balancers</strong>:<br>• Forward requests directly to the backend without modification.<br>• Backend can see client details such as IP addresses.<br>• Suitable for scenarios where client transparency is required.</p><p>In a microservice architecture, you have numerous small services rather than a single monolithic application. Each microservice can be deployed in multiple regions and can have multiple versions. This creates diverse deployment scenarios, and HTTPS load balancing offers the flexibility to manage them effectively.</p><p>To effectively manage and route traffic in a microservice environment, it’s crucial to grasp some key load balancing concepts:</p><p>1. <strong>Backend Configuration</strong>: Defines the backend services and instances that handle incoming traffic.<br>2. <strong>Host and Path Rules</strong>: Dictates how traffic is routed based on URL paths.<br>3. <strong>Frontend Configuration</strong>: Establishes the IP address, protocol, and port that users interact with.</p><p><strong>Creating an HTTPS Load Balancer</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*uTgrtIwwCFe84BrsJf_AXg.png" /></figure><p>selecting HTTPS load balancing, and configuring it to balance traffic from the internet. This setup involves three main components:</p><p>• <strong>Frontend Configuration</strong>: Assigns an IP address, protocol, and port for users to connect to.<br>• <strong>Backend Configuration</strong>: Includes backend services and backends, which are essentially managed instance groups in specific regions.<br>• <strong>Host and Path Rules</strong>: Uses URL maps to determine traffic routing.</p><p><strong>Example Scenario: Multi-Regional Microservices</strong></p><p>Imagine you have two microservices, each deployed in two different regions. Here’s how you can configure your load balancer:</p><p>1. <strong>Backend Services</strong>: Create a backend service for each microservice. For instance, Backend Service A for Microservice A and Backend Service B for Microservice B.<br>2. <strong>URL Map</strong>: Define routing rules such as:<br>• Requests to /serviceA go to Backend Service A.<br>• Requests to /serviceB go to Backend Service B.</p><p>3. <strong>Backends</strong>: Each backend service will have backends representing managed instance groups in different regions.</p><p><strong>Global Routing with Premium Tier</strong></p><p>For optimal performance and global routing, you should use Google Cloud’s Premium networking tier. This allows the load balancer to route user requests to the nearest instance group, ensuring low latency and high availability.</p><p>The Standard tier is a more economical option for load balancing that focuses on regional traffic management rather than global distribution. It provides basic load balancing capabilities and is ideal for applications that do not require the global reach and advanced features of the Premium tier.</p><p><strong>Key Features of Standard Tier</strong></p><ol><li><strong>Regional Traffic Management</strong>: The Standard tier supports regional load balancing, meaning traffic can be routed only within a specific region. This is suitable for applications with a user base primarily located in a single geographic area.</li><li><strong>Cost-Effective</strong>: The Standard tier is designed to be more budget-friendly, making it a good choice for applications that don’t need the advanced features of the Premium tier.</li><li><strong>Forwarding Rules</strong>: In the Standard tier, the IP address assigned to the frontend configuration is regional. This means the forwarding rule and the backend services must be in the same region.</li><li><strong>No Global Routing</strong>: Unlike the Premium tier, the Standard tier does not support global routing. This limits the ability to route traffic to the nearest backend based on the user’s location.</li></ol><p><strong>Multiple Versions of Microservices</strong></p><p>If you have multiple versions of a microservice, such as version 1 and version 2 of Microservice A, you can configure separate backend services for each version. For example:</p><p>• <strong>Microservice A V1</strong>: Backend Service for version 1.<br>• <strong>Microservice A V2</strong>: Backend Service for version 2.<br>• <strong>Microservice B</strong>: A single backend service.</p><p>Then, use URL maps to route traffic based on the version:</p><p>• Requests to /serviceAV1 go to the backend service for version 1.<br>• Requests to /serviceAV2 go to the backend service for version 2.<br>• Requests to /serviceB go to the backend service for Microservice B.</p><p><strong>Flexibility and Scalability</strong></p><p>Google Cloud’s HTTPS load balancing offers a highly flexible and scalable architecture. It allows routing requests globally, supports multiple versions of microservices, and serves users from the nearest backend. This flexibility is vital for maintaining high performance and availability in complex microservice deployments.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=be3daa3ae810" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[GCP:2-Managed Instance Groups [MIG]]]></title>
            <link>https://medium.com/@me.sanjeev3d/gcp-2-managed-instance-groups-mig-5c3077e8c5d3?source=rss-89634bf8d93a------2</link>
            <guid isPermaLink="false">https://medium.com/p/5c3077e8c5d3</guid>
            <category><![CDATA[cloud]]></category>
            <category><![CDATA[gcp]]></category>
            <category><![CDATA[managed-instance-group]]></category>
            <category><![CDATA[auto-scaling-groups]]></category>
            <category><![CDATA[instance]]></category>
            <dc:creator><![CDATA[Sanjeev Gautam]]></dc:creator>
            <pubDate>Tue, 23 Jul 2024 16:51:54 GMT</pubDate>
            <atom:updated>2024-07-23T16:51:54.226Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/706/1*Vu7tG2FlULvgWqlxa0GGDQ.png" /></figure><p><strong>Welcome back to our new section on Instance Groups!</strong></p><p>In this segment, we’ll dive into the world of Instance Groups, exploring how they can help you manage a collection of virtual machine instances effectively. We’ll cover what instance groups are, the different types you can create, and their unique features.</p><p><strong>Understanding Instance Groups</strong></p><p>An instance group is a collection of virtual machine (VM) instances managed as a single entity. This allows you to handle a group of similar VMs that share the same lifecycle as one unit.</p><p><strong>Types of Instance Groups</strong></p><p>There are two primary types of instance groups: Managed Instance Groups (MIG) and Unmanaged Instance Groups.</p><p>1. <strong>Managed Instance Groups (MIG)</strong></p><p>• <strong>Definition</strong>: MIGs are collections of identical VMs created using a template. Each VM in a MIG has the same machine type, image, and configuration.<br>• <strong>Key Features</strong>:</p><p>• <strong>Auto-scaling</strong>: Automatically adjust the number of compute instances based on demand.<br>• <strong>Auto-healing</strong>: Replace failed instances automatically based on health checks.<br>• <strong>Managed Releases</strong>: Deploy new versions without downtime.<br>• <strong>Load Balancing</strong>: Distribute traffic among instances.<br>• <strong>High Availability</strong>: Regional MIGs distribute instances across multiple zones for higher availability.<br>• <strong>Rolling Updates</strong>: Update a subset of instances gradually.<br>• <strong>Canary Deployments</strong>: Test new versions with a small group of instances before full rollout.</p><p>2. <strong>Unmanaged Instance Groups</strong></p><p>• <strong>Definition</strong>: Unlike MIGs, Unmanaged Instance Groups can contain VMs with different configurations, images, and hardware.</p><p>• <strong>Key Features</strong>:</p><p>• <strong>Flexibility</strong>: Suitable for diverse VM configurations.<br>• <strong>Limitations</strong>: Do not support auto-scaling, auto-healing, or managed releases.</p><p><strong>Choosing the Right Instance Group</strong></p><p>• <strong>Managed Instance Groups</strong> are ideal when you need multiple instances of the same type of VM. They offer robust features like auto-scaling, auto-healing, and seamless updates.<br>• <strong>Unmanaged Instance Groups</strong> are recommended only when you require a mix of different VM configurations.</p><p><strong>Instance Group Locations</strong></p><p>Instance groups can be either zonal or regional.</p><p>• <strong>Zonal</strong>: Instances are located in a specific zone.<br>• <strong>Regional</strong>: Instances are distributed across multiple zones within a region, providing higher availability.</p><p><strong>Creating Managed Instance Groups: A Step-by-Step Guide</strong></p><p><strong>Step 1: Create an Instance Template</strong></p><p>The first requirement for creating a managed instance group is an instance template. An instance template is a blueprint for VM instances, defining machine type, image, and configuration.</p><p><strong>Step 2: Configure Autoscaling</strong></p><p>Autoscaling automatically adjusts the number of VM instances based on the load. Here are the key configurations:</p><p>• <strong>Minimum and Maximum Instances</strong>: Define the range of instances.<br>• <strong>Autoscaling Metrics</strong>: Choose metrics for autoscaling, such as CPU utilization, load balancer utilization, or metrics from Google Cloud’s Stackdriver.<br> • <strong>Cooldown Period</strong>: Set a waiting period before re-evaluating autoscaling metrics to prevent frequent scaling actions.<br> • <strong>Scale-in Controls</strong>: Limit the rate at which instances are reduced to avoid sudden drops in capacity.</p><p><strong>Step 3: Configure Auto Healing</strong></p><p>Auto healing replaces unhealthy instances with healthy ones. To set this up:</p><p>• <strong>Health Check</strong>: Create a health check specifying protocol, port, and request path.<br>• <strong>Initial Delay</strong>: Set a delay before the health check to allow the VM to initialize.</p><p><strong>Step 4: Hands-On Demo</strong></p><p>Now, let’s walk through a demo to create a managed instance group.</p><p>1. <strong>Create a VM Instance</strong>: Use an instance template (e.g., my-instance-template) with a custom image.<br>2. <strong>Access Instance Groups</strong>: Navigate to the instance groups section in your cloud console.<br>3. <strong>Create a Managed Instance Group</strong>:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Zd_FV4Y4w8qt7HjVna_1uA.png" /></figure><p>• Select the instance template.<br>• Choose the location (single zone or multiple zones for higher availability).<br>• Configure autoscaling (mode, minimum and maximum instances, metrics).<br> • Enable auto healing and set up health checks.</p><p>4. <strong>Stateless vs. Stateful</strong>:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*5ksipRgRzRneSDxMPaEgDw.png" /></figure><p>• <strong>Stateless</strong>: Ideal for web applications or REST APIs where state persistence isn’t critical.<br>• <strong>Stateful</strong>: Suitable for databases and applications where data persistence is <br>essential.</p><p>In case of unmanaged Instance group you can able to select different template</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*jcBgOK2tOw0gJSUPr8lnYA.png" /></figure><p><strong>Example Configuration</strong></p><p>Here’s an example of creating a managed instance group named my-managed-instance-group:</p><p>• <strong>Instance Template</strong>: my-instance-template<br>• <strong>Location</strong>: Default region and zones.<br>• <strong>Autoscaling</strong>:<br> • Mode: ON<br> • Minimum Instances: 2<br> • Maximum Instances: 2<br> • Metrics: CPU utilization at 60%<br> • Cooldown Period: 60 seconds</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*TwTmvRKkbCcydz6C86wUcA.png" /><figcaption>Auto-scaling mode</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*FybGcn_d5fEN3ToCTfJnrQ.png" /><figcaption>Autoscaling metrics</figcaption></figure><p>There is another feature added by GCP is predictive autoscaling which means it autoscale based on historical data or trends.</p><p>• <strong>Health Check</strong>:</p><p>• Protocol: HTTP<br>• Port: 80<br>• Interval: 5 seconds<br>• Timeout: 5 seconds<br>• Healthy Threshold: Default<br>• Unhealthy Threshold: Default<br>• Initial Delay: 30 seconds</p><p><strong>Create and Monitor</strong></p><p>After configuring the managed instance group, create it and monitor the process.</p><p>After approximately 10 minutes, the creation of our managed instance group was completed successfully. Upon completion, you might see a warning indicating that the number of instances is equal to the maximum number of instances. This simply means the auto-scaler cannot add or remove instances beyond this limit. For our setup, having two instances running at all times with health checks enabled is intentional. In a real-world scenario, you might set a higher maximum number of instances to accommodate varying loads.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*1muM1P2E0KDoPL7pzVgQow.png" /></figure><p><strong>Reviewing the Managed Instance Group Configuration</strong></p><p>On the Managed Instance Group page, you can see key details such as the group name, the number of instances, the template used, and the auto-scaling configuration. Clicking on the instance group name reveals more details, including:</p><p>• <strong>Total Instances</strong>: Shows the number of instances running.<br>• <strong>Location</strong>: Displays the zones where the instances are deployed.<br>• <strong>Health Status</strong>: Indicates the health of the instances.<br>• <strong>Auto Healing</strong>: Confirms that health checks are configured.</p><p>By clicking on the health check link, you can view specific health check details, such as the HTTP request to the root on port 80.</p><p><strong>Instance Details</strong></p><p>Within the instance group, you can see the individual instances, their names, and the zones they are deployed in (e.g., us-central1-f and us-central1-c). Clicking on an instance provides further configuration details. For instance, if you navigate to the Virtual Machine Instances section, you’ll see all VM instances, including those created through the managed instance group and any pre-existing ones.</p><p>Each instance created by the managed instance group will have an external IP address. Accessing these IPs should bring up an HTTP response, confirming the instances are operational.</p><p><strong>Editing the Instance Group</strong></p><p>You have the flexibility to edit the instance group settings:</p><p>• <strong>Auto-Scaling Metrics</strong>: Adjust the metrics used for auto-scaling.<br>• <strong>Health Checks</strong>: Modify the health check configuration.<br>• <strong>Number of Instances</strong>: Change the minimum and maximum number of instances.</p><p>For example, if you change the minimum and maximum number of instances to three, the group will update accordingly, and a third instance will be up and running shortly.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Q6_KrDDF5JIsInV2MhKhIw.png" /><figcaption>update VMs</figcaption></figure><p>Previously, we made a basic update by changing the number of instances and adjusting the auto-scaling configuration to have a minimum and maximum of three instances. This allowed us to see the automatic scaling in action. Beyond this, there are several other updates you can perform on a managed instance group.</p><p><strong>Rolling Updates</strong></p><p>A rolling update involves gradually updating instances in an instance group to a new instance template. This is useful when you need to upgrade the application version. Here’s how it works:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*wB4IJZoxhR5p-t1qUsHTrg.png" /><figcaption>Canary Testing</figcaption></figure><p>1. <strong>Specify the New Template</strong>: Start by defining the new instance template, which references the new application version. Optionally, you can specify a canary testing template.<br>2. <strong>Update Configuration</strong>: Choose how many instances to update at a time. For example, if you have 10 instances, you might update two instances at a time.<br>3. <strong>Canary Testing</strong>: Test a small set of instances (canary instances) with the new template before rolling out to the remaining instances. This ensures the new version works as expected.</p><p><strong>Proactive vs. Opportunistic Updates</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*WM98marBJThayc5k7Nu14w.png" /></figure><p>• <strong>Proactive Updates</strong>: Immediately update all instances.<br>• <strong>Opportunistic Updates</strong>: Update instances only when they are resized, killed, or replaced.</p><p><strong>Configuring Rolling Updates</strong></p><p>1. <strong>Update Start Time</strong>: Decide when the update should start.<br>2. <strong>Max Surge</strong>: Specify how many additional instances can be created during the update. For instance, if you have 10 instances, setting a max surge of two means there will be up to 12 instances running at peak.<br>3. <strong>Max Unavailable</strong>: Determine how many instances can be offline during the update. If you set two instances as the maximum unavailable, the update process will ensure at least eight instances remain active.</p><p><strong>Rolling Restarts or Replacements</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*DmayywANe097JauMjZHzpQ.png" /></figure><p>• <strong>Rolling Restart</strong>: Restart all VM instances without changing the instance template.<br>• <strong>Rolling Replace</strong>: Replace VM instances with new ones using the same template.</p><p><strong>Configuring Rolling Restarts/Replacements</strong></p><p>1. <strong>Max Unavailable</strong>: Define how many instances can be offline during the restart/replacement.<br>2. <strong>Max Surge</strong>: Specify the number of additional instances to create before the replacement process begins.</p><p><strong>Practical Application</strong></p><p>To configure these updates, navigate to the managed instance group’s settings:</p><p>1. <strong>Rolling Update</strong>:<br>• Go to “Update VMs” and select the new instance template.<br>• Optionally configure canary testing by selecting a new template for a subset of instances.<br>• Set update parameters such as max surge and max unavailable.</p><p>2. <strong>Rolling Restart/Replace</strong>:<br>• Choose “Restart or Replace VMs”.<br>• Set the desired operation (restart or replace).<br>• Configure max unavailable and max surge as needed.</p><p><strong>Conclusion</strong></p><p>Creating and managing a managed instance group (MIG) involves several critical configurations, from setting up an instance template to fine-tuning autoscaling and auto-healing settings. By following these steps, you can ensure that your VM instances are managed efficiently, providing high availability and reliability for your applications.</p><p>In our sessions, we explored the practical aspects of managed instance groups, including the creation process, configuration settings, and how to manage and monitor these groups. This foundational understanding prepares you for more advanced configurations and optimizations, which we’ll explore in future sessions.</p><p>We also delved into updating managed instance groups through rolling updates, rolling restarts, and replacements. We discussed proactive and opportunistic updates, as well as how to configure key parameters like max surge and max unavailable. These techniques ensure smooth updates with minimal downtime and operational disruption.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=5c3077e8c5d3" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Securing Your Kubernetes Cluster: Best Practices and Tools]]></title>
            <link>https://medium.com/@me.sanjeev3d/securing-your-kubernetes-cluster-best-practices-and-tools-e7731739141d?source=rss-89634bf8d93a------2</link>
            <guid isPermaLink="false">https://medium.com/p/e7731739141d</guid>
            <dc:creator><![CDATA[Sanjeev Gautam]]></dc:creator>
            <pubDate>Mon, 15 May 2023 17:25:12 GMT</pubDate>
            <atom:updated>2023-05-15T17:25:12.798Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/381/1*WlJ-A5XyQbmdonxzaweHgA.jpeg" /></figure><blockquote><strong>Introduction</strong></blockquote><p>As organizations increasingly adopt Kubernetes for container orchestration, ensuring the security of the cluster becomes paramount. Kubernetes offers various built-in security features and allows integration with external tools to enhance its security posture. In this blog post, we will explore best practices and tools to secure your Kubernetes cluster.</p><p>1. <strong>Secure Cluster Configuration:</strong></p><p>Properly configuring your Kubernetes cluster is the first step towards ensuring its security. Here are some key considerations:</p><p><em>a. Secure API Server:</em><br> — Enable authentication and authorization mechanisms like RBAC (Role-Based Access Control) and webhooks.<br> — Utilize secure communication protocols such as TLS.<br> — Disable insecure APIs and endpoints.</p><p><em>b. Network Policies:</em><br> — Define network policies to control ingress and egress traffic within the cluster.<br> — Limit communication between pods to the necessary services only.</p><p><em>c. Secrets Management:</em><br> — Avoid storing sensitive information (such as passwords or API keys) directly in Pod YAML files.<br> — Utilize Kubernetes Secrets to store and manage sensitive data securely.</p><p><strong>2. RBAC and Access Control:</strong></p><p>Role-Based Access Control (RBAC) provides fine-grained control over cluster resources. Follow these best practices:</p><p><em>a. Principle of Least Privilege:</em><br> — Assign minimal necessary permissions to users and service accounts.<br> — Regularly review and revoke unnecessary privileges.</p><p><em>b. Use Service Accounts:</em><br> — Avoid using default service accounts and create dedicated ones.<br> — Restrict permissions for service accounts based on their requirements.</p><p><strong>3. Pod Security Policies:</strong></p><p>Pod Security Policies (PSPs) define a set of security policies that pods must adhere to. They control the actions a pod can perform. Consider these steps:</p><p><em>a. Enable Pod Security Policies:</em><br> — Ensure PSP admission controller is enabled in the Kubernetes API server.</p><p><em>b. Define Appropriate Policies:</em><br> — Restrict privilege escalation, host namespaces, and capabilities.<br> — Enforce container image security, read-only filesystems, and restricted host paths.</p><p><strong>4. Container Image Security:</strong></p><p>Container images play a crucial role in the security of your Kubernetes workloads. Apply these practices:</p><p><em>a. Use Trusted Images:</em><br> — Scan container images for vulnerabilities using tools like Clair, Trivy, or Anchore.<br> — Regularly update images to incorporate security patches.</p><p><em>b. Implement Image Signing and Verification:</em><br> — Use image signing and verification mechanisms like Notary or Cosign to ensure image integrity.</p><p><strong>5. Network Security:</strong></p><p>Kubernetes network security is essential for protecting cluster communication. Consider these measures:</p><p><em>a. Enable Network Policies:</em><br> — Network policies enforce rules to restrict pod-to-pod communication.<br> — Implement policies to limit traffic based on namespaces, IP addresses, or ports.</p><p><em>b. Use Secure Network Plugins:</em><br> — Choose network plugins that provide encryption, isolation, and network segmentation.<br> — Examples include Calico, Cilium, or Flannel with Network Policy support.</p><p><strong>6. Monitor and Audit Cluster Activity:</strong></p><p>Continuous monitoring and auditing help detect and respond to potential security incidents. Here’s what you can do:</p><p><em>a. Centralized Logging:</em><br> — Collect logs from all Kubernetes components and applications running in the cluster.<br> — Utilize tools like Elasticsearch, Fluentd, and Kibana (EFK stack) or Prometheus and Grafana for monitoring.</p><p><em>b. Security Auditing:</em><br> — Leverage Kubernetes auditing feature to log and monitor cluster activities.<br> — Regularly review audit logs for any suspicious or unauthorized activities.</p><p><strong>Conclusion:</strong></p><p>Securing your Kubernetes cluster is a crucial aspect of maintaining a robust and resilient infrastructure. By following the best practices outlined in this blog post, you can significantly enhance the security posture of your Kubernetes deployments. Remember, security is an ongoing process, so continuously monitor, update, and adapt your security measures to evolving threats.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=e7731739141d" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Python: Boto]]></title>
            <link>https://medium.com/@me.sanjeev3d/python-boto-4e6cb1483d73?source=rss-89634bf8d93a------2</link>
            <guid isPermaLink="false">https://medium.com/p/4e6cb1483d73</guid>
            <category><![CDATA[boto3]]></category>
            <category><![CDATA[boto]]></category>
            <category><![CDATA[automate-aws]]></category>
            <category><![CDATA[aws]]></category>
            <category><![CDATA[python-boto-boto3]]></category>
            <dc:creator><![CDATA[Sanjeev Gautam]]></dc:creator>
            <pubDate>Sat, 28 Mar 2020 09:46:02 GMT</pubDate>
            <atom:updated>2020-03-31T13:56:54.359Z</atom:updated>
            <content:encoded><![CDATA[<h3>Boto: Python Library to Automate AWS</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/477/1*-zinVfCGrvnh_O13Fw3zTg.png" /></figure><p>Boto is a Python package that provides programmatic connectivity to Amazon Web Services (AWS). <a href="https://github.com/boto/boto">Boto</a> which allows you to write some handy scripts to automate simple things like starting or stopping EC2 instances or taking regular snapshots of your servers .</p><h3>Installing Boto on Linux (Ubuntu)</h3><p>1. You’ll first use the yum package manager to install Python and the pip Python package installer:</p><pre>apt-get install python python-devel pip</pre><p>2. Now we will install the Boto package via pip:</p><pre>pip install boto</pre><p>3. Finally, we can test if everything has been successfully installed on our Linux box:</p><ul><li><strong>python -c “import boto; print boto.Version”</strong></li></ul><h3>Use cases</h3><p>Lets start with a basic scenario: launching an EC2 instance</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/706/1*cPKhCbjyDQ7gRF4lXtk1BA.png" /></figure><p>The above python script will connect to Amazon’s <em>us-west</em> region and launch an instance using AMI id: ami-6ac2a85a.</p><p>Scheduling stop instances. Let’s say you now want to stop your test environment instances at a certain time each day. Run:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/621/1*rrV6-PyjyylVmpWX2-tfVA.png" /></figure><p>You can schedule this script to run every day at a particular time by creating a cron job.</p><p>To check all IAM user using boto3 module</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/689/1*xww_3yNNmy0lumoUNcWewQ.png" /></figure><p>Here we using profile_name to define user is being use to execute the task which is configure on console using “<strong>aws configure</strong>” command.</p><p>Alternatively we can use sessions to access different accounts and regions</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/827/1*PI5JJO2dB1eMu49UQPAlSQ.png" /></figure><h3>AWS services supported by Boto</h3><p>At the moment, Boto supports more than 50 Amazon services, running the whole range from compute, database, application, and payments and billing. You can find a <a href="https://pypi.python.org/pypi/boto">complete and current list</a> on the Python.org website.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=4e6cb1483d73" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Git and GitHub]]></title>
            <link>https://medium.com/@me.sanjeev3d/git-and-github-4b406352d2c2?source=rss-89634bf8d93a------2</link>
            <guid isPermaLink="false">https://medium.com/p/4b406352d2c2</guid>
            <category><![CDATA[git]]></category>
            <category><![CDATA[github]]></category>
            <dc:creator><![CDATA[Sanjeev Gautam]]></dc:creator>
            <pubDate>Thu, 12 Dec 2019 10:16:22 GMT</pubDate>
            <atom:updated>2019-12-12T10:16:22.476Z</atom:updated>
            <content:encoded><![CDATA[<blockquote><strong>Version control</strong></blockquote><p>Version control is the management of changes of the document, computer program, large website and other collection of information, these changes are termed as the version.</p><p>Every time make changes in our project it will create a snapshot of the entire project and saves it and this snapshot is known as a different version. snapshot is the entire state of file that time.</p><blockquote><strong>Why we need version control system.?</strong></blockquote><p><strong>Collaboration</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/814/1*XwrFVL_o2GacSVcLryZgUw.png" /></figure><p>Above picture describe why we need version control</p><p><strong>Backup</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/876/1*HHrK533jKLmcMBpRNM39dw.png" /></figure><p><strong>Analyse</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/834/1*laqX0Jt8ul68lqgFWL8oTg.png" /></figure><h3><strong>Git</strong></h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/834/1*HBRk50Ot7OOHncSepk6ZyA.png" /></figure><p>First, we make changes in the local repository and then push to the main repository.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/840/1*g1euiNt-hye1zjcO4SE5gQ.png" /></figure><p><strong>Features Of Git</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/808/1*l3opdSTqH5ZYBCFvq7230A.png" /></figure><ol><li><strong>Distributed</strong>: Allow distributed development of code, every developer has a local copy of entire development history and changes are copied from one repository to another.</li><li><strong>Compatible:</strong> Compatible with existing system and protocols, SVN &amp; SVK repositories can be directly accessed using Git-SVN</li><li><strong>Branching</strong>: It take few second to create &amp; Merge branches, Master branch always contain production quality code</li><li><strong>Speed</strong>: Fetching data from local repository is 100 time faster than local repositories.</li><li><strong>Reliable:</strong> On event of system crash, the lost data can be easily recovered from any of local repositories of the collaborators.</li><li><strong>Secure</strong>: Uses SHA1 to name and identify objects.Every file &amp; Commit is check summed and is retried by its checksum at time of checkout</li></ol><p>What is Repository..?</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/788/1*Z6DjhN6vhWvgn58Cawcgag.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/837/1*NGwV3nWctQkXpq9eS7IfYw.png" /></figure><p><strong>Git Operation And Command</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/810/1*3QKPtmLOTpNYXyfYbHFUsA.png" /></figure><p>Create Repository</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/837/1*DuJHMQcM4KARvS9IgbNDmA.png" /></figure><p>Now central repository is created</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/964/1*3VmyO6peCrbtog2ESM7YvA.png" /></figure><p>It&#39;s time to create the local repository.</p><p>Now to create a local repository, we use the command</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/507/1*39dSR6z__kHT33GHb4_FnA.png" /></figure><p>File inside /home/sanjeev/.git</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/576/1*WNiViJWdhluLna_EwBsfuA.png" /></figure><p>Syncing local repo to the central repository.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/838/1*D612PRaxnh7kCnm_oHgR3g.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/794/1*iYUn5mlMoQS7uODi4cR4iA.png" /></figure><p><strong>Making Changes</strong></p><p>This is the layer between work-space and local repository</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/850/1*2JzbtgjSoQu8hORn9Bfg5A.png" /></figure><p>We create a simple text file inside same git folder that is our workspace, now we have to add same to index to perform git commit</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/729/1*Rg1wQBZNy3JXsNzGUliPOQ.png" /></figure><p>As checked through <strong>git status</strong> file1 is not in an index as it is <strong>untracked files </strong>so we have to add first to index.</p><p># git add file1</p><p>after adding we check status for same, getting below error while commiting</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/687/1*z9BuKSO2mAFdmoNmpKvdFg.png" /></figure><p># git config — global user.email “<a href="/service/mailto:me.sanjeev3d@gmail.com">me.sanjeev3d@gmail.com</a>”<br># git config — global user.name “sanjeev3d”</p><p>To un config same</p><p># git config — global — replace-all user.name “sanjeev3d”<br># git config — global — replace-all user.email “<a href="/service/mailto:me.sanjeev3d@gmail.com">me.sanjeev3d@gmail.com</a>”</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/474/1*-6N6tLLCFMeDUjkh9Pr7Kg.png" /><figcaption>Commited</figcaption></figure><p>Now add two more file and modify file1 too.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/707/1*BwMMDwcC_oMCTT6DVD-J2w.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/450/1*NHBIthfUCHhSemizE_I4pA.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/516/1*V080JjtPOw9m8ltke_8veQ.png" /></figure><p>To check commit detail we check it using</p><p># git log</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/632/1*W2px4NQNanmwobE2fD5BZw.png" /></figure><p>commit having 40 digits unique hexadecimal code with date and author with the commit message.</p><p><strong>Parallel Development: Branching</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/842/1*Vw62IwIo6daO4jWuLFpclA.png" /></figure><p>By default we are working in the master branch we can create another branch to develop some more feature in code using branch and merge them after sanity.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/589/1*_beV060SQEkySJoC7PPGSw.png" /></figure><p>Above we create branch first branch and it inherits all file from the master. switched to another branch using git checkout command.</p><p>Now, File4 id created for firstbranch and commit to it.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/712/1*QssC5igcGR3YSIQr6u8lfg.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/664/1*GfmxAPaLL-6CLjgbYvu0zw.png" /></figure><p><strong>Parallel Development: Merging</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/844/1*GD7y91TOhBK0s7dBoEwbZQ.png" /></figure><h3><strong>Daily use command</strong></h3><p><strong>Parallel Development: Rebasing</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/840/1*7gFVSE4cJjsngM34M8JkmA.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*7SLdJL1wU-qsuSIwBArEBQ.jpeg" /></figure><p><strong>Reverting a single modified file</strong></p><p>Case 1: If the file isn’t committed</p><pre><strong>git checkout filename</strong></pre><p>Case 2: If the file is already commit</p><pre><strong>git checkout 7080e5da </strong><strong>filename</strong></pre><p>This should unstage a &lt;file&gt; for you (without removing or otherwise modifying the file):</p><pre><strong>git reset &lt;file&gt;</strong></pre><p>Actually this command doing<strong> — mixed reset</strong>,after running command file move to working directory from index area.</p><h4><strong>fix a message of a commit I just made</strong></h4><p><strong><em>git commit — amend -m “Add your correct commit message here.”</em></strong></p><h4><strong>Revert</strong></h4><p>As it sounds, the revert command changes all the files for a specific commit back to their state before that commit was completed. Specify the commit that you want to revert:</p><p><strong>git revert &lt;commit_ID&gt;</strong></p><p>It’s important to note the mechanics of this command. The reverted commit is not deleted. Rather, Git creates a new commit with the included files reverted to their previous state.</p><p>So your version control history moves forward while the state of your files moves backwards.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/632/1*PsZdpSKIuayzZeqHf-7hWg.png" /></figure><h4><strong>Reset</strong></h4><p>This option is a little different than a revert. It resets the status of your repo (working HEAD) to an older revision. It’s a true rollback of the state of your repo.</p><p><strong>git reset — hard &lt;commit_ID&gt;</strong></p><p>When you use this option, Git discards any commits between the current state of the repo and the target commit. The branch will then appear to stop at the commit you reset the HEAD to.</p><p><em>Note:</em> <em>Although the commits no longer appear to be a part of your branch history, they are not deleted. They are still stored in Git.</em></p><p><em>As well, be cautious with the — hard option. It will also discard any local changes you have made. If you want to reset to a previous commit, but still have your current changes saved as they are, use the</em><strong><em> — keep </em></strong><em>option instead.</em></p><p><strong>git reset — hard &lt;commit_ID&gt; — keep</strong></p><h4>Rebase</h4><p>Rebasing is often used as an alternative to merging. Rebasing a branch updates one branch with another by applying the commits of one branch on top of the commits of another branch. For example, if working on a feature branch that is out of date with a dev branch, rebasing the feature branch onto dev will allow all the new commits from dev to be included in feature. Here’s what this looks like visually:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/484/1*wmc9_KJH9PsL7Vlefp6qWw.png" /><figcaption>Feature branch before rebasing</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/802/1*DD0YuAlFj2NXTt-w6Vi6Cw.png" /><figcaption>Feature branch after rebasing onto dev</figcaption></figure><p><strong>git rebase feature dev</strong></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=4b406352d2c2" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Kubernetes]]></title>
            <link>https://medium.com/@me.sanjeev3d/kubernetes-f2942739a2c6?source=rss-89634bf8d93a------2</link>
            <guid isPermaLink="false">https://medium.com/p/f2942739a2c6</guid>
            <category><![CDATA[pods]]></category>
            <category><![CDATA[kubernetes]]></category>
            <category><![CDATA[containers]]></category>
            <category><![CDATA[persistent-volume]]></category>
            <dc:creator><![CDATA[Sanjeev Gautam]]></dc:creator>
            <pubDate>Wed, 22 May 2019 18:30:02 GMT</pubDate>
            <atom:updated>2019-05-24T07:29:02.457Z</atom:updated>
            <content:encoded><![CDATA[<h3>Kubernetes Essentials: An Overview</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/800/1*lnIG0B3uU1g4LOeaP5dt6Q.jpeg" /></figure><p>Kubernetes is a platform-agnostic container orchestration tool created by Google and heavily supported by the open source community as a project of the Cloud Native Computing Foundation. It allows you to spin up a number of container instances and manage them for scaling and fault tolerance. It also handles a wide range of management activities that would otherwise require separate solutions or custom code, including request routing, container discovery, health checking, and rolling updates.</p><p><strong><em>Kubernetes is constructed using several components, as follows:</em></strong></p><ul><li><em>Kubernetes master</em></li><li><em>Kubernetes nodes</em></li><li><em>etcd</em></li><li><em>Overlay network (flannel)</em></li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/588/1*7QCC5DO8hoh4ih3GfyzNuQ.png" /></figure><p>These components are connected via a network, as shown in the following screenshot:<br>The preceding image can be summarized as follows:</p><ul><li><em>Kubernetes master connects to etcd via HTTP or HTTPS to store the data. It also connects flannel to access the container application.</em></li><li><em>Kubernetes nodes connect to the Kubernetes master via HTTP or HTTPS to get a command and report the status.</em></li><li><em>Kubernetes nodes use an overlay network (for example, flannel) to make a<br>connection of their container applications.</em></li></ul><p><strong>Kubernetes master</strong><br>Kubernetes master is the main component of Kubernetes cluster. It serves several functionalities, such as the following items:</p><ul><li><em>Authorization and authentication</em></li><li><em>RESTful API entry point</em></li><li><em>Container deployment scheduler to the Kubernetes nodes</em></li><li><em>Scaling and replicating the controller</em></li><li><em>Read and store the configuration</em></li><li><em>Command Line Interface</em></li></ul><p>There are several daemon processes that make the Kubernetes master’s functionality, such as <strong>Kube-apiserver</strong>, <strong>Kube-scheduler</strong>, and <strong>Kube-controller-manager</strong>. Hypercube wrapper launched all of them.</p><p><strong>API server (Kube-apiserver)</strong><br>The API server provides an HTTP- or HTTPS-based RESTful API, which is the hub between Kubernetes components, such as kubectl, scheduler, replication controller, etcd datastore, and kubelet and Kube-proxy, which runs on Kubernetes nodes and so on.</p><p><strong>Scheduler (Kube-scheduler)</strong><br>Scheduler helps to choose which container runs by which nodes. It is a simple algorithm that defines the priority to dispatch and bind containers to nodes, for example:</p><ul><li><em>CPU</em></li><li><em>Memory</em></li><li><em>How many containers are running?</em></li></ul><p><strong>Controller manager (Kube-controller-manager)</strong><br>Controller manager performs cluster operations. For example:</p><ul><li><em>Manages Kubernetes nodes</em></li><li><em>Creates and updates the Kubernetes internal information</em></li><li><em>Attempts to change the current status to the desired status</em></li></ul><p><strong>Command Line Interface (kubectl)</strong><br>After you install Kubernetes master, you can use the Kubernetes Command Line Interface kubectl to control the Kubernetes cluster. For example, <strong>kubectl get cs</strong> returns the status of each component.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/528/1*WQF7jma5BFUbsJwRwH5CKQ.png" /><figcaption>To check the health of all kubernetes component</figcaption></figure><p><strong>Kubernetes node</strong><br>Kubernetes node is a slave node in the Kubernetes cluster. It is controlled by Kubernetes master to run the container application using Docker (<a href="http://docker.com/">http://docker.com</a> ) or rkt ( <a href="http://coreos.com/rkt/docs/latest/">http://coreos.com/rkt/docs/latest/</a> )</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/695/1*a8ysdyVtOAbl9tKleJs3AA.png" /><figcaption>It reflects all nodes in that cluster</figcaption></figure><p>Kubernetes Node consists of two component.</p><p><strong>kubelet</strong><br>kubelet is the main process on Kubernetes node that communicates with Kubernetes master to handle the following operations:</p><ul><li><em>Periodically access the API Controller to check and report</em></li><li><em>Perform container operations</em></li><li><em>Runs the HTTP server to provide simple APIs</em></li></ul><p><strong>Proxy (Kube-proxy)</strong><br>Proxy handles the network proxy and load balancer for each container. It performs to change the Linux iptables rules (nat table) to control TCP and UDP packets across the containers. After starting the Kube-proxy daemon, it will configure iptables rules; you can see iptables -t nat -L or iptables -t nat -S to check the nat table rules, as follows:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/790/1*Ba8TmRTU4mN96Cwyv-BaSg.png" /></figure><p>There are two more components to complement the Kubernetes nodes’ functionalities, the datastore <strong>etcd, </strong>and the <strong>overlay network </strong>flannel.</p><p><strong>etcd</strong><br>The etcd ( <a href="https://coreos.com/etcd/">https://coreos.com/etcd/</a> ) is the distributed key-value datastore. It can be accessed via the RESTful API to perform the CRUD operation over the network. Kubernetes uses etcd as the main datastore.</p><p><strong>Overlay network</strong><br>Network communication between containers is the most difficult part. Because when you start to run the Docker, an IP address will be assigned dynamically; the container application needs to know the peer’s IP address and port number.</p><p>If the container’s network communication is only within the single host, you can use the Docker link to generate the environment variable to discover the peer. However, Kubernetes usually works as a cluster and ambassador pattern or overlay network could help to connect<br>every node. Kubernetes uses an overlay network to manage multiple containers’ communication. For overlay network, Kubernetes has several options, but using flannel is the easier solution.</p><p><em>Flannel also uses etcd to configure the settings and store the status.</em></p><p>To create a container cluster on google cloud</p><p><strong><em># gcloud container clusters create test-cluster</em></strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*xrQLEtDWfDztv48F2bwJ6g.png" /></figure><p><strong><em># kubectl run jenkins — image=jenkins — port=8080</em></strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/579/1*MmF1aE334Efe4xYDXsOP-w.png" /></figure><p><strong><em># kubectl expose deployment jenkins — type=LoadBalancer — port=8080</em></strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/654/1*T95fShvDPdIEfgz8WRxC2Q.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/621/1*1VPnmYrWGAYdbv8kxGHWoQ.png" /></figure><p><strong><em># kubectl get services(after sometime external IP is assigned for user interface)</em></strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/611/1*3pQD2zLim3jNUxo_Q9VFyQ.png" /></figure><p><strong><em># kubectl get all</em></strong></p><p>It provides all pods, service, and deployment with a replica set</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/737/1*hxcUvEhWU-_ul7OM8pCDig.png" /></figure><p>To get some more detail line on which node service is running or ip address</p><p><strong># kubectl get all -o wide</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*_VbKl0o37TQUZerhj77fsg.png" /></figure><p>we check node capacity that how much pods we can create in a single node.</p><p><strong><em># kubectl get nodes -o json | jq ‘.items[] | {name: .metadata.name, capacity: .status.capacity}’</em></strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/859/1*a0furBKMOC0urf7uR1kcQA.png" /><figcaption>To get the detail we first need to install jq using “apt install jq”</figcaption></figure><p>Kubernetes consists of Building block like Container, Pod, Deployment, Service, Ingress</p><p><strong>Container: </strong>Programs running on Kubernetes are packaged as Linux containers. Containers are a widely accepted standard, so there are already many <a href="https://hub.docker.com/explore/">pre-built images</a> that can be deployed on Kubernetes.</p><p>Any program and all its dependencies can be bundled up into a single file and then shared on the internet. Anyone can download the container and deploy it on their infrastructure with very little setup required.</p><p><strong>Pod: </strong>Kubernetes doesn’t run containers directly; instead it wraps one or more containers into a higher-level structure called a <a href="https://kubernetes.io/docs/concepts/workloads/pods/pod/">pod</a>. Any containers in the same pod will share the same resources and local network. Containers can easily communicate with other containers in the same pod as though they were on the same machine while maintaining a degree of isolation from others.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/713/1*KfXtK0w5zQGTNTlI-OuiHw.png" /></figure><p>Pods can hold multiple containers, but you should limit yourself when possible. Because pods are scaled up and down as a unit, all containers in a pod must scale together, regardless of their individual needs. This leads to wasted resources and an expensive bill.</p><p><strong>Deployments: </strong>Although pods are the basic unit of computation in Kubernetes, they are not typically directly launched on a cluster. Instead, pods are usually managed by one more layer of abstraction: the deployment.</p><p>A deployment’s primary purpose is to declare how many replicas of a pod should be running at a time. When a deployment is added to the cluster, it will automatically spin up the requested number of pods, and then monitor them. If a pod dies, the deployment will automatically re-create it.</p><p>Using a deployment, you don’t have to deal with pods manually. You can just declare the desired state of the system, and it will be managed for you automatically.</p><p><strong>Services: </strong>A Kubernetes <strong>Service</strong> is an abstraction which defines a logical set of <strong>Pods</strong> and a policy by which to access them - sometimes called a micro-service.</p><p>Service is of three type</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/638/1*Z2xFE7x5iVjTSQ8zWziUBQ.jpeg" /></figure><p><strong><em>ClusterIP: </em></strong>A ClusterIP service is the default Kubernetes service. It gives you a service inside your cluster that other apps inside your cluster can access. There is no external access.</p><p>Using Kubernetes proxy and <strong>ClusterIP</strong>: The default Kubernetes ServiceType is ClusterIp, which exposes the Service on a cluster-internal IP. To reach the ClusterIp from an external source, you can open a Kubernetes proxy between the external source and the cluster. This is usually only used for development.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/502/1*bPIWACsqArSI-ommIAAK0Q.jpeg" /></figure><p>The YAML for a ClusterIP service looks like this:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/701/1*2ytHgXnxhZ8NskinG2qH0g.png" /></figure><p><strong><em>NodePort</em></strong></p><p>A NodePort service is the most primitive way to get external traffic directly to your service. NodePort, as the name implies, opens a specific port on all the Nodes (the VMs), and any traffic that is sent to this port is forwarded to the service.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/738/1*yVcKlFd4DqcKNleV5JcBDg.jpeg" /></figure><p>The YAML for a NodePort service looks like this:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/653/1*cX7UaL0UxoOLaQKZyGHeQQ.png" /></figure><p>Basically, a NodePort service has two differences from a normal “ClusterIP” service. First, the type is “NodePort.” There is also an additional port called the NodePort that specifies which port to open on the nodes. If you don’t specify this port, it will pick a random port.</p><p><strong><em>LoadBalancer</em></strong></p><p>A LoadBalancer service is the standard way to expose a service to the internet. On GKE, this will spin up a <strong>Network Load Balancer</strong> that will give you a single IP address that will forward all traffic to your service.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/716/1*IAT04vP3bnag_rHN8_v20A.jpeg" /></figure><p>The big downside is that each service you expose with a LoadBalancer will get its own IP address, and you have to pay for a LoadBalancer per exposed service, which can get expensive!</p><p><strong>Ingress: </strong>Created a cluster of nodes, and launch deployments of pods onto the cluster. There is one last problem to solve, however: allowing external traffic to your application.</p><p>By default, Kubernetes provides isolation between pods and the outside world. If you want to communicate with a service running in a pod, you have to open up a channel for communication. This is referred to as ingress.</p><p>Ingress isn’t a type of Sevice, but rather an object that acts as a reverse proxy and single entry-point to your cluster that routes the request to different services. The most basic Ingress is the<strong> </strong><a href="https://github.com/kubernetes/ingress-nginx"><strong>NGINX Ingress Controller</strong></a>, where the NGINX takes on the role of reverse proxy, while also functioning as SSL.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/600/1*xIClkNXgCxJPE6S50TyvaA.png" /></figure><p><strong>Example</strong>: Creating nginx deployment and service</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/872/1*ghwtZeB_aW_hAFrYuQaqNg.png" /></figure><p>Deployment is created for nginx where replicas are 2 means create two similar pods and strategy type is Recreate means if pod get die it recreate the same again. To apply we run below command</p><p><strong><em># kubectl create -f nginx-deploy.yaml</em></strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/759/1*9vNEcb2wQN4n73bLjdnsIA.png" /></figure><p>Service created of the same nginx with — type=LoadBalancer</p><p><strong><em># kubectl create -f nginx-srv.yaml</em></strong></p><p>we can update deployment by changing in the file and through below command.</p><p><strong><em># kubectl apply -f nginx-deploy.yaml</em></strong></p><h3><strong>Sidecar container</strong></h3><p>A sidecar container is generally used to monitor something on the primary application and perform some task on that information alleviating the primary of that responsibility.</p><p>This container checks every five seconds to a file where the app container is writing a random number. If the number is greater than ‘7’ it displays a good job. The container could collect all the numbers and send them off, or do any number of tasks. The point is to take care of the information so that it doesn’t have to be handled by the primary app.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*H2B26I9xmSiR1JedH8Mafw.png" /></figure><p>Here two container running on the same pod where the app is primary container and num-checker is secondary which act as a sidecar container.</p><p><strong># kubectl log -f my-pod</strong></p><p>Getting below output when checking the log of sidecar container</p><p><strong># kubectl logs -f my-pod app num-checker</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/496/1*DSsioDRMq9m1wMA6JGDZ9g.png" /></figure><h3><strong>Helm</strong></h3><p><a href="https://helm.sh/">Helm</a>, the package manager for Kubernetes, is a useful tool for installing, upgrading and managing applications on a Kubernetes cluster. Helm packages are called <em>charts</em>.</p><p><strong>Installation</strong></p><p>While several <a href="https://github.com/helm/helm/blob/master/docs/install.md">methods to install Helm</a> exists, the simplest way to install Helm is to run Helm’s installer script in a terminal:</p><pre>curl <a href="https://raw.githubusercontent.com/kubernetes/helm/master/scripts/get">https://raw.githubusercontent.com/kubernetes/helm/master/scripts/get</a> | bash</pre><p><strong>Initialization</strong></p><p>After installing helm on your machine, initialize Helm on your Kubernetes cluster:</p><p>Set up a <a href="https://kubernetes.io/docs/tasks/configure-pod-container/configure-service-account/">ServiceAccount</a> for use by tiller.</p><pre>kubectl --namespace kube-system create serviceaccount tiller</pre><p>Give them ServiceAccount full permissions to manage the cluster.</p><pre>kubectl create clusterrolebinding tiller --clusterrole cluster-admin --serviceaccount=kube-system:tiller</pre><p>Initialize helm and tiller.</p><pre>helm init --service-account tiller --wait</pre><p>Now verify Helm using</p><pre>helm version</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/864/1*hmTYX3RxWuy3agecZZqdeA.png" /></figure><p><strong>Helm Auto-completions</strong></p><pre>$ helm completion bash<br>$ source &lt;(helm completion bash)</pre><p>Above command enable helm auto-completion</p><h3>Persistent Volumes</h3><p>The <strong>PersistentVolume</strong> the subsystem provides an API for users and administrators that abstracts details of how storage is provided from how it is consumed. To do this we introduce two new API resources: <strong>PersistentVolume</strong> and <strong>PersistentVolumeClaim</strong>.</p><p>A <strong>PersistentVolume</strong> (PV) is a piece of storage in the cluster that has been provisioned by an administrator. It is a resource in the cluster just like a node is a cluster resource. PVs are volume plugins like Volumes but have a lifecycle independent of any individual pod that uses the PV. we share this volume using NFS, Host Path, ISCSI</p><p>While <strong>PersistentVolumeClaims</strong> allow a user to consume abstract storage resources, it is common that users need <strong>PersistentVolumes</strong>with varying properties, such as performance, for different problems.</p><p>First, we create PersistentVolume</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/684/1*NuDHlKzxw37liJaLjuqd8Q.png" /></figure><p>In this PersistentVolume.yaml volume is mounting from hostPath and PersistentVolumeReclaimPolicy is Delete means data is also being deleted after the container is being deleted if we want to keep data even after container get deleted</p><p><strong><em>PersistentVolumeReclaimPolicy</em></strong></p><p>Current reclaim policies are:</p><ul><li>Retain — manual reclamation</li><li>Recycle — basic scrub (<strong>rm -rf /thevolume/*</strong>)</li><li>Delete — associated storage asset such as AWS EBS, GCE PD, Azure Disk, or OpenStack Cinder volume is deleted</li></ul><p>Currently, only NFS and HostPath support recycling. AWS EBS, GCE PD, Azure Disk, and Cinder volumes support deletion.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/781/1*SshJyGEonov029X4Z_T4Mw.png" /></figure><p>Second, we create PersistentVolumeClaim</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/533/1*KL_D285abBO37k4euGga3g.png" /></figure><p>In this, we claim the required amount of resources from PersistentVolume which is being created by the administrator.</p><p>The access modes are:</p><ul><li>ReadWriteOnce — the volume can be mounted as read-write by a single node</li><li>ReadOnlyMany — the volume can be mounted read-only by many nodes</li><li>ReadWriteMany — the volume can be mounted as read-write by many nodes</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/653/1*Rn1rQTfKSm78TSTgKkaobQ.png" /></figure><p>Here we check that status of PVC is in Bound state which means that that much of resources is reserved.</p><p>Now we create pod using the same PVC</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/828/1*3ZwLloOJt4DUOFg-RpViGw.png" /></figure><p>After this, if we check PersistentVolume, PersistentVolumeClaim then we get volume field is reflecting that is being used by nginx pod which created.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/982/1*mhpp6MayIm1fkMFSYLwByw.png" /></figure><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=f2942739a2c6" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Continuous Deployment: Automated Software Delivery]]></title>
            <link>https://medium.com/@me.sanjeev3d/continuous-deployment-automated-software-delivery-5236f462d8b4?source=rss-89634bf8d93a------2</link>
            <guid isPermaLink="false">https://medium.com/p/5236f462d8b4</guid>
            <category><![CDATA[jenkins]]></category>
            <category><![CDATA[continuous-integration]]></category>
            <category><![CDATA[deployment]]></category>
            <category><![CDATA[continuous-delivery]]></category>
            <dc:creator><![CDATA[Sanjeev Gautam]]></dc:creator>
            <pubDate>Thu, 16 May 2019 13:09:07 GMT</pubDate>
            <atom:updated>2019-05-18T06:25:16.282Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/882/1*9iFTQLhii5jBQHhNeWyzqQ.png" /></figure><p>Continuous deployment is a strategy for software releases wherein any code commit that passes the automated testing phase is automatically released into the production environment, making changes that are visible to the software’s users.</p><p>Whereas Traditional approach is Manual deployments which is error-prone. This is because it involves humans doing stuff, if anything <em>can</em> go wrong, it <em>will</em> go wrong. Important steps in a release can be accidentally missed, faults that occur during a release may not be spotted, the incorrect versions of software can be shipped and broken software ends up going live. If you are lucky, you’ll be able to recover from this quickly. If you are unlucky, well… it’s pretty embarrassing at best.</p><p>Automated deployments don’t suffer from variability. Once configured, the process is set and will be the same every time a release is initiated. If it works the first time you deploy your software into a given environment, it’ll work the 100th time.</p><h3><strong>CD approach</strong></h3><p>Continuous delivery is a software engineering <strong>approach</strong> in which teams produce software in short cycles, ensuring that the software can be reliably released at any time and, when releasing the software, doing so manually. It aims at building, testing, and releasing software with greater speed and frequency.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/458/1*SIaPVl9X_zVVWlCuwx8rFA.png" /></figure><h3><strong>Why CD…?</strong></h3><p><strong><em>Lower risk.</em></strong><em> </em>The number one reason I like to deploy each new feature as soon as it is done is that it lowers the risk. Every time you change the software there is a risk that bugs are introduced. If the change deployed is small, there is less code to look through in case of a problem. If you only deploy new software every three weeks, there is a lot more code that could be causing a problem. If a problem can’t be found or fixed quickly, it is also a lot easier to revert a small deploy than a large deploy.</p><p><strong><em>Features reach customers faster.</em></strong> All things being equal, the faster a feature reaches the customer, the better. Having a feature ready for production, but not deploying it, is wasteful.</p><p><strong><em>Faster feedback.</em></strong> The sooner the customer starts using the new feature, the sooner you hear what works, what doesn’t work, and what improvements they would like. Very often, the next features to be developed is not known until the customer has tried out the current feature and provided feedback. Therefore, the sooner it is deployed, the sooner you can find and work out the remaining problems in the code.</p><h3><strong>Stages in Continuous deployment</strong></h3><p><strong><em>Code Analysis: A</em></strong>nalysis of the code is for security vulnerabilities, and comparing it against a set of coding rules meant to keep your code readable and maintainable according to team standards.</p><p>Running code analysis can, and should, be done at multiple points, both on the developer workstation and/or through a branch build of the checked-in code.</p><p>Tool use for code analysis is <a href="https://www.sonarqube.org/"><strong>SonarQube</strong></a> <em>the most powerful code quality tool</em> with a lot of measures and filters.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*vyak9g7DfsR9OOwnpuHQcw.png" /><figcaption>Integration of sonarQube</figcaption></figure><ol><li>Developers code in their IDEs and use <a href="https://sonarlint.org/">SonarLint</a> to run the local analysis.</li><li>Developers push their code into their favorite SCM: git, SVN, Bitbucket.</li><li>The Continuous Integration Server triggers an automatic build, and the execution of the SonarScanner required to run the SonarQube analysis.</li><li>The analysis report is sent to the SonarQube Server for processing.</li><li>SonarQube Server processes and stores the analysis report results in the SonarQube Database and displays the results in the UI.</li><li>Developers review, comment, challenge their Issues to manage and reduce their Technical Debt through the SonarQube UI.</li><li>Managers receive Reports from the analysis. Ops use JMX to monitor SonarQube Server.</li></ol><p><strong>Build: Build</strong> means to <strong>Compile the project, </strong>it<strong> </strong>refers to the process that converts files and other assets into a software product in its final or consumable form. The build may include:</p><ul><li>compiling source files</li><li>packaging compiled files into compressed formats (such as jar, zip)</li><li>producing installers</li><li>creating or updating of database schema or data</li></ul><p><strong>Testing: </strong>Once the build phase is over, then you move on to the <strong>testing phase</strong>. In this phase, we have various kinds of testing. One of them is the <em>unit test</em> (where we test the chunk/unit of software or for its sanity test), testing phase again comes into action after deployment for a sanity test.</p><p><strong>Deploy: Deploy</strong> means to <strong>Publish the output, </strong>it<strong> </strong>is a process through which the code/build is moved from one environment to another. Best practice of deploying code is to deploy in different stages like dev, staging and then to the production environment.</p><p><strong>Tool use in Continuous deployment</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/676/1*a_BZUKUJPEA3GFxZOmLjfg.png" /></figure><ul><li><strong>Jira</strong> Software for issue tracking</li><li><strong>Git</strong> for version control</li><li><strong>Bitbucket</strong> for repository management</li><li><strong>Jenkins </strong>for the build, test, and deploy automation</li><li><strong>Ansible</strong> for configuration management on cross-host</li><li><strong>Docker</strong> for containerization tool.</li><li><strong>Kubernetes</strong> as an Orchestration tool.</li><li><strong>Terraform</strong> Use as Infra as a code tool.</li></ul><h3><strong>Case Study: FINTECH Project</strong></h3><p><strong>Requirement</strong></p><p>This project includes running of dozen of micro-services in Kubernetes(k8) Cluster along with the integration of other components of GCP which include CloudSQL, PubSub, Storage Bucket, Stack Driver, Container Registry, Cloud Endpoint. The entry point to Kubernetes Cluster should be an Nginx Load Balancer.</p><p>Everything deployed through CI/CD pipeline using Jenkins. In CI, a new k8 cluster will be deployed using terraform and all the micro-services will be deployed using Helm Chart.</p><p>On the other hand, in CD, changes will be firstly deployed on Staging cluster and Production Cluster.</p><p>Apart from this, there should be an auto-scalability in pod and node level in all the k8 clusters.</p><p><strong>Pipeline Structure to deploy Infrastructure</strong></p><p>This project is used to build full infrastructure on Google Kubernetes Engine. On every run, it builds full infrastructure, in CI with includes the deployment of k8s cluster, deployment of CloudSQL, deployment of micro-services and Nginx load balancer using Helm Chart.</p><p>In CD, pipeline updates k8 cluster in case of any changes from the desired state. In subsequent stages, it updates the micro-services pods accordingly in Staging. If pipeline passes from Staging, it will automatically deploy it on production.</p><blockquote><strong><em>CI stages</em></strong></blockquote><ul><li><strong>Infra Build: (Using Terraform)</strong></li></ul><p>This stage Initialize the terraform workspace</p><p>Create an instance and install all necessary packages for Redis.</p><p>Create Gcloudsql server with Mysql 5.7. It sets up a password for root and proxy user.</p><p>Create a CI cluster with defined node and machine type. Apart from this it creates secrets for cloud_proxy, Pub-sub and initializes helm in the cluster.</p><ul><li><strong>SQL table setup:</strong></li></ul><p>In this stage, a new database is created in cloudsql using cloud_proxy. After database creation, all necessary tables are being created in the same DB. The main purpose of this stage is to setup database and tables before micro-services start on the k8s cluster.</p><ul><li><strong>Helm Deployment:</strong></li></ul><p>In this stage, helm installs Ingress Load balancer first from defined chart. Then it starts all micro-services container on CI k8s cluster and configures ingress resource which is used to rewrite the request.</p><ul><li><strong>Test Stage:</strong></li></ul><p>This stage checks the micro-services pod status. If any of the microservices is in restarted, crashed or in loopback state, it breaks the pipeline.</p><ul><li><strong>Cleanup Stage:</strong></li></ul><p>This stage destroys all resource created by terraform. This stage always runs even if the pipeline fails to avoid manual interaction with CI resources.</p><blockquote><strong><em>CD Stages</em></strong></blockquote><ul><li><strong>Staging Cluster Setup: (Using Terraform)</strong></li></ul><p>This stage checks the staging resources which includes the Staging-redis-server, staging-cloudsql, staging-cluster, helm Initialize in k8s and all required secrets ( Pub/sub, proxy-user, etc.) If any resource is missing then terraform fix only that resource.</p><ul><li><strong>Staging SQL table setup:</strong></li></ul><p>This stage sets up the staging cloudsql DB and tables. It creates the database/ tables if not exist. So that database will be up before micro-services deployment.</p><ul><li><strong>Staging Helm Deployment:</strong></li></ul><p>Pipeline pauses in this stage and wait for user input button. There is [ Yes / No ] button if user select yes button then all microservices is updated along with Staging ingress load balancer. If the deployment is not present in the kubernetes cluster then it installs all microservices. If a user selects No button then no action will perform.</p><ul><li><strong>Staging Rollback:</strong></li></ul><p>Again pipeline will pause in this stage and waits for user [Yes/No] button. If something went wrong on staging after updating of micro-services then this will rollback all changes and set the state of all micro-services which was before the update. This stage only runs if the user selects the yes button. If select no button then no action will perform.</p><ul><li><strong>Production Cluster setup:</strong></li></ul><p>In this stage terraform checks all production resources, secrets, helm. If everything is present then no action will be performed. If there will be something missing then terraform will create only that missing resource. It also manages the cloudsql failover replication, setup backup time and enables backup.</p><ul><li><strong>Production SQL table setup:</strong></li></ul><p>This stage sets up the production of cloudsql DB and tables. It created the database/ tables if not exist. So that database up before micro-services deployment.</p><ul><li><strong>Helm production Deployment:</strong></li></ul><p>Helm will check the SHA of images on GCR and only upgrades microservices with new images on GCR.</p><ul><li><strong>Rollback Production changes:</strong></li></ul><p>Pipeline pauses on this stage and wait for user input with [Yes/no] button. If something went wrong then the user just need to go to Jenkins UI and respective pipeline. Just click on yes button, then helm will rollback micro-services to the previous state before update.</p><ul><li><strong>Final Stage:</strong></li></ul><p>In this stage, Email notification sends to defined group with the status of this deployment.</p><h3><strong>Deployment Strategy</strong></h3><p>To meet frequent deployment challenges face earlier, application and infrastructure teams should devise and adopt a deployment strategy suitable for their use case.</p><blockquote><strong>Blue-Green Deployment</strong></blockquote><p>This is a fail-safe process. In this method, two identical production environments work in parallel.</p><p>One is the currently-running production environment receiving all user traffic (depicted as Blue). The other is a clone of it, but idle (Green). Both use the same database back-end and app configuration:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/880/0*NW5M4nfNO66OI7hd.png" /></figure><p>The new version of the application is deployed in the green environment and tested for functionality and performance. Once the testing results are successful, application traffic is routed from blue to green. Green then becomes the new production.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/880/0*SqXwrviudi2QCF3S.png" /></figure><p>If there is an issue after green becomes live, traffic can be routed back to blue.</p><p>In a blue-green deployment, both systems use the same persistence layer or database back end. It’s essential to keep the application data in sync, but a mirrored database can help achieve that.</p><p>Once green becomes live, you can shut down or recycle the old blue instances. You might deploy a newer version on those instances and make them the new green for the next release. Blue-green deployments rely on traffic routing.</p><blockquote><strong>Canary Deployment</strong></blockquote><p>Canary deployment is like blue-green, except it’s more risk-averse. Instead of switching from blue to green in one step, you use a phased approach.</p><p>With the canary deployment, you deploy a new application code in a small part of the production infrastructure. Once the application is signed off for release, only a few users are routed to it. This minimizes any impact.</p><p>With no errors reported, the new version can gradually roll out to the rest of the infrastructure. The image below demonstrates canary deployment:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/880/0*1UhMU6MuLBMqkjSQ.png" /></figure><p>The main challenge of canary deployment is to devise a way to route some users to the new application. Also, some applications may always need the same group of users for testing, while others may require a different group every time.</p><p>Consider a way to route new users by exploring several techniques:</p><ul><li>Exposing internal users to the canary deployment before allowing external user access;</li><li>Basing routing on the source IP range;</li><li>Releasing the application in specific geographic regions;</li><li>Using an application logic to unlock new features to specific users and groups. This logic is removed when the application goes live for the rest of the users.</li></ul><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=5236f462d8b4" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Apache Webserver]]></title>
            <link>https://medium.com/@me.sanjeev3d/https-medium-com-me-sanjeev3d-apache-webserver-a0d7f8167063?source=rss-89634bf8d93a------2</link>
            <guid isPermaLink="false">https://medium.com/p/a0d7f8167063</guid>
            <category><![CDATA[web-server]]></category>
            <category><![CDATA[apache-webserver]]></category>
            <category><![CDATA[apache]]></category>
            <category><![CDATA[devops]]></category>
            <dc:creator><![CDATA[Sanjeev Gautam]]></dc:creator>
            <pubDate>Thu, 10 Jan 2019 19:11:52 GMT</pubDate>
            <atom:updated>2019-01-14T19:52:36.055Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/370/1*i2yB0JFGwq3wcEzse-yl6A.jpeg" /></figure><p>Apache is the most widely used web server software. Developed and maintained by Apache Software Foundation, Apache is an open source software available for free. It runs on 67% of all web servers in the world. It is fast, reliable, and secure. It can be highly customized to meet the needs of many different environments by using extensions and modules.</p><p>Apache breaks down its functionality and components into individual units that can be customized and configured independently. The basic unit that describes an individual site or domain is called a virtual host. Virtual hosts allow one server to host multiple domains or interfaces by using a matching system.</p><p>Each domain that is configured will direct the visitor to a specific directory holding that site’s information, without ever indicating that the same server is also responsible for other sites.</p><h3>Features of Apache Server</h3><ul><li>Apache server is a free and an open source web server.</li><li>It can be installed on all operating systems like Linux, Windows, Unix, FreeBSD, Solaris, Mac OS X etc.</li><li>It is a powerful, flexible, HTTP/1.1 compliant web server.</li><li>This server is highly configurable and extensible with third-party modules.</li><li>It provides complete source code and comes with an unrestricted license.</li><li>Apache supports some of the capabilities like CGI (Common Gateway Interface) and SSI (Server Side Includes), URL redirection, loading modules support, user authentication, proxy caching abilities etc.</li></ul><h3>Installation and configuration on Ubuntu</h3><p>Apache webserver is already available in our repository, so we can download simply by single command.</p><p># apt-get install apache2</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/416/1*VGTbu3yaTrcYIP_IXFzSyA.png" /><figcaption>version oy apache2 server</figcaption></figure><p>After instalation, we check webserver is enabled or not enable using<strong><em> systemctl enable apache2</em></strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/905/1*5M4tLR87gcVDFBHS-3c_7w.png" /><figcaption>apache 2 is in running state</figcaption></figure><p>we can check using the browser <a href="http://192.168.1.110/">http://192.168.1.110/</a></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/805/1*ocVarhcIN5V04IF0ohnDnQ.png" /><figcaption>apache2 default page get loaded.</figcaption></figure><p>After installation check the /etc/apache2 directory</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/630/1*i1VvCyxBCAaF4-l_n-HoTw.png" /><figcaption>structure of /etc/apache2</figcaption></figure><blockquote><strong>Configuration FilesDescription</strong></blockquote><p><strong><em>Content</em></strong></p><ul><li><strong>/var/www/html</strong>: The actual web content, which by default only consists of the default Apache page we saw earlier, is served out of the /var/www/html directory. This can be changed by altering Apache configuration files.</li></ul><p><strong><em>Server Configuration</em></strong></p><ul><li><strong>/etc/apache2</strong>: The Apache configuration directory. All of the Apache configuration files reside here.</li><li><strong>/etc/apache2/apache2.conf</strong>: The main Apache configuration file. This can be modified to make changes to the Apache global configuration. This file is responsible for loading many of the other files in the configuration directory.</li><li><strong>/etc/apache2/ports.conf</strong>: This file specifies the ports that Apache will listen on. By default, Apache listens on port 80 and additionally listens on port 443 when a module providing SSL capabilities is enabled.</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/695/1*2OLOPgdRrgvmjxxJ6Av8wg.png" /></figure><ul><li><strong>/etc/apache2/sites-available:</strong> The directory where per-site “Virtual Hosts” can be stored. Apache will not use the configuration files found in this directory unless they are linked to the <strong>sites-enabled </strong>directory. Typically, all server block configuration is done in this directory and then enabled by linking to the other directory with the <strong>a2ensite </strong>command.</li><li><strong>/etc/apache2/sites-enable</strong>: The directory where enabled per-site “Virtual Hosts” are stored. Typically, these are created by linking to configuration files found in the <strong>sites-available </strong>directory with the <strong>a2ensites</strong>. Apache reads the configuration files and links found in this directory when it starts or reloads to compile a complete configuration.</li><li><strong>/etc/apache2/conf-available</strong>, <strong>/etc/apache2/conf-enabled</strong>: These directories have the same relationship as the <strong>sites-available</strong> and<strong> sites-enabled</strong> directories, but are used to store configuration fragments that do not belong in a Virtual Host. Files in the <strong>conf-availabl</strong>e directory can be enabled with the<strong> a2enconf</strong> command and disabled with the <strong>a2disconf </strong>command.</li><li><strong>/etc/apache2/mods-available</strong>, <strong>/etc/apache2/mods-enabled</strong>: These directories contain the available and enabled modules, respectively. Files in ending in <strong>.load</strong> contain fragments to load specific modules, while files ending in <strong>.conf</strong> contain the configuration for those modules. Modules can be enabled and disabled using the <strong>a2enmod</strong> and <strong>a2dismod</strong> command.</li></ul><p><strong>Server Logs</strong></p><ul><li><strong>/var/log/apache2/access.log</strong>: By default, every request to your web server is recorded in this log file unless Apache is configured to do otherwise.</li><li><strong>/var/log/apache2/error.log</strong>: By default, all errors are recorded in this file. The <strong>LogLevel</strong> directive in the Apache configuration specifies how much detail the error logs will contain.</li></ul><p>we can configure SSL using</p><p><strong><em>vim /etc/apache2/sites-available/default-ssl.conf</em></strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/811/1*j250tTRwvPsHCa3N5HcVmA.png" /></figure><p>To change in a server CGI script, we can change using</p><p><strong><em>vim /etc/apache2/conf-available/serve-cgi-bin.conf</em></strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/704/1*IHDWQyX-gcghszcJHJH4cQ.png" /></figure><blockquote><strong>Configuration Directives in apache2.conf</strong></blockquote><p>The main configuration details for your Apache server are held in the “<strong>/etc/apache2/apache2.conf</strong>” file.</p><p>This file is divided into three main sections: configuration for the global Apache server process, the configuration for the default server, and configuration of Virtual Hosts.</p><p>In Ubuntu and Debian, the majority of the file is for global definitions, and the configuration of the default server and virtual hosts is handled at the end, by using the “Include …” directive.</p><p>There are a number of different “Include” statements. These load module definitions, the <strong>ports.conf</strong> document, the specific configuration files in the “<strong>conf.d/</strong>” directory, and finally, the Virtual Host definitions in the “<strong>sites-enabled/</strong>” directory.</p><p><strong>Global Configuration Section</strong></p><p>This section is used to configure some options that control how Apache works as a whole.</p><p><strong><em>Timeout: </em></strong>By default, this parameter is set to “300”, which means that the server has a maximum of 300 seconds to fulfill each request. This is probably too high for most setups and can safely be dropped to something between 30 and 60 seconds.</p><p><strong><em>KeepAlive: </em></strong>This option, if set to “On”, will allow each connection to remain open to handle multiple requests from the same client. If this is set to “Off”, each request will have to establish a new connection, which can result in significant overhead depending on your setup and traffic situation.</p><p><strong><em>KeepAliveTimeout: </em></strong>This setting specifies how long to wait for the next request after finishing the last one. If the timeout threshold is reached, then the connection will die. Means that the next time content is requested, the server will establish a new connection.</p><p><strong><em>MPM Configuration: </em></strong>specifies the configuration of the MPM (Multi-Processing Module) options.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/416/1*FEqUxFwpTaojk-dPzBCAag.png" /><figcaption>apache2 -l</figcaption></figure><p><strong>AllowOverride:</strong> directive is used to allow the use of .htaccess within the web server to allow overriding of the <strong>Apache</strong> config on a per-directory basis. … That’s why it only works when you have <strong>AllowOverride</strong> All because you are telling the webserver to allow the use of a .htaccess file which CI uses.</p><p><strong>Exploring the Default Virtual Host File</strong></p><p><strong><em># sudo vim /etc/apache2/sites-available/000-default.conf</em></strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/758/1*Q-jcKoBEcN4RCzXjiH_k2g.png" /></figure><p>The default Virtual Host is configured to handle any request on port 80, the standard http port. This is defined in the declaration header where it says “*:80”, meaning port 80 on any interface.</p><p>The “DocumentRoot” option specifies where the content that is requested for this Virtual Host will be located. The default Virtual Host is set up to serve content out of the “/var/www” directory on Ubuntu.</p><p><strong>Enabling Sites and Modules in Apache</strong></p><p>To automatically create a symbolic link in the “sites-enabled” directory to an existing file in the “sites-available” directory, issue the following command:</p><p><strong># a2ensite Vitual_Host_FileName</strong></p><p>After enabling a site, issue the following command to tell Apache to re-read its configuration files, allowing the change to propagate:</p><p><strong># systemctl restart apache2</strong></p><p>Modules can be enabled or disabled by using the <strong>“a2enmod” </strong>and <strong>“a2dismod”</strong> commands respectively. They work in the same way as the “site” versions of these commands.</p><blockquote><strong>Lab Section to setup own website</strong></blockquote><p>Create a virtual host in sites-available for the specific website.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/587/1*4_bGEtvkxLFUJkbdkUDNKw.gif" /></figure><p>Here we create <strong>sanjeev-apache.conf </strong>in site available and we have to create a symlink for same in the sites-enabled directory.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/685/1*UXiWNaqmEM_m1sgYVbqSxg.png" /><figcaption>a2ensite is used to create a symlink</figcaption></figure><p><em># systemctl reload apache2</em></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/931/1*SlPqco1o1wndCRCQfSH2OQ.png" /></figure><p>After taking reload to apache2 service symlink start working.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/626/1*wo0UqiI5BgCjtY3dUGjxcA.png" /><figcaption>virtual host created for sanjeev-apache.net</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/509/1*nGkThO_6K9Dz72RBaEjDkA.png" /><figcaption>created document root file in /var/www/test</figcaption></figure><p>Make an entry in /etc/hosts to resolve ip address to domain name locally</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/574/1*P2EBIQT7_yZGWoARXb04vA.gif" /><figcaption>hostname entry in /etc/hosts</figcaption></figure><p>Now check same using <a href="http://sanjeev-apache.net/"><strong>http://sanjeev-apache.net/</strong></a></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*UbQzM9h7JIb3FzHuKMkLGA.png" /></figure><h4>.htaccess</h4><p>.htaccess is a configuration file for use on web servers running the Apache Web Server software. When a <strong>.htaccess</strong> file is placed in a directory which is in turn ‘loaded via the Apache Web Server’, then the .htaccess file is detected and executed by the Apache Web Server software.</p><p>These .htaccess files can be used to alter the configuration of the Apache Web Server software to <strong>enable/disable</strong> additional functionality and features that the Apache Web Server software has to offer. These facilities include basic redirect functionality</p><p>we can also create a .htaccess file using a text editor like vim, gedit. example:</p><p><strong>To Activate a .htaccess file</strong></p><p>To activate .htaccess file we have to alter the <strong>AllowOveride</strong> setting to all as shown below.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/597/1*3R5M9CArDYvvDApUk3wb6Q.gif" /></figure><p><strong>Five Common Uses for a .htaccess Page</strong></p><ol><li><strong>Mod_Rewrite:</strong> one of the most useful facets of the .htaccess file is mod_rewrite. You can use the space in the .htaccess file to designate and alter how URLs and web pages on your sites are displayed to your users.</li><li><strong>Authentication</strong>: Although using the .htaccess file does not require as many permissions as accessing the apache2.conf file would require, we can still make effective changes to a site. Once such change is to require a password to access certain sections of the webpage.</li></ol><p>The .htaccess passwords are kept in a file called .htpasswd. Go ahead and create and save that file, being sure to store it somewhere other than the web directory, for security reasons.</p><p>You should use the space inside the .htpasswd file to write in the name and passwords of all the users that you want to have access to the protected part of the site.</p><p>You can use <a href="http://www.joeswebtools.com/security/htpasswd-generator/">this useful site</a> to generate the username and encrypted password pair. If the username of your authorized user is <strong>jsmith</strong> and password is <strong>“awesome”</strong>, the pair would look like this: <strong>jsmith:VtweQU73iyETM</strong>. You can paste as many lines as needed into the .htpasswd file, but be sure that every user gets their own line.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/644/1*paa8VhmbHdJFzHSthvlr0A.png" /></figure><ul><li><strong>AuthUserFile:</strong> This line designates the server path to the .htpasswd file.</li><li><strong>AuthGroupFile:</strong> This line can be used to convey the location of the .htgroup. As we have not created such a file, we can leave /dev/null in place.</li><li><strong>AuthName:</strong> This is text that will be displayed at the password prompt. You can put anything here.</li><li><strong>AuthType: </strong>This refers to the type of authentication that will be used to check the passwords. The passwords are checked via HTTP and the keyword Basic should not be changed.</li><li><strong>Require valid-user:</strong> This line represents one of two possibilities. “Require valid-user,” tells the .htaccess file that there are several people who should be able to log into the password protected area. The other option is to use the phrase “require user <em>username</em>” to indicate the specific permitted person.</li></ul><p><strong>3. Custom Error Pages</strong>: the .htaccess file additionally allows you to create custom error pages for your site. Some of the most common errors are:</p><ul><li>400 Bad Request</li><li>401 Authorization Required</li><li>403 Forbidden Page</li><li>404 File not Found</li><li>500 Internal Error</li></ul><p>To make a page look friendlier and to provide more information to the site visitor than the default server error page offers, you can use the .htaccess file to create custom error pages.</p><p>I’m going to create a 404 page in this tutorial. However, you can substitute that error for whatever you prefer:</p><p>Once you have created and uploaded the desired error page, you can go ahead and designate its location in the .htaccess file.</p><p><strong><em>ErrorDocument 404 /new404.html</em></strong></p><p>Keep in mind that the Apache looks for the 404 pages located within the site’s root. If you placed the new error page in a deeper subdirectory, you need to include that in the line, making it look something like this:</p><p><strong><em>ErrorDocument 404 /error_pages/new404.html</em></strong></p><p><strong>4. Mime Types:</strong> In cases where your site features some application files that your server was not set up to deliver, you can add MIME types to your Apache server in the .htaccess file with the following code.</p><p><strong><em>AddType audio/mp4a-latm .m4a</em></strong></p><p>Be sure to replace application and file extension with the Mime Type that you want to support.</p><p><strong>5. SSI</strong>: Server Side Includes are a great time-saver on a website. One of the most common uses of SSI is to update a large number of pages with some specific data, without having to update each page individually (for example, if you want to change a quotation at the bottom of a page).</p><p>To enable SSI, type the following code into your .htaccess file.</p><p><strong><em>AddType text/html .shtml<br>AddHandler server-parsed .shtml</em></strong></p><p>These three lines have the effect of telling the .htaccess that .shtml files are valid, with the second line specifically making the server parse all files ending in .shtml for any SSI commands.</p><p>However, if you have many .html pages that you are not eager to rename with .shtml extensions, you can use another tactic to parse them for SSI commands, the XBitHack.</p><p>Adding this line to the .htaccess file makes Apache check all the html files with the appropriate permissions for Server Side Includes.</p><p><strong><em>XBitHack on</em></strong></p><p>To make a page eligible for the XBitHack, use this command:</p><p><strong><em>chmod +x pagename.html</em></strong></p><h4>Some more Configuration Directives in httpd.conf</h4><p><strong>AccessFileName</strong></p><p>AccessFileName names the file which the server should use for access control information in each directory. The default is <strong>.htaccess.</strong></p><p>Immediately after the AccessFileName directive, a set of Files tags apply access control to any file beginning with a .ht.</p><p><strong>Action</strong></p><p>The Action specifies a MIME content type and CGI script pair so that when a file of that media type is requested, a particular CGI script is executed.</p><p><strong>AddDescription</strong></p><p>When using <strong>FancyIndexing</strong> as an IndexOptions parameter, the AddDescription directive can be used to display user-specified descriptions for certain files or file types in a server-generated directory listing.</p><p><strong>AddEncoding</strong></p><p>AddEncoding names file name extensions which should specify a particular encoding type. AddEncoding can also be used to instruct some browsers to uncompress certain files as they are downloaded.</p><p><strong>AddHandler</strong></p><p>AddHandler maps file extensions to specific handlers. For example, the cgi-script handler can be matched with the extension .cgi to automatically treat a file ending with .cgi as a CGI script. The following is a sample AddHandlerdirective for the .cgi extension.</p><pre><strong>AddHandler cgi-script .cgi</strong></pre><p>This directive enables CGIs outside of the cgi-bin to function in any directory on the server which has the ExecCGI option within the directories container.</p><p>In addition to CGI scripts, the AddHandler directive is used to process server-parsed HTML and image-map files.</p><p><strong>AddLanguage</strong></p><p>AddLanguage associates file name extensions with specific languages. This directive is useful for Apache HTTP Servers which serve content in multiple languages based on the client Web browser’s language settings.</p><p><strong>AddType</strong></p><p>Use the AddType directive to define or override a default MIME type and file extension pairs. The following example directive tells the Apache HTTP Server to recognize the .tgz file extension:</p><pre><strong>AddType application/x-tar .tgz</strong></pre><p><strong>Allow</strong></p><p>Allow specifies which client can access a given directory. The client can be all, a domain name, an IP address, a partial IP address, a network/netmask pair, and so on. The DocumentRoot directory is configured to Allow requests from all, meaning everyone has access.</p><p><strong>AllowOverride</strong></p><p>The AllowOverride directive sets whether any Options can be overridden by the declarations in a <strong>.htaccess</strong> file. By default, both the root directory and the DocumentRoot are set to allow no .htaccess overrides.</p><p><strong>BrowserMatch</strong></p><p>The BrowserMatch directive allows the server to define environment variables and take appropriate actions based on the User-Agent HTTP header field — which identifies the client’s Web browser type. By default, the Web server uses BrowserMatch to deny connections to specific browsers with known problems and also to disable keepalives and HTTP header flushes for browsers that are known to have problems with those actions.</p><p><strong>Cache Directives</strong></p><p>The following, however, is a list of some of the more important cache-related directives.</p><ul><li><strong>CacheEnable</strong> — Specifies whether the cache is a disk, memory, or file descriptor cache. By default CacheEnableconfigures a disk cache for URLs at or below /.</li><li><strong>CacheRoot</strong> — Specifies the name of the directory containing cached files. The default CacheRoot is the /var/httpd/proxy/ directory.</li><li><strong>CacheSize </strong>— Specifies how much space the cache can use in kilobytes. The default CacheSize is 5 KB.</li></ul><p>The following is a list of some of the other common cache-related directives.</p><ul><li><strong>CacheMaxExpire </strong>— Specifies how long HTML documents are retained (without a reload from the originating Web server) in the cache. The default is 24 hours (86400 seconds).</li><li>C<strong>acheDefaultExpire</strong> — Specifies the expiry time in hours for a document that was received using a protocol that does not support expiry times. The default is set to 1 hour (3600 seconds).</li><li><strong>NoProxy</strong> — Specifies a space-separated list of subnets, IP addresses, domains, or hosts whose content is not cached. This setting is most useful for Intranet sites.</li></ul><p><strong>CacheNegotiatedDocs</strong></p><p>By default, the Web server asks proxy servers not to cache any documents which were negotiated on the basis of content (that is, they may change over time or because of the input from the requester). If CacheNegotiatedDocs is set to on, this function is disabled and proxy servers are allowed to cache such documents.</p><p><strong>CustomLog</strong></p><p>CustomLog identifies the log file and the log file format. By default, the access log is recorded to the<strong><em> /var/log/apache2/access.log</em></strong> file while errors are recorded in the <strong><em>/var/log/apache2/error.log</em></strong> file.</p><p>The default CustomLog format is the combined log file format, as illustrated here:</p><pre><strong><em>remotehost rfc931 user date &quot;request&quot; status bytes referrer user-agent</em></strong></pre><p><strong>DefaultType</strong></p><p>DefaultType sets a default content type for the Web server to use for documents whose MIME types cannot be determined. The default is text/plain.</p><p><strong>Deny</strong></p><p>Deny works similar to Allow, except it specifies who is denied access. The DocumentRoot is not configured to Deny requests from anyone by default.</p><p><strong>Directory</strong></p><p>&lt;Directory /path/to/directory&gt; and &lt;/Directory&gt; tags create a container used to enclose a group of configuration directives which apply only to a specific directory and its subdirectories. Any directive which is applicable to a directory may be used within Directory tags.</p><p>By default, very restrictive parameters are applied to the root directory (/), using the Options (refer to <a href="http://web.mit.edu/rhel-doc/5/RHEL-5-manual/Deployment_Guide-en-US/s1-apache-config.html#s2-apache-options">Options</a>) andAllowOverride directives. Under this configuration, any directory on the system which needs more permissive settings has to be explicitly given those settings.</p><p>The Directory container can be also be used to configure additional cgi-bin directories for server-side applications outside of the directory specified in the <strong>ScriptAlias</strong> directive.</p><p>To accomplish this, the Directory container must set the ExecCGI option for that directory.</p><p>For example, if CGI scripts are located in /home/my_cgi_directory, add the following Directory container to the httpd.conf file:</p><pre><strong>&lt;Directory /home/my_cgi_directory&gt;     Options +ExecCGI &lt;/Directory&gt;</strong></pre><p>Next, the AddHandler directive must be uncommented to identify files with the .cgi extension as CGI scripts.</p><p><strong>DirectoryIndex</strong></p><p>The DirectoryIndex is the default page served by the server when a user requests an index of a directory by specifying a forward slash (/) at the end of the directory name.</p><p>When a user requests the page <a href="http://example/this_directory/,">http://<strong><em>example</em></strong>/<strong><em>this_directory</em></strong>/,</a> they get either the DirectoryIndex page, if it exists, or a server-generated directory list. The default for DirectoryIndex is <strong>index.html.</strong> The server tries to find either of these files and returns the first one it finds. If it does not find one of these files and Options <strong>Indexes </strong>is set for that directory, the server generates and returns a listing, in HTML format, of the subdirectories and files within the directory, unless the directory listing feature is turned off.</p><p><strong>DocumentRoot</strong></p><p>DocumentRoot is the directory which contains most of the HTML files which are served in response to requests. The default DocumentRoot, for both the non-secure and secure Web servers, is the /var/www/html directory. For example, the server might receive a request for the following document:</p><pre><a href="http://sanjeev-apache.net/"><strong>http://sanjeev-apache.net/</strong></a></pre><p>The server looks for the following file in the default directory:</p><pre><strong>/var/www/html/index.html</strong></pre><p><strong>ErrorDocument</strong></p><p>The ErrorDocument directive associates an HTTP response code with a message or a URL to be sent back to the client. By default, the Web server outputs a simple and usually cryptic error message when an error occurs. The ErrorDocumentdirective forces the Web server to instead output a customized message or page.</p><p><strong>HostnameLookups</strong></p><p>HostnameLookups can be set to on, off, or double. If HostnameLookups is set to on, the server automatically resolves the IP address for each connection. Resolving the IP address means that the server makes one or more connections to a DNS server, adding processing overhead. If HostnameLookups is set to double, the server performs a double-reverse DNS look up adding even more processing overhead.</p><p>To conserve resources on the server, HostnameLookups is set to off by default.</p><p><strong>IfModule</strong></p><p>&lt;IfModule&gt; and &lt;/IfModule&gt; tags create a conditional container which are only activated if the specified module is loaded. Directives within the IfModule container are processed under one of two conditions. The directives are processed if the module contained within the starting &lt;IfModule&gt; tag is loaded. Or, if an exclamation point ! appears before the module name, the directives are processed only if the module specified in the &lt;IfModule&gt; tag is <em>not</em> loaded.</p><p><strong>Include</strong></p><p>Include allows other configuration files to be included at runtime.</p><p>The path to these configuration files can be absolute or relative to the ServerRoot.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/496/1*rG3RUuo_GgnWV21k7x79cQ.png" /></figure><p>For the server to use individually packaged modules, such as mod_ssl, mod_perl, and php, the following directive must be included in Section 1: Global Environment of apache2.conf:</p><pre><strong>Include conf.d/*.conf</strong></pre><p><strong>IndexIgnore</strong></p><p>IndexIgnore lists file extensions, partial file names, wildcard expressions, or full file names. The Web server does not include any files which match any of those parameters in server-generated directory listings.</p><p><strong>LanguagePriority</strong></p><p>LanguagePriority sets precedence for different languages in case the client Web browser has no language preference set.</p><p><strong>Listen</strong></p><p>The Listen command identifies the ports on which the Web server accepts incoming requests. By default, the Apache HTTP Server is set to listen to port 80 for non-secure Web communications and (in the ssl.conf file which defines any secure servers) to port 443 for secure Web communications.</p><p>If the Apache HTTP Server is configured to listen to a port under 1024, only the root user can start it. For port 1024 and above, httpd can be started as a regular user.</p><p>The Listen directive can also be used to specify particular IP addresses over which the server accepts connections.</p><p><strong>Location</strong></p><p>The &lt;Location&gt; and &lt;/Location&gt; tags create a container in which access control based on URL can be specified.</p><p>For instance, to allow people connecting from within the server’s domain to see status reports, use the following directives:</p><pre><strong>&lt;Location /server-status&gt;  <br> SetHandler server-status <br> Order deny,allow     <br> Deny from all      <br> Allow from <em>&lt;.sanjeev-apache.net&gt;</em> <br>&lt;/Location&gt;</strong></pre><p>Replace <strong><em>&lt;.sanjeev-apache.net&gt;</em></strong> with the second-level domain name for the Web server.</p><p>To provide server configuration reports (including installed modules and configuration directives) to requests from inside the domain, use the following directives:</p><pre><strong>&lt;Location /server-info&gt;     <br>SetHandler server-info     <br>Order deny,allow     <br>Deny from all     <br>Allow from <em>&lt;.example.com&gt;</em> <br>&lt;/Location&gt;</strong></pre><p>Again, replace <strong><em>&lt;.sanjeev-apache.net&gt;</em></strong> with the second-level domain name for the Web server.</p><p><strong>LogFormat</strong></p><p>The LogFormat directive configures the format of the various Web server log files. The actual LogFormat used depends on the settings given in the CustomLog directive.</p><p>The following are the format options if the CustomLog directive is set to combined:</p><p>%h (remote host’s IP address or hostname)</p><p>Lists the remote IP address of the requesting client. If HostnameLookups is set to on, the client hostname is recorded unless it is not available from DNS.</p><p>%l (rfc931)</p><p>Not used. A hyphen — appears in the log file for this field.</p><p>%u (authenticated user)</p><p>Lists the username of the user recorded if authentication was required. Usually, this is not used, so a hyphen -appears in the log file for this field.</p><p>%t (date)</p><p>Lists the date and time of the request.</p><p>%r (request string)</p><p>Lists the request string exactly as it came from the browser or client.</p><p>%s (status)</p><p>Lists the HTTP status code which was returned to the client host.</p><p>%b (bytes)</p><p>Lists the size of the document.</p><p>%\”%{Referer}i\” (referrer)</p><p>Lists the URL of the webpage which referred the client host to Web server.</p><p>%\”%{User-Agent}i\” (user-agent)</p><p>Lists the type of Web browser making the request.</p><p><strong>LogLevel</strong></p><p>LogLevel sets how verbose the error messages in the error logs are. LogLevel can be set (from least verbose to most verbose) to emerg, alert, crit, error, warn, notice, info, or debug. The default LogLevel is warn.</p><p><strong>MaxKeepAliveRequests</strong></p><p>This directive sets the maximum number of requests allowed per persistent connection. The Apache Project recommends a high setting, which improves the server’s performance. MaxKeepAliveRequests is set to 100 by default, which should be appropriate for most situations.</p><p><strong>Options</strong></p><p>The Options directive controls which server features are available in a particular directory. For example, under the restrictive parameters specified for the root directory, Options is only set to the FollowSymLinks directive. No features are enabled, except that the server is allowed to follow symbolic links in the root directory.</p><p>By default, in the DocumentRoot directory, Options is set to include Indexes and FollowSymLinks. Indexes permit the server to generate a directory listing for a directory if no DirectoryIndex (for example, index.html) is specified.FollowSymLinks allows the server to follow symbolic links in that directory.</p><p><strong>Order</strong></p><p>The Order directive controls the order in which allow and deny directives are evaluated. The server is configured to evaluate the Allow directives before the Deny directives for the DocumentRoot directory.</p><p><strong>PidFile</strong></p><p>PidFile names the file where the server records its process ID (PID). By default the PID is listed in /var/run/apache2.pid.</p><p><strong>Redirect</strong></p><p>When a webpage is moved, Redirect can be used to map the file location to a new URL. The format is as follows:</p><pre>Redirect /<strong><em>&lt;old-path&gt;</em></strong>/<strong><em>&lt;file-name&gt;</em></strong> http://<strong><em>&lt;current-domain&gt;</em></strong>/<strong><em>&lt;current-path&gt;</em></strong>/<strong><em>&lt;file-name&gt;</em></strong></pre><p>In this example, replace <strong><em>&lt;old-path&gt;</em></strong> with the old path information for <strong><em>&lt;file-name&gt;</em></strong> and <strong><em>&lt;current-domain&gt;</em></strong> and <strong><em>&lt;current-path&gt;</em></strong> with the current domain and path information for <strong><em>&lt;file-name&gt;</em></strong>.</p><p>In this example, any requests for <strong><em>&lt;file-name&gt;</em></strong> at the old location is automatically redirected to the new location.</p><p>For more advanced redirection techniques, use the mod_rewrite module included with the Apache HTTP Server. For more information about configuring the mod_rewrite module.</p><p><strong>ScriptAlias</strong></p><p>The ScriptAlias directive defines where CGI scripts are located. Generally, it is not good practice to leave CGI scripts within the DocumentRoot, where they can potentially be viewed as text documents. For this reason, a special directory outside of the DocumentRoot directory containing server-side executables and scripts is designated by the ScriptAliasdirective. This directory is known as a cgi-bin and is set to /var/www/cgi-bin/ by default.</p><p><strong>ServerName</strong></p><p>ServerName specifies a hostname and port number (matching the Listen directive) for the server. The ServerNamedoes not need to match the machine’s actual hostname.</p><p>The following is a sample ServerName directive:</p><pre>ServerName <a href="http://www.sanjeev-apache.net/">www.sanjeev-apache.net:80</a></pre><p>When specifying a ServerName, be sure the IP address and server name pair are included in the<strong> /etc/hosts </strong>file.</p><p><strong>ServerRoot</strong></p><p>The ServerRoot directive specifies the top-level directory containing website content. By default, ServerRoot is set to <strong>“/etc/apache2”</strong> for both secure and non-secure servers.</p><p><strong>VirtualHost</strong></p><p>&lt;VirtualHost&gt; and &lt;/VirtualHost&gt; tags create a container outlining the characteristics of a virtual host. The VirtualHost container accepts most configuration directives.</p><p>A commented VirtualHost container is provided in apache2.conf, which illustrates the minimum set of configuration directives necessary for each virtual host.</p><p><em>&lt;VirtualHost *:80&gt;<br> ServerAdmin</em><a href="/service/mailto:admin@sanjeev-apache.net"><em>admin@sanjeev-apache.net</em></a><em><br> ServerName sanjeev-apache.net<br> ServerAlias</em><a href="http://www.sanjeev-apache.net/"><em>www.sanjeev-apache.net</em></a><em><br> DocumentRoot /var/www/test/index.html<br> ErrorLog ${APACHE_LOG_DIR}/error.log<br> CustomLog ${APACHE_LOG_DIR}/access.log combined<br>&lt;/VirtualHost&gt;</em></p><p><strong>Rewrite URLs with mod_rewrite</strong></p><p>Enabling mod_rewrite</p><pre><strong>sudo a2enmod rewrite</strong></pre><p>Now take apache2.service restart</p><p>Setting up .htaccess: An <strong>.htaccess</strong> file allows us to modify our rewrite rules without accessing server configuration files. For this reason, <strong>.htaccess </strong>is critical to your web application’s security.</p><p>For enabling .htaccess file in the configuration we need to mention some condition in <strong>/etc/apache2/sites-available/sanjeev-apache.net</strong> as shown below and then take apache2 service restart.</p><pre><strong>&lt;VirtualHost *:80&gt;<br>    &lt;Directory /var/www/html&gt;<br>        Options Indexes FollowSymLinks MultiViews<br>        AllowOverride All<br>        Require all granted<br>    &lt;/Directory&gt;</strong></pre><pre><strong>    . . .<br>&lt;/VirtualHost&gt;</strong></pre><p>Now, create the <strong>.htaccess</strong> file in the web root.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/562/1*9Y4ZGvbY5qCML3Ai62lo_Q.gif" /><figcaption>cd /var/www/test/.htaccess</figcaption></figure><p><strong>vim /var/www/html/.htaccess</strong></p><pre><strong>RewriteEngine on<br>RewriteRule ^about$ about.html [NC]</strong></pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/993/1*CkFbG2KxzeeemZay_MZbzA.png" /><figcaption>redirect to about section using mod_rewrite module</figcaption></figure><p>In this case, ^about$ is the pattern, about.html is the substitution, and [NC] is a flag. Our example uses a few characters with special meaning:</p><ul><li><strong>^</strong> indicates the start of the URL after <strong>localhost/about</strong></li><li><strong>$</strong> indicates the end of the URL.</li><li><strong>about</strong> matches the string “about”.</li><li><strong>about.html</strong> is the actual file that the user accesses.</li><li>[NC] is a flag that makes the rule case insensitive.</li></ul><p><strong>Simplifying Query Strings with RewriteRule</strong></p><pre><strong>RewriteRule ^shirt/summer$ results.php?item=shirt&amp;season=summer [QSA]</strong></pre><p>The<strong> shirt/summer</strong> is explicitly matched in the requested address and Apache is told to serve <strong>results.php?item=shirt&amp;season=summer</strong> instead.</p><p>To make the rule more generic, we can use the regular expression to match parts of the original address and use those parts in a substitution pattern. The modified rule will then look as follows:</p><pre><strong>RewriteRule ^([A-Za-z0-9]+)/(summer|winter|fall|spring) results.php?item=$1&amp;season=$2 [QSA]</strong></pre><p>The first regular expression group in parenthesis matches a string containing alphanumeric characters and numbers like <strong>shirt</strong> or <strong>pants</strong> and saves the matched fragment as the <strong>$1</strong> variable. The second regular expression group in parenthesis matches exactly <strong>summer, </strong><strong>winter, </strong><strong>fall,</strong> or <strong>spring</strong>, and similarly saves the matched fragment as <strong>$2.</strong></p><p><strong>Adding Conditions with Logic Using RewriteConds</strong></p><p>The RewriteCond directive lets us add conditions to our rewrite rules to control when the rules will be processed. All RewriteConds abide by the following format:</p><p>General RewriteCond structure</p><pre><strong>RewriteCond TestString Condition [Flags]</strong></pre><ul><li><strong>RewriteCond</strong> specifies the <strong>RewriteCond</strong> directive.</li><li><strong>TestString</strong> is the string to test against.</li><li><strong>Condition</strong> is the pattern or condition to match.</li><li><strong>Flags</strong> are optional parameters that may modify the condition and evaluation rules.</li></ul><p>If a <strong>RewriteCond</strong> evaluates to true, the <strong>RewriteRule</strong> immediately following will be considered. If it won&#39;t, the rule will be discarded. Multiple <strong>RewriteCond</strong> may be used one after another and, with the default behavior, all must evaluate to true for the following rule to be considered.</p><p>As an example, let’s assume you would like to redirect all requests to non-existent files or directories on your site back to the home page instead of showing the standard <strong>404 Not Found</strong> error page. This can be achieved with following conditions rules:</p><p><strong><em>vim /var/www/test/.htaccess</em></strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/478/1*lKcbYfzG4bZSn_6vns9T1A.png" /></figure><p><strong>Enabling Necessary Apache Modules</strong></p><p>The modules we need are <strong>mod_proxy</strong> itself and several of its add-on modules, which extend its functionality to support different network protocols. Specifically, we will use:</p><ul><li><strong>mod_proxy</strong>, the main proxy module Apache module for redirecting connections; it allows Apache to act as a gateway to the underlying application servers.</li><li><strong>mod_proxy_http</strong>, which adds support for proxying HTTP connections.</li><li><strong>mod_proxy_balancer</strong> and <strong>mod_lbmethod_byrequests</strong>, which add load balancing features for multiple backend servers.</li></ul><p>To enable these four modules, execute the following commands in succession.</p><ul><li><em>sudo a2enmod proxy</em></li><li><em>sudo a2enmod proxy_http</em></li><li><em>sudo a2enmod proxy_balancer</em></li><li><em>sudo a2enmod lbmethod_byrequests</em></li></ul><p>To put these changes into effect, restart Apache.</p><ul><li><em>sudo systemctl restart apache2</em></li></ul><p>Modifying the Default Configuration to Enable Reverse Proxy</p><p><strong>Example 1</strong> — Reverse Proxying a Single Backend Server</p><p>Replace all the contents within <strong>VirtualHost</strong> block with the following, so your configuration file looks like this:</p><p>/etc/apache2/sites-available/000-default.conf</p><pre><strong>&lt;VirtualHost *:80&gt;<br>    ProxyPreserveHost On</strong></pre><pre><strong>ProxyPass / http://127.0.0.1:8080/<br>    ProxyPassReverse / http://127.0.0.1:8080/<br>&lt;/VirtualHost&gt;</strong></pre><p>If you followed along with the example servers in Step 2, use 127.0.0.1:8080 as written in the block above. If you have your own application servers, use their addresses instead.</p><p>There are three directives here:</p><ul><li><strong>ProxyPreserveHost</strong> makes Apache pass the original <strong>Host</strong> header to the backend server. This is useful, as it makes the backend server aware of the address used to access the application.</li><li><strong>ProxyPass</strong> is the main proxy configuration directive. In this case, it specifies that everything under the root URL (/) should be mapped to the backend server at the given address. For example, if Apache gets a request for <strong>/example</strong>, it will connect to <strong>http://your_backend_server/example</strong> and return the response to the original client.</li><li><strong>ProxyPassReverse</strong> should have the same configuration as <strong>ProxyPass</strong>. It tells Apache to modify the response headers from backend server. This makes sure that if the backend server returns a location redirect header, the client&#39;s browser will be redirected to the proxy address and not the backend server address, which would not work as intended.</li></ul><p>To put these changes into effect, restart Apache.</p><ul><li>sudo systemctl restart apache2</li></ul><p>Now, if you access <strong>http://your_server_ip</strong> in a web browser, you will see your backend server response instead of standard Apache welcome page. If you followed Step 2, this means you&#39;ll see Hellow world!.</p><p><strong>Example 2</strong> — Load Balancing Across Multiple Backend Servers</p><p>If you have multiple backend servers, a good way to distribute the traffic across them when proxying is to use load balancing features of mod_proxy.</p><p>Replace all the contents within the VirtualHost block with the following, so your configuration file looks like this:</p><p>/etc/apache2/sites-available/000-default.conf</p><pre><strong>&lt;VirtualHost *:80&gt;<br>&lt;Proxy balancer://mycluster&gt;<br>    BalancerMember http://127.0.0.1:8080<br>    BalancerMember http://127.0.0.1:8081<br>&lt;/Proxy&gt;</strong></pre><pre><strong>ProxyPreserveHost On</strong></pre><pre><strong>ProxyPass / balancer://mycluster/<br>    ProxyPassReverse / balancer://mycluster/<br>&lt;/VirtualHost&gt;</strong></pre><p>The configuration is similar to the previous one, but instead of specifying a single backend server directly, we’ve used an additional Proxy block to define multiple servers. The block is named <strong>balancer://mycluster</strong> (the name can be freely altered) and consists of one or more BalancerMembers, which specify the underlying backend server addresses. The <strong>ProxyPass</strong> and <strong>ProxyPassReverse</strong> directives use the load balancer pool named <strong>mycluster</strong> instead of a specific server.</p><p>If you followed along with the example servers in Step 2, use <strong>127.0.0.1:8080</strong> and <strong>127.0.0.1:8081</strong>for the <strong>BalancerMember</strong> directives, as written in the block above. If you have your own application servers, use their addresses instead.</p><p><strong>Create a Self-Signed SSL Certificate</strong></p><p>TLS, or transport layer security, and its predecessor SSL, which stands for secure sockets layer, are web protocols used to wrap normal traffic in a protected, encrypted wrapper.</p><p>Using this technology, servers can send traffic safely between the server and clients without the possibility of the messages being intercepted by outside parties. The certificate system also assists users in verifying the identity of the sites that they are connecting with.</p><p>Step 1: Create SSL certificate</p><p><em># sudo openssl req -x509 -nodes -days 365 -newkey rsa:2048 -keyout /etc/ssl/private/apache-selfsigned.key -out /etc/ssl/certs/apache-selfsigned.crt</em></p><p>After this, we need to make entry of some required detail as shown below.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/880/1*O2N9L-2kTrga3pzkvQo5xQ.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/662/1*-HcJTNsin7Snes6r_f10yA.png" /><figcaption>apache-selfsigned.key created</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/573/1*ObJAlmhEmAEuERNHgqkhaA.png" /><figcaption>apache-selfsigned.crt created</figcaption></figure><ul><li><strong>OpenSSL</strong>: This is the basic command line tool for creating and managing OpenSSL certificates, keys, and other files.</li><li><strong>req</strong>: This subcommand specifies that we want to use X.509 certificate signing request (CSR) management. The “X.509” is a public key infrastructure standard that SSL and TLS adhere to for its key and certificate management. We want to create a new X.509 cert, so we are using this subcommand.</li><li>-<strong>x509</strong>: This further modifies the previous subcommand by telling the utility that we want to make a self-signed certificate instead of generating a certificate signing request, as would normally happen.</li><li>-<strong>nodes</strong>: This tells OpenSSL to skip the option to secure our certificate with a passphrase. We need Apache to be able to read the file, without user intervention, when the server starts up. A passphrase would prevent this from happening because we would have to enter it after every restart.</li><li>-<strong>days</strong> <strong>365</strong>: This option sets the length of time that the certificate will be considered valid. We set it for one year here.</li><li>-<strong>newkey</strong> <strong>rsa</strong>:<strong>2048</strong>: This specifies that we want to generate a new certificate and a new key at the same time. We did not create the key that is required to sign the certificate in a previous step, so we need to create it along with the certificate. The<strong> rsa:2048</strong> portion tells it to make an RSA key that is 2048 bits long.</li><li>-<strong>keyout</strong>: This line tells OpenSSL where to place the generated private key file that we are creating.</li><li>-<strong>out</strong>: This tells OpenSSL where to place the certificate that we are creating.</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/893/1*Zlu8WSwW1QelzaPBUgwF9Q.png" /></figure><p>Now we have to enable <strong>/etc/apache2.sites-available/default-ssl.conf </strong>using<strong> a2ensite </strong>and<strong> ssl </strong>module using <strong>a2enmod.</strong></p><p><strong>MPM (Multi-Processing Module)</strong></p><p>Apache uses one of following MPM (Multi-Processing Module) for handling incoming requests and processes them. Both have their own working type. Below is some basic details about both MPM and there working.</p><p><strong>Prefork MPM:</strong></p><p>Prefork MPM launches multiple child processes. Each child process handle one connection at a time.</p><p>Prefork uses high memory in comparison to worker MPM. Prefork is the default MPM used by Apache2 server. Preform MPM always runs few minimum (MinSpareServers) defined processes as spare, so new requests do not need to wait for new process to start.</p><p><strong>Worker MPM:</strong></p><p>Worker MPM generates multiple child processes similar to prefork. Each child process runs many threads. Each thread handles one connection at a time.</p><p>In sort Worker MPM implements a hybrid multi-process multi-threaded server. Worker MPM uses low memory in comparison to Prefork MPM.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/799/1*FrwxvQOFlL9Nlfzrk1Wj8g.png" /></figure><p><strong><em>StartServers</em></strong></p><ol><li>This directive set the number of thread created at the time of startup.</li><li>The number of thread is dynamically controlled depending upon load.</li></ol><p><strong><em>ThreadPerChild</em></strong></p><ol><li>This directive set number of threads created by each child process</li><li>child create these thread at startup and never create more.</li></ol><p><strong><em>ThreadLimit</em></strong></p><ol><li>set upper limit on the configurable number of thread per child process.</li><li>If the thread limit is set higher than ThreadPerChild than, extra unused shared memory will be allocated.</li><li>If both ThreadLimit and ThreadPerChild are set to value higher than the system can handle, Apache may not start or system can become unstable.</li></ol><p><strong><em>ServerLimit</em></strong></p><ol><li>The upper limit on the number of processes.</li><li>If ServerLimit is 10, ThreadLimit is 10, than MaxClient is 100</li></ol><p><strong><em>MaxRequestPerChild</em></strong></p><ol><li>The MaxRequestPerChild directive set the limit on the number of requests that an individual child process will handle.</li><li>After MaxRequestPerChild request, a child process will die.</li><li>If MaxRequestPerChild is 0, then the process will never expire.</li></ol><p><strong>Event MPM:</strong></p><p>Event MPM is introduced in Apache 2.4, It is pretty similar to worker MPM but it designed for managing high loads.</p><p>This MPM allows more requests to be served simultaneously by passing off some processing work to supporting threads. Using this MPM Apache tries to fix the ‘keep alive problem’ faced by other MPM. When a client completes the first request then the client can keep the connection open, and send further requests using the same socket, which reduces connection overload.</p><p>~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=a0d7f8167063" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[MongoDB]]></title>
            <link>https://medium.com/@me.sanjeev3d/https-medium-com-me-sanjeev3d-mongodb-c538d9169418?source=rss-89634bf8d93a------2</link>
            <guid isPermaLink="false">https://medium.com/p/c538d9169418</guid>
            <category><![CDATA[nosql]]></category>
            <category><![CDATA[mongodb]]></category>
            <category><![CDATA[database]]></category>
            <category><![CDATA[mongo]]></category>
            <dc:creator><![CDATA[Sanjeev Gautam]]></dc:creator>
            <pubDate>Tue, 01 Jan 2019 21:58:54 GMT</pubDate>
            <atom:updated>2019-01-23T23:36:12.484Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Ce0gUe0LbnhL7ebnDGTp5w.png" /></figure><p>Mongodb is open-source No-SQL database document database used commonly in modern web applications which is also schemaless. Mongodb work on the cross-platform. It deals with the idea of data accumulation and report. MongoDB works on the concept of collection and document. default port 27017</p><p><strong><em>Database</em></strong></p><p>The database is a physical compartment for data accumulation, every database has its own set of the document on the record framework.</p><p><strong><em>Collection</em></strong></p><p>Collections are a gathering of MongoDB archives. It is similar to the table in RDBMS. A <em>collection</em> may store documents those who are not same in structure, because of a schema-free database.</p><p><strong><em>Document</em></strong></p><p><em>The document</em> is the unit of storing data in a MongoDB database. It is similar to the record in RDBMS. In the below code, it is shown that two MongoDB documents, belongs to same collection, storing data of different structures.</p><p>{“tutorial” : “NoSQL”} {“topic_id” : 7}</p><blockquote><strong>Installation and configuration on Ubuntu</strong></blockquote><ol><li><strong>Adding Mongodb repository</strong></li></ol><p><strong><em>$ </em></strong><em>sudo apt-key adv --keyserver hkp://keyserver.ubuntu.com:80 --recv EA312927</em></p><p><strong>2. create a list file for MongoDB or add a file to the source list</strong></p><p><em>$ echo “deb http://repo.mongodb.org/apt/ubuntu xenial/mongodb-org/3.2 multiverse” | sudo tee /etc/apt/sources.list.d/mongodb-org-3.2.list</em></p><p><strong>3. Update the package list.</strong></p><p><em>$ sudo apt-get update</em></p><p><strong>4. Installing and Verifying MongoDB</strong></p><p><em>$ sudo apt-get install mongodb-org</em></p><p><em>$ sudo systemctl start mongod</em></p><p><em>$ sudo systemctl status mongod</em></p><p>A relationship between RDBMS and Mongodb</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/672/1*wBajLzHtKHUIyqANxhsoAQ.png" /></figure><blockquote><strong>Advantages of MongoDB over RDBMS</strong></blockquote><p><strong>Schema-less:</strong> MongoDB is a document database in which one collection holds different documents. The number of fields, content, and size of the document can differ from one document to another.</p><p>Structure of a single object is clear.</p><p>No complex joins</p><p>Deep query-ability. MongoDB supports dynamic queries on documents using a document based query language that’s nearly as powerful as SQL</p><p>Tuning</p><p>Ease of scaleout: MongoDB is easy to scale.</p><p>After installing mongodb on ubuntu we can enter into mongo shell using <strong>“mongo” </strong>and check all db command using<strong> “db.help()”</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/724/1*Qed3fnST8DLxIaKKCduoZA.png" /><figcaption>entering mongo shell we check mongo command</figcaption></figure><p>To check mongo db stats using “<strong>db.stats()</strong>”</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/328/1*KS2d_s_JNHs0Ay19VIQZOw.png" /><figcaption>mongodb status</figcaption></figure><blockquote><strong>MongoDB: Create Database</strong></blockquote><p>MongoDB <strong>use DATABASE_NAME</strong> is used to create the database. The command will create a new database if it doesn’t exist, otherwise, it will return the existing database.</p><p><strong>Example:</strong> <em>&gt; use mytestdb<br>switched to db mytestdb</em></p><p><strong>To check in which db currently working.</strong></p><p><em>&gt; db<br>mytestdb</em></p><p><strong>To check the database list.</strong></p><p><em>&gt; show dbs<br>local 0.000GB</em></p><p>mytestdb is not in db list because we need to insert at least one collection to reflect inside dbs list.</p><p><em>&gt; db.song.insert({“name”:”sanjeev_collection”})<br>WriteResult({ “nInserted” : 1 })</em></p><p>Now <strong>mytestdb</strong> is reflecting</p><p><em>&gt; show dbs<br>local 0.000GB<br></em><strong><em>mytestdb 0.000GB</em></strong></p><h3>dropDatabase() Method</h3><p>MongoDB <strong>db.dropDatabase()</strong> command is used to drop a existing database.</p><p><em>&gt; db.dropDatabase()<br>{ “dropped” : “mytestdb”, “ok” : 1 }</em></p><p>if we do not select any database to delete then it deletes default database.</p><h3>createCollection() Method</h3><p>MongoDB <strong>db.createCollection(name, options)</strong> is used to create collection.</p><p><strong>Basic Syntax:</strong> db.createCollection(“name”, option)</p><p><em>&gt; db.createCollection(“mycollection”)<br>{ “ok” : 1 }</em></p><p><strong>Inserting in existing collection</strong></p><p><em>&gt; db.mycollection.insert({“name”:”sanjeev”, “address”:”gurgaon”})<br>WriteResult({ “nInserted” : 1 })</em></p><p><strong>create and insert in collection at same time</strong></p><p><em>&gt; db.createCollection(“testcollection”, {“Name”:”Abhishek”, “M.number”:”9162342798&quot;, “Address”:”Patna”})<br>{ “ok” : 1 }</em></p><h3>drop() Method</h3><p>MongoDB’s <strong>db.collection.drop()</strong> is used to drop a collection from the database.</p><p>First, check collection in db</p><p><em>&gt; show collections<br>collection_test<br>mycollection<br>testcollection</em></p><p>Then delete <strong>collection_test</strong> using below command.</p><p><em>&gt; db.collection_test.drop()<br>true</em></p><blockquote><strong>MongoDB Datatype</strong></blockquote><p>MongoDB supports many datatypes.</p><ul><li><strong>String</strong> − This is the most commonly used datatype to store the data. String in MongoDB must be UTF-8 valid.</li><li><strong>Integer</strong> − This type is used to store a numerical value. An integer can be 32 bit or 64 bit depending upon your server.</li><li><strong>Boolean</strong> − This type is used to store a boolean (true/ false) value.</li><li><strong>Double</strong> − This type is used to store floating point values.</li><li><strong>Min/ Max keys</strong> − This type is used to compare a value against the lowest and highest BSON elements.</li><li><strong>Arrays</strong> − This type is used to store arrays or list or multiple values into one key.</li><li><strong>Timestamp</strong> − timestamp. This can be handy for recording when a document has been modified or added.</li><li><strong>Object</strong> − This datatype is used for embedded documents.</li><li><strong>Null</strong> − This type is used to store a Null value.</li><li><strong>Symbol</strong> − This datatype is used identically to a string; however, it’s generally reserved for languages that use a specific symbol type.</li><li><strong>Date </strong>− This datatype is used to store the current date or time in UNIX time format. You can specify your own date time by creating the object of Date and passing day, month, a year into it.</li><li><strong>Object ID</strong> − This datatype is used to store the document’s ID.</li><li><strong>Binary data</strong> − This datatype is used to store binary data.</li><li><strong>Code</strong> − This datatype is used to store JavaScript code into the document.</li><li><strong>Regular expression</strong> − This datatype is used to store regular expression.</li></ul><h3>Insert document in MongoDB collection</h3><p>Insert data into MongoDB collection, you need to use MongoDB’s <strong>insert()</strong> or <strong>save()</strong> method.</p><p><strong>Basic syntax:</strong> db.COLLECTION_NAME.insert(document)</p><p>&gt; <em>db.mycollection.insert({ <br>title: ‘About Sanjeev’, <br>description: ‘Devops Engineer’,<br>by: ‘Fan of Sanjeev’,<br>url: ‘</em><a href="http://medium.com/sanjeev3d.com&#39;"><em>http://medium.com/sanjeev3d.com&#39;</em></a><em>, <br>tags: [‘devops’, ‘engineer’, ‘linuxguy’], <br>likes: 100 })<br>WriteResult({ “nInserted” : 1 })</em></p><p>Here mycollection is collection name which already exixst if it not exist it will automatically created.</p><p>In the inserted document, if we don’t specify the<strong> _id </strong>parameter, then MongoDB assigns a unique <strong>ObjectId </strong>for this document.</p><p><strong>_id</strong>: ObjectId(4 bytes timestamp, 3 bytes machine id, 2 bytes process id,<br> 3 bytes incrementer)</p><p>To insert multiple documents</p><p><em>&gt; db.testcollection.save([ { title: ‘About Abhishek’, description: ‘Graduate’, by: ‘Fan of Abhishek’, like: 150 }, { title: ‘About Rahul’, description: ‘Electronics engineer’, by: ‘Fan of Rahul’, comment: [ { liked: 100, followed: ‘manyone’ }]}])<br>BulkWriteResult({<br> “writeErrors” : [ ],<br> “writeConcernErrors” : [ ],<br> “nInserted” : 2,<br> “nUpserted” : 0,<br> “nMatched” : 0,<br> “nModified” : 0,<br> “nRemoved” : 0,<br> “upserted” : [ ]<br>})</em></p><h3>find() Method</h3><p>To query data from MongoDB collection, you need to use MongoDB’s <strong>find()</strong> method.</p><p><strong>Basic syntax:</strong> db.collection_name.find()</p><p><strong>find()</strong> method will display all the documents in a non-structured way, like below.</p><p><em>&gt; db.testcollection.find()<br>{ “_id” : ObjectId(“5c2bba6a244c9573845bc587”), “title” : “About Abhishek”, “description” : “Graduate”, “by” : “Fan of Abhishek”, “like” : 150 }<br>{ “_id” : ObjectId(“5c2bba6a244c9573845bc588”), “title” : “About Rahul”, “description” : “Electronics engineer”, “by” : “Fan of Rahul”, “comment” : [ { “liked” : 100, “followed” : “manyone” } ] }</em></p><h3>pretty() Method</h3><p>To display the results in a formatted way, you can use <strong>pretty()</strong> method.</p><p><strong>Basic syntax: </strong>db.mycollection.find().pretty()</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/512/1*Kz_AucDbNIWVk4vucOc9zw.png" /></figure><h4><strong>Query on basis of some condition</strong></h4><p><strong>Equality: </strong>db.testcollection.find({“by”:”Fan of Rahul”}).pretty()</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/547/1*I_ylFVidDgwLvesrBgvP_w.png" /></figure><p><strong>Less than:</strong></p><p><strong>Example:</strong> db.testcollection.find({“like”:{$lt:1000}}).pretty()</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/544/1*w57AuU706o9Y4QYhzsgH5Q.png" /></figure><p><strong>Less than equals:</strong> db.testcollection.find({“like”:{$lte:150}}).pretty()</p><p><strong>Greater Than:</strong> db.testcollection.find({“likes”:{$gt:50}}).pretty()</p><p><strong>Greater Than Equals:</strong> db.testcollection.find({“likes”:{$gte:50}}).pretty()</p><p><strong>Not Equals:</strong> db.mycol.find({“likes”:{$ne:50}}).pretty()</p><p><strong>AND in MongoDB</strong></p><p><strong>Example:</strong> db.testcollection.find({$and:[{“by”:”Fan of Abhishek”},{“like”:{$gte:150}}]}).pretty()</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/806/1*9etR_y6tvAcL_8ErI6rlew.png" /><figcaption>using AND conditioning.</figcaption></figure><p>The query is reflected if both condition in find match</p><p><strong>OR in MongoDB</strong></p><p>Example: db.testcollection.find({$or:[{“by”:”Fan of Rahul”},{“by”:”Fan of Abhishek”}]}).pretty()</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/837/1*UgyTwqWDajmjgVBEPHZ9tA.png" /></figure><p>If any of two condition match query is reflected</p><h3>MongoDB Update() Method</h3><p>Example: db.testcollection.update({“by”:”Fan of Abhishek”},{$set:{“by”:”Abhishek”}})</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/741/1*RnABdse0kAd5tIwO5MjjVA.gif" /></figure><p>By default, MongoDB will update only a single document. To update multiple documents, you need to set a parameter ‘multi’ to true.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/733/1*ww8N3X_roq_vCbjXriVj8g.png" /><figcaption>Here all get modified if the condition match</figcaption></figure><h3>remove() Method</h3><p>MongoDB’s <strong>remove()</strong> method is used to remove a document from the collection. remove() method accepts two parameters. One is deletion criteria and second is the justOne flag.</p><ul><li><strong>deletion criteria</strong> − (Optional) deletion criteria according to documents will be removed.</li><li><strong>justOne</strong> − (Optional) if set to true or 1, then remove only one document.</li></ul><p><strong>Basic Syntax:</strong> db.COLLECTION_NAME.remove(DELLETION_CRITTERIA)</p><p><em>&gt; db.testcollection.remove({“by”:”Abhishek”})<br>WriteResult({ “nRemoved” : 1 })</em></p><p><strong>To remove only one</strong></p><p><strong>Basic syntax:</strong> db.COLLECTION_NAME.remove(DELETION_CRITERIA,1)</p><p>db.testcollection.remove({“Name”:”sanjeev”},1)</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*SNOf7jNwcKO7UgjfiSMTnQ.gif" /></figure><h4>More About find() Method</h4><p>In MongoDB, when you execute <strong>find()</strong> method, then it displays all fields of a document. To limit this, you need to set a list of fields with value 1 or 0. 1 is used to show the field while 0 is used to hide the fields.</p><p><strong>Basic syntax:</strong> db.COLLECTION_NAME.find({},{KEY:1})</p><p>From sample:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/493/1*0lTGdIQa9UIiXm4PTok3Iw.png" /></figure><p>We find some specific field of document.</p><p>db.testcollection.find({},{“Book”:1,”Author”:1,”_id”:0})</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/560/1*a-_jx0GHzSnUbFm6iX3yhw.png" /></figure><h3>Limit() Method</h3><p>To limit the records in MongoDB, you need to use <strong>limit()</strong> method. The method accepts one number type argument, which is the number of documents that you want to be displayed.</p><p><strong>Basic syntax:</strong> db.COLLECTION_NAME.find().limit(NUMBER)</p><p>db.testcollection.find({},{“Book”:1,”Author”:1,”_id”:0}).limit(2)</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/657/1*9F-RVL9j0y8-i-mtxGqOcQ.png" /></figure><h3>Skip() Method</h3><p>Apart from limit() method, there is one more method <strong>skip()</strong> which also accepts number type argument and is used to skip the number of documents.</p><p>Basic syntax: db.COLLECTION_NAME.find().limit(NUMBER).skip(NUMBER)</p><p>db.testcollection.find({},{“Book”:1,”Author”:1,”_id”:0}).limit(2).skip(1)</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/697/1*YTSXl938S_NbmoZuvhlpTw.png" /><figcaption>Skipped 1st entry and reflect 2nd and 3rd .</figcaption></figure><h3>sort() Method</h3><p>To sort documents in MongoDB, you need to use <strong>sort()</strong> method. The method accepts a document containing a list of fields along with their sorting order. To specify sorting order 1 and -1 are used. 1 is used for ascending order while -1 is used for descending order.</p><p>Basic Syntax: db.COLLECTION_NAME.find().sort({KEY:1})</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/589/1*tNs0F4VHVFDY6_Pn_nazLw.png" /></figure><h3>Indexing</h3><p>For this we first create sample collection with 10000 of object using below command.</p><p>for(i=0; i&lt;10000; ++i){<br>… db.post.insert({stdent_id:i,name:”Sanjeev”})<br>… }</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/615/1*8juIGzULmC_hliiQWZYohQ.png" /></figure><p>Now if i do query for student id 9000 then it will take sometime because it check overall through object.</p><p>To find only one exact one query.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/642/1*AF1k_ett0FWOuCHHirxIhA.png" /></figure><p>To create the index for unique Id.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/614/1*S1sBIpjSq_ZHvv5ujLi_bA.png" /></figure><p>We always create indexing for unique id, in order to remove index we put dropIndex in place of ensureIndex</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/452/1*NVEy3EYZZ95RQrBdH61f7w.png" /></figure><h3>MongoDB — Aggregation</h3><p>Aggregations operations process data records and return computed results. Aggregation operations group values from multiple documents together, and can perform a variety of operations on the grouped data to return a single result. In SQL count(*) and with group by is an equivalent of mongodb aggregation.</p><p>Basic Syntax: db.COLLECTION_NAME.aggregate(AGGREGATE_OPERATION)</p><p>Example, we have below objects in the document.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/576/1*qm6QLDtRr4xZLzeqeJXAsg.png" /></figure><p>we perform aggregation on above objects in collections.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/707/1*_VuAxtPfeQ1Ugi5uoOXPTA.png" /></figure><p>HERE we getting aggregated data for the female and male student, if in the above query we put 2 then it obtain the result by double.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/762/1*R2zQ78hKVVHtVQggUoXFGg.png" /><figcaption>Perform some more task like Min and Max on age value</figcaption></figure><h4>Aggregation Pipeline</h4><p>n UNIX command, shell pipeline means the possibility to execute an operation on some input and use the output as the input for the next command and so on. MongoDB also supports the same concept in the aggregation framework. There is a set of possible stages and each of those is taken as a set of documents as an input and produces a resulting set of documents (or the final resulting JSON document at the end of the pipeline). This can then, in turn, be used for the next stage and so on.</p><p>Following are the possible stages in the aggregation framework −</p><ul><li><strong>$project</strong> − Used to select some specific fields from a collection.</li><li><strong>$match</strong> − This is a filtering operation and thus this can reduce the amount of documents that are given as input to the next stage.</li><li><strong>$group</strong> − This does the actual aggregation as discussed above.</li><li><strong>$sort</strong> − Sorts the documents.</li><li><strong>$skip</strong> − With this, it is possible to skip forward in the list of documents for a given amount of documents.</li><li><strong>$limit</strong> − This limits the amount of documents to look at, by the given number starting from the current positions.</li><li><strong>$unwind</strong> − This is used to unwind document that are using arrays. When using an array, the data is kind of pre-joined and this operation will be undone with this to have individual documents again. Thus with this stage, we will increase the amount of documents for the next stage.</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/538/1*i3smevCEWK_iAze3JJcTAw.png" /><figcaption>pipeline query</figcaption></figure><h4>Mongod Replication</h4><p><strong>ReplicaSet</strong></p><p>MongoDB achieves replication by the use of replica set. A <strong>replica set</strong> is a cluster of MongoDB database servers that implements master-slave (primary-secondary) replication. In a replica, one node is a primary node that receives all write operations. All other instances, such as secondaries, apply operations from the primary so that they have the same data set. Replica set can have only one primary node. Replica sets also fail over automatically, so if one of the members becomes unavailable, a new primary host is elected and your data is still accessible. That means, when a primary replica fails, the replica set automatically conducts an election process to determine which secondary should become the primary.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/527/1*NqJLKd21njfLiKeKuZ6t-w.png" /></figure><h4>Steps to remember</h4><ol><li>First, we need to create three virtual machines.</li><li>Set the hostname of each instance.</li><li>Configure the host files in path <strong>/etc/hosts.</strong></li><li>To create a replica set, you’ll need at least three instances with MongoDB installed.</li><li>Initialize the master(<strong>primary</strong>) database server and add the <strong>secondary </strong>servers as slaves.</li></ol><p>we have three mongo server as shown below</p><p><em>192.168.33.90 dbmanager<br>192.168.33.91 dbnode1<br>192.168.33.92 dbnode2</em></p><p>Install mongo on all and then make entry of all server on every server /etc/hosts file</p><p>Now configure replica set using configuration file /etc/mongod.conf</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/832/1*-jAmtQBM4QFPEHNTjTJ1-A.gif" /></figure><p>Make a similar entry on all mongo server, then take Restart MongoDB on all nodes.</p><p>we will create the replica set. We will use the ‘<strong>dbmanager</strong>’ server as ‘<strong>PRIMARY</strong>’ node, and ‘<strong>dbnode1</strong>’ and ‘<strong>dbnode2</strong>’ as ‘<strong>SECONDARY</strong>’ nodes.</p><p>Login to dbmanager server and start mongo shell and initiate replica set using</p><pre><strong><em>rs.initiate()</em></strong></pre><p>Make sure ‘ok’ value is 1.</p><p>Now add the ‘dbnode1’ and ‘dbnode2’ nodes to the replica sets.</p><pre><strong><em>rs.add(&quot;dbnode1&quot;)<br>rs.add(&quot;dbnode2&quot;)</em></strong></pre><p>Next, check the replica sets status with the rs query below.</p><pre><strong><em>rs.status()<br>rs.isMaster()</em></strong></pre><p>Now we can check using writing on primary db (dbmanager) and will be replicated at all replica node.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=c538d9169418" width="1" height="1" alt="">]]></content:encoded>
        </item>
    </channel>
</rss>