<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Venkatesh R on Medium]]></title>
        <description><![CDATA[Stories by Venkatesh R on Medium]]></description>
        <link>https://medium.com/@tatankavenkat?source=rss-67c92e804e41------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/1*R1G_DE69XQ4z__GLa4NbfQ.jpeg</url>
            <title>Stories by Venkatesh R on Medium</title>
            <link>https://medium.com/@tatankavenkat?source=rss-67c92e804e41------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Sat, 21 Nov 2026 08:14:08 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@tatankavenkat/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[Setting Up MongoDB Replication on GCP Compute Engine — Live Migration from On-Prem or Another Cloud]]></title>
            <link>https://medium.com/niveus-solutions/setting-up-mongodb-replication-on-gcp-compute-engine-live-migration-from-on-prem-or-another-cloud-b245712d657b?source=rss-67c92e804e41------2</link>
            <guid isPermaLink="false">https://medium.com/p/b245712d657b</guid>
            <category><![CDATA[data-modernization]]></category>
            <category><![CDATA[gcp]]></category>
            <category><![CDATA[mongodb]]></category>
            <category><![CDATA[cloud-computing]]></category>
            <dc:creator><![CDATA[Venkatesh R]]></dc:creator>
            <pubDate>Thu, 10 Apr 2025 09:36:58 GMT</pubDate>
            <atom:updated>2025-04-30T10:09:20.495Z</atom:updated>
            <content:encoded><![CDATA[<h3>Setting Up MongoDB Replication on GCP Compute Engine — Live Migration from On-Prem or Another Cloud</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*KeqwpavQESETvDkshpauwg.png" /></figure><p>MongoDB’s replication mechanism provides a powerful way to ensure high availability and data redundancy by synchronising data across multiple servers in a <strong>replica set</strong>. In this guide, we walk you through setting up a MongoDB replica set on <strong>Google Cloud Platform (GCP) Compute Engine</strong>, with a focus on <strong>live migration</strong> from an existing MongoDB instance — whether it’s hosted on-premises or on another cloud vendor.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/775/1*4ddK3JeMmp6zEkKgfqNEZA.png" /></figure><h3>What is a MongoDB Replica Set?</h3><p>A <strong>replica set</strong> in MongoDB is a group of mongod instances that maintain the same dataset. Replica sets provide:</p><ul><li><strong>High availability</strong></li><li><strong>Automatic failover</strong></li><li><strong>Data redundancy</strong></li></ul><p>A typical setup includes:</p><ul><li><strong>1 Primary</strong>: Accepts all write operations.</li><li><strong>2 Secondaries</strong>: Maintain copies of the primary’s data set. In case of primary failure, a new primary is elected via a voting process.</li></ul><p><strong>Recommended Reading:</strong></p><ul><li><a href="https://www.mongodb.com/docs/manual/replication/">Replication Concepts — MongoDB Docs</a></li><li><a href="https://www.mongodb.com/docs/manual/tutorial/convert-standalone-to-replica-set/">Convert Standalone to Replica Set</a></li><li><a href="https://www.mongodb.com/docs/manual/tutorial/expand-replica-set/">Expand Replica Set</a></li><li><a href="https://www.mongodb.com/docs/v5.0/core/replica-set-architectures/">Replica Set Architectures</a></li></ul><p>📺 <strong>Video Overview:</strong><a href="https://www.youtube.com/watch?v=gChzfhVGqp8"> MongoDB Replication Deep Dive</a></p><h3>Architecture Overview</h3><p>We will implement a <strong>P-S-S (Primary-Secondary-Secondary)</strong> replica set across three GCP VMs:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/740/0*b3WFh4q6jrNk7zR2" /></figure><p>powReplicaSet:</p><p>├── mongod1 (Primary) [Port: 27017]</p><p>├── mongod2 (Secondary) [Port: 27027]</p><p>└── mongod3 (Secondary) [Port: 27037]</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/660/0*rHTuK0933TuMYQd6" /></figure><p>Each instance will be capable of becoming the <strong>primary</strong> in the event of failure via MongoDB’s built-in <strong>voting mechanism</strong>.</p><h3>Pre-requisites</h3><ol><li>GCP Project with billing enabled.</li><li>Three Compute Engine VMs (Ubuntu 20.04 or 22.04).</li><li>Ingress firewall rules for MongoDB ports (27017, 27027, 27037).</li><li>Existing MongoDB source instance (on-prem or cloud).</li><li>SSH access to all nodes.</li></ol><h3>Step-by-Step Setup</h3><h3>1️⃣ Install MongoDB on All Nodes</h3><p>sudo apt update</p><p>sudo apt install -y mongodb-server-core</p><p>Alternatively, follow the<a href="https://www.mongodb.com/docs/manual/tutorial/install-mongodb-on-ubuntu/"> official MongoDB Ubuntu install guide</a>.</p><h3>2️⃣ Create Data Directories</h3><p>sudo mkdir -p /mongos/db1 /mongos/db2 /mongos/db3</p><p>sudo chown -R mongodb:mongodb /mongos</p><h3>3️⃣ Start MongoDB Instances</h3><p>mongod — port 27017 — dbpath /mongos/db1 — replSet powReplicaSet — bind_ip_all — fork — logpath /var/log/mongodb1.log</p><p>mongod — port 27027 — dbpath /mongos/db2 — replSet powReplicaSet — bind_ip_all — fork — logpath /var/log/mongodb2.log</p><p>mongod — port 27037 — dbpath /mongos/db3 — replSet powReplicaSet — bind_ip_all — fork — logpath /var/log/mongodb3.log</p><h3>4️⃣ Connect to Primary Node</h3><p>mongosh — port 27017</p><h3>5️⃣ Initialize Replica Set</h3><p>rs.initiate({</p><p>_id: “powReplicaSet”,</p><p>members: [</p><p>{ _id: 0, host: “your-vm1-ip:27017” },</p><p>{ _id: 1, host: “your-vm2-ip:27027” },</p><p>{ _id: 2, host: “your-vm3-ip:27037” }</p><p>]</p><p>})</p><p>Check status:</p><p>rs.status()</p><h3>6️⃣ Optional: Add/Remove Members</h3><p>rs.add(“new-hostname:port”)</p><p>rs.remove(“hostname:port”)</p><h3>Live Migration from Existing MongoDB</h3><p>For live migration:</p><ol><li><strong>Enable replication on source instance</strong> (if not already).</li><li>Use <strong>mongodump and mongorestore</strong> for data transfer, or use <strong>initial sync</strong> if direct connection is possible.</li><li>Add the new GCP primary to the source replica set (if possible), promote it, then gracefully remove old members.</li><li>Ensure <strong>oplog</strong> is enabled on the source to maintain consistency during sync.</li><li>Validate data consistency after migration.</li></ol><h3>Test the Setup</h3><p>You can use <strong>MongoDB Compass</strong> to connect using:</p><p>mongodb://&lt;vm1-ip&gt;:27017,&lt;vm2-ip&gt;:27027,&lt;vm3-ip&gt;:27037/?replicaSet=powReplicaSet</p><p>📘 Learn more:</p><ul><li><a href="https://www.mongodb.com/docs/manual/reference/connection-string/">MongoDB Connection String Reference</a></li><li><a href="https://www.mongodb.com/products/compass">MongoDB Compass Docs</a></li></ul><h3>Replica Set Voting Considerations</h3><p>In a 3-member replica set, MongoDB uses a <strong>majority vote system</strong>:</p><ul><li>Must have a majority (i.e., 2 out of 3 nodes) available to elect a primary.</li><li>Ensure all members have voting rights unless configured otherwise.</li></ul><p>Without a majority, a new primary <strong>cannot</strong> be elected, which could lead to write downtime.</p><h3>Stopping a Node (For Testing)</h3><p># Stop primary manually</p><p>ps aux | grep mongod</p><p>kill &lt;pid&gt;</p><p>Then observe automatic failover via rs.status() on remaining nodes.</p><h3>Summary</h3><p>✅ Setup 3-node MongoDB replica set on GCP Compute Engine<br> ✅ Migrated data from existing on-prem/cloud MongoDB<br> ✅ Verified automatic failover and high availability<br> ✅ Configured replica-aware connection string</p><h3>Resources &amp; References</h3><ul><li><a href="https://www.mongodb.com/docs/manual/replication/">Official MongoDB Replication Docs</a></li><li><a href="https://www.digitalocean.com/community/tutorials/how-to-configure-a-mongodb-replica-set-on-ubuntu-20-04">DigitalOcean MongoDB Replica Set Tutorial</a></li><li><a href="https://hevodata.com/learn/mongodb-compass-ubuntu/">MongoDB Compass on Ubuntu</a></li><li><a href="https://www.mongodb.com/docs/mongodb-shell/">MongoDB Shell Installation</a></li></ul><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=b245712d657b" width="1" height="1" alt=""><hr><p><a href="https://medium.com/niveus-solutions/setting-up-mongodb-replication-on-gcp-compute-engine-live-migration-from-on-prem-or-another-cloud-b245712d657b">Setting Up MongoDB Replication on GCP Compute Engine — Live Migration from On-Prem or Another Cloud</a> was originally published in <a href="https://medium.com/niveus-solutions">Niveus Solutions Stories</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Enhancing Cloud Storage Security with Google Cloud Storage (GCS) IP Filtering]]></title>
            <link>https://medium.com/niveus-solutions/enhancing-cloud-storage-security-with-google-cloud-storage-gcs-ip-filtering-81740748d511?source=rss-67c92e804e41------2</link>
            <guid isPermaLink="false">https://medium.com/p/81740748d511</guid>
            <category><![CDATA[cloud-computing]]></category>
            <category><![CDATA[gcp-security-operations]]></category>
            <category><![CDATA[google-cloud-storage]]></category>
            <category><![CDATA[cloud-security]]></category>
            <category><![CDATA[gcp]]></category>
            <dc:creator><![CDATA[Venkatesh R]]></dc:creator>
            <pubDate>Wed, 09 Apr 2025 08:13:20 GMT</pubDate>
            <atom:updated>2025-04-09T09:30:32.116Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*IykgiG3XTX6PSAAjK3Ql6A.png" /></figure><p>In today’s cloud-native world, network security remains a cornerstone of a strong cybersecurity strategy. While Identity and Access Management (IAM) offers robust control over who can access what, adding a layer of network-level restrictions can significantly enhance your security posture. Enter <strong>Google Cloud Storage (GCS) IP Filtering</strong> — a feature that allows administrators to restrict access to buckets based on client IP addresses.</p><p>This blog explores the concept of GCS IP filtering, demonstrates a proof of concept (PoC), discusses internal and external use cases, highlights how it complements <strong>VPC Service Controls (VPC SC)</strong>, and outlines some key limitations to keep in mind.</p><h3>A. Proof of Concept and Use Cases of Internal and External Access</h3><h3>What is GCS IP Filtering?</h3><p>GCS IP filtering enables administrators to allow or deny access to buckets based on the source IP address of the incoming request. This security control helps enforce policies at the network layer — an extra shield beyond IAM permissions.</p><h3>Proof of Concept (PoC)</h3><p>Here’s how you can test IP filtering on a GCS bucket using a simple allowlist:</p><ol><li><strong>Identify your public IP address</strong>:</li></ol><p><strong>curl ifconfig.me</strong></p><p># Example output: 122.174.194.176</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/701/1*_ho1236CJl2-qQ5lxcaLeA.png" /></figure><ol><li><strong>Create the IP filter configuration file</strong>:</li></ol><p>#create the IP filtering config file</p><p>ipfilter.json (content)</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/471/1*Xou-FR58zU5SwowGjA3B3Q.png" /></figure><p>All Public IP addresses</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/483/1*tE6dwFnQZx8IagNMTt8aWQ.png" /></figure><p>#Apply the IPfiltering on the gcs storage</p><p><strong>MC&gt;:~ (gcs-ip-fileter-poc)$ gcloud alpha storage buckets update gs://BUCKET_NAME — ip-filter-file=ipfilter.json</strong></p><p><strong>Updating gs://BUCKET_NAME/…</strong></p><p><strong>Completed 1</strong></p><p>#Perform cp task on the gcs storage from Public IP 1</p><p><strong>MC&gt;:~$ gsutil cp test.txt gs://BUCKET_NAME/</strong></p><p><strong>Copying file://test.txt [Content-Type=text/plain]…</strong></p><p><strong>/ [1 files][ 0.0 B/ 0.0 B]</strong></p><p><strong>Operation completed over 1 object.</strong></p><p>#Perform cp task on the gcs storage from different IP not in the list</p><p><strong>MC2&gt;:~$ gsutil cp test.txt gs://BUCKET_NAME/</strong></p><p><strong>Copying file://test.txt [Content-Type=text/plain]…</strong></p><p><strong>AccessDeniedException: 403 There is an IP filtering condition that is preventing access to the resource.</strong></p><h3>Summarizing the POC</h3><ol><li><strong>Apply the configuration</strong>:</li></ol><p>gcloud alpha storage buckets update gs://BUCKET_NAME — ip-filter-file=ipfilter.json</p><p><strong>2. Test access from allowed and disallowed IPs</strong>:</p><p><strong>From an allowed IP</strong>, you’ll see the copy operation succeed:</p><p>gsutil cp test.txt gs://BUCKET_NAME/</p><p><strong>From a disallowed IP</strong>, access is blocked:</p><p>AccessDeniedException: 403 There is an IP filtering condition that is preventing access…</p><p>For more on this setup, refer to<a href="https://cloud.google.com/storage/docs/create-ip-filter"> Google Cloud’s official documentation</a>.</p><h3>Internal Use Cases</h3><ul><li><strong>Restrict access to corporate networks</strong>: Only allow access from known office IPs or VPN ranges.</li><li><strong>Secure internal compute resources</strong>: Limit GCS access to certain GCE instances or internal services.</li><li><strong>Isolate dev/test environments</strong>: Enforce access boundaries by environment using network location.</li></ul><h3>External Use Cases</h3><ul><li><strong>Grant access to partners</strong>: Use allow lists to permit trusted third-party IPs.</li><li><strong>Restrict access to fixed client locations</strong>: Limit data access to specific office branches or static IP.</li><li><strong>Block malicious traffic</strong>: Deny access from suspicious or untrusted IP addresses.</li></ul><h3>GCS IP Filtering vs. VPC Service Controls (VPC SC)</h3><p>Though both IP filtering and VPC SC are used to secure GCS, they function differently and can be used together for layered security.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*e9HuDI7A3pO7Rec6GZ61Gg.png" /></figure><p>Working Together: A Defence-in-Depth Approach</p><ul><li><strong>VPC SC</strong> can establish a perimeter, ensuring resources aren’t accessed from outside the trusted network.</li><li><strong>GCS IP Filtering</strong> can then narrow access further, allowing only specific IPs — even within the perimeter — to access sensitive buckets.</li></ul><p>This multi-layered model helps prevent accidental exposure while supporting external collaborations securely.</p><h3><strong>Limitations of GCS IP Filtering</strong></h3><p>Despite its benefits, GCS IP filtering has several constraints:</p><ul><li><strong>Public IPs only</strong>: Private VPC IPs won’t work unless routed via a NAT gateway.</li><li><strong>No IPv6 support</strong>: Only IPv4 addresses and CIDRs are currently supported.</li><li><strong>No private IP granularity</strong>: Cannot restrict access to individual internal resources based on private IP.</li><li><strong>Header dependency</strong>: Requests through proxies/load balancers may obscure source IP without correct header forwarding.</li><li><strong>Operational overhead</strong>: Managing numerous IP filters across many buckets can be tedious.</li><li><strong>Global rule application</strong>: Rules apply globally, not regionally.</li><li><strong>Bypass potential</strong>: Malicious actors may spoof allowed IPs or compromise systems within trusted ranges.</li><li><strong>Not a replacement for IAM</strong>: GCS IP filtering doesn’t replace identity-based authentication and permission checks.</li></ul><p><strong>Note:</strong> Always combine IP filtering with IAM, encryption, and VPC SC for comprehensive security.</p><h3>Appendix:</h3><h4>Managing Permissions and IP Filtering</h4><h4>Step-by-Step: Creating and Binding a Custom Role</h4><ol><li><strong>Create the role with exempt permissions</strong>:</li></ol><p>gcloud iam roles create CustomStorageRole \</p><p>— project=PROJECT_NAME \</p><p>— title=”Custom Storage Role” \</p><p>— description=”Role with storage.buckets.exemptFromIpFilter permission” \</p><p>— permissions=storage.buckets.exemptFromIpFilter \</p><p>— stage=GA</p><ol><li><strong>Bind the role to a user or service account via the IAM console</strong>.</li><li><strong>To remove IP filtering from a bucket</strong>:</li></ol><p>gcloud alpha storage buckets update gs://BUCKET_NAME — clear-ip-filter</p><p><strong>3. For updates or new configurations</strong>, consult<a href="https://cloud.google.com/storage/docs/update-ip-filter#create-or-update-ip-filtering"> this guide</a>.</p><p>GCS IP Filtering is a powerful yet lightweight way to enhance access control based on network location. Whether you’re trying to limit access to specific offices, secure sensitive development environments, or share data with trusted partners, it offers a simple mechanism to bolster your bucket-level defences. While it shouldn’t replace broader security controls like IAM or VPC SC, it plays an important role in a <strong>defence-in-depth</strong> strategy.</p><p>Explore more:</p><ul><li><a href="https://cloud.google.com/storage/docs/ip-filtering-overview#limitations">IP Filtering Overview</a></li><li><a href="https://cloud.google.com/vpc-service-controls">VPC Service Controls</a></li><li><a href="https://cloud.google.com/iam">Google Cloud IAM</a></li></ul><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=81740748d511" width="1" height="1" alt=""><hr><p><a href="https://medium.com/niveus-solutions/enhancing-cloud-storage-security-with-google-cloud-storage-gcs-ip-filtering-81740748d511">Enhancing Cloud Storage Security with Google Cloud Storage (GCS) IP Filtering</a> was originally published in <a href="https://medium.com/niveus-solutions">Niveus Solutions Stories</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Mastering Kong API Gateway Setup on GKE | Architecture & Best Practices]]></title>
            <link>https://medium.com/niveus-solutions/kong-detail-setup-d56c3915f6da?source=rss-67c92e804e41------2</link>
            <guid isPermaLink="false">https://medium.com/p/d56c3915f6da</guid>
            <category><![CDATA[cloud-computing]]></category>
            <category><![CDATA[kong-ui]]></category>
            <category><![CDATA[infra-modernization]]></category>
            <category><![CDATA[gcp]]></category>
            <category><![CDATA[kong]]></category>
            <dc:creator><![CDATA[Venkatesh R]]></dc:creator>
            <pubDate>Wed, 09 Oct 2024 11:35:39 GMT</pubDate>
            <atom:updated>2025-03-16T18:48:24.082Z</atom:updated>
            <content:encoded><![CDATA[<h3><strong>Kong Detail Setup Part 2</strong></h3><h3>Architecture</h3><p>In a typical Kong API Gateway architecture deployed on Google Kubernetes Engine (GKE), Kong Manager (Kong UI) does not directly communicate with the Kong Gateway Ingress.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*pra1HsPcNwBwU9DVuQekOg.png" /></figure><p>Here’s a breakdown of the communication flow:</p><p><strong>Components</strong></p><p>Kong Manager (Kong UI): Provides a user interface for managing Kong Gateway configuration.</p><p>Kong Gateway Control Plane: Manages the overall configuration of the Kong Gateway Data Plane (handling external API traffic).</p><p>Kong Gateway Data Plane: Processes incoming requests from external clients, applying policies and routing them to backend services as defined by the configuration.</p><p>Kong Gateway Ingress: Exposes the Kong Gateway Data Plane to the external network via Kubernetes Ingress resources.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*ypheflB-ftp9cxTQ" /></figure><p><strong>Communication Flow</strong></p><p>1. User Interaction: Users interact with Kong Manager through a web browser.</p><p>2. Kong Manager (HTTPS): Kong Manager communicates with the Kong Gateway Control Plane using the Kong Admin API (typically over HTTPS).</p><p>3. Kong Admin API: The Kong Admin API provides an interface for managing Kong Gateway configuration (routes, plugins, etc.).</p><p>4. Control Plane Updates: Based on user actions in Kong Manager, the Kong Gateway Control Plane updates the configuration of the Kong Gateway Data Plane.</p><p>5. Data Plane Processing: The Kong Gateway Data Plane receives incoming requests, applies routing and policies based on the updated configuration, and forwards them to backend services.</p><p>6. External Traffic: External clients interact with the Kong Gateway Data Plane through the Kong Gateway Ingress.</p><p><strong>Key Points</strong></p><p>- Kong Manager and Kong Gateway Ingress serve different purposes:</p><p>- Kong Manager: Configuration management (user-facing).</p><p>- Kong Gateway Ingress: Routing external traffic (exposed to the internet).</p><p>- They indirectly communicate through the Kong Gateway Control Plane, which bridges the gap between management and data processing.</p><p><strong>User Interaction (External)</strong></p><p>User — — — — HTTPS (Optional) — — → Web Browser</p><p>GKE Infrastructure (Internal):</p><p>Web Browser — — — — → (Optional) Internal Load Balancer (ILB) — — — — →</p><p>Kong Manager (Optional Namespace)</p><p>Kong Manager Communication (Internal):</p><p>Kong Manager — — — — — REST API (Kong Admin) (Optional HTTPS) — — →</p><p>Kong Gateway Control Plane</p><p>Kong Gateway Functionality:</p><p>- Within GKE Cluster:</p><p>Kong Gateway Control Plane — — — — → Kong Gateway Data Plane (Internal Traffic)</p><p>- API Routes</p><p>- Plugins</p><p>- Configuration Management</p><p>Exposed to External Network:</p><p>External Clients — — — — → (Optional) GKE Load Balancer (External Traffic) — — → Kong Gateway Ingress — — → Kong Gateway Data Plane</p><p>Additional Information:</p><p>Kong Manager (Optional Namespace): This placement improves security and isolation.</p><p>HTTPS (Optional): Secures communication between components.</p><p>Kong Admin API: Interface for managing Kong Gateway.</p><p>Kong Gateway Ingress: Exposes the Kong Gateway Data Plane to external clients.</p><p><strong>Explanation</strong></p><p>1. Users interact with Kong Manager through a web browser (HTTPS optional) over the internet.</p><p>2. Optionally, an internal load balancer (ILB) can be used to restrict access to Kong Manager within your GKE network.</p><p>3. Kong Manager communicates with the Kong Gateway Control Plane using the Kong Admin API (internal traffic).</p><p>4. Within the GKE cluster, the Kong Gateway Control Plane manages the configuration of the Kong Gateway Data Plane, including API routes, plugins, and overall configuration.</p><p>5. External clients interact with the Kong Gateway Data Plane (exposed to the external network) through the Kong Gateway Ingress (optional GKE load balancer can be used for additional control).</p><p>6. Kong Gateway Ingress routes external traffic to the Kong Gateway Data Plane based on configuration.</p><p>This diagram emphasizes the separate roles of Kong Manager (configuration management) and Kong Gateway (traffic processing) while highlighting the internal communication between these components within the GKE cluster. The Kong Gateway Ingress serves as the entry point for external traffic.</p><h4>Prerequisites:</h4><p>- A functioning GKE cluster.</p><p>- `kubectl` installed and configured to interact with your GKE cluster.</p><p>- Helm package manager installed on your local machine.</p><p>Steps:</p><h4>1. Add the Kong Helm Repository:</h4><p>```bash</p><p>helm repo add kong <a href="https://charts.konghq.com/">https://charts.konghq.com</a></p><p>helm repo update</p><p>```</p><h4>2. Install Kong Gateway (if not already installed):</h4><ul><li>Refer to the Kong documentation for detailed instructions on installing Kong Gateway in your GKE cluster using Helm</li><li><a href="https://docs.konghq.com/kubernetes-ingress%20/controller/latest/install/cloud/gke/">https://docs.konghq.com/kubernetes-ingress /controller/latest/install/cloud/gke/</a></li></ul><h4>3. Install Kong Manager:</h4><p>- Create a separate namespace for Kong Manager (recommended):</p><p>```bash</p><p>kubectl create namespace kong-manager</p><p>```</p><p>- Create a values file (`values-manager.yaml`) to customize Kong Manager’s configuration :</p><p>```yaml</p><p># Replace with your domain name (if using a custom domain)</p><p>manager:</p><p>env:</p><p>admin_gui_url: https://UI-kong-manager.&lt;yourdomain&gt;.com</p><p># Enable Kong Manager authentication</p><p>enterprise:</p><p>rbac:</p><p>enabled: true</p><p>admin_gui_auth: basic-auth</p><p>```</p><h4>- Install Kong Manager using Helm:</h4><p>```bash</p><p>helm install kong-manager kong/kong \</p><p>— namespace kong-manager \</p><p>[PATH_TO_values-manager.yaml] # If using a custom values file</p><p>```</p><p>4. <strong>Configure Kong Manager Authentication :</strong></p><p>- If you enabled Kong Manager authentication in the `values-manager.yaml` file, you’ll need to create basic authentication credentials:</p><p>```bash</p><p>kubectl create secret generic kong-manager-credentials \</p><p>— namespace kong-manager \</p><p>— from-literal=username=kong_admin \</p><p>— from-literal=password=&lt;your_desired_password&gt;</p><p>```</p><p>- Replace `&lt;your_desired_password&gt;` with a strong password.</p><p>Creating an Internal Load Balancer (ILB) for Kong Manager using values.yaml</p><p>This example demonstrates how to configure an Internal Load Balancer (ILB) for Kong Manager using a values.yaml file, assuming you’re using Helm to deploy Kong on Google Kubernetes Engine (GKE).</p><p>Important Note:</p><p>The specific structure of your values.yaml file might vary depending on the Helm chart you’re using. This example provides a general guide, and you may need to adjust it based on your chosen chart.</p><p>Values.yaml Configuration:</p><p>```yaml</p><p># Service configuration for Kong Manager</p><p>kong:</p><p># Enable internal load balancer for Kong Manager</p><p>service:</p><p>type: LoadBalancer # Change to “LoadBalancer” for ILB</p><p># Optional: Specify a service name for Kong Manager</p><p>name: kong-manager # Adjust if needed</p><p># Ingress configuration (optional, for external access)</p><p># ingress:</p><p># enabled: false # Disable external access by default</p><p># Additional configurations for Kong Manager (if needed)</p><p># …</p><p>```</p><p><strong>Explanation:</strong></p><p>1. `kong.service.type`: This key-value pair sets the service type for Kong Manager. Change the value from `ClusterIP` (default) to `LoadBalancer` to create an internal load balancer.</p><p>2. `kong.service.name` (Optional): This key-value pair allows you to specify a custom service name for Kong Manager.</p><p><strong>Deploying with Helm:</strong></p><p>Once you’ve updated your values.yaml file with the desired settings, run the following command to deploy Kong with the internal load balancer configuration:</p><p>```bash</p><p>helm install kong — namespace &lt;your-namespace&gt; -f values.yaml</p><p>```</p><p><strong>Accessing Kong Manager:</strong></p><p>To access Kong Manager after deployment, use the internal IP address assigned to the service. You can retrieve the IP address using the following command:</p><p>```bash</p><p>kubectl get service kong-manager (or your custom service name) -n &lt;your-namespace&gt; -o jsonpath=’{.status.loadBalancer.ingress[0].ip}’</p><p>```</p><p>Additional Considerations:</p><p>- Remember to update `&lt;your-namespace&gt;` with the namespace where you’re deploying Kong.</p><p>- If you’re using a custom service name (e.g., `kong-manager`), replace it accordingly.</p><p>- This example disables external access through the Ingress section (commented out). Adjust these settings based on your specific requirements.</p><p>5. Access Kong Manager:</p><p>- Obtain the IP address for Kong Manager:</p><p>```bash</p><p>kubectl get ingress -n kong-manager</p><p>```</p><p>- In your web browser, navigate to the Kong Manager URL (e.g., `https://&lt;ingress_ip&gt;` or `https://UI-kong-manager.domainname.com` if using a custom domain).</p><p>- Use the credentials you configured (default: username `kong_admin`, password set during installation) to log in.</p><h4>Interaction between Kong Manager and Kong API Gateway:</h4><p>- Kong Manager provides a graphical user interface (GUI) for managing your Kong API Gateway deployed in GKE.</p><p>- You can use Kong Manager to:</p><p>- Create and manage APIs (register upstream services).</p><p>- Define routes and plugins for routing traffic to your APIs.</p><p>- Configure authentication, authorization, and rate limiting policies.</p><p>- Monitor API traffic and performance.</p><p>- Kong Manager communicates with the Kong Gateway control plane through the Kong API to manage these configurations.</p><p>Secure your Kong Manager deployment by considering HTTPS, firewalls, and proper access control.</p><p>- Explore advanced Kong Manager features like plugins, custom code, and integrations.</p><p>- Refer to the Kong documentation for detailed configuration options and best practices: <a href="https://docs.konghq.com/gateway-operator/latest/get-started/konnect/install/">https://docs.konghq.com/gateway-operator/latest/get-started/konnect/install/</a></p><h4>Kong Manager with SSL Certificate in GKE</h4><h4>Components:</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/682/0*RppPxLDT7cbkemGY" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*S8XzTxkhN5m4Q6OR" /></figure><p>Ref : <a href="https://docs.konghq.com/gateway/3.6.x/kong-manager/">https://docs.konghq.com/gateway/3.6.x/kong-manager/</a></p><h4><strong>GKE Cluster: Manages containerized workloads, including Kong Gateway and Kong Manager deployments.</strong></h4><h4><strong>Kong Gateway Deployment: Runs the Kong API Gateway within the GKE cluster, managing API traffic.</strong></h4><p>Kong Manager Deployment (Optional Namespace): Deployed in GKE, offering a GUI for managing the Kong Gateway.</p><p>cert-manager Deployment: Signed Certificates / Manages SSL certificate issuance and renewal using Let’s Encrypt.</p><h4>External DNS Provider : (e.g., Cloud DNS) Manages DNS records for your domain name if using a custom domain for Kong Manager.</h4><p>User: Accesses Kong Manager through a web browser.</p><h4>Data Flow:</h4><p>1. User: Accesses Kong Manager in their web browser, typically using a URL like `https://UI-kong-manager.yourdomain.com`.</p><p>2. Web Browser: Sends a HTTPS request to Kong Manager’s Ingress IP or DNS name.</p><p>3. GKE Load Balancer: Routes the HTTPS request to the Kong Manager service based on configuration.</p><p>4. Kong Manager Ingress: Matches the request based on domain name and path and routes it to the Kong Manager deployment.</p><p>5. Kong Manager Deployment:</p><p>- Verifies the SSL certificate presented by the Ingress using the trusted certificate authority</p><p>- If valid, it serves the Kong Manager user interface (UI).</p><p>6. User: Interacts with the Kong Manager UI to manage APIs, routes, plugins, etc.</p><p>7. Kong Manager (control plane)</p><p>- Communicates with the Kong Gateway control plane to configure APIs, routes, and other settings.</p><p>8. cert-manager:</p><p>- Manages certificate issuance and renewal Signed Certificate..</p><p>- Interacts with the External DNS Provider to update DNS records for certificate challenges.</p><p><strong>Prerequisites:</strong></p><p>The Admin API should be enabled for the Kong Manager</p><p>Enable Kong Manager via docker if the data plane not able to fetch the details:</p><ol><li>Enable the Admin Api as mentioned in the above steps.</li><li>Download the latest docker image of the Kong.</li><li>Pass the admin url to the docker image.</li><li><a href="https://docs.konghq.com/gateway/latest/kong-manager/enable/">Run the Docker Image.</a></li><li>Access the Docker image via a static Url.</li></ol><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=d56c3915f6da" width="1" height="1" alt=""><hr><p><a href="https://medium.com/niveus-solutions/kong-detail-setup-d56c3915f6da">Mastering Kong API Gateway Setup on GKE | Architecture &amp; Best Practices</a> was originally published in <a href="https://medium.com/niveus-solutions">Niveus Solutions Stories</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Kafka UI: A Visual Companion for Your Kafka Cluster]]></title>
            <link>https://medium.com/niveus-solutions/kafka-ui-a-visual-companion-for-your-kafka-cluster-4c413535f700?source=rss-67c92e804e41------2</link>
            <guid isPermaLink="false">https://medium.com/p/4c413535f700</guid>
            <category><![CDATA[kafka-ui]]></category>
            <category><![CDATA[cloud-computing]]></category>
            <category><![CDATA[gcp]]></category>
            <category><![CDATA[infra-modernization]]></category>
            <category><![CDATA[kafka]]></category>
            <dc:creator><![CDATA[Venkatesh R]]></dc:creator>
            <pubDate>Wed, 09 Oct 2024 09:31:29 GMT</pubDate>
            <atom:updated>2025-03-16T18:53:16.998Z</atom:updated>
            <content:encoded><![CDATA[<h3>Kafka UI with Apache Kafka on VM Part 3</h3><p><strong>Introduction</strong></p><p>Kafka, a distributed streaming platform, has become a cornerstone of modern data pipelines. Its ability to handle massive volumes of data in real-time makes it ideal for a wide range of applications. However, managing a Kafka cluster can be complex, especially as it grows in size and complexity. This is where a Kafka UI becomes invaluable.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*UTE5cxY_fE-pZ2EIpPnKyQ.png" /></figure><p><strong>What is a Kafka UI?</strong></p><p>A Kafka UI is a web-based interface that provides a visual representation of your Kafka cluster. It simplifies tasks such as:</p><ul><li><strong>Topic management:</strong> Creating, deleting, and listing topics</li><li><strong>Consumer management:</strong> Viewing consumer groups, offsets, and lag</li><li><strong>Producer management:</strong> Sending messages to topics</li><li><strong>Cluster monitoring:</strong> Tracking metrics like broker health, throughput, and latency</li></ul><p>By providing a user-friendly interface, a Kafka UI can significantly reduce the time and effort required to manage a Kafka cluster.</p><p><strong>Why is a Kafka UI Needed?</strong></p><p><strong>Simplified Management:</strong></p><ul><li><strong>Centralized view:</strong> A Kafka UI provides a single, centralized location to manage all aspects of your Kafka cluster.</li><li><strong>Intuitive interface:</strong> The visual interface makes it easy to understand and interact with the cluster.</li><li><strong>Reduced complexity:</strong> Complex tasks like managing consumer groups and offsets become much simpler.</li></ul><p><strong>Improved Troubleshooting:</strong></p><ul><li><strong>Real-time monitoring:</strong> Kafka UIs often provide real-time monitoring of key metrics, making it easier to identify and address issues.</li><li><strong>Visual diagnostics:</strong> Visualizations can help pinpoint problems more quickly.</li></ul><p><strong>Enhanced Collaboration:</strong></p><ul><li><strong>Shared access:</strong> Multiple team members can access the UI to collaborate on cluster management.</li><li><strong>Improved communication:</strong> The visual interface can facilitate better communication and understanding.</li></ul><p><strong>Increased Efficiency:</strong></p><ul><li><strong>Automation:</strong> Some Kafka UIs offer automation features, such as creating topics or assigning consumers, which can save time.</li><li><strong>Reduced errors:</strong> The visual interface can help prevent errors and inconsistencies.</li></ul><p><strong>Choosing the Right Kafka UI</strong></p><p>There are several Kafka UIs available, each with its own features and strengths. When choosing a UI, consider the following factors:</p><ul><li><strong>Functionality:</strong> Does the UI provide the features you need, such as topic management, consumer management, and monitoring?</li><li><strong>Ease of use:</strong> Is the UI intuitive and easy to learn?</li><li><strong>Integration:</strong> Can the UI integrate with other tools in your ecosystem, such as monitoring systems or alerting services?</li><li><strong>Scalability:</strong> Can the UI handle the size and complexity of your Kafka cluster?</li></ul><p><strong>Architecture:</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/485/0*p0Y-I3Ygf1_OCb8e" /></figure><p><strong>Components:</strong></p><p>User</p><p>Web Browser</p><p>VM Firewall</p><p>Kafka UI</p><p>ZooKeeper (on VM)</p><p>Kafka Broker (on VM)</p><p>Kafka Producer (External Clients)</p><p>Kafka Consumer (External Clients)</p><p><strong>Data Flow:</strong></p><p>1. User — — HTTPS — → Web Browser</p><p>2. Web Browser — — → VM Firewall — — -&gt; Kafka UI</p><p>- If Kafka UI is on the VM, this connection wouldn’t exist.</p><p>3. Kafka UI — — REST API — → ZooKeeper (on VM)</p><p>- If Kafka UI is on a separate server, the connection would be external.</p><p>4. Kafka UI — — REST API — → Kafka Broker (on VM -</p><p>If Kafka UI is on a separate server, the connection would be external.</p><p>5. Kafka Producer (External Clients) — — -&gt; Kafka Broker (on VM)</p><p>6. Kafka Broker (on VM) — — (Message Routing &amp; Replication) — — → Kafka Consumer (External Clients)</p><p><strong>key phrases used</strong></p><p>` — — HTTPS — — -&gt;`: User interaction with web browser (HTTPS optional)</p><p>` — — →`: Optional component or connection</p><p>` — — -&gt;`: Data flow</p><p>``: Denotes an optional component</p><p><strong>Kafka UI :</strong></p><p>General</p><p>Clone/Copy Chart to your working directory Execute command</p><p>helm repo add kafka-ui <a href="https://provectus.github.io/kafka-ui-charts">https://provectus.github.io/kafka-ui-charts</a></p><p>helm install kafka-ui kafka-ui/kafka-ui</p><p><em>Passing Kafka-UI configuration as Dict</em></p><p>Create values.yml file</p><p>Install by executing command</p><p>helm install helm-release-name charts/kafka-ui -f values.yml</p><p><em>Passing configuration file as ConfigMap</em></p><p>Create config map</p><p>yamlApplicationConfig:</p><p>kafka:</p><p>clusters:</p><p>- name: yaml</p><p>bootstrapServers: kafka-cluster-broker-endpoints:9092</p><p>auth:</p><p>type: disabled</p><p>management:</p><p>health:</p><p>ldap:</p><p>enabled: false</p><p><em>This ConfigMap will be mounted to the Pod</em></p><p>Install by executing the command</p><p>helm install helm-release-name charts/kafka-ui — set yamlApplicationConfigConfigMap.name=”kafka-ui-configmap”,yamlApplicationConfigConfigMap.keyName=”config.yml”</p><p><strong>Passing environment variables as ConfigMap</strong></p><p>Create config map</p><p>apiVersion: v1</p><p>kind: ConfigMap</p><p>metadata:</p><p>name: kafka-ui-helm-values</p><p>data:</p><p>KAFKA_CLUSTERS_0_NAME: “kafka-cluster-name”</p><p>KAFKA_CLUSTERS_0_BOOTSTRAPSERVERS: “kafka-cluster-broker-endpoints:9092”</p><p>AUTH_TYPE: “DISABLED”</p><p>MANAGEMENT_HEALTH_LDAP_ENABLED: “FALSE”</p><p>Install by executing the command</p><p>helm install helm-release-name charts/kafka-ui — set existingConfigMap=”kafka-ui-helm-values”</p><p>Clone the repo:</p><p><a href="https://github.com/provectus/kafka-ui">https://github.com/provectus/kafka-ui</a></p><p>Access the Application:</p><p>Then access the web UI at<a href="http://localhost:8080/"> http://VMIP:8080</a></p><p>Resources:</p><p>- Kafka UI Configuration: <a href="https://docs.kafka-ui.provectus.io/configuration/helm-charts/quick-start">https://docs.kafka-ui.provectus.io/configuration/helm-charts/quick-start</a></p><p>Set the basic Authentication:</p><p>yamlApplicationConfig:</p><p>auth:</p><p>type: login_form</p><p>spring:</p><p>security:</p><p>user:</p><p>name: admin</p><p>password: &lt;Password&gt;</p><p><a href="https://github.com/provectus/kafka-ui/issues/3426">https://github.com/provectus/kafka-ui/issues/3426</a></p><p><strong>Connecting to a GCP application using a private IP from your laptop involves several steps:</strong></p><p><strong>1. Setting up the GCP environment:</strong></p><ul><li><strong>Private IP</strong>: Ensure your GCP application has a private IP address assigned. This might involve creating a Cloud SQL instance with a private IP or configuring a Compute Engine VM without assigning a public IP.</li><li><strong>VPC Peering (Optional):</strong> If your application resides in a different VPC network than your laptop, you might need to set up VPC peering to allow communication between the networks.</li><li><strong>Firewall Rules:</strong> Configure firewall rules on the GCP side to allow incoming traffic from your laptop’s IP address on the specific port your application uses.</li></ul><p><strong>2. Local User Authentication:</strong></p><ul><li><strong>Local User Accounts:</strong> Make sure your GCP application supports local user authentication and has appropriate accounts created for access.</li><li><strong>Service Accounts (Not Recommended):</strong> Using service accounts for local access is not recommended as it requires additional configuration and might not be suitable for interactive use.</li></ul><p><strong>3. Routing (Optional):</strong></p><ul><li><strong>Static Routes (If necessary):</strong> If your laptop isn’t on the same network as the GCP application, you might need to configure static routes on your laptop to direct traffic to the application’s private IP address through the appropriate gateway.</li></ul><p><strong>4. Connecting from your Laptop:</strong></p><ul><li><strong>Client Software:</strong> Use the appropriate client software for your application. For example, a MySQL workbench for a Cloud SQL instance or an SSH client for a Compute Engine VM.</li><li><strong>Private IP and Local Username:</strong> Connect to the application using the private IP address obtained in step 1 and the local username and password configured in step 2.</li></ul><p><strong>Here are some resources that might be helpful:</strong></p><ul><li><strong>Connecting to Cloud SQL with Private IP:</strong><a href="https://cloud.google.com/sql/docs/mysql/connect-instance-private-ip"> https://cloud.google.com/sql/docs/mysql/connect-instance-private-ip</a></li><li><strong>Compute Engine with IAP (for local user access):</strong><a href="https://mentorcruise.com/blog/how-to-connect-to-gcp-compute-engine-with-private-ip-via-iap-90c85/"> https://mentorcruise.com/blog/how-to-connect-to-gcp-compute-engine-with-private-ip-via-iap-90c85/</a></li></ul><p>Exposing applications directly to the internet using private IPs is not recommended for security reasons. Consider using a bastion host or a VPN connection for a more secure approach.</p><p><strong>Conclusion</strong></p><p>A Kafka UI can be a valuable asset for any organization using Kafka. By simplifying management, improving troubleshooting, enhancing collaboration, and increasing efficiency, a Kafka UI can help you get the most out of your Kafka cluster.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=4c413535f700" width="1" height="1" alt=""><hr><p><a href="https://medium.com/niveus-solutions/kafka-ui-a-visual-companion-for-your-kafka-cluster-4c413535f700">Kafka UI: A Visual Companion for Your Kafka Cluster</a> was originally published in <a href="https://medium.com/niveus-solutions">Niveus Solutions Stories</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Setting Up Kong UI and Kafka UI: A Comprehensive Guide.]]></title>
            <link>https://medium.com/niveus-solutions/setting-up-kong-ui-and-kafka-ui-a-comprehensive-guide-d9bfcd0dc757?source=rss-67c92e804e41------2</link>
            <guid isPermaLink="false">https://medium.com/p/d9bfcd0dc757</guid>
            <category><![CDATA[kong-api-gateway]]></category>
            <category><![CDATA[cloud-computing]]></category>
            <category><![CDATA[infra-modernization]]></category>
            <category><![CDATA[kafka]]></category>
            <category><![CDATA[kong]]></category>
            <dc:creator><![CDATA[Venkatesh R]]></dc:creator>
            <pubDate>Tue, 08 Oct 2024 08:22:24 GMT</pubDate>
            <atom:updated>2025-03-16T18:56:52.453Z</atom:updated>
            <content:encoded><![CDATA[<h3>Setting Up Kong UI and Kafka UI: A Comprehensive Guide</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*gJXQP9VgKKvT63ElKSs0Vg.png" /></figure><p><strong>Introduction</strong></p><p>Kong and Kafka are two powerful tools that have become indispensable in modern microservices architectures. Kong, a high-performance open-source API gateway, provides a single point of entry for all API requests, offering features like rate limiting, authentication, and load balancing. Kafka, a distributed streaming platform, is ideal for handling real-time data pipelines and event-driven applications.</p><p>In this blog post, we’ll guide you through the process of setting up Kong UI and Kafka UI, providing a step-by-step approach and an illustrative architecture diagram.</p><p><strong>Architecture Diagram</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/636/0*hhNq4JKdHmahKfm_" /></figure><p><strong>Prerequisites</strong></p><p>Before we begin, ensure you have the following:</p><ul><li>A running Docker instance.</li><li>Basic knowledge of Docker and Docker Compose.</li><li>A Kong cluster (if not already set up).</li></ul><h4><strong>Step-by-Step Guide</strong></h4><p>Set Up Kong UI</p><p><a href="https://docs.konghq.com/gateway/latest/kong-manager/"><strong>Kong UI Official Documentation.</strong></a></p><p>Docker based setup. <a href="https://docs.konghq.com/gateway/latest/kong-manager/enable/">Kong UI Official</a></p><p><strong>Kong UI repository: </strong>git clone <a href="https://github.com/Kong/kong-manager.git">https://github.com/Kong/kong-manager.git</a></p><blockquote>docker exec -i &lt;KONG_CONTAINER_ID&gt; /bin/sh -c “export KONG_ADMIN_GUI_PATH=’/’; export KONG_ADMIN_GUI_URL=’<a href="http://localhost:8002/manager&#39;">http://localhost:8002/manager&#39;</a>; kong reload; exit”</blockquote><blockquote>Replace <a href="http://kong:8001/">http://kong:8001</a> with the actual URL of your Kong Admin API.</blockquote><h4>Set Up Kafka UI</h4><ul><li><strong>Download the Kafka UI JAR:</strong></li></ul><blockquote><strong>Run the Kafka UI JAR: </strong>java -jar kafka-ui-x.x.x.jar Replace x.x.x with the actual version of Kafka UI.</blockquote><ul><li><strong>Configure Kafka UI:</strong></li><li>Open the Kafka UI web interface.</li><li>Provide the necessary Kafka broker addresses and other configuration details.</li></ul><p><strong>Accessing the UIs</strong></p><ul><li><strong>Kong UI:</strong> Access the Kong UI at <a href="http://localhost:8001.">http://localhost:8001.</a></li><li><strong>Kafka UI:</strong> Access the Kafka UI at the specified URL in the console output.</li></ul><p><strong>Additional Tips</strong></p><ul><li>For production environments, consider using a container orchestration platform like Kubernetes to manage and scale your Kong UI and Kafka UI deployments.</li><li>Explore the features and capabilities of both Kong UI and Kafka UI to optimize your API management and data streaming workflows.</li></ul><p>By leveraging Kong UI and Kafka UI, we can enhance the efficiency and reliability of your microservices architecture.</p><p><strong>Conclusion</strong></p><p>By following these steps, you’ve successfully set up Kong UI and Kafka UI, providing you with valuable tools for managing your API gateway and Kafka cluster.</p><p>Upcoming Part 2 of this series contains the detailed Kong API UI setup</p><p>Upcoming Part 3 of this series contains the detailed Kafka UI setup</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=d9bfcd0dc757" width="1" height="1" alt=""><hr><p><a href="https://medium.com/niveus-solutions/setting-up-kong-ui-and-kafka-ui-a-comprehensive-guide-d9bfcd0dc757">Setting Up Kong UI and Kafka UI: A Comprehensive Guide.</a> was originally published in <a href="https://medium.com/niveus-solutions">Niveus Solutions Stories</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Secure Cloud SQL Access with IAM Authentication: A Guide]]></title>
            <link>https://medium.com/niveus-solutions/securely-accessing-cloud-sql-with-iam-authentication-a-step-by-step-guide-499887001ef3?source=rss-67c92e804e41------2</link>
            <guid isPermaLink="false">https://medium.com/p/499887001ef3</guid>
            <category><![CDATA[cloud-security]]></category>
            <category><![CDATA[cloud-services]]></category>
            <category><![CDATA[cloud-sql]]></category>
            <category><![CDATA[cloud-computing]]></category>
            <category><![CDATA[gcp]]></category>
            <dc:creator><![CDATA[Venkatesh R]]></dc:creator>
            <pubDate>Sat, 07 Sep 2024 04:22:14 GMT</pubDate>
            <atom:updated>2024-12-20T11:41:49.292Z</atom:updated>
            <content:encoded><![CDATA[<h3>Securely Accessing Cloud SQL with IAM Authentication: A Step-by-Step Guide</h3><p>This blog post guides you through setting up IAM authentication for accessing your Cloud SQL instance. This method enhances security by leveraging Google Cloud identities instead of passwords.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*OPR523v4tqv2DdFLHMXr8Q.png" /></figure><p><strong>Prerequisites:</strong></p><ul><li>A Google Cloud Platform (GCP) project with a Cloud SQL instance.</li><li>IAM user with the cloudsql.instanceUser and cloudsql.client roles assigned.</li><li>Jump server (optional) for connecting from environments with restricted access.</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*BLRCMbSj1VeLLLMUcBZbxw.jpeg" /></figure><p><strong>Enabling IAM Authentication and Creating Database User:</strong></p><ol><li><strong>Enable IAM Authentication:</strong> Navigate to your Cloud SQL instance in the GCP Console and locate the “Settings” section. Under “IAM Authentication,” set the flag to “ON” (as shown in [Image])</li></ol><figure><img alt="" src="https://cdn-images-1.medium.com/max/929/0*egF46mmA9BTIjOiC" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/472/0*lglk7L1hTYYVLQo8" /></figure><ol><li><strong>Create Database (Optional):</strong> If necessary, create a database using your preferred SQL language commands.</li><li><strong>Create IAM User:</strong> Grant access by creating a user with the format user@emailid in your Cloud SQL instance. Set the user type to CLOUD_IAM_USER (as shown in Image)</li></ol><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*0NJ6Z20-2wO-p__K" /></figure><p><strong>Ref:</strong> <a href="https://cloud.google.com/sql/docs/postgres/iam-authentication#automatic">https://cloud.google.com/sql/docs/postgres/iam-authentication#automatic</a></p><p><strong>Installing Cloud SQL Proxy:</strong></p><ol><li><strong>Download Proxy:</strong> Use curl to download the Cloud SQL proxy for your operating system (shown in the provided code snippet).</li><li><strong>Set Permissions:</strong> Make the downloaded file executable using chmod +x cloud-sql-proxy.</li></ol><p><strong>Connecting from Local Machine (Public IP):</strong></p><ol><li><strong>Run Proxy:</strong> Execute the following command, replacing placeholders with your details:</li></ol><p>./cloud-sql-proxy — address 127.0.0.1 — port 1234 prj-name:region:db-name — auto-iam-authn</p><ol><li><strong>Connect to Database:</strong> Use your preferred SQL client (e.g., psql) to connect with IAM authentication:</li></ol><p>psql “host=127.0.0.1 port=5432 dbname=dbname user=user@emailid sslmode=disable”</p><p><strong>Connecting from Local Machine (Private IP) or Jump Server:</strong></p><p><strong>Option 1: Using Jump Server (VPN):</strong></p><ol><li><strong>Connect to VPN:</strong> Establish a VPN connection to your GCP network if your local system has a private IP.</li><li><strong>Run Proxy on Jump Server:</strong> Similar to the local connection, execute the proxy command on the jump server, specifying — private-ip (refer to the code snippet).</li></ol><p><strong>Option 2: Using Cloud IAP (No VPN):</strong></p><ol><li><strong>Enable IAP for Cloud SQL:</strong> Follow the official guide (link provided) to enable Cloud IAP for your Cloud SQL instance.</li><li><strong>SSH with IAP Tunneling:</strong> Use the gcloud compute ssh command with the — tunnel-through-iap flag to establish a secure connection to your jump server (refer to the code snippet).</li></ol><p><strong>Connecting from Jump Server:</strong></p><ol><li><strong>Run Proxy:</strong> Execute the proxy command on the jump server with — private-ip and — auto-iam-authn flags (refer to the code snippet).</li><li><strong>Connect to Database:</strong> Use your preferred SQL client to connect with IAM authentication, specifying the jump server’s private IP address as the host (refer to the provided code snippet).</li></ol><p><strong>Troubleshooting:</strong></p><p>The provided error message (ACCESS_TOKEN_SCOPE_INSUFFICIENT) indicates insufficient permissions for the Cloud SQL proxy. Here’s how to resolve it:</p><ol><li><strong>Verify Authentication:</strong> Ensure you are logged in to the GCP Console with the appropriate IAM user.</li><li><strong>Check Scopes:</strong> Make sure the user has the cloudsql.instanceUser and cloudsql.client roles assigned.</li><li><strong>Re-run Authentication:</strong> If using the gcloud CLI, re-run gcloud auth login to refresh your credentials.</li></ol><p><strong>Additional Notes:</strong></p><ul><li>Consider installing the “Connect to server” plugin for your IDE (like VSSC) for a more convenient connection experience.</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/960/0*ZCvO4FmBdyWBOFHh" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*bpBwdjQB5eiEmBKpPKaKSw.jpeg" /></figure><ul><li>Refer to the provided references for detailed information on Cloud SQL IAM authentication, creating database users, and installing the Cloud SQL proxy.</li></ul><p>By following these steps and addressing potential issues, We can securely access your Cloud SQL instance leveraging IAM authentication for enhanced security.</p><p><a href="https://cloud.google.com/blog/products/databases/application-security-with-cloud-sql-iam-database-authentication">Cloud SQL IAM authentication from Kubernetes and Google Cloud provided connectors | Google Cloud Blog</a></p><p><strong>Useful Commands</strong></p><h4>Enabling IAM Authentication and Creating Database User</h4><p><strong>URL:</strong><a href="https://cloud.google.com/sql/docs/mysql/create-edit-iam-instances"> https://cloud.google.com/sql/docs/mysql/create-edit-iam-instances</a></p><p><strong>Commands:</strong></p><ul><li><strong>Set IAM Authentication flag to ON:</strong> This can be done in the GCP Console under the “Settings” section of your Cloud SQL instance.</li></ul><p><strong>Create database:</strong></p><p>CREATE DATABASE dbname;</p><p><strong>Create IAM user:</strong></p><p>CREATE USER user@emailid CLOUD_IAM_USER;</p><p><strong>Installing Cloud SQL Proxy</strong></p><p><strong>URL:</strong><a href="https://cloud.google.com/sql/docs/mysql/connect-instance-auth-proxy"> https://cloud.google.com/sql/docs/mysql/connect-instance-auth-proxy</a></p><p><strong>Command:</strong></p><p>curl -o cloud-sql-proxy <a href="https://storage.googleapis.com/cloud-sql-connectors/cloud-sql-proxy/v2.11.4/cloud-sql-proxy.linux.amd64">https://storage.googleapis.com/cloud-sql-connectors/cloud-sql-proxy/v2.11.4/cloud-sql-proxy.linux.amd64</a></p><p>chmod +x cloud-sql-proxy</p><p><strong>Connecting from Local Machine (Public IP)</strong></p><p><strong>Commands:</strong></p><p>./cloud-sql-proxy — address 127.0.0.1 — port 1234 prj-name:region:db-name — auto-iam-authn</p><p><strong>Connecting from Local Machine (Private IP) or Jump Server</strong></p><p>psql “host=127.0.0.1 port=5432 dbname=dbname user=user@emailid sslmode=disable”</p><p><strong>URL:</strong><a href="https://cloud.google.com/sql/docs/mysql/connect-instance-private-ip"> https://cloud.google.com/sql/docs/mysql/connect-instance-private-ip</a></p><p><strong>Commands:</strong></p><ul><li><strong>Run proxy on jump server (with private IP):</strong></li></ul><p>./cloud-sql-proxy — address 127.0.0.1 — port portno prj-name:region:db-name — private-ip — auto-iam-authn</p><p><strong>SSH with IAP tunneling:</strong></p><p>gcloud compute ssh vmname — tunnel-through-iap</p><p><strong>Connecting from Jump Server</strong></p><p><strong>Commands:</strong></p><ul><li><strong>Run proxy on jump server:</strong></li></ul><p>./cloud-sql-proxy — address 127.0.0.1 — port portno prj-name:region:db-name — private-ip — auto-iam-authn</p><ul><li><strong>Connect to database</strong></li></ul><p>psql “host=dbIP port=portno dbname=dbname user=user@emailid sslmode=disable”</p><p><strong>Additional References</strong></p><ul><li><strong>Creating IAM users:</strong><a href="https://cloud.google.com/sql/docs/mysql/create-edit-iam-instances"> https://cloud.google.com/sql/docs/mysql/create-edit-iam-instances</a></li><li><strong>Installing PostgreSQL on Ubuntu:</strong><a href="https://phoenixnap.com/kb/how-to-install-postgresql-on-ubuntu"> https://phoenixnap.com/kb/how-to-install-postgresql-on-ubuntu</a></li><li><strong>Installing MySQL on Ubuntu:</strong><a href="https://www.digitalocean.com/community/tutorials/how-to-install-mysql-on-ubuntu-22-04"> https://www.digitalocean.com/community/tutorials/how-to-install-mysql-on-ubuntu-22-04</a></li></ul><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=499887001ef3" width="1" height="1" alt=""><hr><p><a href="https://medium.com/niveus-solutions/securely-accessing-cloud-sql-with-iam-authentication-a-step-by-step-guide-499887001ef3">Secure Cloud SQL Access with IAM Authentication: A Guide</a> was originally published in <a href="https://medium.com/niveus-solutions">Niveus Solutions Stories</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Navigate DevOps Tools — Azure DevOps, AWS, GCP, Open Source]]></title>
            <link>https://tatankavenkat.medium.com/navigate-devops-tools-azure-devops-aws-gcp-open-source-36ec50184262?source=rss-67c92e804e41------2</link>
            <guid isPermaLink="false">https://medium.com/p/36ec50184262</guid>
            <category><![CDATA[aws]]></category>
            <category><![CDATA[gcp]]></category>
            <category><![CDATA[azure]]></category>
            <category><![CDATA[devops]]></category>
            <category><![CDATA[cloud-computing]]></category>
            <dc:creator><![CDATA[Venkatesh R]]></dc:creator>
            <pubDate>Fri, 05 Jul 2024 12:39:38 GMT</pubDate>
            <atom:updated>2024-09-27T06:01:31.522Z</atom:updated>
            <content:encoded><![CDATA[<h3>Navigate DevOps Tools — Azure DevOps, AWS, GCP, Open Source</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/500/0*6mHl3t2c7OqSVkkH.jpg" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1000/0*SQumeVn3z9ZEbz24.jpg" /></figure><p>As the DevOps landscape continues to shift rapidly, choosing the right tools for your software delivery pipeline can be daunting. With numerous options available, each offering unique capabilities and integrations, finding the right fit for your team’s needs is crucial. This blog post explores a comprehensive range of solutions, including Azure DevOps, AWS services, GCP Cloud Build, open-source options, and third-party tools, to help you make an informed decision on the best DevOps toolset for your organization. Whether you’re seeking a centralized platform, a modular approach, or open-source flexibility, understanding the key features and benefits of each toolset will empower you to optimize your DevOps processes and achieve faster, more reliable software delivery.</p><p>The right DevOps tools not only streamline your development processes but also enhance collaboration, automate workflows, and ensure high-quality code deployment. This guide is designed to help you navigate through the myriad of DevOps solutions available, providing insights into their core functionalities, advantages, and real-world applications. By leveraging the right toolset tailored to your specific needs, you can transform your DevOps strategy, reduce time-to-market, and improve overall operational efficiency. Dive in to discover how each tool can play a pivotal role in your software development lifecycle and position your organization for success.</p><h3>Centralized Powerhouse: Azure DevOps</h3><p><strong>Azure DevOps</strong> is Microsoft’s all-in-one suite for managing the software development lifecycle. Its comprehensive set of tools is designed to cater to every aspect of the DevOps process, making it a go-to solution for many organizations. Here are some detailed aspects of its offerings:</p><h3>Overview and Key Features</h3><ol><li><strong>Testing:<br></strong>Automated and Manual Testing: Azure Test Plans provides a rich set of tools for manual, exploratory, and automated testing. This ensures that your code is thoroughly tested before deployment, maintaining high quality and reliability.</li><li><strong>Collaboration:</strong><br>Integrated Communication: Azure DevOps tools promote seamless collaboration with features like work item tracking, team chat, and integrated feedback mechanisms. This ensures that all stakeholders are aligned, and any issues can be promptly addressed.</li></ol><h3>AWS Modular Approach: Build, Deploy, and Beyond</h3><p><strong>Amazon Web Services</strong> offers a suite of specialized DevOps tools in AWS that provide flexibility and scalability. Here’s a deeper look into its modular approach.</p><h3>Overview and Key Features</h3><h3>Comparison with Azure DevOps</h3><p>While Azure DevOps provides an all-in-one solution, AWS’s approach is more modular, allowing users to pick and choose the services they need. This can be particularly advantageous for organizations already deeply integrated into the AWS ecosystem, as it allows for a more tailored and flexible setup. Each <strong>DevOps tool in AWS</strong> can be integrated as needed, providing a customizable and scalable solution.</p><h3>GCP Cloud Build: Streamlined CI/CD in Google Cloud</h3><p><strong>Google Cloud (GCP) Cloud Build</strong> is a serverless CI/CD platform that offers a streamlined way to build, test, and deploy applications within the GCP ecosystem. As a crucial part of the <strong>Google Cloud (GCP) DevOps Tools</strong> suite, Cloud Build provides numerous features to enhance the efficiency and scalability of your <a href="https://niveussolutions.com/setting-up-ci-cd-pipelines-best-practices-case-studies/"><strong>CI/CD pipelines</strong></a>. Here’s an in-depth look at its features.</p><h3>Overview and Key Features</h3><h3>Open Source Options: Flexibility and Customization</h3><p>When choosing an open-source CI/CD tool, there are several factors to consider. Flexibility and customization are key advantages of open-source solutions, allowing you to tailor the tool to your specific needs and integrate it with other tools in your DevOps pipeline. However, the responsibility for setup, maintenance, and security updates also falls on your team. Here’s a closer look at two popular open-source options for your CI/CD pipeline:</p><h3>Jenkins</h3><p><strong>Jenkins</strong> is one of the most widely used open-source automation servers, known for its flexibility and extensive plugin ecosystem. Here’s a closer look:</p><h3>GitHub Actions</h3><p><strong>GitHub Actions</strong> brings CI/CD capabilities directly into the GitHub ecosystem. Here’s an exploration of its features:</p><h3>Comparison of what’s been offered by the Vendors</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/726/0*lfiYqr0dcDAzICyb" /></figure><h3>Choosing the Right Toolset</h3><p>Selecting the appropriate DevOps tools depends on various factors, including your team’s expertise, existing infrastructure, project requirements, and budget. Here’s a comprehensive guide to help you make an informed decision:</p><h3>Conclusion</h3><p>The landscape of DevOps tools is vast and ever-evolving. By understanding the strengths and weaknesses of various solutions, you can build a DevOps toolkit that streamlines your software delivery process, optimizes efficiency, and empowers your team to deliver high-quality software faster. Consider factors like your team’s technical expertise, existing cloud infrastructure, and budget when making your choice. Remember, the most important aspect is selecting tools that cater to your specific project requirements and development workflow.</p><p><em>Originally published at </em><a href="https://niveussolutions.com/navigating-the-devops-toolset-azure-aws-gcp/"><em>https://niveussolutions.com</em></a><em> on July 5, 2024.</em></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=36ec50184262" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Optimizing Your Cloud SQL: Fine-Tuning, Sizing, and Long-Running Queries]]></title>
            <link>https://medium.com/niveus-solutions/optimizing-your-cloud-sql-fine-tuning-sizing-and-long-running-queries-a7b0e172fdf6?source=rss-67c92e804e41------2</link>
            <guid isPermaLink="false">https://medium.com/p/a7b0e172fdf6</guid>
            <category><![CDATA[mysql]]></category>
            <category><![CDATA[gcp]]></category>
            <category><![CDATA[data-modernization]]></category>
            <category><![CDATA[postgresql]]></category>
            <category><![CDATA[database]]></category>
            <dc:creator><![CDATA[Venkatesh R]]></dc:creator>
            <pubDate>Fri, 05 Jul 2024 06:20:49 GMT</pubDate>
            <atom:updated>2025-03-16T21:29:26.535Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*liyBT3ByWcZPpEByI9RlGA.jpeg" /><figcaption><a href="https://www.freepik.com/free-photo/system-backup-database-integration_18122697.htm#query=database&amp;position=13&amp;from_view=keyword&amp;track=sph&amp;uuid=26b78eff-3e6f-40c0-bb6d-b8dfaed4c46c">Image by rawpixel.com on Freepik</a></figcaption></figure><p>Cloud databases offer scalability and flexibility, but for optimal performance, you need to fine-tune your databases and size them appropriately.</p><p>This blog post will guide you through:</p><ul><li><strong>Fine-tuning databases:</strong> Learn how to adjust parameters for better performance in MySQL, PostgreSQL, and MSSQL.</li><li><strong>Cloud SQL sizing:</strong> Discover strategies for right-sizing your Cloud SQL instances based on your workload.</li><li><strong>Identifying long-running queries:</strong> Explore tools to find slow queries in each database system (Query Store).</li><li><strong>Read replica best practices:</strong> Understand the do’s and don’t’s of using read replicas with large databases and data lag.</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*7ULVE0dwvdCk-NvhuEulNQ.png" /></figure><p><strong>Fine-Tuning Your Database Engine</strong></p><p>Each database engine offers parameters you can adjust to optimize performance for your specific workload. Here’s a glimpse into what you can fine-tune:</p><ul><li><strong>Buffer and cache sizes:</strong> Allocate memory efficiently for frequently accessed data.</li><li><strong>Connection pool settings:</strong> Manage the number of connections to prevent bottlenecks.</li><li><strong>Query optimizer statistics:</strong> Ensure the optimizer uses accurate data for efficient query plans.</li></ul><p>Fine-tuning requires a good understanding of your database system and workload characteristics. Experimentation and monitoring are key!</p><p><strong>Cloud SQL Sizing: Right-fit Your Instance</strong></p><p>Cloud SQL offers various machine types and configurations. Choosing the right size depends on factors like:</p><ul><li><strong>Expected read/write throughput:</strong> How much data will be accessed and modified concurrently?</li><li><strong>Data storage requirements:</strong> How much data needs to be stored?</li><li><strong>Budgetary constraints:</strong> Consider the cost-performance trade-off of different configurations.</li></ul><p><strong>Identifying Long-Running Queries</strong></p><p>Slow queries can significantly impact performance. Here are some tools to identify them:</p><ul><li><strong>MySQL:</strong> Use the SHOW PROCESSLIST command or the built-in query performance insight features.</li><li><strong>PostgreSQL:</strong> Leverage the EXPLAIN command or the powerful query store feature.</li><li><strong>MSSQL:</strong> Utilize the <a href="https://learn.microsoft.com/en-us/sql/relational-databases/performance/monitoring-performance-by-using-the-query-store?view=sql-server-ver16">Query Store</a> functionality to analyze long-running queries.</li></ul><p>By identifying and optimizing slow queries, you can significantly improve database responsiveness.</p><p><strong>Read Replicas: Do’s and Don’ts with Large Databases</strong></p><p>Read replicas offer a read-only copy of your production database, ideal for scaling read traffic. However, with large databases and data lag, some considerations exist:</p><ul><li><strong>Do:</strong> Use read replicas for reporting and analytics to offload read pressure from your primary database.</li><li><strong>Don’t:</strong> Perform write operations on read replicas. They are for read-only purposes.</li><li><strong>Do:</strong> Be aware of data lag and its potential impact on the replica’s data freshness.</li><li><strong>Don’t:</strong> Rely solely on read replicas for critical operations that require real-time data.</li></ul><p>Data lag is inherent in read replicas. Choose an acceptable lag based on your use case.</p><p>By following these tips and using the available tools, you can fine-tune your Cloud SQL databases, size them appropriately, identify and optimize slow queries, and effectively leverage read replicas for optimal performance in your cloud environment.</p><p><strong>Useful References:</strong></p><p><a href="https://cloud.google.com/sql/docs/mysql/best-practices">MySQL Best Practices</a></p><p><a href="https://cloud.google.com/sql/docs/postgres/best-practices">PostgreSQL Best Practices</a></p><p><a href="https://medium.com/plumbersofdatascience/gcp-cloud-sql-best-practices-to-follow-in-2023-d26c92f7d8b1">MSSQL Best Practices</a></p><p><a href="https://learn.microsoft.com/en-us/sql/relational-databases/performance/monitoring-performance-by-using-the-query-store?view=sql-server-ver16">Query Store</a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=a7b0e172fdf6" width="1" height="1" alt=""><hr><p><a href="https://medium.com/niveus-solutions/optimizing-your-cloud-sql-fine-tuning-sizing-and-long-running-queries-a7b0e172fdf6">Optimizing Your Cloud SQL: Fine-Tuning, Sizing, and Long-Running Queries</a> was originally published in <a href="https://medium.com/niveus-solutions">Niveus Solutions Stories</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Container Orchestration Showdown: Azure Kubernetes Service (AKS) vs. Azure Batch]]></title>
            <link>https://tatankavenkat.medium.com/container-orchestration-showdown-azure-kubernetes-service-aks-vs-azure-batch-383f4bb1ab83?source=rss-67c92e804e41------2</link>
            <guid isPermaLink="false">https://medium.com/p/383f4bb1ab83</guid>
            <category><![CDATA[cloud-computing]]></category>
            <category><![CDATA[azure-kubernetes-service]]></category>
            <category><![CDATA[kubernetes]]></category>
            <category><![CDATA[container-orchestration]]></category>
            <category><![CDATA[azure]]></category>
            <dc:creator><![CDATA[Venkatesh R]]></dc:creator>
            <pubDate>Sat, 29 Jun 2024 02:53:54 GMT</pubDate>
            <atom:updated>2024-06-29T02:54:53.671Z</atom:updated>
            <content:encoded><![CDATA[<p>In the realm of cloud-based container orchestration, two prominent Azure services emerge: Azure Kubernetes Service (AKS) and Azure Batch. While both deal with containers, they cater to distinct use cases. This blog post dives into their functionalities, guiding you towards the ideal choice for your project.</p><p><strong>Understanding Azure Kubernetes Service (AKS)</strong></p><p>AKS is a managed Kubernetes service. Kubernetes, an open-source system, automates container deployment, scaling, and management. AKS simplifies Kubernetes deployment and management by handling the underlying infrastructure, allowing you to focus on your containerized applications.</p><p><strong>When to Choose AKS</strong></p><ul><li><strong>Microservices Architecture:</strong> AKS excels at orchestrating complex applications built on microservices, where independent containerized services collaborate to form a larger application.</li><li><strong>State Management:</strong> AKS is well-suited for applications requiring stateful deployments, where containers maintain data across restarts. Databases and message queues are prime examples.</li><li><strong>Horizontal Scaling:</strong> AKS seamlessly scales your containerized applications horizontally based on demand, ensuring optimal performance during traffic spikes.</li><li><strong>Advanced Orchestration:</strong> If your project demands intricate container orchestration with features like service discovery, load balancing, and health checks, AKS is the way to go.</li></ul><p><strong>Azure Batch: For Large-Scale Parallel Processing</strong></p><p>Azure Batch caters to large-scale, parallel, and high-performance computing (HPC) workloads. It provisions virtual machines on-demand, executes batch jobs across these VMs, and then deallocates them when finished, optimizing resource utilization.</p><p><strong>When to Choose Azure Batch</strong></p><ul><li><strong>Independent Tasks:</strong> Azure Batch shines when your workload consists of independent tasks that can run in parallel on separate VMs. This is ideal for scientific simulations, data processing pipelines, and rendering jobs.</li><li><strong>Short-Lived Tasks:</strong> Batch excels at handling short-lived tasks that don’t require persistent state across container restarts.</li><li><strong>Cost Optimization:</strong> For workloads with sporadic use cases, Azure Batch’s on-demand VM provisioning can be more cost-effective than continuously running AKS clusters.</li></ul><p><strong>AKS vs. Vanilla Kubernetes: A Tabular Comparison</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/714/1*e0V2y_BWjQj9QUurrxIqQQ.png" /></figure><p><strong>Choosing the Right Tool</strong></p><p>The choice between AKS and Azure Batch boils down to your project’s requirements. If you need intricate container orchestration for stateful microservices applications, AKS is the champion. For large-scale, parallel, and short-lived tasks with cost optimization in mind, Azure Batch reigns supreme.</p><p><strong>In Conclusion</strong></p><p>Both AKS and Azure Batch are valuable tools within the Azure ecosystem. Understanding their strengths and ideal use cases empowers you to make informed decisions for your containerized workloads. By selecting the right tool, you can optimize your development process, ensure cost-effectiveness, and achieve superior application performance.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=383f4bb1ab83" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Implementing GCP Organization Policies with Terraform]]></title>
            <link>https://tatankavenkat.medium.com/implementing-gcp-organization-policies-with-terraform-72b0ebf07039?source=rss-67c92e804e41------2</link>
            <guid isPermaLink="false">https://medium.com/p/72b0ebf07039</guid>
            <category><![CDATA[cloud-computing]]></category>
            <category><![CDATA[cloud-security]]></category>
            <category><![CDATA[cloud]]></category>
            <category><![CDATA[policy]]></category>
            <category><![CDATA[gcp]]></category>
            <dc:creator><![CDATA[Venkatesh R]]></dc:creator>
            <pubDate>Thu, 13 Jun 2024 04:24:08 GMT</pubDate>
            <atom:updated>2024-10-04T06:02:02.793Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/500/0*6FTXi-zcuwWSwASt.jpg" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1000/0*_Jvah0fnx0nYIPBP.jpg" /></figure><p>While Google Cloud Platform (GCP) empowers you with a vast arsenal of services to build and deploy innovative solutions, managing this powerhouse platform effectively is crucial. If left unchecked, it can lead to security vulnerabilities, compliance issues, and potential cost overruns. To ensure a smooth and secure journey, you need GCP organization policies — a set of essential tools that act as the guiding principles for your cloud environment. Think of them as the guardrails that keep your cloud kingdom secure, efficient, and compliant. In this blog we’ll delve into the critical aspects of managing Google Cloud Platform (GCP) effectively by harnessing the power of GCP organization policies and Terraform</p><p>This blog explores the power of GCP organization policies and how Terraform can streamline their creation and enforcement. Terraform enables you to define and manage your cloud infrastructure declaratively, allowing for automated provisioning, configuration, and management of resources across various cloud providers, including Google Cloud Platform (GCP). By leveraging these tools together, you can establish a well-governed cloud environment that fosters security, agility, and cost-effectiveness.</p><h3>What are GCP Organization Policies?</h3><p>Imagine your cloud environment as a vibrant apartment complex. Just like a well-run apartment building fosters a sense of community and a safe environment for residents, managing cloud resources requires well-defined <strong>GCP organization policies</strong>. These policies act as the building’s instruction manual, dictating how tenants (your cloud resources) interact within the shared spaces (cloud services).</p><p>Here’s how <strong>GCP organization policy best practices</strong> translate to everyday cloud environment management:</p><ul><li><strong>Limiting Who Has a Key:</strong> Restrict public IP access by limiting VMs with external IPs. This is similar to requiring tenants to have a key fob to access the building’s main entrance. Only authorized residents (VMs) can freely come and go.</li><li><strong>Keeping the Bike Room Locked:</strong> Enforce Public Access Prevention (PAP) to safeguard your Cloud Storage data. Think of the bike room being locked, requiring authorization to access stored items (data). No one can just grab a bike (data) without permission.</li><li><strong>Specifying Who Can Visit Tenant Apartments:</strong> Define allowed external IPs for VMs. This is like clarifying which guests a tenant can allow into their apartment. Only approved visitors (external IPs) can connect to a specific VM (apartment).</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/720/0*5vyOnWYiw4TJoGDE" /></figure><p>Reference from gcp documentation org policy overview</p><h3>Terraform in Action: Crafting Powerful Organization Policies for GCP</h3><p><a href="https://niveussolutions.com/terraform-enterprise-minimize-cloud-waste-spend/">Terraform</a> acts as the command center for your cloud infrastructure, allowing you to define and manage everything in code, including not only the resources themselves (virtual machines, storage buckets, etc.) but also the policies that govern their behavior. Think of organization policies as the rulebook for your cloud environment, dictating how resources can interact with shared services Here’s where Terraform GCP organization policies come into play.</p><p>Terraform uses Google Cloud Platform’s Organization Policy API to interact with GCP and enforce these policies. This creates a powerful one-two punch: Terraform lets you define your infrastructure as code, ensuring consistency and repeatability, while also codifying the policies that govern its operation. This translates to a secure, efficient, and well-managed cloud environment, just like a clear set of rules keeps an apartment building running smoothly.</p><h3>Do’s &amp; Don’ts in Organizational Policy</h3><p>Here’s a simplified workflow on implementing Terraform for the organizational policy:</p><ul><li><strong>Define the Policy:</strong> Write the organization policy configuration in a Terraform file.</li><li><strong>Plan and Apply:</strong> Run terraform plan to preview the changes, then terraform apply to create the policy in GCP.</li><li><strong>Relaxing the Rules:</strong> When is it Okay?</li></ul><p>While organization policies are crucial for security and efficiency, there may be situations where they can feel restrictive, especially during development. As such, it’s important to loosen the reins with caution. Here are some reasons to consider a temporary relaxation of policies:</p><ul><li><strong>Short-term Access for Testing:</strong> Imagine needing to test a new feature that requires a temporary connection to an external service. A policy might normally block this external access. In this case, a temporary exemption can be granted for the specific test environment, allowing you to conduct thorough testing within a defined timeframe and limited scope. This ensures you get the necessary testing done while minimizing potential security risks.</li><li><strong>Legacy Workloads:</strong> Sometimes, older systems might not be compatible with the stricter security protocols enforced by new policies. For these situations, creating exemptions can be a viable option. However, it’s crucial to develop a clear migration plan. This plan should outline a timeline and steps for upgrading the legacy system to comply with the organization policies. This ensures long-term security while giving you time to address compatibility issues with older systems.</li></ul><p>Remember, loosening restrictions should always be a thoughtful decision with well-defined parameters.</p><h3>Before You Terraform: A Process for Enterprises</h3><p>Implementing Terraform organization policies is a critical step towards a secure and efficient cloud environment, but it’s important to take a measured approach. Here’s a recommended process for enterprises to ensure a smooth and successful rollout:</p><ol><li><strong>Identify Needs:</strong> This initial phase involves a thorough assessment of your organization’s security posture and compliance requirements. What regulations does your industry adhere to? What level of data protection is necessary for your workloads? By clearly defining these needs, you can establish a solid foundation for crafting effective organization policies.</li><li><strong>Define Policies:</strong> With your needs identified, it’s time to translate them into actionable policies. Leverage Terraform to codify these policies as code, ensuring consistency and repeatability. This could involve restricting public IP access for VMs, enforcing Public Access Prevention (PAP) for Cloud Storage buckets, or defining allowed external IPs for specific resources. Remember, these policies act as the guardrails that govern how your cloud resources interact with GCP services.</li><li><strong>Testing:</strong> Before unleashing the policies on your entire cloud environment, it’s wise to conduct a thorough testing phase. Create a secure, isolated environment that mimics your production setup. Here, you can introduce sample workloads and test how they interact with the newly defined policies. This helps identify any potential conflicts or unintended consequences before they impact critical systems.</li><li><strong>Phased Rollout:</strong> Even with testing, a full-scale deployment can be risky. Therefore, a phased rollout is highly recommended. Start by applying the policies to non-critical environments, allowing you to observe their impact on day-to-day operations. This staged approach minimizes disruption and allows for adjustments if needed. As confidence builds, you can gradually extend the policies to encompass more critical workloads.</li><li><strong>Monitoring and Review:</strong> Don’t just set it and forget it! Continuously monitor your cloud environment to ensure ongoing compliance with the organization policies. Utilize GCP’s monitoring tools to track policy adherence and identify any potential violations. Furthermore, schedule regular reviews of your organization policies. The cloud landscape is constantly evolving, so revisiting and adapting your policies is essential to maintain optimal security and efficiency.</li></ol><h3>Use cases &amp; Benefits</h3><p>Here’s a real-life example incorporating restricting a specific Google Compute Engine (GCE) SKU and enforcing a specific region for creating VMs:</p><p><strong>Real-Life Example: Controlling Compute Costs and Regional Deployments</strong></p><p>Imagine your company wants to optimize its cloud spending on Google Compute Engine (GCE) VMs. You can implement GCP organization policies to achieve this. Here’s how:</p><ul><li><strong>Policy 1:</strong> Restricting GCE SKU: Let’s say you want to limit VMs to a specific tier of machine types, like standard machines (e.g., n1-standard-1) for basic workloads. An organization policy can be created to disallow creating VMs with machine types from higher tiers (e.g., n2-highmem-8) unless explicitly justified. This ensures cost-effectiveness by preventing users from accidentally or unknowingly provisioning expensive VMs.</li><li><strong>Policy 2:</strong> Enforcing Regional Deployment: For regulatory purposes, your company might require all data to reside in a specific geographic location. An organization policy can be enforced to restrict VM creation to a specific GCP region (e.g., Asia-South1) that complies with your regulations. This ensures data sovereignty and adherence to compliance requirements.</li></ul><p>By implementing these policies, you achieve:</p><ul><li><strong>Cost Optimization:</strong> By limiting VM SKUs, you restrict resource allocation, potentially leading to lower cloud bills.</li><li><strong>Regulatory Compliance:</strong> Enforcing specific regions for VM deployment ensures your data stays within the required geographic boundaries.</li></ul><p>Terraform empowers you to automate the creation and enforcement of organization policies within your GCP environment. This eliminates the need for manual configuration, a process prone to errors and inconsistencies. Instead, you can define your policies directly within Terraform configuration files. This approach treats policies as code, ensuring version control, simplified collaboration, and consistent application across your deployments. Within these configurations, you can specify granular controls, such as permitted SKUs (machine types) for VMs or valid regions for deployments. Terraform, then, seamlessly interacts with GCP’s Organization Policy API to translate your defined policies into actionable configurations within GCP. This automation not only saves you valuable time and resources, but also guarantees consistent policy enforcement across your entire organization.</p><h3>Conclusion: Building a Secure and Efficient Cloud with GCP Organization Policies and Terraform</h3><p>By effectively leveraging GCP organization policies and Terraform, you can establish a well-governed cloud environment that fosters security, agility, and cost-effectiveness. GCP organization policies act as the guardrails, defining the rules for how your cloud resources interact with GCP services. Terraform empowers you to codify these policies as code, ensuring consistency, repeatability, and automated enforcement across your entire GCP organization.</p><p>The recommended process outlined in this guide provides a roadmap for implementing organization policies thoughtfully. Remember, striking a balance between security and development agility is key. By following these steps and best practices, you can empower your organization to thrive in the cloud while maintaining a secure and compliant environment.</p><p>So, take control of your cloud house! Keep it orderly and efficient with the power of GCP organization policies and Terraform.</p><p><em>Originally published at </em><a href="https://niveussolutions.com/gcp-organization-policies-with-terraform/"><em>https://niveussolutions.com</em></a><em> on June 13, 2024.</em></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=72b0ebf07039" width="1" height="1" alt="">]]></content:encoded>
        </item>
    </channel>
</rss>