<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by SADA, An Insight Company on Medium]]></title>
        <description><![CDATA[Stories by SADA, An Insight Company on Medium]]></description>
        <link>https://medium.com/@sada-engineering?source=rss-6bc96cd41286------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/1*fCfPB6aeEP2gkRnD5IdgNA.png</url>
            <title>Stories by SADA, An Insight Company on Medium</title>
            <link>https://medium.com/@sada-engineering?source=rss-6bc96cd41286------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Fri, 20 Nov 2026 09:33:28 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@sada-engineering/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[Stop being the human API: A three-stage productivity flywheel]]></title>
            <link>https://engineering.sada.com/stop-being-the-human-api-a-three-stage-productivity-flywheel-a1c99262caae?source=rss-6bc96cd41286------2</link>
            <guid isPermaLink="false">https://medium.com/p/a1c99262caae</guid>
            <category><![CDATA[ai-architecture]]></category>
            <category><![CDATA[command-line-interface]]></category>
            <category><![CDATA[generative-ai-tools]]></category>
            <category><![CDATA[ai]]></category>
            <category><![CDATA[devops]]></category>
            <dc:creator><![CDATA[SADA, An Insight Company]]></dc:creator>
            <pubDate>Tue, 09 Dec 2025 19:48:56 GMT</pubDate>
            <atom:updated>2026-01-22T00:17:29.737Z</atom:updated>
            <content:encoded><![CDATA[<p><strong>Author: </strong>Prabha Arivalagan, Sr. AI Architect, SADA, An Insight Company</p><p>A guide for AI and software engineers, DevOps practitioners, and backend developers who work across multiple systems and want to eliminate context-switching overhead.</p><p><strong>TL;DR</strong></p><p>Command Line Interface (CLI) AI assistants eliminate context-switching by orchestrating Terminal, GitHub, GCloud, and your notes from one place. Configure universal access once, then let AI coordinate everything while your knowledge compounds. The result: get your work done in hours, not days.</p><p><strong>Data privacy note:</strong> CLI assistants send your code and prompts to cloud providers for processing. Choose the right authentication method and configure privacy settings to ensure your internal documents aren’t used for model training. See the “Choosing your CLI assistant” section for guidance.</p><p><strong>About tool examples:</strong> This post uses Gemini CLI for all examples, but the productivity patterns apply to any CLI assistant (Claude Code, Copilot CLI, etc.). While file paths, configuration folder names, and syntax will differ slightly between tools, you can easily translate the concepts to your tool of choice.</p><h3>The 1-day task that now takes 1 hour</h3><p><strong>Tuesday morning standup:</strong> “Our customer sentiment classifier, which uses Gemini, just started acting erratically this morning. We’re seeing inconsistent results: The same product review is classified as ‘positive’ at 9 AM and ‘negative’ at noon. Quality Assurance (QA) is blocking the release.”</p><h3>The old way (what I used to do): Methodical but fragmented debugging across multiple tools</h3><p>After poking around for a while, the debugging path (in hindsight) looked something like:</p><ul><li>Write a test script to call the Application Programming Interface (API) 50 times with the same review, confirming inconsistent outputs.</li><li>Check Cloud Run logs to see actual model calls and responses.</li><li>Inspect the model configuration file to identify which model is being used.</li><li>Review Git history to pinpoint <em>when</em> and <em>why</em> the model configuration changed.</li><li>Check Vertex AI to confirm the fine-tuned model still exists.</li><li>Review fine-tuning job details to understand the model’s training history.</li><li>Search my notes to find context about why we fine-tuned in the first place.</li><li>Test both models to compare their behavior and consistency.</li><li>Examine the commit message to understand the developer’s intent behind the change.</li></ul><p>Each step required switching tools, losing context, and manually correlating information. The inefficiency isn’t in any one step — it’s in the exhausting process of connecting the dots across these disparate data sources.”</p><h3>The new way: AI-accelerated debugging</h3><p>I stayed entirely within my terminal, working with my CLI assistant. Through a series of prompts, the assistant orchestrated the entire diagnostic process:</p><ul><li><strong>Wrote and ran a test script</strong> to verify the current 60% consistency against the expected 98%.</li><li><strong>Checked Git history</strong> to discover that someone had switched from our fine-tuned model to the base Gemini flash model three days prior.</li><li><strong>Searched my notes</strong> to understand the original reason for fine-tuning (using 10k product-specific reviews for consistent sentiment).</li><li><strong>Confirmed</strong> that our fine-tuned model still existed in Vertex AI.</li></ul><p><strong>Total time: One hour. </strong>The problem was diagnosed, fixed, and deployed by 10 AM.</p><p><strong>The real win was speed. The enabler was delegation.</strong> I stopped being the human glue connecting all these systems. The assistant did the grunt work; I remained focused on making critical decisions.</p><pre>The ! prefix is key here, run any shell command, and the output becomes part of<br> the assistant&#39;s context. No switching windows, no copy-pasting.</pre><h3>The real problem: context-switching nightmare</h3><p>Developers spend the majority of their work time on activities beyond writing code. The rest of the time? We’re hopping between systems: Terminal → Browser → Notes → GitHub → GCloud Console → Back to Terminal. Each switch is a break in flow.</p><p>You become the human API, manually translating and cross-referencing information between systems, and acting as the search engine across dozens of files and notes.</p><p>Most of us have built solid organizational systems — from knowledge management tools to detailed READMEs. But even perfect organization cannot solve two core challenges:</p><h3>Challenge 1: Unpredictable knowledge connections</h3><p>You can’t predict what knowledge will be relevant to future problems. Today’s meeting notes, for instance, may become crucial for next quarter’s architecture. Project A’s caching solution might inform Project B’s rate-limiting strategy, which then requires connecting to new features in Redis that haven’t even been released yet. Retrieving that information and connecting the dots burns massive brain cycles, even when you know exactly where everything is filed.</p><h3>Challenge 2: Fragmented execution</h3><p>Finding the solution is only half the battle. Then you need to:</p><ul><li>Implement the code fix and test it.</li><li>Create a Git commit with context pulled from three different sources.</li><li>Raise a GitHub issue linking back to the original incident.</li><li>Update GCloud configurations.</li><li>Generate architecture diagrams for the team review.</li><li>Document the decisions for future reference.</li><li>Create a Pull Request (PR) with meaningful descriptions.</li></ul><p>Each action means switching tools, copying context, and reformatting information. By the time you’ve bounced between seven systems, you’ve lost 30 minutes just to tool-switching overhead.</p><h3>The hidden challenge: The socialization bottleneck</h3><p>Even after solving the problem, you need to communicate it. Architecture decisions need diagrams for stakeholder review. Solutions need documentation for team understanding. Fixes need clear PR descriptions for code review. Each communication requires translating technical details into different formats for different audiences. The feedback cycle stretches because creating these artifacts takes hours.</p><p><strong>What if you could orchestrate all of this from one place?</strong> Not just finding information, but connecting it, executing on it, and communicating it — <strong>all without leaving your terminal?</strong></p><h3>My solution: A three-stage productivity flywheel with CLI assistants</h3><p>Many developers mistakenly view CLI assistants as mere coding aids. In reality, they are command centers for modern development, capable of orchestrating information gathering, executing commands, generating documentation, and eliminating the constant tool-switching that fragments our focus.</p><p>After experimenting with various AI assistant approaches, I’ve developed a workflow using AI-powered CLI assistants (primarily Gemini CLI and Claude Code) that has fundamentally transformed how I work. This isn’t about comparing tools or arguing which is “best,”; it’s about sharing a practical workflow that made me more productive in my day-to-day tasks. (Note: You’ll still use your IDE for visual editing while the CLI orchestrates tasks. See ‘Physical setup’ for details.)</p><p><strong>The key insight: context switching is the silent productivity killer.</strong> CLI assistants eliminate it by orchestrating all your systems from one place, handling the tedious work of searching, connecting, and executing while you focus on judgment and decisions. The work of bridging information across tools was always necessary and always manual. With CLI assistants, what used to require hours of manual tool-switching now happens in minutes of focused execution.</p><p>The power of this approach comes from building momentum, like a flywheel, where each rotation makes the next one easier and faster:</p><ol><li><strong>Stage 1: Universal Access</strong> — Configuration and folder structure that makes all your knowledge accessible from anywhere.</li><li><strong>Stage 2: Intelligent Orchestration</strong> — Coordinating systems and generating insights from your knowledge base.</li><li><strong>Stage 3: Compounding Knowledge</strong> — Capture and organize learnings that compound over time.</li></ol><figure><img alt="" src="https://cdn-images-1.medium.com/max/512/1*vMv1pcgT_iGQamj4W79Brw.jpeg" /><figcaption><em>Generated using Gemini 3 Pro image mode</em></figcaption></figure><h3>The three-stage productivity flywheel</h3><p>The power of this framework isn’t in any single stage; it’s in how each stage feeds into the next. Stage 1 (Universal Access) enables Stage 2 (Orchestration). Stage 2 generates insights that Stage 3 captures. And Stage 3 enriches what’s available for the next round of Stage 2. It’s a true productivity flywheel.</p><p>Here’s how it compounds: During orchestration, you generate insights. These get captured into your knowledge base. Next time you orchestrate, you have more to draw from. <strong>Yesterday’s discoveries become today’s accelerators.</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/512/1*CpDG-vTJe1uUsRHHotDhUA.png" /></figure><h4>Stage 1: Universal Access — getting the flywheel started</h4><p>Before you can automate or synthesize, you need access to your information. CLI assistants access what you explicitly provide: your local notes, meeting transcripts, knowledge base, project files, and Git repositories, not enterprise systems like Confluence or Slack (unless you set up additional integrations).</p><p>This is where CLI assistants fundamentally differ from traditional automation tools: they can access your captured knowledge across all projects through a configuration hierarchy.</p><h3>How the configuration hierarchy works</h3><p>CLI assistants respect a configuration hierarchy where settings cascade from general to specific, with more specific configurations taking precedence:</p><pre>~/<br>├── notes/               # Your knowledge base<br>│ ├── daily/             # Daily logs and journals<br>│ └── technologies/      # Tech deep-dives and learnings<br>├── projects/<br>│ ├── project-a/<br>│ │ └── .gemini/         # Project-specific config<br>│ ├── project-b/<br>│ │ └── .gemini/<br>│ └── project-c/<br>│ └── .gemini/<br>└── .gemini/              # Global config (applies everywhere)<br>├── settings.json         # Access, models, preferences<br>└── GEMINI.md             # Global context and instructions</pre><p>This works through additive context merging: Your global GEMINI.md and project GEMINI.md files are concatenated together, giving the assistant access to both your universal knowledge and project-specific context. Meanwhile, project settings.json overrides global settings for access configurations. The result is a unified view of all your captured knowledge and project context.</p><h3>Cross-project knowledge access</h3><p>Your global configuration sets universal instructions and preferences that apply across all projects, including paths to your knowledge base</p><p>By “knowledge base,” I mean simple text-based notes — just markdown files in folders on your computer.</p><pre>~/notes/ # Start simple, organize later!<br><br>├── Daily-Notes/<br>│ ├── 2024/<br>│ │ ├── 10-October/<br>│ │ │ ├── 2024–10–15.md      # Fine-tuned model training and validation results<br>│ │ │ └── 2024–10–22.md      # Team architecture review<br>│ │ └── 11-November/<br>│ └── 2025/<br>├── Projects/<br>│ ├── sentiment-classifier/<br>│ │ ├── architecture.md<br>│ │ ├── model-selection.md     # Why we fine-tuned<br>│ │ ├── fine-tuning-history.md # Training data and results<br>│ │ └── troubleshooting.md<br>│ └── data-pipeline/<br>│ ├── patterns.md<br>│ └── performance-tuning.md<br>├── Technologies/<br>│ ├── GCP/<br>│ │ ├── cloud-logging.md<br>│ │ └── vertex-ai.md<br>│ ├── Databases/<br>│ │ ├── postgres-patterns.md<br>│ │ └── redis-debugging.md<br>│ └── APIs/<br>│ └── gemini-integration.md<br>└── Patterns/<br>├── Authentication/<br>│ └── oauth-flows.md<br>├── ML-Engineering/<br>│ ├── model-consistency.md<br>│ └── fine-tuned-model-management.md  # Best practices<br>└── Debugging/<br>└── ml-drift-patterns.md</pre><p><strong>What tool do you use for this?</strong> I use Obsidian (<a href="https://x.com/karpathy/status/1761467904737067456?lang=en">inspired by Andrej Karpathy’s post</a>), it’s essentially an Integrated Development Environment (IDE) for markdown files that makes viewing and organizing notes prettier with features like calendar view and graph connections. But here’s the key: you don’t need Obsidian to start! Your knowledge base is just plain .md files in folders. VSCode, Notepad, or any text editor works perfectly. I started with ~/notes/ and random .md(markdown) files. Obsidian just adds a nicer interface later if you want it, and it comes with useful plugins. The CLI assistant reads markdown natively either way, no special tools required, no vendor lock-in, just plain text that works forever.</p><p>Your ~/.gemini/GEMINI.md file provides instructions about your structure. What you include depends on your stage:</p><h3>OPTION 1: Just getting started (perfectly fine!)</h3><pre>~/.gemini/GEMINI.md<br><br>## Knowledge Base Access<br><br>My notes are located at: `/Users/username/notes/`<br><br>Search this folder for any relevant information from my past work.</pre><h3>OPTION 2: Evolved structure (what you’ll build over time)</h3><pre>~/.gemini/GEMINI.md<br><br>## Knowledge Base Access<br><br>My notes are located at: `/Users/username/notes/`<br><br>### Folder Navigation Instructions<br>- <strong>**Daily-Notes/**</strong>: Search here for work logs, debugging sessions, meeting <br>notes by date<br><br>- <strong>**Projects/**</strong>: Look for project-specific patterns and past implementations<br><br>- <strong>**Technologies/**</strong>: Reference for tool-specific knowledge and best practices<br><br>- <strong>**Patterns/**</strong>: Reusable solutions and architectural patterns<br><br>### When to Access Each Location<br><br>- For debugging similar issues: Search Daily-Notes/ and Patterns/Debugging/<br><br>- For ML model issues: Check Projects/*/model-selection.md and <br>Patterns/ML-Engineering/<br><br>- For understanding past decisions: Search Projects/*/architecture.md and <br>Daily-Notes/<br><br>- For implementation examples: Check Projects/ and Patterns/<br><br>- For tool-specific help: Reference Technologies/<br><br>- For recent context: Always check Daily-Notes/ from the last 30 days<br><br>- For architectural decisions: Search Projects/*/architecture.md files<br><br>### Search Priority<br>1. First check Daily-Notes from recent weeks for fresh context<br><br>2. Then search relevant Patterns/ for proven solutions<br><br>3. Finally check Projects/ for specific implementations</pre><p>Now, whether you’re in project-A/, project-B/, or client-xyz/, the CLI assistant always knows how to access your entire knowledge base. Every project automatically inherits this capability. The instructions and configuration travel with you.</p><p>This foundation is what enables everything else. Without universal access, you’d be limited to the current project’s context. With it, you have all your captured experiences and learnings at your fingertips.</p><p>Want to go further? CLI extensions let you add custom tools, integrate your favorite APIs, and create shortcuts for repetitive workflows. <a href="https://geminicli.com/docs/extensions/getting-started-extensions/">Learn more about CLI extensions</a>.</p><h4>Stage 2: Intelligent Orchestration — spinning the wheel</h4><p>With universal access established, CLI assistants become powerful orchestration engines. This isn’t about simple automation; it’s about connecting dots across your entire knowledge landscape to generate insights and solutions.</p><p>From manual system-hopping to intelligent orchestration</p><p>Speed comes from delegation. Make the CLI assistant do the grunt work while you focus on making the right decisions.</p><p><strong><em>The manual way: you are the human API</em></strong></p><p>Picture yourself debugging that model consistency issue. You open Terminal, run a grep command, and copy the error. Switch to the browser, paste it into Google, and scan the results. Open your notes, search for the error pattern, and find a related entry. Go back to the browser, check Vertex AI’s release notes. Search through meeting notes across different folders. Return to Terminal to try a fix.</p><p>Each system speaks a different language. You’re constantly translating, copying, pasting, and reformatting. Your brain is the only thing connecting these isolated islands of information.</p><p><strong><em>The orchestrated way: CLI as universal conductor</em></strong></p><p>Now watch the same task performed with a CLI assistant. <strong>One command orchestrates everything</strong>: “Find timeout errors in logs, check my notes for similar patterns, search the web for recent API changes, connect the dots.”</p><p>The assistant speaks to each system in its native language — grep for logs, markdown for notes, web APIs for search — and brings you a <strong>unified answer</strong>.</p><ul><li>No context switching.</li><li>No manual copying.</li><li>No translation overhead.</li></ul><p>The systems that were isolated islands are now an orchestra, and your CLI assistant is the conductor. You can watch this in action and interrupt as needed, giving a new direction to steer the investigation.</p><p><strong>The difference is clear:</strong> You spend less time on logistics (switching, searching, connecting) and more time on actual problem-solving. The tedious work of bridging systems disappears. You focus on what matters: validating solutions, making judgment calls, and improving the fix.</p><p><strong>Real-world orchestration in action</strong></p><p>Beyond debugging, this orchestration approach transforms all engineering tasks. Here’s how it applies to architecture design.</p><p><strong>Example: Data pipeline architecture</strong></p><p>Last month’s challenge: Design architecture for 1M events/day.</p><p><strong>Single prompt:</strong></p><pre>Review meeting notes in <br>~/notes/Projects/data-pipeline/meetings/2024–10-stakeholder-requirements.md<br><br>Search my knowledge base at ~/notes/Projects/data-pipeline/ for similar patterns<br><br>Check ~/notes/Technologies/Databases/kafka-patterns.md for past implementations<br><br>Research current Kafka best practices for high-throughput systems<br><br>Generate architecture document with:<br><br>- Complete technical design for 1M events/day<br><br>- Mermaid sequence diagram showing data flow<br><br>- Mermaid architecture diagram for team review<br><br>- Trade-off analysis table for stakeholder discussion<br><br>Format for easy sharing in tomorrow&#39;s architecture review meeting</pre><p><strong>Result in 3 hours (vs 3 days manually)</strong></p><p>The CLI assistant accelerates complex tasks, delivering the following in just three hours:</p><ul><li>Complete architecture with Mermaid diagrams</li><li>Past patterns adapted to current requirements</li><li>Latest 2025 Kafka optimizations from web research</li><li>Clearly defined trade-offs addressing stakeholder concerns</li></ul><p>The power is iteration. When a stakeholder mentions, “We discussed event sourcing last Tuesday,” I can immediately update with @meeting-notes/2024–10–22.md and fresh web research in minutes instead of hours.</p><p><strong>The diagram iteration story</strong></p><p>This illustrates a bigger shift in how I socialize technical designs with my team. In the past, complex architecture diagrams created painful feedback bottlenecks. This involved spending 2–3 hours in diagramming tools, sharing with the team, waiting for feedback, spending another hour updating, and repeating. The slow iteration often meant settling for a “good enough” solution rather than thoroughly exploring and validating alternatives.</p><p>Now, I generate diagrams instantly by leveraging my notes. I simply prompt: “Create a Mermaid sequence diagram based on @architecture-notes/event-flow.md and @meeting-notes/2024–10–15.md.” The CLI assistant pulls context from my documented decisions and discussions, creating accurate diagrams in seconds.</p><p>This speed changes everything. I can share a draft workflow or architectural diagram with the team immediately and incorporate feedback via quick chats instead of scheduling multiple follow-up meetings. We reach consensus fast because iteration is frictionless. Once the team agrees on the design, I recreate it in Lucidchart for polished documentation with better visual appeal.</p><p><strong>Common orchestration patterns</strong></p><p>Beyond architecture, this unified workflow transforms daily development tasks:</p><p><strong>PR creation with full context:</strong></p><pre>&quot;Create a PR from the last 3 commits, including context from<br><br>@project-notes/decisions/Architecture Decision Record (ADR)-002.md about why <br>we chose this approach&quot;</pre><p><strong>Result: </strong>A high-quality PR with a meaningful description that connects the code changes directly to the original architecture decisions.</p><p><strong>Debugging with historical context:</strong></p><pre>&quot;This error looks familiar. Search my notes for similar stack trace and their <br>solutions, then suggest a fix&quot;</pre><p><strong>Result:</strong> Instant access to past debugging sessions and proven solutions.</p><p><strong>Test generation from multiple sources:</strong></p><pre>&quot;Generate test cases based on requirements in @requirements.md and <br>@Daily-notes/2025–11–01/scalability-discussion-transcripts.md<br><br>and edge cases we discussed in my yesterday&#39;s notes&quot;</pre><p><strong>Result: </strong>Comprehensive tests that reflect both specs and team discussions.</p><h4>Stage 3: Compounding Knowledge — building unstoppable momentum</h4><p>With the flywheel spinning from Stages 1 and 2 (Universal Access and Intelligent Orchestration), something remarkable happens: your knowledge begins to compound. Every project, every debugging session, every decision makes you more effective at the next challenge.</p><p><strong>The bidirectional knowledge flow</strong></p><p>With your notes organized in markdown files (whether using Obsidian, VSCode, or any text editor), CLI assistants create a powerful bidirectional flow:</p><p><strong>Reading from your past</strong> (knowledge retrieval):</p><pre>&quot;Find all authentication patterns I&#39;ve implemented&quot;<br><br>&quot;What was the root cause when we had similar performance issues?&quot;<br><br>&quot;Show me past decisions about database migrations&quot;</pre><p><strong>Writing to your future</strong> (knowledge capture):</p><pre>&quot;Add today&#39;s debugging solution to my knowledge base&quot;<br><br>&quot;Create a note about this caching pattern for future reference&quot;<br><br>&quot;Update my notes with lessons learned from this incident&quot;</pre><p>Watch how knowledge compounds across projects:</p><p><strong>Project A (January): </strong>Implement Open Authorization (OAuth) with Proof Key for Code Exchange (PKCE) flow</p><ul><li><strong>Challenge: </strong>Complex token refresh logic</li><li><strong>Solution:</strong> Custom middleware pattern</li><li><strong>Capture:</strong> “Save this OAuth implementation pattern to my notes.”</li></ul><p><strong>Project B (March):</strong> New client needs authentication</p><ul><li><strong>Query: </strong>“How did we handle OAuth token refresh before?”</li><li><strong>Result: </strong>Instant access to the solution saved in my general notes</li><li><strong>Improvement:</strong> Adapt pattern with new learnings</li><li><strong>Capture: </strong>“Update OAuth note with rate limiting improvements.”</li></ul><p><strong>Project C (June):</strong> Authentication audit</p><ul><li><strong>Query: </strong>“Show all auth patterns with their trade-offs.”</li><li><strong>Result: </strong>Complete history of implementations</li><li><strong>Synthesis: </strong>Generate best practices document</li><li><strong>Capture:</strong> “Create comprehensive auth playbook.”</li></ul><p>Each project doesn’t just benefit from past knowledge; it enriches the knowledge base for future projects. You’re not just remembering solutions; you’re building a compounding system of expertise.</p><h4>Daily logs vs. permanent knowledge</h4><p>The system distinguishes between two types of knowledge:</p><p><strong>Daily notes </strong>(your professional diary with calendar navigation): Think of these as your engineering logbook. I organize them by date (using Obsidian’s calendar plugin), making it trivial to jump to any day’s work. These aren’t ephemeral throwaway notes; they’re your searchable history:</p><ul><li>Today’s debugging sessions with full stack traces</li><li>Meeting decisions with context</li><li>Problem-solving attempts (failed and successful)</li><li>Work-in-progress thoughts and observations</li></ul><p><strong>Permanent knowledge</strong> (curated and refined patterns):</p><ul><li>Proven architectural patterns</li><li>Technical deep dives and investigations</li><li>Post-mortems with lessons learned</li><li>Domain expertise and best practices</li></ul><p>Knowledge capture is just as frictionless as retrieval. Mid-debugging, you can simply say:</p><ul><li>“Save this stack trace and solution to today’s daily note.”</li><li>“This pattern is gold, create a permanent note on connection pooling strategies.”</li></ul><p>The CLI assistant handles the file creation, formatting, and organization. You stay in flow. To maintain a high-quality knowledge base, you can ask the AI assistant to summarize information you’ve read. Once you are satisfied, you can then ask it to save the refined content in your daily notes or long-term knowledge base.</p><p>The CLI assistant also helps convert valuable daily discoveries into permanent knowledge. For example: “This debugging approach worked well. Extract it from today’s notes and create a permanent pattern document in <strong>@data/patterns/data-pipeline</strong> folder.”</p><p>The result? After six months, you don’t just have six months more experience locked in fading memory; you have six months of AI-searchable notes and documented problem-solving that makes every future task easier. Your productivity doesn’t just improve linearly; it compounds.</p><h3>Putting it all together: sample workflow</h3><p>Watch how the three stages of the flywheel work together in a real debugging session:</p><p><strong>The physical setup:</strong></p><p>Let me clarify the actual development environment — this isn’t a terminal-only setup. You’ll have your VSCode open for viewing and editing code, reviewing documents, examining Mermaid diagrams, and navigating your project structure visually. Your terminal runs alongside where you interact with the CLI assistant to orchestrate actions, search knowledge, and approve changes.</p><p>The CLI assistant bridges both: it understands your VSCode context (current files, project structure) while you maintain visual control in the IDE and approval control in the terminal. You get the visual richness of an IDE for complex work (reviewing diffs, understanding architecture diagrams, and editing code), combined with the orchestration power of CLI assistants for connecting information across systems.</p><p><strong>Starting the investigation</strong></p><p>Open terminal in your project and prompt: “Summarize recent sentiment classifier inconsistency from project-notes and check my notes for fine-tuned model troubleshooting patterns.”</p><p>⇒ Assistant instantly bridges project context with historical knowledge.</p><p><strong>Finding the root cause</strong></p><p>Prompt: “Search my Daily-Notes for model selection decisions, cross-reference with Projects/sentiment-classifier/fine-tuning-history.md, check git history for model configuration changes.”</p><p>⇒ One prompt orchestrates local search, git analysis, and historical context.</p><p><strong>Implementing the fix</strong></p><p>Prompt: “Find the fine-tuned model endpoint in Projects/sentiment-classifier/ and update the config.”</p><p>⇒ Endpoint found, model reverted to correct endpoint.</p><p><strong>Closing the loop</strong></p><p>Prompt: “Generate a PR description for this fix and add an entry to Projects/sentiment-classifier/issues.md with today’s date.”</p><p>⇒ PR submitted, issue logged with fix date, incident closed.</p><p><strong>Notice the pattern: </strong>Every prompt leverages both project-specific context AND accumulated knowledge. The assistant maintains boundaries (client details stay in project folders) while maximizing knowledge reuse.</p><h3>Making it part of your rhythm</h3><p>The key to adopting this system is starting small and building habits:</p><ul><li><strong>Before starting any task</strong>: “What similar problems have I solved? Check my notes.”</li><li><strong>When stuck</strong>: “Search my Daily-Notes and the web for this error pattern.”</li><li><strong>After solving something tricky</strong>: “Save this solution to my notes for future me.”</li><li><strong>During reviews</strong>: “Generate a diagram to explain this architecture.”</li><li><strong>For monotonous tasks</strong>: “Create a commit message from these changes” or “Generate test cases based on this implementation.”</li></ul><p>Start with one or two prompts that save you the most time. As you see the value, you’ll naturally expand your usage. The goal isn’t to hand over control; it’s to eliminate the tedious work of searching, switching, and formatting so you can focus on solving problems and making high-value decisions.</p><h3>Critical success factors</h3><p>Powerful tools require disciplined use. I’ve learned some critical lessons about maintaining control while maximizing benefit.</p><h3>Maintain oversight and control</h3><p>Here’s the truth about CLI assistants: They’re incredibly smart but prone to making silly mistakes. They might confidently suggest consolidating two “duplicate” validation functions that look identical but serve different purposes (e.g., one for strict user input, one for lenient API responses). Merging them would break production. They may also recommend complex abstractions when a simple function would work better.</p><p><strong>You must manually review and approve every suggestion</strong>. Don’t let the AI assistant drive your code development; it will easily overengineer and overcomplicate solutions. You drive the agent. Keep it on a tight leash. Don’t let it wander on its own.</p><ul><li>Never auto-approve code changes or documentation updates — review everything.</li><li>Push back on overengineering. Use prompts like: “Give me the simplest solution that works.”</li><li>You make the decisions; the assistant does the legwork.</li></ul><h3>Getting started today</h3><p>Ready to transform your workflow? Here’s what actually makes the difference:</p><ul><li>How well you organize your knowledge base (even a simple ~/notes/ folder beats nothing).</li><li>How precise your instructions are in your configuration file (e.g., GEMINI.md)</li><li>How disciplined you are about capturing learnings back into your notes.</li><li>How specific your prompts are when asking the assistant for help.</li></ul><p><strong>Starting from zero?</strong> Here’s your minimal viable setup (MVS):</p><ol><li>Create ~/notes/ folder</li><li>Add instructions in GEMINI.md pointing to this new folder.</li><li>Save your first debugging solution as a .md file.</li><li>See the benefit: Observe the time savings when you face your second similar problem.</li></ol><h3>Choosing your CLI assistant</h3><p>I’m going to save you from a pointless debate: It doesn’t matter which CLI assistant you pick. I use both Gemini CLI and Claude Code depending on which computer I’m on, and they both transform my workflow equally well.</p><p>It really comes down to setup. If your Gemini CLI is configured correctly and you have an organized knowledge base, you’re going to get way more done than with a poorly configured Claude Code, and the opposite is just as true.</p><ul><li>Pick whichever tool you can get running the fastest.</li><li>Spend your energy on configuration and organization, not on comparison shopping.</li><li>The “best” CLI assistant is the one you actually use, configured intelligently for your workflow.</li></ul><p><strong>Important: Configure privacy settings properly</strong></p><p>Make sure you configure enterprise AI privacy settings that respect the sensitive nature of your data. Check your CLI assistant’s documentation for authentication methods and privacy configurations that prevent your code and internal documents from being used for model training.</p><h3>When this approach doesn’t work</h3><p>Let’s be honest about when CLI assistants fail:</p><p><strong>When you become a lazy prompter</strong></p><p>Don’t be lazy. “Fix this bug” is not a prompt; it’s wishful thinking. These tools reward precision. Spent a few minutes writing and reviewing prompts. It is worth it. For example, “Debug the timeout error in payment-service.js line 234, check if it’s related to the connection pool settings we discussed yesterday.” The quality of output directly correlates with the effort you put into the prompt. Garbage in, garbage out.</p><p><strong>When you forget they’re sycophants</strong></p><p>AI assistants are trained to be cheerleaders, not critics. Every user’s code is “elegant.” Every approach is “insightful.” Yours isn’t special, they say this to everyone.</p><p>Don’t fall for it. Be skeptical. One trick that works: use critical prompts. Instead of “Does this look good?” ask “What’s wrong with this approach?” Instead of “Is this optimal?” ask “What edge cases did you miss?” You’ll get far more useful feedback when you ask them to find problems rather than validate solutions.</p><p>Challenge everything. When your assistant says, “This solution is optimal,” ask “What could go wrong?” When it generates code, ask “What edge cases did you miss?” Treat it like you’re mentoring an eager junior who wants to impress you but doesn’t fully grasp the consequences. Push back. Question assumptions. Never accept the first answer without scrutiny.</p><p><strong>When you stop feeding the system</strong></p><p>The compound effect only works if you keep adding to your knowledge base. That brilliant debugging session becomes worthless if you don’t capture the solution. That architectural insight is lost if you don’t document it. The system is only as good as the knowledge you feed it. Skip the capture step, and you’re just using an expensive search tool.</p><p>The key is recognizing that these tools augment, not replace, your expertise. They excel at connecting information but require your judgment to validate and improve outputs.</p><h3>Conclusion</h3><p>AI CLI assistants represent a fundamental shift in how we work as engineers. They don’t fix broken organizations or replace good tools; they eliminate the context-switching nightmare that consumes half our day.</p><p><strong>Here’s the honest value proposition: </strong>The work of connecting information — bridging past solutions with current problems, finding patterns across projects, and adapting old fixes to new challenges — was always part of engineering. CLI assistants make what used to require hours of manual searching and switching happen in minutes of orchestrated execution.</p><p>The true measure of these tools isn’t making information accessible (good organization already does that). It’s making complex connections practical. When today’s problem requires insights from three past projects, two meeting notes, current API docs, and production logs, CLI assistants orchestrate that discovery fast enough to be routine rather than prohibitive.</p><p>The future of engineering isn’t all about AI replacing developers; it’s about AI handling the tedious grunt work that was always necessary but previously consumed most of our mental energy. You stop being the human API between systems and start being the engineer who solves problems.</p><h3>Upgrade your workflow: Build your AI foundation</h3><p>You’ve learned that the true measure of productivity is not writing more code, but eliminating context-switching and mastering the art of orchestration and knowledge capture.</p><p>This requires a disciplined approach to configuration and setup — the foundational “ground work.”</p><p>Ready to build your three-stage productivity flywheel with enterprise-grade expertise?</p><p>SADA Ground School is an on-demand event designed to give your team the foundational knowledge needed to securely implement and scale next-generation AI solutions, including developer tools like Gemini CLI.</p><p>Stop being the human API. Optimize the setup that makes AI work for you.</p><p>→ <a href="https://events.sada.com/e/sada-ground-school-2025?_gl=1*1tawjqd*_gcl_au*MTk5ODQzODE0Mi4xNzU3NjIyMjkw*_ga*MTg0NDI3NzUxMC4xNzQxODE2MTgw*_ga_S0YP6DHS42*czE3NjUyMzE1MjEkbzMwMCRnMSR0MTc2NTIzMTY4NyRqNTkkbDAkaDA.">Watch the SADA Ground School on-demand now to build your AI foundation.</a></p><h3>About Prabha Arivalagan</h3><p>Prabha Arivalagan is a Senior AI Architect at SADA, An Insight company. With over 16 years of experience in AI and Data Analytics, he specializes in building AI agents, AI-based search, and RAG solutions. He focuses on driving tangible business value by prioritizing meaningful impact over industry hype. He has a keen interest in AI productivity tools and agents, leveraging them to help customers and himself work more efficiently. When he isn’t designing AI and cloud architectures, Prabha enjoys hiking, spending quiet time in the woods, and reading non-fiction.</p><h3>Resources</h3><p><strong>Getting started</strong></p><ul><li><a href="https://addyo.substack.com/p/gemini-cli-tips-and-tricks">Gemini CLI Tips and Tricks</a> — Comprehensive Gemini CLI guide</li><li><a href="https://obsidian.md/">Obsidian</a> — Local-first knowledge management</li></ul><p><strong>Further reading</strong></p><ul><li><a href="https://docs.anthropic.com/">Anthropic’s Claude documentation</a></li><li><a href="https://github.com/features/copilot/cli">Copilot CLI documentation</a></li></ul><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=a1c99262caae" width="1" height="1" alt=""><hr><p><a href="https://engineering.sada.com/stop-being-the-human-api-a-three-stage-productivity-flywheel-a1c99262caae">Stop being the human API: A three-stage productivity flywheel</a> was originally published in <a href="https://engineering.sada.com/">The SADA Engineering Blog</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[IngressNightmare: SADA’s perspective on the Ingress-nginx vulnerability]]></title>
            <link>https://engineering.sada.com/ingressnightmare-sadas-perspective-on-the-ingress-nginx-vulnerability-04c3307b5161?source=rss-6bc96cd41286------2</link>
            <guid isPermaLink="false">https://medium.com/p/04c3307b5161</guid>
            <category><![CDATA[kubernetes-security]]></category>
            <category><![CDATA[kubernetes]]></category>
            <category><![CDATA[nginx-ingress]]></category>
            <category><![CDATA[cloud-security]]></category>
            <category><![CDATA[gke]]></category>
            <dc:creator><![CDATA[SADA, An Insight Company]]></dc:creator>
            <pubDate>Thu, 27 Mar 2025 20:42:47 GMT</pubDate>
            <atom:updated>2025-04-24T14:47:40.982Z</atom:updated>
            <content:encoded><![CDATA[<p><strong>Author:</strong> Jake Luna, Staff Cloud Security Engineer, SADA</p><p><strong><em>A new vulnerability in ingress-nginx has been found, allowing for remote code execution (RCE) and potential complete Kubernetes cluster takeover. This attack vector has been assigned a CVSS v3.1 base score of 9.8 and requires immediate and thorough mitigation.</em></strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/800/1*ZHpfP3cqf_hSvBsjjsXW_A.jpeg" /></figure><h3>What you need to know</h3><p>Ingress controllers are used by Kubernetes to perform reverse proxy or load balancing. The Ingress concept lets you map traffic to different backends based on rules you define via the Kubernetes API. Ingress may provide load balancing, SSL termination, and name-based virtual hosting.</p><p>Please reference <a href="https://kubernetes.io/blog/2025/03/24/ingress-nginx-cve-2025-1974/">Ingress-nginx CVE-2025–1974: What You Need to Know | Kubernetes</a> for the official Kubernetes communications on the vulnerability.</p><h3>1. Immediate actions and assessment for mitigation:</h3><ul><li><strong>Identify affected clusters:</strong> The very first step is to identify all Kubernetes clusters within your infrastructure that are running Ingress-NGINX by running this kubectl command within your clusters:</li></ul><pre>kubectl get pods - all-namespaces - selector app.kubernetes.io/name=ingress-nginx</pre><p>This inventory should include production, staging, development, and any other environments. Determine the specific versions of ingress-nginx in use across these clusters, as the vulnerability affects certain version ranges.</p><ul><li><strong>Understand the exposure:</strong> Based on the network accessibility of your ingress-nginx admission controllers, assess the immediate exposure of each identified cluster. If the admission controllers are reachable from outside the cluster network or from untrusted internal networks without any form of authentication (as indicated in the vulnerability description), <strong>THE RISK IS EXTREMELY HIGH</strong>.</li><li><strong>Consider immediate network restrictions (temporary mitigation):</strong> As a temporary measure to reduce the attack surface while a proper fix is implemented, consider implementing network policies within your Kubernetes clusters or firewall rules at the network level to restrict access to the ingress-nginx admission controller pods. Ideally, access should be limited to only those internal components within the control plane that legitimately need to interact with the admission controller. However, remember that this is a compensating control and does not address the root cause of the unauthenticated access.</li><li><strong>Review audit logs:</strong> Immediately examine the audit logs of your Kubernetes control plane and the ingress-nginx controller pods for any suspicious activity or indicators of compromise related to the admission controller. Look for unexpected requests to the admission controller endpoints or unusual behavior within the NGINX controller pods.</li></ul><h3>2. Implement patches/upgrades:</h3><ul><li><strong>Apply the official fix:</strong> The most critical step is to apply the official patch or upgrade to a version of ingress-nginx that resolves this vulnerability. This vulnerability is fixed in ingress-nginx Controller <strong>version 1.12.1 and 1.11.5</strong>.</li><li><strong>Automate patch management:</strong> If you have automated patch management systems in place for your Kubernetes components and container images, ensure that the fix is prioritized and rolled out through these systems rapidly. Automation helps reduce the window of vulnerability and ensures consistent application of the patch across your environment.</li><li><strong>Thoroughly test the patch in non-production environments:</strong> Before deploying the patch or upgrade to production clusters, rigorously test it in non-production (staging, testing, or development) environments that closely mirror your production setup. This testing should include functional testing of your ingress configurations and security testing to verify that the vulnerability has been effectively remediated without introducing new issues.</li><li><strong>Follow secure deployment practices:</strong> When deploying the patched version, adhere to secure deployment practices, such as using automated deployment pipelines (CI/CD) and infrastructure as code (IaC) to ensure consistency and reduce manual errors.</li></ul><h3><em>In context: Applicability to GKE/Autopilot</em></h3><p>It’s important to understand how this applies to Google Kubernetes Engine (GKE). While GKE provides a robust and managed Kubernetes environment, using GKE or even GKE Autopilot does not automatically protect you from vulnerabilities like this ingress-nginx issue.</p><ul><li><strong>GKE Standard:</strong> In standard GKE clusters, you have more control over the underlying nodes and deployed applications, including the Ingress controllers. Therefore, you are responsible for patching and securing your ingress-nginx deployments.</li><li><strong>GKE Autopilot:</strong> Even in Autopilot, where Google manages the underlying infrastructure, you are still responsible for the security of your application deployments. If you deploy ingress-nginx, you must ensure it is patched and configured securely. Autopilot simplifies some operational aspects, but it doesn’t absolve you of your application security responsibilities.</li></ul><p>SADA’s best practice is to leverage Google Cloud Load Balancer in front of Kubernetes clusters. This is the approach we advocate for scalability, availability, and enhanced security. While you can configure a Kubernetes service of type LoadBalancer to provision an external load balancer, SADA views this as a higher-risk configuration, especially in light of this vulnerability, if not implemented with robust security measures.</p><h3>Post-resolution: Next steps for cluster hardening</h3><p>Following the implementation of initial mitigation measures for this vulnerability, organizations should consider additional strategic security enhancements to fortify the overall security posture of their GKE clusters.</p><h3>Bolster security of ingress-nginx configurations</h3><ul><li><strong>Apply the principle of least privilege:</strong> Even after patching, ensure that the ingress-nginx controller pods and the admission controller have the minimum necessary privileges to perform their functions. Review the RBAC (Role-Based Access Control) configurations within your Kubernetes clusters to verify that the ingress-nginx components do not have overly permissive permissions.</li><li><strong>Secure NGINX configuration:</strong> Regularly review and harden the NGINX configuration used by the Ingress controller based on security best practices. This includes limiting the use of potentially risky directives and ensuring proper security headers are configured.</li><li><strong>Utilize secure base images:</strong> Ensure that the container images used for your ingress-nginx controller are based on minimal and regularly updated base images that have been scanned for vulnerabilities. Consider using distroless images to reduce the attack surface.</li></ul><h3>Strengthen network security</h3><ul><li><strong>Implement Network Policies:</strong> Enforce network segmentation within your Kubernetes clusters using Network Policies to restrict network traffic between different pods and namespaces. These policies should limit the lateral movement of potential attackers in case the ingress-nginx controller is compromised. Specifically, restrict inbound traffic to the admission controller pods to only authorized components within the cluster.</li><li><strong>Firewall rules:</strong> Review and refine Google Cloud Firewall rules to limit external access to your Kubernetes clusters and the underlying infrastructure.</li></ul><h3>Enhance monitoring and detection</h3><ul><li><strong>Centralized logging with Cloud Logging:</strong> SADA strongly recommends ensuring that comprehensive audit logs from your GKE control plane, ingress-nginx controller pods, and admission controller are collected and stored in Google Cloud Logging. Cloud Logging provides a centralized and scalable platform for log management and analysis, which is crucial for effective security monitoring in GKE.</li><li><strong>Real-time monitoring and alerting with Cloud Logging:</strong> Implement real-time monitoring and alerting within Cloud Logging for suspicious activities, unexpected access attempts, or anomalous behavior related to the ingress-nginx admission controller and controller pods. SADA helps clients configure alerts in Cloud Logging to notify security teams immediately of any potential exploitation attempts, enabling proactive incident response.</li><li><strong>Anomaly detection considerations:</strong> SADA advises that you consider using anomaly detection capabilities, which can be integrated with Cloud Logging, to identify deviations from normal traffic patterns and behavior on the admission controller and within the NGINX pods.</li><li><strong>Integration with Security Command Center and SIEM:</strong> Forward relevant logs and alerts from Cloud Logging to Google Cloud’s Security Command Center for centralized security visibility and threat detection. SADA also helps clients integrate Cloud Logging with their existing Security Information and Event Management (SIEM) systems for centralized analysis, correlation with other security events, and comprehensive incident management.</li></ul><h3>Ensure robust vulnerability management</h3><ul><li><strong>Regular vulnerability scanning:</strong> SADA recommends implementing automated vulnerability scanning for your container images (using tools like Artifact Registry vulnerability scanning), GKE nodes, and other infrastructure components on a regular basis. Ensure that these scans include checks for known vulnerabilities like the one disclosed in Ingress-NGINX.</li><li><strong>Prioritize remediation with Security Command Center:</strong> Based on the severity of identified vulnerabilities and the potential impact, prioritize remediation efforts according to a defined vulnerability remediation schedule. SADA advises leveraging Google Cloud’s Security Command Center to gain centralized visibility into vulnerabilities and prioritize remediation based on risk scoring and potential impact within your GCP environment. Critical and high-severity vulnerabilities, especially those with known exploits, should be addressed with the highest urgency.</li></ul><h3>Prepare for incident response scenarios</h3><ul><li><strong>Update incident response plan:</strong> Review and update your organization’s incident response plan to include specific procedures for handling potential exploitation of Kubernetes vulnerabilities, including RCE on Ingress controllers and cluster takeover scenarios.</li><li><strong>Conduct incident response drills:</strong> Perform regular incident response drills and tabletop exercises to ensure that your security teams are prepared to effectively detect, contain, eradicate, recover from, and learn from security incidents involving Kubernetes.</li><li><strong>Establish clear communication channels:</strong> Ensure clear communication channels and escalation paths between your security operations, application development, and infrastructure teams for reporting and responding to security incidents.</li></ul><h3>Consider leveraging long-term security strategies</h3><ul><li><strong>Shift left on security:</strong> Integrate security practices earlier in the software development lifecycle (SDLC) and infrastructure as code (IaC) pipelines (“shift left”). This includes incorporating security scanning (SAST, DAST, IaC scanning) and policy enforcement into your CI/CD pipelines to identify and prevent vulnerabilities before they reach production.</li><li><strong>Secure software supply chain:</strong> Implement measures to secure your software supply chain, including verifying the integrity of base images, dependencies, and third-party components used in your ingress-nginx deployments. Use tools like Software Bill of Materials (SBOM) to track and manage these components.</li><li><strong>Policy as code:</strong> Implement policy as code (PaC) using tools like Open Policy Agent (OPA) or HashiCorp Sentinel to define and enforce security policies for your Kubernetes configurations, including ingress objects and admission controller settings.</li><li><strong>Zero trust architecture:</strong> Continue to move towards a zero trust security model where trust is never assumed, and all access is explicitly verified. This includes strong identity verification, micro-segmentation, and continuous monitoring.</li><li><strong>Regular security audits and penetration testing:</strong> Conduct regular security audits and penetration testing of your Kubernetes infrastructure, including the ingress-nginx components, by independent third parties to identify potential weaknesses and vulnerabilities.</li></ul><p>SADA offers a <a href="https://info.sada.com/cloud-security-confidence-assessment">comprehensive security assessment</a> to help, and we also offer 24/7 Incident Response Services if you’ve experienced a breach or observed suspicious activity in your environment. Call <strong>800–930–3475</strong> for assistance.</p><h3>Appendix</h3><h3>Item 1: Network security examples for ingress-nginx Mitigation</h3><h3>Background</h3><p>As SADA helps our clients navigate security challenges, we emphasize a layered approach. In the context of the ingress-nginx vulnerability, restricting network access is a critical compensating control. This appendix provides examples of how to implement network policies within Kubernetes and firewall rules at the network level to limit potential attack vectors.</p><h3>Kubernetes Network Policies</h3><p>Kubernetes Network Policies allow you to control traffic flow at the pod and namespace level. SADA strongly recommends implementing these to micro-segment your Kubernetes environment.</p><h3>A. Example: Restricting access to ingress-nginx admission controller pods</h3><p>This example demonstrates how to restrict inbound traffic to the ingress-nginx admission controller pods, allowing only authorized components within the cluster to communicate with them.</p><pre>apiVersion: networking.k8s.io/v1<br>kind: NetworkPolicy<br>metadata:<br>name: restrict-ingress-nginx-admission-controller<br>namespace: ingress-nginx # Adjust if your ingress-nginx is in a different namespace<br>spec:<br>podSelector:<br>matchLabels:<br>app.kubernetes.io/name: ingress-nginx<br>app.kubernetes.io/component: admission-webhook # Assuming this label identifies your admission controller pods<br>ingress:<br>- from:<br>- namespaceSelector:<br>matchLabels:<br>kubernetes.io/metadata.name: kube-system # Or the namespace of authorized control plane components<br>ports:<br>- protocol: TCP<br>port: 443 # Or the port your admission controller listens on<br>egress:<br>- to:<br>- podSelector: {} # Allow all egress from the admission controller pods</pre><p><strong>Explanation:</strong></p><ul><li><strong>podSelector:</strong> This selects the ingress-nginx admission controller pods based on their labels. You will need to adjust these labels to match your specific ingress-nginx deployment.</li><li><strong>ingress:</strong> This section defines the allowed inbound traffic.</li><li><strong>from: </strong>We are allowing traffic only from the kube-system namespace (where control plane components often reside). SADA recommends tailoring this to the <em>specific</em> namespaces or pods that need access, adhering to the principle of least privilege.</li><li><strong>ports:</strong> This specifies that only TCP traffic on port 443 (or the appropriate port) is allowed.</li><li><strong>egress:</strong> This example allows all egress traffic from the admission controller pods. SADA advises that, in a production environment, you should restrict this further to only the necessary outbound connections.</li></ul><p><strong>Important considerations:</strong></p><ul><li><strong>Namespace:</strong> Adjust the <em>namespace</em> field to match where your ingress-nginx controller is deployed.</li><li><strong>Labels:</strong> Ensure the <em>podSelector</em> labels accurately match the labels used by your ingress-nginx admission controller pods. Use the <em>kubectl get pods — namespace &lt;ingress-nginx-namespace&gt; — show-labels</em> command to verify.</li><li><strong>Authorized components:</strong> Carefully determine which components need access to the admission controller and adjust the <em>namespaceSelector</em> or <em>podSelector</em> accordingly. SADA can assist in identifying these authorized components.</li><li><strong>Testing:</strong> SADA emphasizes the importance of thoroughly testing Network Policies in a non-production environment before applying them to production. Incorrectly configured policies can disrupt critical communication.</li></ul><h3>II. Firewall rules (Google Cloud example)</h3><p>If you are using Google Kubernetes Engine (GKE), Google Cloud Firewall Rules provide an additional layer of network security. SADA often utilizes these in conjunction with Kubernetes Network Policies for defense in depth.</p><h3>A. Example: Restricting external access to ingress-nginx controllers</h3><p>This example demonstrates how to restrict external access to the ingress-nginx controllers themselves.</p><pre>gcloud compute firewall-rules create restrict-ingress-nginx-external-access \<br> - allow=tcp:&lt;ingress-nginx-port&gt; \<br> - source-ranges=&lt;allowed-source-ranges&gt; \<br> - target-tags=&lt;ingress-nginx-node-tags&gt; \<br> - description=&quot;Restrict external access to ingress-nginx controllers&quot;</pre><p><strong>Explanation:</strong></p><ul><li>— <strong>allow:</strong> This specifies the allowed protocol and port (e.g., TCP and the port your ingress-nginx controller is listening on, such as 80 or 443).</li><li><strong>— source-ranges:</strong> This limits the source IP address ranges that are allowed to access the ingress-nginx controllers. SADA recommends restricting this to only necessary external sources, or ideally, routing external traffic through Google Cloud Load Balancer. If you are using Google Cloud Load Balancer, you would typically allow traffic from the load balancer’s IP ranges.</li><li><strong>— target-tags: </strong>This applies the firewall rule to instances with specific network tags. You’ll need to tag the nodes where your ingress-nginx controllers are running. SADA can help you with proper tagging strategies.</li></ul><p><strong>Important considerations:</strong></p><ul><li><strong>Node tags:</strong> Properly tagging your GKE nodes is crucial for applying firewall rules effectively.</li><li><strong>Source ranges:</strong> Exercise extreme caution when defining — source-ranges. Allowing overly broad ranges can negate the security benefit. SADA advocates for the principle of least privilege here.</li><li><strong>Google Cloud Load Balancer:</strong> As previously mentioned, SADA’s best practice is to use Google Cloud Load Balancer. In that case, you would configure the firewall rules to allow traffic from the load balancer to your ingress-nginx controllers, and the load balancer would handle external access control.</li><li><strong>Automation:</strong> SADA recommends managing firewall rules as Infrastructure as Code (IaC) using tools like Terraform to ensure consistency and auditability.</li></ul><h3>III. Additional guidance from SADA</h3><ul><li><strong>Defense in depth:</strong> SADA strongly advises using a combination of Kubernetes Network Policies and firewall rules for a robust defense-in-depth strategy.</li><li><strong>Regular review:</strong> Network security rules should be regularly reviewed and audited. SADA can provide ongoing security assessments to ensure your configurations remain secure.</li><li><strong>Automation:</strong> SADA promotes the use of automation for managing network security configurations. This reduces the risk of human error and ensures consistent enforcement.</li><li><strong>Zero trust:</strong> These network security measures align with the principles of zero trust, which SADA champions. By explicitly verifying and limiting network access, you reduce the impact of potential compromises.</li></ul><h3>Item 2: Log review in Google Cloud</h3><h3>1. GKE audit logs</h3><p>GKE audit logs record a chronological sequence of activities within your GKE clusters. These logs can provide valuable insights into who did what, when, and where.</p><ul><li><strong>What to look for:</strong> Focus on audit logs related to the admission controller. Look for any unauthorized attempts to modify admission controller configurations or unusual API calls that might indicate exploitation.</li><li><strong>Google Cloud tools:<br>Cloud Logging:</strong> GKE audit logs are integrated with Cloud Logging. You can use Cloud Logging’s powerful filtering and querying capabilities to search for suspicious activity. For example, you might filter logs by specific users, namespaces, or API verbs (e.g., <strong>CREATE, UPDATE, DELETE</strong>) related to the admission controller.<br><strong>Example query:</strong> To find attempts to modify admission webhooks, you could use a query like:</li></ul><pre>resource.type=&quot;gke_cluster&quot;<br>logName=&quot;projects/&lt;YOUR_PROJECT_ID&gt;/logs/cloudaudit.googleapis.com%2Factivity&quot;<br>protoPayload.methodName=&quot;admissionregistration.k8s.io/v1.MutatingWebhookConfiguration.Update&quot;</pre><ul><li>SADA advises that you adapt this query to your specific setup and the admission webhook configurations you are monitoring.</li></ul><h3>2. ingress-nginx controller pod logs</h3><p>Container logs provide detailed information about what’s happening inside your ingress-nginx controller pods.</p><ul><li><strong>What to look for:</strong> Examine the NGINX logs for any unexpected requests, especially to the admission controller endpoints. Look for unusual HTTP methods, abnormal request sizes, or requests from unknown IP addresses (if you have implemented network restrictions).</li><li><strong>Google Cloud Tools:<br>Cloud Logging:</strong> GKE container logs are also integrated with Cloud Logging. You can correlate these logs with GKE audit logs for a more complete picture.<br><strong>Example query:</strong> To look for unusual POST requests to the admission controller endpoint (assuming it’s <em>/admission-endpoint</em>), you might use a query like:</li></ul><pre>resource.type=&quot;k8s_container&quot;<br>resource.labels.container_name=&quot;&lt;your-ingress-nginx-controller-container-name&gt;&quot;<br>logName=&quot;projects/&lt;YOUR_PROJECT_ID&gt;/logs/containers/&lt;your-ingress-nginx-controller-container-name&gt;&quot;<br>httpRequest.requestMethod=&quot;POST&quot;<br>httpRequest.requestUrl.contains=&quot;/admission-endpoint&quot;</pre><ul><li>SADA recommends customizing this query based on your ingress-nginx configuration and endpoint.</li></ul><h3>3. Network monitoring</h3><p>Monitoring network traffic can help you detect suspicious connections to and from your ingress-nginx controllers.</p><ul><li><strong>What to look for:</strong> Look for unusual traffic patterns, such as connections from unexpected sources, excessive traffic volume, or connections to unusual ports.</li><li><strong>Google Cloud tools:<br>VPC Flow Logs:</strong> VPC Flow Logs record information about the IP traffic going to and from your GKE nodes. You can analyze these logs to identify suspicious network activity.<br><strong>Cloud IDS (Intrusion Detection System):</strong> Cloud IDS can detect threats in your GKE network. It can identify malicious activity, such as intrusion attempts or malware. SADA often recommends Cloud IDS for enhanced network security monitoring.<br><strong>Security Command Center:</strong> Google Cloud’s Security Command Center provides a central dashboard for viewing and managing security findings across your Google Cloud environment, including your GKE clusters. It can help you identify and prioritize security risks related to your ingress-nginx deployments.</li></ul><p><strong>Important considerations from SADA:</strong></p><ul><li><strong>Centralized logging and SIEM integration:</strong> SADA strongly advises centralizing your logs (GKE audit logs, container logs, VPC Flow Logs) in Cloud Logging and integrating them with your Security Information and Event Management (SIEM) system. This enables effective correlation, analysis, and alerting.</li><li><strong>Alerting:</strong> Configure alerts based on your log queries and network monitoring to notify your security team of any suspicious activity in real-time. SADA emphasizes the importance of proactive alerting for timely incident response.</li><li><strong>Automation:</strong> Automate the analysis of logs and network traffic where possible. Use Cloud Logging’s API or other automation tools to regularly scan for suspicious patterns and trigger alerts.</li><li><strong>Expert assistance:</strong> SADA’s security experts can provide tailored guidance on setting up effective monitoring and detection for your GKE environment, including customized log queries, alerting rules, and SIEM integration.</li><li>If you think you may have been compromised, please reach out to our Incident Response team by calling 800–950–3475. Our response team can help with forensics to determine the depth of an attack and the response to breaches.</li></ul><p><strong>Not compromised but want to assess your security posture? Schedule a </strong><a href="https://info.sada.com/cloud-security-confidence-assessment"><strong>Cloud Security Confidence Assessment</strong></a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=04c3307b5161" width="1" height="1" alt=""><hr><p><a href="https://engineering.sada.com/ingressnightmare-sadas-perspective-on-the-ingress-nginx-vulnerability-04c3307b5161">IngressNightmare: SADA’s perspective on the Ingress-nginx vulnerability</a> was originally published in <a href="https://engineering.sada.com/">The SADA Engineering Blog</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Bridging the gap: validating data across data warehouses with data validation testing]]></title>
            <link>https://engineering.sada.com/bridging-the-gap-validating-data-across-data-warehouses-with-data-validation-testing-5cd9e68e73bf?source=rss-6bc96cd41286------2</link>
            <guid isPermaLink="false">https://medium.com/p/5cd9e68e73bf</guid>
            <category><![CDATA[google-cloud-platform]]></category>
            <category><![CDATA[data-warehouse]]></category>
            <category><![CDATA[data]]></category>
            <category><![CDATA[testing]]></category>
            <category><![CDATA[data-validation]]></category>
            <dc:creator><![CDATA[SADA, An Insight Company]]></dc:creator>
            <pubDate>Wed, 08 May 2024 13:02:07 GMT</pubDate>
            <atom:updated>2024-05-08T15:47:47.823Z</atom:updated>
            <content:encoded><![CDATA[<p><strong><em>Authors:</em></strong><em> </em><a href="https://www.linkedin.com/in/mina-botros-569a4726/"><em>Mina Botros</em></a><em>, Senior Data Platform Engineer, SADA; </em><a href="https://www.linkedin.com/in/s-c-n/"><em>Samuel Njoku</em></a><em>, Associate Manager of Infrastructure Modernization, SADA; </em><a href="https://www.linkedin.com/in/rv1448/"><em>Rahul Vangala</em></a><em>, Senior Data Engineer, SADA</em></p><h3>Introduction</h3><p>Managing multiple data warehouses is a common challenge that organizations face today. Two popular choices are Snowflake and BigQuery, each boasting its own strengths and catering to specific needs. However, this multi-warehouse setup can introduce a new challenge: ensuring data consistency between them. This is where the <a href="https://github.com/GoogleCloudPlatform/professional-services-data-validator">Data Validation Tool </a>(DVT) comes in, acting as a crucial bridge between Snowflake and BigQuery. The tool is not limited to just those two data warehouse systems but can work with a <a href="https://cloud.google.com/blog/products/databases/automate-data-validation-with-dvt">variety of databases</a>.</p><h3>High-level overview</h3><p>DVT, by Google, is an open-source Python command-line tool specifically designed for data warehouse, database, or data lake migrations. It automates comparing data between source and target tables to ensure they match after each migration step. DVT supports various data sources like BigQuery and Cloud SQL and works across different environments.</p><p>The tool uses the powerful <a href="https://ibis-project.org/">Ibis framework</a> to build portable data frames that are agnostic to backend frameworks.</p><h3>What is DVT and why is it important for Snowflake-BigQuery integration?</h3><p>DVT is a data quality practice that involves comparing data across different sources to identify and rectify discrepancies. In the context of Snowflake and BigQuery, DVT helps ensure that the data being replicated or migrated between the two platforms is accurate and complete. This is critical for several reasons:</p><ul><li><strong>Improved decision-making:</strong> Inconsistent data can lead to misleading insights and inaccurate reports, hindering effective decision-making.</li><li><strong>Enhanced trust in data:</strong> DVT builds trust in the data’s integrity, allowing stakeholders to confidently rely on it for analysis and reporting.</li><li><strong>Reduced business risks:</strong> Data inconsistencies can lead to operational issues and financial losses. DVT helps mitigate these risks by proactively identifying and resolving data quality problems.</li></ul><h3>How can DVT be implemented for Snowflake-BigQuery validation?</h3><p>There are several approaches to DVT for Snowflake-BigQuery integration. Here are some popular methods:</p><ul><li><strong>Direct table comparisons:</strong> This involves querying corresponding tables in both Snowflake and BigQuery and comparing the results row-by-row. While straightforward, this method can be inefficient for large datasets.</li><li><strong>Data profiling:</strong> Profiling tools can be used to analyze the statistical properties of data in both platforms, such as column distributions and null values. This can help identify potential discrepancies before performing detailed comparisons.</li><li><strong>Schema validations:</strong> Schema validations help ensure that the structure of your data in a source system matches the structure in a target system. It compares column names and data types, ignoring case sensitivity.</li><li><strong>Custom query row validations:</strong> This approach allows you to compare the individual rows of data between a source and target system, even when working with custom queries instead of entire tables.</li><li><strong>Ad hoc SQL exploration queries directly within the validation process:</strong> This saves you time and keeps your workflow streamlined.</li></ul><p>Code examples can be found on the <a href="https://github.com/GoogleCloudPlatform/professional-services-data-validator/tree/develop/samples">GitHub repository</a>.</p><h3>DVT in action: a retail data warehouse migration</h3><p>Now that we’ve laid out what DVT is and why it’s important for Snowflake-BigQuery integration, let’s dive into a real-world use case. SADA recently helped an enterprise retail customer build a parallel data warehouse in BigQuery. This project aimed to:</p><ul><li>Build a more robust data warehouse with improved data management capabilities.</li><li>Build a more insightful data warehouse providing accurate and timely business insights.</li><li>Decommission an existing Snowflake-based data warehouse.</li></ul><h3>Challenge: ensuring data consistency during a complex migration</h3><p>In this project, we aimed to replace the data warehouse built in Snowflake with the new and improved data warehouse built in BigQuery. The data warehouse in Snowflake has been used for a number of years and is well-trusted by its consumers.</p><p>We needed the ability to prove that data is consistent between those two systems in cases when no improvements were made during the development process. Moreover, we needed to determine if the logical data transformations and data aliasing could still be validated using DVT in cases where enhancements over the data model in Snowflake were needed and implemented in the ETL pipelines feeding data into BigQuery.</p><p>The data loading patterns into Snowflake didn’t match that of BigQuery, therefore data was not expected to be identical at all times. Both systems needed to be in consistent alignment within a small margin of error.</p><p>DVT played a central role in our User Acceptance Testing in allowing us to get the assurance that data matches between those two systems and if there were valid acceptable reasons when it didn’t. Moreover, the cross-data warehouse validation testing enabled us to uncover, prioritize, and resolve a number of defects that were not previously discovered.</p><h3>Key considerations for data validation tool selection</h3><p>Now, let’s discuss some of the key pillars we focused on to identify the right tool for the job.</p><h4>Incremental testing</h4><p>Incremental testing, within the context of ETL processes, is a strategy where testing is conducted in small, manageable segments during the development of data pipelines and production runs to conduct data quality checks. Furthermore, the incremental testing strategy is also applied post-deployment of the pipeline into production as part of the migration process. This allows for the collection of metrics from each incremental job run, providing ongoing insights into data quality and comparison with the source system which we migrated from.</p><h4>Elimination of manual testing</h4><p>This automation incorporates incremental testing and automated comparison of data between the Snowflake and BigQuery warehouse systems as part of the migration process, aiming to mitigate any discrepancies between the systems. Furthermore, the incremental strategy must adapt to both known differences and potential improvements during the comparison between the source and target systems. Considering the near-real-time data warehouse refresh every 15 minutes, it’s imperative to develop a scalable method for running these comparisons without relying heavily on manual intervention.</p><h4>Easy identification of discrepancies</h4><p>As part of this incremental testing, ongoing tests are essential to compare new outputs produced in both the source and target systems. It is crucial to establish a reliable method for identifying differences at the table level, and column level, and for metrics related to data within these entities, such as the number of nulls, the number of non-null values, and the total number of rows. Moreover, there should be a capability to generate these metrics in a way that allows for direct comparison, producing outputs in the same format across both systems <strong>(apples-to-apples comparisons)</strong>. With all these guards in place, we need the right way to identify error cases post-identification of discrepancies.</p><h4>Tracking data quality metrics over time</h4><p>DVT helps you ensure data quality and reliability by scanning for issues it identifies inconsistencies, anomalies, and errors to pinpoint problems. Moreover, it tracks trends and changes over time, helping you spot areas needing improvement in terms of data quality. Besides that it could be integrated with alerting capabilities to be notified when quality metrics fall outside defined thresholds, allowing for quick action. Additionally, it enhances collaboration and shares reports with stakeholders for a shared understanding and data-driven decisions by storing its results in a database for easy access. We stored the DVT test results in BigQuery to be able to demonstrate the history of defects and to analyze the results further for reporting purposes.</p><h4>Continuous integration and deployment</h4><p>Google DVT isn’t directly used for Continuous Integration and Deployment (CI/CD) itself, as its focus lies on data validation. However, it can be a valuable asset within a CI/CD pipeline for ensuring data health and integrity throughout the deployment process. That can be achieved by integrating DVT checks into your CI pipeline to automatically validate data after code changes, deployments, or data migrations, catching issues early and preventing them from reaching production. Moreover, continuous data validation with DVT increases confidence in deployments by ensuring data consistency and quality across environments. Lastly, with automated DVT reports, pinpointing data-related issues during deployments becomes significantly faster, minimizing downtime and rollback complexities. GitHub Actions was used to create an environment and execute the DVT tests based on triggering or through manual execution.</p><h3>How data discrepancies between Snowflake and BigQuery were addressed</h3><p>While the DVT tool strives for bit-for-bit data accuracy, acknowledging inherent platform differences between Snowflake and BigQuery was crucial. Here’s how we tackled some key discrepancies:</p><ol><li>Threshold percentage tolerances:</li></ol><ul><li>Instead of strict equality checks, we implemented configurable threshold tolerances for specific data types. These thresholds accounted for known discrepancies, like timestamp rounding variations, without compromising data integrity.</li><li>For example, a marginal tolerance on floating-point comparisons allowed for minor rounding differences while flagging significant deviations.</li></ul><p>2. Rounding discrepancies:</p><ul><li>We identified known rounding issues between BigQuery and Snowflake decimal data types. To address this, we employed custom data transformation functions during comparison.</li><li>These functions normalized data representation before comparison, ensuring consistent interpretation regardless of origin platform. For instance, rounding both values to a specific decimal place before comparison eliminated minor rounding discrepancies.</li></ul><p>3. Intuitive discrepancy identification:</p><ul><li>We went beyond just flagging discrepancies. The DVT tool generated detailed reports highlighting which specific data points exceeded the configured thresholds.</li><li>This granular insight helped pinpoint the exact areas of divergence, enabling deeper investigation and targeted remediation efforts.</li></ul><p>These measures ensured efficient and accurate data comparison across Snowflake and BigQuery, striking a balance between strict data validation and acknowledging platform peculiarities.</p><p>Sample validations can be found on the <a href="https://github.com/GoogleCloudPlatform/professional-services-data-validator/blob/develop/docs/examples.md">GitHub repository</a>.</p><p>Additional details are provided in the sections below:</p><h4>Deployment environments</h4><p>The setup included three main deployment environments, development, staging, and production. DVT was configured to be connected to each of those environments. The same test cases could be executed against any of those environments through a simple configuration setup.</p><p>User Acceptance Testing was performed in a separate staging GCP project. The data in the staging environment was compared to the corresponding data in Snowflake using DVT. When data inconsistencies were discovered, the defects would be fixed in the lower development environment.</p><h4>Establishing data warehouse connections</h4><p>Before running validations, DVT requires a source and target connection to be set up. These connections can be stored locally or in a Google Cloud Storage (GCS) bucket. Using a GCS bucket is recommended for storing these connections as it allows you to limit access to these connections as credentials are stored in plain text.</p><p>To set up a connection between the source and target data warehouses, the DVT binary (<a href="https://github.com/GoogleCloudPlatform/professional-services-data-validator">google-pso-data-validator</a>) needs to be installed on the platform it will be executed from. For this, you will need to have Python 3.8 or above installed.</p><p>Validation results can be printed to stdout by default or outputted to a BigQuery table (recommended). We chose to use a BigQuery results table, which was created as part of our initial setup.</p><p>Additionally, we chose to automate our DVT validations using a CI/CD pipeline. These were short-lived runs executed as GitHub Actions pipelines triggered by our data management workflows. More on this later.</p><h4>Creating data validation connections</h4><p>DVT allows users to create connections using the CLI. These connections will automatically be saved either to ~/.config/google-pso-data-validator/ or a directory specified by the environment variable PSO_DV_CONFIG_HOME.</p><p>To store the connections in a GCS bucket, we add the GCS directory path to the PSO_DV_CONFIG_HOME environment variable. The following is a sample:</p><p>export PSO_DV_CONFIG_HOME=gs://my-bucket/my/connections/path/</p><p>Note that if this path is set, query validation configs will also be saved here.</p><h4>Source connection</h4><p>Snowflake is our source data warehouse, but DVT supports the other data warehouses as seen <a href="https://github.com/GoogleCloudPlatform/professional-services-data-validator/blob/develop/docs/connections.md#list-supporting-connection-types">here</a>.</p><p>Note that DVT supports Google Cloud Secret Manager for storing and referencing secrets in your connection configuration.</p><p>The following sample CLI command shows you how to create a Snowflake connection:</p><pre>data-validation connections add<br> - secret-manager-type GCP<br> - secret-manager-project-id GCP_SECRET_PROJECT_ID<br> - connection-name SNOWFLAKE_CONN_NAME<br> - user SNOWFLAKE_USER<br> - password SNOWFLAKE_PASSWORD<br> - account SNOWFLAKE_ACCOUNT<br> - database SNOWFLAKE_DATABASE/SCHEMA<br> - connect-args CONNECT_ARGS</pre><h4>Creating a target connection</h4><p>BigQuery is our target data warehouse so a target connection is also needed for BigQuery.</p><p>Note that the User/Service account needs the following BigQuery permissions to run DVT:</p><ul><li>bigquery.jobs.create (BigQuery JobUser role)</li><li>bigquery.readsessions.create (BigQuery Read Session User)</li><li>bigquery.tables.get (BigQuery Data Viewer)</li><li>bigquery.tables.getData (BigQuery Data Viewer)</li></ul><p>And the following permissions when storing validation results in BigQuery:</p><ul><li>bigquery.tables.update (BigQuery Data Editor)</li><li>bigquery.tables.updateData (BigQuery Data Editor)</li></ul><p>The following sample CLI command shows you how to create a BigQuery connection:</p><pre>data-validation connections add<br> - secret-manager-type GCP<br> - secret-manager-project-id GCP_SECRET_PROJECT_ID<br> - connection-name BIGQUERY_CONN_NAME<br> - project-id GCP_PROJECT<br> - google-service-account-key-path PATH_TO_SA_KEY</pre><h3>Star schema validation</h3><h4>Fact table validation</h4><p>The fact table was validated by aggregating data over several months to compare key metrics across the two data warehouse systems. A combination of row and column-based comparisons were used for validation depending on the conditions for each of the columns. Some of the columns could be easily compared across the systems, in which case row-based comparisons were used. In other cases, some columns were expected to deviate due to how each of the data warehouses performed numerical rounding, in which case a column-based comparison with a margin of error was used.</p><p>The fact tables’ DVT tests were written as YAML files to describe the source, target, transformation, and comparison criteria. Here is a sample of the row-based tests which were used.</p><p>(Note that in the below sample, source refers to the data warehouse in Snowflake while the target refers to data warehouse in BigQuery.)</p><pre>result_handler:<br> project_id: [GCP PROJECT ID]<br> table_id: dvt_data_validator.results<br> type: BigQuery<br># reference to the Snowflake Source System Connection<br>source: sf<br># reference to the BigQuery Source System Connection<br>target: bq<br>validations:<br> # this is the list of aggregations which will be applied to each of the comparison fields<br>- aggregates:<br> # Local currency columns<br> - field_alias: sum_gross_sales_loc<br>   source_column: GROSSSALESLOC<br>   target_column: GROSSSALESLOC<br>   type: sum<br> - field_alias: sum_net_sales_loc<br>   source_column: NETSALESLOC<br>   target_column: NETSALESLOC<br>   type: sum<br> - field_alias: sum_tax_loc<br>   source_column: TAXLOC<br>   target_column: TAXLOC<br>   type: sum<br> - field_alias: sum_cost_loc<br>   source_column: COSTLOC<br>   target_column: COSTLOC<br>   type: sum<br><br><br> # GBP currency columns<br> - field_alias: sum_tax_g<br>   source_column: TAXG<br>   target_column: TAXG<br>   type: sum<br> - field_alias: sum_gross_sales_g<br>   source_column: GROSSSALESG<br>   target_column: GROSSSALESG<br>   type: sum<br> - field_alias: sum_net_sales_g<br>   source_column: NETSALESG<br>   target_column: NETSALESG<br>   type: sum<br> - field_alias: sum_cost_g<br>   source_column: COSTG<br>   target_column: COSTG<br>   type: sum<br><br><br>   # Sales Quantity<br> - field_alias: sum_sales_qty<br>   source_column: SALESQUANT<br>   target_column: SALESQUANT<br>   type: sum<br><br><br> # this describes the transformations which is done to rollup the transaction data time to the year and month granularity<br> calculated_fields:<br> - depth: 0<br>   field_alias: transaction_t<br>   source_calculated_columns:<br>   - END_DATETIME<br>   target_calculated_columns:<br>   - transaction_date_time<br>   type: custom<br>   ibis_expr: ibis.expr.types.TemporalValue.strftime<br>   params:<br>   - format_str: &#39;%Y-%m&#39;<br> # filter the rows which are retrieved from the source and target systesm. Currently filtering on the past three month<br> # excluding current month<br> filters:<br> - source: t1.&quot;transaction_t&quot;&gt;=to_varchar(ADD_MONTHS(CURRENT_DATE, -3), &#39;YYYY-MM&#39;)<br>   target: transaction_t&gt;=CAST(DATE_SUB(CURRENT_DATE(&#39;UTC&#39;), INTERVAL 3 MONTH) AS STRING FORMAT(&#39;YYYY-MM&#39;))<br>   type: custom<br> - source: t1.&quot;transaction_t&quot;&lt;to_varchar(CURRENT_DATE, &#39;YYYY-MM&#39;)<br>   target: transaction_t&lt;cast(CURRENT_DATE(&#39;UTC&#39;) as string format(&#39;YYYY-MM&#39;))<br>   type: custom<br> # group by the transaction time field<br> grouped_columns:<br> - cast: null<br>   field_alias: transaction_t<br>   source_column: transaction_t<br>   target_column: transaction_t<br> # below is the list of column names to be used for comparison from each of the source and target systems<br> comparison_fields:<br><br><br> # Local currency columns<br> - cast: null<br>   field_alias: gross_sales_loc<br>   source_column: GROSS_SALES_LOC<br>   target_column: GrossSalesLoc<br> - cast: null<br>   field_alias: net_sales_loc<br>   source_column: NET_SALES_LOC<br>   target_column: NetSalesLoc<br> - cast: null<br>   field_alias: tax_loc<br>   source_column: TAX_LOC<br>   target_column: TaxLoc<br> - cast: null<br>   field_alias: cost_loc<br>   source_column: COST_LOC<br>   target_column: CostLoc<br><br><br> # GBP currency columns<br> - cast: null<br>   field_alias: tax_g<br>   source_column: TAX_G<br>   target_column: TaxG<br> - cast: null<br>   field_alias: gross_sales_g<br>   source_column: GROSS_SALES_G<br>   target_column: GrossSalesG<br> - cast: null<br>   field_alias: net_sales_g<br>   source_column: NET_SALES_G<br>   target_column: NetSalesG<br> - cast: null<br>   field_alias: cost_g<br>   source_column: COST_G<br>   target_column: CostG<br><br><br> # Sales Quantity<br> - cast: null<br>   field_alias: sales_quantity<br>   source_column: SALES_QTY<br>   target_column: SalesQuantity<br><br><br> # include both failures and successes<br> filter_status: null<br> format: table<br> labels:<br>   - !!python/tuple<br>     - domain<br>     - sales<br>   - !!python/tuple<br>     - scope<br>     - fact_sales<br> # the primary key for this table is the transaction time, which is essentially rolled up to the year and month<br> primary_keys:<br> - cast: null<br>   field_alias: transaction_t<br>   source_column: transaction_t<br>   target_column: transaction_t<br> random_row_batch_size: null<br> schema_name: FACT<br> table_name: SALES<br> target_schema_name: BQ_DATASET<br> target_table_name: SALES_FACT_TABLE<br># apply row-based comparison based on the primary key for each row with zero error margin<br> threshold: 0.0<br> type: Row<br> use_random_rows: false</pre><p>On the other hand, in cases where we knew that there were numerical rounding differences between Snowflake and BigQuery, we applied column-based comparison with a small margin of error represented as the ‘threshold’ parameter towards the end of the YAML file below:</p><pre>result_handler:<br> project_id: [GCP PROJECT ID]<br> table_id: dvt_data_validator.results<br> type: BigQuery<br># reference to the Snowflake Source System Connection<br>source: sf<br># reference to the BigQuery Source System Connection<br>target: bq<br>validations:<br>  # this is the list of aggregations which will be applied to each of the comparison fields<br>- aggregates:<br> - field_alias: sum_gross_sales_e<br>   source_column: GROSS_SALES_E<br>   target_column: GrossSalesE<br>   type: sum<br> - field_alias: sum_net_sales_e<br>   source_column: NET_SALES_E<br>   target_column: NetSalesE<br>   type: sum<br> - field_alias: sum_tax_e<br>   source_column: TAX_E<br>   target_column: TaxE<br>   type: sum<br> - field_alias: sum_cost_e<br>   source_column: COST_E<br>   target_column: CostE<br>   type: sum<br> filter_status: null<br> # this describes the transformations which is done to rollup the transaction data time to the year and month granularity<br> calculated_fields:<br>   - depth: 0<br>     field_alias: transaction_t<br>     source_calculated_columns:<br>       - END_DATETIME<br>     target_calculated_columns:<br>       - transaction_date_t<br>     type: custom<br>     ibis_expr: ibis.expr.types.TemporalValue.strftime<br>     params:<br>       - format_str: &#39;%Y-%m&#39;<br> # filter the rows which are retrieved from the source and target system. Currently filtering on the past three month<br> # excluding current month<br> filters:<br>   - source: t1.&quot;transaction_t&quot;&gt;=to_varchar(ADD_MONTHS(CURRENT_DATE, -3), &#39;YYYY-MM&#39;)<br>     target: transaction_t&gt;=CAST(DATE_SUB(CURRENT_DATE(&#39;UTC&#39;), INTERVAL 3 MONTH) AS STRING FORMAT(&#39;YYYY-MM&#39;))<br>     type: custom<br>   - source: t1.&quot;transaction_t&quot;&lt;to_varchar(CURRENT_DATE, &#39;YYYY-MM&#39;)<br>     target: transaction_t&lt;cast(CURRENT_DATE(&#39;UTC&#39;) as string format(&#39;YYYY-MM&#39;))<br>     type: custom<br> # below is the list of column names to be used for comparison from each of the source and target systems<br> comparison_fields:<br>   - cast: null<br>     field_alias: gross_sales<br>     source_column: GROSS_SALES_EURO<br>     target_column: GrossSalesE<br>   - cast: null<br>     field_alias: net_sales<br>     source_column: NET_SALES_E<br>     target_column: NetSalesE<br>   - cast: null<br>     field_alias: tax<br>     source_column: TAX_E<br>     target_column: TaxE<br>   - cast: null<br>     field_alias: cost<br>     source_column: COST_E<br>     target_column: CostE<br> # group by the transaction time field<br> grouped_columns:<br>   - cast: null<br>     field_alias: transaction_t<br>     source_column: transaction_t<br>     target_column: transaction_t<br> format: table<br> labels:<br>   - !!python/tuple<br>     - domain<br>     - sales<br>   - !!python/tuple<br>     - scope<br>     - fact_sales<br> random_row_batch_size: null<br> schema_name: FACT<br> table_name: SALES<br> target_schema_name: BQ_DATASET<br> target_table_name: SALES_FACT_TABLE<br> # below is the threshold that was configured to account for business-acceptable discrepancies<br> threshold: 0.00003<br> type: Column<br> use_random_rows: false</pre><p>The above validation test YAML files were executed through the following DVT CLI bash commands:</p><pre># Fact Sales Validation Tests<br># Validate the fact sales local and GBP currency metrics<br>data-validation configs \<br>run -c ../validations/sales/fact_sales/fact_sales_table_validations_local_and_gbp_metrics.yaml<br># Validate the fact sales euro metrics metrics<br>data-validation configs \<br>run -c ../validations/sales/fact_sales/fact_sales_table_validations_euro_metrics.yaml</pre><h4>Dimension tables validation</h4><p>Dimensions required a distinct validation strategy compared to that of the fact tables. To streamline the process, the team opted for crafting SQL queries tailored to each data warehouse system, eliminating the need for a YAML file in this instance.</p><p>Below, we’ll illustrate two use cases: one involving a direct comparison between corresponding dimension tables, and another necessitating transformations for a similarity comparison.</p><p>Firstly, let’s examine a straightforward use case involving the Brand Type Dimension table. To extract data from Snowflake compatible with the enhancements implemented in BigQuery, we had to transform the Snowflake data to match the following format:</p><pre>SELECT<br>DATE(END_DATETIME) DATEKEY,<br>case [CASES LISTED HERE] AS BRANDTYPE,<br>SUM(GROSS_SALES_GBP) GROSSSALES<br>FROM SALES_FACT_TABLE FS<br>JOIN PRODUCT_DIMENSION_TABLE AS P<br>ON FS.PRODUCTKEY=P.PRODUCTKEY<br>WHERE DATE(END_DATETIME) = DATEADD(DAY, -1, CURRENT_DATE())<br>GROUP BY DATEKEY,BRANDTYPE<br>ORDER BY DATEKEY,BRANDTYPE;</pre><p>In BigQuery, the query looked as follows:</p><pre>SELECT<br>DATE(END_DATETIME) DATEKEY,<br>case [CASES LISTED HERE] AS BRANDTYPE,<br>SUM(GROSS_SALES_GBP) GROSSSALES<br>FROM SALES_FACT_TABLE FS<br>JOIN PRODUCT_DIMENSION_TABLE AS P<br>ON FS.PRODUCTKEY=P.PRODUCTKEY<br>WHERE DATE(END_DATETIME) = DATEADD(DAY, -1, CURRENT_DATE())<br>GROUP BY DATEKEY,BRANDTYPE<br>ORDER BY DATEKEY,BRANDTYPE;</pre><p>Note that both queries compare only the previous day of data and not the entirety of data in each of the tables. This has been mainly scoped down to control costs and runtime on both data warehouses.</p><p>In order to use the above SQL files, we ran the following DVT CLI bash command:</p><pre>## Fact Sales and Dim Brand Type Validation for a single day<br># dim_brand_type column-by-column comparison<br>data-validation validate custom-query row \<br> - source-query-file ../validations/sales/dim_brand_type/source_query.sql \<br> - target-query-file ../validations/sales/dim_brand_type/target_query.sql \<br>-sc sf \<br>-tc bq \<br> - comparison-fields BRANDTYPE,GROSSSALES \<br>-pk brandtype \<br>-bqrh [GCP PROJECT ID].dvt_data_validator.results \<br> - filter-status &#39;fail&#39;</pre><p>In the command above, we pass in the following parameters:</p><ul><li>The relative paths of each of the SQL files</li><li>The source and target connection names referencing each of the data warehouses</li><li>The comparison fields to be used across the systems</li><li>The primary key of each of the tables which will be used by DVT to join and compare the results</li><li>The BigQuery project, dataset, and table name where DVT will write its test output results to</li><li>The filter status of which results would be written to BigQuery, in this case, its filtered to just the failure cases</li></ul><p>On the other hand, a straightforward example is the Product Gender Dimension which matched in logic in each of the data warehouses and only needed some column aliasing to be able to successfully perform the comparison.</p><p>In Snowflake, the query looked as follows:</p><pre>SELECT<br>DATE(END_DATETIME) DATEKEY,<br>PRODUCTGENDERAGE AS PRODUCTGENDERAGE,<br>PRODUCTGENDERGROUP AS PRODUCTGENDERGROUP,<br>PRODUCTGENDERMWK AS PRODUCTGENDERSECTION,<br>SUM(GROSS_SALES_GBP) GROSSSALES<br>FROM SALES_FACT_TABLE fs<br>JOIN PRODUCT_DIMENSION_TABLE as p<br>ON fs.productkey=p.productkey<br>WHERE DATE(end_datetime) = dateadd(day, -1, current_date())<br>GROUP BY DATEKEY, PRODUCTGENDERAGE, PRODUCTGENDERGROUP, PRODUCTGENDERSECTION<br>ORDER BY DATEKEY, PRODUCTGENDERAGE, PRODUCTGENDERGROUP, PRODUCTGENDERSECTION;</pre><p>In BigQuery, the query looked as follows:</p><pre>SELECT<br>DATE(TRANSACTIONDATETIME) DATEKEY,<br>PRODUCTGENDERAGE,<br>PRODUCTGENDERGROUP,<br>PRODUCTGENDERSECTION,<br>SUM(GROSSSALESGBP) GROSSSALES<br>FROM SALES_FACT_TABLE fs<br>JOIN PRODUCT_GENDER_DIMENSION_TABLE AS pg<br>ON fs.productgenderbk = pg.productgenderbk<br>WHERE DATE(transactiondatetime) = date_sub(current_date(), interval 1 day)<br>GROUP BY DATEKEY, PRODUCTGENDERAGE, PRODUCTGENDERGROUP, PRODUCTGENDERSECTION<br>ORDER BY DATEKEY, PRODUCTGENDERAGE, PRODUCTGENDERGROUP, PRODUCTGENDERSECTION;</pre><p>In order to use the above SQL files, we ran the following DVT CLI bash command:</p><pre>## Fact Sales and Dim Brand Type Validation for a single day<br># dim_product_gender column-by-column comparison<br>data-validation validate custom-query row \<br> - source-query-file ../validations/sales/dim_product_gender/source_query.sql \<br> - target-query-file ../validations/sales/dim_product_gender/target_query.sql \<br>-sc sf \<br>-tc bq \<br> - comparison-fields PRODUCTGENDERAGE,PRODUCTGENDERGROUP,PRODUCTGENDERSECTION,GROSSSALES \<br>-pk productgenderage \<br>-bqrh gcp_project_1.dvt_data_validator.results \<br> - filter-status &#39;fail&#39;</pre><h3>CI/CD automation</h3><p>To automate DVT data validations, we defined a GitHub Actions pipeline that runs each data validation step required per environment. GitHub Actions is a continuous integration and continuous delivery (CI/CD) platform that allows you to automate your build, test, and deployment pipeline.</p><h4>Below is a diagram depicting the workflow a data engineer would follow when triggering a data validation:</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*YAd4Zi5f4TfEgHkn" /></figure><h4>DVT GitHub Actions workflow</h4><p>A developer needs to run a new data validation or update an existing one:</p><ol><li>Create a feature branch in your data validation GitHub repository</li><li>Makes changes and pushes changes to a remote feature branch → Triggers DVT data validation GitHub Actions workflow:</li><li>If the DVT GitHub Actions pipeline fails → Developer continues to work on changes and fixes</li><li>If the DVT GitHub Actions pipeline passes → Developer creates a new PR to the Main branch</li><li>Developer creates a PR to Main branch → Triggers DVT data validation workflow:</li><li>If the DVT GitHub Actions pipeline run fails → Developer continues to work on changes and fixes</li><li>If the DVT GitHub Actions pipeline run passes → PR is Approved and Merged to the Main branch</li></ol><p>You can add a step to store the connections in a GCS bucket or locally. If you choose to store them locally, add a step to delete the config folder after the data validation step completes to ensure it doesn’t persist.</p><h3>Result</h3><p>Using DVT, we systematically uncovered and resolved all system discrepancies in an iterative manner. With each defect addressed, we reran validation tests to prevent regression, facilitated by seamless integration with the GitHub Actions pipeline. Each successful execution marked progress toward our project goals and eventual closure.</p><p>In the User Acceptance Testing environment, we identified issues, troubleshooted them, resolved them in a lower development environment, retested with DVT, promoted to higher environments, and repeated the process.</p><p>This rigorous approach instilled confidence in our ability to avoid regressions despite subsequent defect fixes, reassuring the customer that quality remained uncompromised throughout the project lifecycle.</p><h3>Conclusion: choosing the optimal approach for DVT testing</h3><p>The optimal DVT approach for your Snowflake-BigQuery integration will depend on several factors, such as the volume and complexity of your data, the frequency of data movement, and your budget. Consider these factors when making your decision:</p><ul><li><strong>Data volume:</strong> For smaller datasets, direct table comparisons might be sufficient. However, for larger datasets, profiling tools or dedicated DVT tools might be more efficient.</li><li><strong>Data complexity:</strong> If your data is complex with intricate relationships and transformations, data quality rules or specialized DVT tools might be necessary to capture all potential discrepancies.</li><li><strong>Frequency of data movement:</strong> If data is frequently replicated or migrated between Snowflake and BigQuery, an automated DVT solution might be more efficient and cost-effective.</li></ul><p>Data consistency between Snowflake and BigQuery is crucial for organizations relying on both platforms. Implementing a robust DVT strategy is essential for ensuring data quality, improving decision-making, and mitigating risks associated with inconsistent data. By carefully evaluating your needs and choosing the right DVT approach, you can build a bridge of trust between your Snowflake and BigQuery data, empowering your organization to confidently navigate the ever-evolving data landscape.</p><h3>Additional tips</h3><ul><li>For best results, integrate your DVT tests with the ETL/ELT pipelines to get recurring test results and early detection of discrepancies. DVT tests can be integrated with <a href="https://cloud.google.com/composer?hl=en">Cloud Composer</a>, <a href="https://cloud.google.com/functions?hl=en">Cloud Functions</a>, and <a href="https://cloud.google.com/run?hl=en">Cloud Run</a>.</li><li>Consider creating multiple levels of DVT testing with varying levels of granularity depending on the use case. Integrate lightweight DVT tests with the data pipelines for cheap and quick validation of scoped data sets and time ranges to balance cost with efficiency. Integrate more detailed tests to cover wider ranges of data with a CI/CD pipeline that can be either manually or automatically triggered based on your use case.</li><li>Document your DVT process clearly to ensure consistency and repeatability. Having a well-documented DVT test suite will help the team scale and replicate test cases systematically to reach conformity in testing approaches.</li><li>Leveraging data governance practices to ensure data quality is a priority across the organization. DVT testing in its various ways of implementation will impact project timelines and will need to be accounted for to ensure successful completion of projects in a timely manner.</li></ul><p>By following these tips and implementing a well-defined DVT strategy, you can effectively validate data between Snowflake and BigQuery, unlocking the full potential of your data-driven initiatives.</p><h4>About <a href="https://www.linkedin.com/in/mina-botros-569a4726/">Mina Botros</a></h4><p>Mina Botros is a seasoned Senior Data Platform Engineer at SADA, An Insight company, boasting over a decade of experience in the IT industry, primarily focused on data engineering. Proficient in Python, Java, and Scala, Mina has extensive expertise in Google Cloud Dataflow and holds certifications including AWS Certified Solutions Architect — Associate (SAA), Databricks Certified Data Engineer Professional, Certified Secure Software Engineer Associate, and Google Cloud Professional Data Engineer. Additionally, Mina possesses a solid background in machine learning.</p><h4>About <a href="https://www.linkedin.com/in/s-c-n/">Samuel Njoku</a></h4><p>Samuel Njoku is a visionary manager, cloud expert, and AI enthusiast with a proven track record of leading high-performance distributed teams. He excels in delivering cutting-edge cloud solutions and managing large-scale infrastructure systems. Samuel is passionate about leveraging data and technology to drive innovation and growth. He is skilled in building and motivating teams, as well as providing mentorship and guidance. Samuel holds various certifications, including Professional Cloud Architect, Professional Cloud Security Engineer, Professional Cloud Database Engineer, and Professional Machine Learning Engineer. He is also an active member of the tech community and frequently speaks at industry events.</p><h4>About <a href="https://www.linkedin.com/in/rv1448/">Rahul Vangala</a></h4><p>Rahul Vangala is a Senior Data Engineer at SADA, An Insight company, with over 10 years of experience in designing and implementing analytical warehouses and data platforms. With deep expertise in managing high-volume data and applying advanced tuning techniques for SQL and SPARK. He has primarily worked in the banking, healthcare, and digital marketing domains. He is skilled in Python, Scala, Java, Google Cloud Dataproc, Google Cloud Dataflow, Google Cloud Composer, and BigQuery, and holds a Google Cloud Professional Data Engineer certification.</p><h4><strong>About SADA</strong></h4><p>SADA, An Insight company, is a global cloud consulting and professional services market leader providing solutions powered by Google Cloud. Since 2000, we’ve been transforming organizations across various sectors — healthcare, finance, media, entertainment, retail, manufacturing, and the public sector — by addressing their most complex challenges. In addition to being Google’s leading partner for generative AI solutions, SADA’s expertise includes Infrastructure &amp; App Modernization, Cloud Security, Data Analytics, Productivity &amp; Collaboration, and Location Intelligence.</p><p>Rely on our proven delivery methods, honed over thousands of successful engagements, to accelerate your momentum and realize value faster than ever. With 3000+ completed projects in the Google technology ecosystem and 500+ Google Professional Certifications, we guarantee lasting results. When it comes to Google Cloud, we’re not an add-on, we’re a must-have, driving the business performance of our clients with its power. Visit <a href="https://sada.com/">SADA.com</a> to learn more.</p><p>If you’re interested in becoming a part of the SADA team, please visit our <a href="https://sada.com/careers-cloud-computing-business-solutions">careers page</a>.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=5cd9e68e73bf" width="1" height="1" alt=""><hr><p><a href="https://engineering.sada.com/bridging-the-gap-validating-data-across-data-warehouses-with-data-validation-testing-5cd9e68e73bf">Bridging the gap: validating data across data warehouses with data validation testing</a> was originally published in <a href="https://engineering.sada.com/">The SADA Engineering Blog</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Introduction]]></title>
            <link>https://engineering.sada.com/introduction-4f1cb55a8908?source=rss-6bc96cd41286------2</link>
            <guid isPermaLink="false">https://medium.com/p/4f1cb55a8908</guid>
            <category><![CDATA[google-cloud-platform]]></category>
            <category><![CDATA[kubernetes]]></category>
            <category><![CDATA[gke]]></category>
            <category><![CDATA[infrastructure]]></category>
            <category><![CDATA[kubernetes-engine]]></category>
            <dc:creator><![CDATA[SADA, An Insight Company]]></dc:creator>
            <pubDate>Wed, 21 Feb 2024 14:17:11 GMT</pubDate>
            <atom:updated>2024-02-22T22:32:50.112Z</atom:updated>
            <content:encoded><![CDATA[<h3>A guide to setting up GKE multi-cluster Gateway</h3><p>Author: Vinita Joshi, Senior Cloud Infrastructure Engineer, SADA</p><h3>Introduction</h3><p>In Google Kubernetes Engine (GKE), the Ingress feature serves to expose services outside the cluster, establishing a layer 7 load balancer. However, traditional Ingress implementations pose certain limitations. Specifically, they mandate that Ingress and its associated services reside in the same namespace. Additionally, all routes and rules are confined within a single YAML file, often leading to complexity when managing multiple rules and annotations. This complexity further escalates when multiple teams share the Ingress resource, risking route disruptions caused by changes made by individual teams.</p><p>The GKE Gateway addresses these challenges effectively. It enables cross-namespace routing, allowing the Gateway to exist in its dedicated namespace — a manageable space for platform teams. Application teams can independently manage their routes and share the Gateway. The GKE Gateway is Google Kubernetes Engine’s implementation of the Kubernetes Gateway API for Cloud Load Balancing. The Kubernetes Gateway API, overseen by <a href="https://github.com/kubernetes/community/tree/master/sig-network">SIG Network</a>, is an open-source project comprising diverse resources that model service networking within Kubernetes.</p><p>There are two versions of GKE Gateway controller:</p><ul><li>Single cluster — manages single-cluster Gateways for a single GKE cluster.</li><li>Multi-cluster — manages multi-cluster Gateways for one or more GKE clusters in different regions.</li></ul><p>I will cover the multi-cluster Gateway setup as part of this blog.</p><h3>What is GKE Gateway?</h3><p>The GKE Gateway serves as a resource defining an access point. Each GKE Gateway is linked to a Gateway Class, specifying the type of Gateway Controller. Within GKE, various gateway classes facilitate provisioning under GCP load balancers:</p><ul><li>Global External L7 Load Balancer</li><li>Regional External L7 Load Balancer</li><li>Regional Internal L7 Load Balancer</li><li>Global External L7 Load Balancer for multiple clusters</li></ul><p>This article focuses on setting up Global External L7 Load Balancer for multiple clusters using GKE Gateway. In this example, we’ll set up an external, multi-cluster Gateway to perform load balancing across two GKE clusters, specifically for internet traffic.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/970/0*qGgqAN_-_zw2-beO.png" /></figure><h3>Setting up the environment</h3><ul><li>Deploy two GKE clusters, ‘gke-us-east-1’ in the ‘us-east-1’ region and ‘gke-us-west-1’ in the ‘us-west-1’ region. Register both clusters to the project’s fleet and ensure that workload identity is enabled for each.</li><li>Enable multi-cluster services in the project’s fleet.</li></ul><pre>gcloud container fleet muti-cluster-services enable — project=&lt;project_name&gt;</pre><ul><li>Enable the multi-cluster Gateway controller on the config cluster. The config cluster is the GKE cluster in which the Gateway and Route resources are deployed. It is a central place that controls routing across the clusters. In this setup, ‘gke-us-east-1’ is considered a config cluster.</li></ul><pre>gcloud container fleet ingress enable --config-membership=gke-us-east-1 --project=&lt;project_name&gt;</pre><ul><li>Give the network viewer role required for multi-cluster-service.</li></ul><pre>resource &quot;google_project_iam_member&quot; &quot;network_viewer_role_mcs&quot; {<br>  project = &quot;&lt;project-name&gt;&quot;<br>  role    = &quot;roles/compute.networkViewer&quot;<br>  member  = &quot;serviceAccount:&lt;project-name&gt;.svc.id.goog[gke-mcs/gke-mcs-importer]&quot;<br>}</pre><ul><li>Grant the Container Admin role to the GCP Multicluster Ingress service account.</li></ul><pre>resource &quot;google_project_iam_member&quot; &quot;container_admin_role_mcs&quot; {<br>  project = &quot;&lt;project-name&gt;&quot;<br>  role    = &quot;roles/container.admin&quot;<br>  member  = &quot;serviceAccount:service-&lt;project-number&gt;@gcp-sa-multiclusteringress.iam.gserviceaccount.com&quot;<br>}</pre><h3>Setting up the demo application</h3><p>Deploy the demo application in both GKE clusters, ‘gke-us-east-1’ and ‘gke-us-west-1’, using the provided manifests.</p><pre>kind: Namespace<br>apiVersion: v1<br>metadata:<br>  name: demo<br>---<br>apiVersion: apps/v1<br>kind: Deployment<br>metadata:<br>  name: demo<br>  namespace: demo<br>spec:<br>  replicas: 2<br>  selector:<br>    matchLabels:<br>      app: demo<br>      version: v1<br>  template:<br>    metadata:<br>      labels:<br>        app: demo<br>        version: v1<br>    spec:<br>      containers:<br>      - name: whereami<br>        image: us-docker.pkg.dev/google-samples/containers/gke/whereami:v1.2.20<br>        ports:<br>          - containerPort: 8080</pre><h3>Configuring services</h3><p>With the demo application now running across both clusters, we’ll expose and export the applications by deploying Services and ServiceExports to each cluster. ServiceExport is a custom resource which is mapped to a Kubernetes Service. It exports the endpoints of that Service to all clusters registered to the fleet.</p><p>Another custom resource, ServiceImport, is automatically generated by the multi-cluster Service controller for each ServiceExport resource. Multi-cluster Gateways utilize ServiceImports as logical identifiers for a Service.</p><p>Deploy below Service and ServiceExport resources in ‘gke-us-east-1’.</p><pre>apiVersion: v1<br>kind: Service<br>metadata:<br>  name: demo<br>  namespace: demo<br>spec:<br>  selector:<br>    app: demo<br>  ports:<br>  - port: 8080<br>    targetPort: 8080<br>---<br>kind: ServiceExport<br>apiVersion: net.gke.io/v1<br>metadata:<br>  name: demo<br>  namespace: demo<br>---<br>apiVersion: v1<br>kind: Service<br>metadata:<br>  name: demo-east-1<br>  namespace: demo<br>spec:<br>  selector:<br>    app: demo<br>  ports:<br>  - port: 8080<br>    targetPort: 8080<br>---<br>kind: ServiceExport<br>apiVersion: net.gke.io/v1<br>metadata:<br>  name: demo-east-1<br>  namespace: demo</pre><p>Deploy below Service and ServiceExport resources in ‘gke-us-west -1<strong><em>’</em></strong>.</p><pre>apiVersion: v1<br>kind: Service<br>metadata:<br>  name: demo<br>  namespace: demo<br>spec:<br>  selector:<br>    app: demo<br>  ports:<br>  - port: 8080<br>    targetPort: 8080<br>---<br>kind: ServiceExport<br>apiVersion: net.gke.io/v1<br>metadata:<br>  name: demo<br>  namespace: demo<br>---<br>apiVersion: v1<br>kind: Service<br>metadata:<br>  name: demo-west-1<br>  namespace: demo<br>spec:<br>  selector:<br>    app: demo<br>  ports:<br>  - port: 8080<br>    targetPort: 8080<br>---<br>kind: ServiceExport<br>apiVersion: net.gke.io/v1<br>metadata:<br>  name: demo-west-1<br>  namespace: demo</pre><p>After deploying Services and ServiceExports, verify that ServiceImports are created for each cluster.</p><pre>kubectl get serviceimports —-namespace demo</pre><p>The expected output should resemble to</p><pre># gke-us-west-1<br>NAME        AGE<br>demo        1m30s<br>demo-west-1 1m30s<br># gke-us-east-1<br>NAME        AGE<br>demo        1m30s<br>demo-east-1 1m30s</pre><h3>Configuring Gateways and HTTPRoute</h3><p>Once the application and services are deployed, we can configure Gateway and HTTPRoute in the config cluster. To distribute traffic across both clusters, we’ll utilize the Gateway Class ‘gke-l7-global-external-managed-mc’ for creating an External Load Balancer.</p><pre>kind: Gateway<br>apiVersion: gateway.networking.k8s.io/v1beta1<br>metadata:<br>  name: external-http<br>  namespace: demo<br>spec:<br>  gatewayClassName: gke-l7-global-external-managed-mc<br>  listeners:<br>  - name: http<br>    protocol: HTTP<br>    port: 80<br>    allowedRoutes:<br>      kinds:<br>      - kind: HTTPRoute</pre><p>Deploying the Gateway will initiate the creation of a Load Balancer, an external IP address, a forwarding rule, two backend services, and a health check.</p><p>Deploy the following HTTPRoute resource in the config cluster (gke-us-east-1).</p><pre>kind: HTTPRoute<br>apiVersion: gateway.networking.k8s.io/v1beta1<br>metadata:<br>  name: public-demo-route<br>  namespace: demo<br>  labels:<br>    gateway: external-http<br>spec:<br>  hostnames:<br>  - &quot;demo.example.com&quot;<br>  parentRefs:<br>  - name: external-http<br>  rules:<br>  - matches:<br>    - path:<br>        type: PathPrefix<br>        value: /west<br>    backendRefs:<br>    - group: net.gke.io<br>      kind: ServiceImport<br>      name: demo-west-1<br>      port: 8080<br>  - matches:<br>    - path:<br>        type: PathPrefix<br>        value: /east<br>    backendRefs:<br>      - group: net.gke.io<br>        kind: ServiceImport<br>        name: demo-east-1<br>        port: 8080<br>  - backendRefs:<br>    - group: net.gke.io<br>      kind: ServiceImport<br>      name: demo<br>      port: 8080</pre><p>When the HTTPRoute is deployed, requests to /west are directed to pods running in the ‘gke-us-west-1’ cluster, requests to /east are directed to pods running in the ‘gke-us-east-1’ cluster, and any other paths are routed to pods in either cluster based on their health, capacity, and proximity to the requesting client.</p><p>If the traffic is sent to the root path of the domain the load balancer will send traffic to the closest region.</p><pre>curl -H “host:demo.example.com” http://&lt;EXTRENAL_IP_ADDRESS__OF_GATEWAY&gt;</pre><pre>The sample output shows that the request was served from the pod running in gke-us-west-1 cluster.</pre><pre>{<br>“cluster_name” : “gke-us-west-1”,<br>“zone” : “us-west1-a”,<br>“host_header” : “demo.example.com”,<br>“node_name”   : “gke-gke-us-west-1-default-pool-2367899-5y53.c.&lt;project-id&gt;.internal”,<br>“project_id” : “&lt;project-id&gt;”,<br>“timestamp”  : “2024-01-30T14:23:14”,<br>}</pre><p>If the traffic is sent to /east path then the request is served from the pod running in gke-us-east-1 cluster.</p><pre>curl -H “host:demo.example.com” http://&lt;EXTRENAL_IP_ADDRESS_OF_GATEWAY&gt;/east</pre><pre>The sample output shows that the request was served from the pod running in gke-us-east-1 cluster.</pre><pre>{<br>“cluster_name” : “gke-us-east-1”,<br>“zone” : “us-east1-b”,<br>“host_header” : “demo.example.com”,<br>“node_name”   : “gke-gke-us-east-1-default-pool-678899-5p93.c.&lt;project-id&gt;.internal”,<br>“project_id” : “&lt;project-id&gt;”,<br>“timestamp”  : “2024-01-30T14:25:14”,<br>}</pre><p>If the traffic is sent to /west path then the request is served from the pod running in gke-us-west-1 cluster.</p><pre>curl -H “host:demo.example.com” http://&lt;EXTRENAL_IP_ADDRESS_OF_GATEWAY&gt;/west</pre><pre>The sample output shows that the request was served from the pod running in gke-us-west-1 cluster.</pre><pre>{<br>“cluster_name” : “gke-us-west-1”,<br>“zone” : “us-west1-a”,<br>“host_header” : “demo.example.com”,<br>“node_name”   : “gke-gke-us-west-1-default-pool-2367899-5y53.c.&lt;project-id&gt;.internal”,<br>“project_id” : “&lt;project-id&gt;”,<br>“timestamp”  : “2024-01-30T14:26:14”,<br>}</pre><h3>Securing the Gateway</h3><p>If you’re deploying the Gateway in a production environment, make sure that the Gateway can be secured by applying Gateway policies.</p><ul><li>For the domain ‘demo.example.com,’ create an SSLCertificate resource and include it in the Gateway configuration as outlined below:</li></ul><pre>kind: Gateway<br>apiVersion: gateway.networking.k8s.io/v1beta1<br>metadata:<br>  name: external-http<br>  namespace: demo<br>spec:<br>  gatewayClassName: gke-l7-global-external-managed-mc<br>  listeners:<br>  - name: https<br>    protocol: HTTPS<br>    port: 443<br>    tls:<br>      mode: Terminate<br>      options:<br>        networking.gke.io/pre-shared-certs: demo-example-com<br>    allowedRoutes:<br>      kinds:<br>      - kind: HTTPRoute</pre><ul><li>Google Cloud Armor security policies protect the application from web-based attacks. These policies, composed of rules filtering traffic based on conditions like incoming request parameters such as IP address, IP range, region code, or request headers, provide enhanced protection. Once implemented, GCPBackendPolicy can be applied to Kubernetes Services.</li></ul><pre>apiVersion: networking.gke.io/v1<br>kind: GCPBackendPolicy<br>metadata:<br>  name: demo<br>  namespace: demo<br>spec:<br>  default:<br>    securityPolicy: cloud-armor-policy<br>  targetRef:<br>    group: &quot;&quot;<br>    kind: ServiceImport<br>    name: demo<br>    namespace: demo</pre><h3>Conclusion</h3><p>The GKE Gateway represents a powerful evolution beyond traditional Ingress. Its role-oriented design surpasses the constraints of conventional Ingress, offering versatile application across namespaces. This article aimed to provide a clear understanding of configuring the GKE gateway for multiple clusters.</p><h3>References</h3><ul><li><a href="https://cloud.google.com/kubernetes-engine/docs/how-to/deploying-multi-cluster-gateways#deploy-gateway">https://cloud.google.com/kubernetes-engine/docs/how-to/deploying-multi-cluster-gateways#deploy-gateway</a></li><li><a href="https://gateway-api.sigs.k8s.io/">https://gateway-api.sigs.k8s.io/</a></li><li><a href="https://cloud.google.com/kubernetes-engine/docs/how-to/secure-gateway">https://cloud.google.com/kubernetes-engine/docs/how-to/secure-gateway</a></li></ul><h3>About Vinita Joshi</h3><p>Vinita, a Senior Cloud Infrastructure Engineer, thrives on crafting solutions on the Google Cloud Platform, boasting extensive expertise in Kubernetes.</p><h3>About SADA, An Insight company</h3><p>SADA is a professional services market leader and solutions provider of Google Cloud. Since 2000, SADA has helped organizations of every size in healthcare, media, entertainment, retail, manufacturing, and the public sector solve their most complex digital transformation challenges. With offices in North America, India, the UK, and Armenia providing sales, customer support, and professional services, SADA has become Google’s leading partner for generative AI solutions. SADA’s expertise also includes Infrastructure Modernization, Cloud Security, and Data Analytics. A 6x Google Cloud Partner of the Year award winner with 10 Google Cloud Specializations, SADA was recognized as a Niche Player in the 2023 Gartner® Magic Quadrant™ for Public Cloud IT Transformation Services. SADA is a 15x honoree of the Inc. 5000 list of America’s Fastest-Growing Private Companies and has been named to Inc. Magazine’s Best Workplaces four years in a row. Learn more at <a href="http://www.sada.com/">www.sada.com</a>.</p><p>If you’re interested in becoming a part of the SADA team, please <a href="https://sada.com/careers-cloud-computing-business-solutions">visit our careers page</a>.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=4f1cb55a8908" width="1" height="1" alt=""><hr><p><a href="https://engineering.sada.com/introduction-4f1cb55a8908">Introduction</a> was originally published in <a href="https://engineering.sada.com/">The SADA Engineering Blog</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Develop a cloud-agnostic app to connect Amazon S3 and Google Cloud Storage buckets]]></title>
            <link>https://engineering.sada.com/develop-a-cloud-agnostic-app-to-connect-amazon-s3-and-google-cloud-storage-buckets-4ca9bb7f66ee?source=rss-6bc96cd41286------2</link>
            <guid isPermaLink="false">https://medium.com/p/4ca9bb7f66ee</guid>
            <category><![CDATA[google-cloud-storage]]></category>
            <category><![CDATA[google-cloud-platform]]></category>
            <category><![CDATA[cloud-agnostic]]></category>
            <category><![CDATA[amazon-s3]]></category>
            <category><![CDATA[multi-cloud]]></category>
            <dc:creator><![CDATA[SADA, An Insight Company]]></dc:creator>
            <pubDate>Wed, 17 Jan 2024 14:02:19 GMT</pubDate>
            <atom:updated>2024-01-17T14:02:19.405Z</atom:updated>
            <content:encoded><![CDATA[<p><em>By Kumaraswami Muthuswami, SADA Senior Cloud Infrastructure Engineer</em></p><p>It’s become increasingly common to transition between cloud platforms such as GCP, AWS, and Azure. These shifts are driven by various factors including cost considerations, performance optimization, and the unique features offered by each cloud provider.</p><p>When companies switch from one cloud provider to another, they’re able to leverage numerous cloud-agnostic tools and technologies in the market that assist in migrating infrastructure, data, and CI/CD pipelines seamlessly. However, application migration presents a more complex challenge. It often entails either a complete app refactoring or redesign due to varying dependencies or API discrepancies between cloud platforms.</p><p>There are various approaches to tackle this challenge, such as utilizing <a href="https://docs.spring.io/spring-boot/docs/current/api/org/springframework/boot/autoconfigure/condition/ConditionalOnBean.html">conditional beans in Spring Boot</a>, a method widely employed in such scenarios. However, I’ll illustrate an exceptionally effective method we recently employed for a client. They sought to migrate their AWS-based app to GCP while simultaneously maintaining cross-cloud access (hybrid mode) until the transition was complete.</p><p>The strategy involved implementing a common framework that dynamically determines dependencies based on the cloud platform where the app is deployed. This framework enables the creation of a cloud-agnostic app capable of running seamlessly in both cloud environments, allowing access to resources across clouds in hybrid mode. To achieve this, we segregated the implementation into a separate project, which functions as a library integrated into the microservices. This library intelligently determines the required cloud resources based on environment configurations at runtime.</p><p>Let’s dive into the details:</p><h4>High-level diagram</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*0HNMhdP-YFJ92sZh" /></figure><p>Design and implementation</p><h4>Common library</h4><p>We’ll create a common library for microservices to ensure uniform implementations and streamlined dependency management. Its purpose is to establish a consistent framework across all microservices. This library:</p><ul><li>Inherits crucial Spring Boot dependencies to maintain consistency.</li><li>Manages dependencies for microservices, enabling centralized updates from a single location when necessary.</li><li>Encompasses standard beans, utilities, and endpoints essential across all microservices.</li><li>Contains optional utilities that can be utilized based on specific needs.</li><li>Offers common configuration classes to enforce uniform and standardized implementations.</li></ul><h4>Design pattern</h4><p>The solution is designed using the factory design pattern.</p><p>The factory design pattern advocates coding for interfaces rather than implementations. By abstracting the instantiation of actual implementation classes from client code, this pattern enhances the robustness of our codebase, reduces coupling, and simplifies extensibility.</p><h4>Tech stack</h4><ol><li>Java</li><li>Spring Boot</li><li>Google Cloud Storage (GCS)</li><li>Amazon Simple Storage Service (Amazon S3)</li><li>Maven</li></ol><h4>Implementation</h4><p>The solution involves crafting a generic BucketService interface within the common library, featuring two distinct implementations: one utilizing the Amazon S3 client and the other employing the GCS client. To facilitate this, a builder was developed to accept a newly created BucketConfig, a typesafe configuration, and dynamically output the required BucketService implementation at runtime.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*_XCAoXYaI7TWIqsT" /></figure><pre>@Bean<br>  public BucketService bucketService() {<br>    return new BucketServiceBuilder(typesafeConfig.getBucket()).build();<br>  }</pre><p>The implementation of BucketService incorporates the following functionalities to meet these requirements:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*PQFztkpqKxBnNzPJ9wYcog.png" /></figure><p>In this example, the application utilizes TypesafeConfig to manage environment variables. Within TypesafeConfig, the application incorporates a POJO (Plain Old Java Object) class named BucketConfig.java. This class includes various fields such as bucketName, keyPrefix, key, gcpProjectId, and awsRegion to encapsulate different configuration details.</p><p>The BucketConfig class serves to hold specific values based on the designated cloud environment. For instance, the Typesafe configuration file will contain the awsRegion detail for AWS deployment and the gcpProjectId for GCP deployment.</p><p>When the application passes the BucketConfig object to the common-lib, the BucketService, implemented within the common-lib, retrieves the values from the BucketConfig. Subsequently, based on these values, the BucketService dynamically constructs the connection object. For example, if the bucketConfig contains the gcpProjectId, the BucketService constructs the GCS (Google Cloud Storage) bucket object. Conversely, if the awsRegion is present in the bucketConfig, it builds the S3 bucket object accordingly.</p><h4>Code snippet</h4><p>TypeSafeConfig</p><pre>public class BucketConfig {<br> <br>  /* The name of the bucket to use */<br>  private String bucketName;<br><br>  /* The prefix applied to all objects, often used as a &quot;folder&quot; */<br>  @Optional<br>  private String keyPrefix;<br><br>  /* The key for the object in the bucket */<br>  @Optional<br>  private String key;<br><br>  /* The GCP Project ID for the gcs client */<br>  @Optional<br>  private String gcpProjectId;<br><br>  /* The AWS region for the S3 client */<br>  @Optional<br>  private String awsRegion;</pre><p>Bean Creation</p><pre> @Bean<br>    public BucketService bucketService() {<br>      return new BucketServiceBuilder(typesafeConfig.getBucket()).build();<br>    }</pre><p>BuckerServiceBuilder</p><pre>public BucketService build() {<br>    if (!Strings.isNullOrEmpty(bucketConfig.getGcpProjectId())) {<br>      Storage storage = StorageOptions.newBuilder().setProjectId(bucketConfig.getGcpProjectId()).build().getService();<br>      return new GCSBucketService(storage);<br>    } else {<br>      AmazonS3 amazonS3 = AmazonS3ClientBuilder.standard().withRegion(bucketConfig.getAwsRegion()).build();<br>      return new S3BucketService(amazonS3);<br>    }<br>  }</pre><p>BucketService</p><pre>public interface BucketService {<br>/**<br>   * Puts an object into the given bucket, within the given &quot;folder&quot; with the<br>   * given content value.<br>   * <br>   * @param bucketName the name of the bucket<br>   * @param folder     the name of the &quot;folder&quot; to add the object to<br>   * @param file       the name of the object in the &quot;folder&quot;<br>   * @param content    the content of the object as {@link String}<br>   */<br>  void putObject(String bucketName, String folder, String file, String content);<br><br>/**<br>   * Lists all objects within the given bucket.<br>   * <br>   * @param bucketName the name of the bucket<br>   * <br>   * @return a List of Strings with the object keys in the bucket<br>   */<br>  List&lt;String&gt; listObjects(String bucketName);<br><br>}</pre><p>GCSBucketService</p><pre>public class GCSBucketService implements BucketService {<br><br>  private Storage storage;<br><br>  final static private String GCS_DOMAIN = &quot;storage.googleapis.com&quot;;<br><br>  public GCSBucketService(Storage storage) {<br>    this.storage = storage;<br>  }<br><br>@Override<br>  public void putObject(String bucketName, String key, InputStream content, String contentType) {<br>    try {<br>      BlobInfo blobInfo = BlobInfo.newBuilder(BlobId.of(bucketName, key))<br>          .setContentType(contentType)<br>          .setCacheControl(&quot;no-cache&quot;)<br>          .build();<br><br>      storage.createFrom(blobInfo, content);<br>    } catch (IOException e) {<br>      log.error(&quot;Error putting object to GCS bucket: &quot; + bucketName + &quot; at key: &quot; + key, e);<br>    }<br>  }<br><br>@Override<br>  public List&lt;String&gt; listObjects(String bucketName) {<br>    List&lt;String&gt; keys = new ArrayList&lt;&gt;();<br>    Page&lt;Blob&gt; blobs = storage.list(bucketName);<br><br>    for (Blob blob : blobs.iterateAll()) {<br>      keys.add(blob.getName());<br>    }<br><br>    return keys;<br>  }<br>}</pre><p>S3BucketService</p><pre>public class S3BucketService implements BucketService {<br><br>  private AmazonS3 amazonS3;<br><br>  public S3BucketService(AmazonS3 amazonS3) {<br>    this.amazonS3 = amazonS3;<br>  }<br><br>@Override<br>  public void putObject(String bucketName, String folder, String file, String content) {<br>    putObject(bucketName, getKey(folder, file), content);<br>  }<br><br>@Override<br>  public List&lt;String&gt; listObjects(String bucketName) {<br>    List&lt;String&gt; keys = new ArrayList&lt;&gt;();<br>    ObjectListing objectList = amazonS3.listObjects(bucketName);<br><br>    do {<br>      for (S3ObjectSummary objectSummary : objectList.getObjectSummaries()) {<br>        keys.add(objectSummary.getKey());<br>      }<br><br>      objectList = amazonS3.listNextBatchOfObjects(objectList);<br>    } while (objectList.isTruncated());<br><br>    return keys;<br>  }</pre><p>Environment Config: S3 Bucket</p><pre>bucketConfig {<br>  bucketName: &lt;BUCKET_NAME&gt;<br>  gcpProject: &quot;&quot;<br>  key: &lt;BUCKET_NAME&gt;<br>  awsRegion: &lt;REGION&gt;<br>}</pre><p>Environment Config: GCS Bucket</p><pre>bucketConfig {<br>  bucketName: &lt;BUCKET_NAME&gt;<br>  gcpProject: &lt;GCP_BUCKET_PROJECT_ID&gt;<br>  key: &lt;BUCKET_NAME&gt;<br>  awsRegion: &quot;&quot;<br>}</pre><h3>Advantages</h3><ol><li>Reusability: The solution’s components can be utilized across various contexts and applications.</li><li>Easy maintenance: Components are designed in a way that allows for straightforward upkeep and updates.</li><li>Configurability: The solution can be easily configured and adapted to different environments or requirements.</li><li>Streamlined app migration: Simplifies the process of migrating applications with reduced coding efforts.</li><li>Cloud agnosticism: Enables operation across multiple cloud platforms without being tied to any specific provider.</li><li>Hybrid mode compatibility: Supports running applications in a hybrid mode, accessing resources across cloud environments.</li><li>Multiple cloud provider support: Offers the flexibility to incorporate implementations for multiple cloud providers as needed.</li><li>Loosely coupled</li></ol><h4>Conclusion</h4><p>As mentioned earlier, numerous methods exist for establishing connections to cross-cloud resources at runtime. However, this particular approach stands out due to its ease of implementation, maintenance, seamless integration with applications requiring minimal code alterations, and effortless switching between various cloud resources. Should you have any questions or comments, please feel free to share them. Your feedback is greatly appreciated.</p><p><strong>About Kumaraswami</strong></p><p>With over 17 years of experience as a Senior Engineer, Kumaraswami possesses extensive expertise in designing and implementing applications using Java/J2EE and Python. He is adept at crafting UI applications through Angular and React, establishing robust data pipelines, and architecting Google Cloud infrastructure.</p><p>Kumaraswami excels in orchestrating seamless migrations from on-premises systems to Google Cloud Platform (GCP), empowering organizations to leverage cloud-based solutions effectively.</p><p>His proficiency spans Java/J2EE, Python, Ruby, Groovy, Angular, React, Terraform, Chef, Helm, Docker, Jenkins, GitCI, Kubernetes, Apache Kafka, SQL, and cloud-native technologies. He particularly specializes in harnessing Google Cloud tools such as GKE, App Engine, CloudRun, Cloud Functions, DataFlow, Cloud Build, Secret Manager, VPC Service Control, IAM, GAR, VPC Service Control, BigQuery, Cloud SQL, GCS, and Pub/Sub.</p><h4><strong>About SADA, An Insight company</strong></h4><p>SADA, An Insight company, is a global cloud consulting and professional services market leader providing solutions powered by Google Cloud. Since 2000, SADA has helped organizations in healthcare, media, entertainment, retail, manufacturing, and the public sector solve their most complex digital transformation challenges. SADA has offices in North America, India, the UK, and Armenia, providing sales, customer support, and professional services. In addition to being Google’s leading partner for generative AI solutions, SADA’s expertise includes Infrastructure Modernization, Cloud Security, and Data Analytics. A 6x Google Cloud Partner of the Year award winner with 10 Google Cloud Specializations, SADA was recognized as a Niche Player in the 2023 Gartner® Magic Quadrant™ for Public Cloud IT Transformation Services. SADA is a 15x honoree of the <em>Inc. 5000</em> list of America’s Fastest-Growing Private Companies and has been named to <em>Inc</em>.<em> Magazine</em>’s Best Workplaces four years in a row. Learn more: <a href="http://www.sada.com/">www.sada.com</a>.</p><p>If you’re interested in becoming a part of the SADA team, please <a href="https://sada.com/careers-cloud-computing-business-solutions">visit our careers page</a>.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=4ca9bb7f66ee" width="1" height="1" alt=""><hr><p><a href="https://engineering.sada.com/develop-a-cloud-agnostic-app-to-connect-amazon-s3-and-google-cloud-storage-buckets-4ca9bb7f66ee">Develop a cloud-agnostic app to connect Amazon S3 and Google Cloud Storage buckets</a> was originally published in <a href="https://engineering.sada.com/">The SADA Engineering Blog</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Unleashing data insights with Google’s Duet AI, zero code required]]></title>
            <link>https://engineering.sada.com/unleashing-data-insights-with-googles-duet-ai-zero-code-required-6cfef39a220d?source=rss-6bc96cd41286------2</link>
            <guid isPermaLink="false">https://medium.com/p/6cfef39a220d</guid>
            <category><![CDATA[google-duet-ai]]></category>
            <category><![CDATA[duet-ai]]></category>
            <category><![CDATA[ai]]></category>
            <category><![CDATA[google-cloud-platform]]></category>
            <category><![CDATA[duet]]></category>
            <dc:creator><![CDATA[SADA, An Insight Company]]></dc:creator>
            <pubDate>Wed, 03 Jan 2024 14:01:33 GMT</pubDate>
            <atom:updated>2024-01-03T16:57:49.604Z</atom:updated>
            <content:encoded><![CDATA[<p>By Simon Margolis, SADA Associate CTO, AI &amp; ML</p><p>Google’s Duet AI can help a lot of people accomplish a diverse set of tasks, ranging from email generation to code assistance to troubleshooting cloud architectures. In this blog, we’ll walk through the process of leveraging a few different versions of Duet AI in order to help us gain novel insights from a large dataset without writing any code. You’ll see how we can explore the data, answer our questions, and visualize our results, all without any data engineering experience required!</p><h4>Getting grounded: understanding our dataset</h4><p>The first thing we need to do is access our dataset in Google BigQuery Studio. There are numerous Duet AI integrations within the Studio, but for now we’re going to work with the “Codey” model to help us better understand our data.</p><p>For the purposes of this example, we’ll work with the public dataset available in BigQuery called bigquery-public-data.austin_311.311_service_requests. This dataset, as the name suggests, contains information about 311 phone calls in the city of Austin, Texas. As the schema (Figure 1–1) shows, the data includes information such as the type of service request made to the city, the time it took place, and in some cases, the geographical location of the request.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*NsD981hJCWrslP3X" /><figcaption>Figure 1–1 The schema of the data we’re working with</figcaption></figure><p>The first thing we’ll do is get an understanding of what our data actually contains. For example, let’s learn how many records contain geolocation data. To do this, we’ll simply ask Duet AI in English to give us the relevant SQL code for the table we’re looking at.</p><p><em>Prompt:</em></p><p>How many records in the table bigquery-public-data.austin_311.311_service_requests contain geolocation data?</p><p>Duet SQL:</p><pre>SELECT<br>CAST(count(*) as BIGNUMERIC)<br>FROM<br>`bigquery-public-data.austin_311.311_service_requests`<br>WHERE<br>ST_GEOGPOINT(longitude, latitude) IS NOT NULL</pre><p>This SQL returns the number 1710986 which tells us that we have over 1.5 million records with geolocation information. We may want to look at where service requests are coming from, and possibly understand trends regarding the geolocation of the request and the type of request. Let’s use a few more prompts to gain a better understanding of the data.</p><p><em>Prompt:</em></p><p>In the table bigquery-public-data.austin_311.311_service_requests, what are the top 4 most common complaint descriptions?</p><p>Duet SQL:</p><pre>SELECT<br> complaint_description,<br> COUNT(*) AS num_complaints<br>FROM<br> `bigquery-public-data.austin_311.311_service_requests`<br>GROUP BY<br> complaint_description<br>ORDER BY<br> num_complaints DESC<br>LIMIT<br> 4</pre><p>Result:</p><pre>[{<br> &quot;complaint_description&quot;: &quot;Austin Code - Request Code Officer&quot;,<br> &quot;num_complaints&quot;: &quot;213566&quot;<br>}, {<br> &quot;complaint_description&quot;: &quot;Traffic Signal - Maintenance&quot;,<br> &quot;num_complaints&quot;: &quot;98681&quot;<br>}, {<br> &quot;complaint_description&quot;: &quot;Loose Dog&quot;,<br> &quot;num_complaints&quot;: &quot;80168&quot;<br>}, {<br> &quot;complaint_description&quot;: &quot;ARR Missed Recycling&quot;,<br> &quot;num_complaints&quot;: &quot;55380&quot;<br>}]</pre><p>This is already showing me some interesting trends. I’d like to dive deeper into the “Loose Dog” occurrences since that surprised me as a top 4 complaint! Let’s see if we can learn more about where these are taking place.</p><p><em>Prompt:</em></p><p>In the table bigquery-public-data.austin_311.311_service_requests, what are the top 5 locations with “Loose Dog” in the complaint description? Also include the latitude and longitude.</p><p>Duet SQL:</p><pre>SELECT<br> incident_address,<br> latitude,<br> longitude,<br> COUNT(*) AS frequency<br>FROM<br> `bigquery-public-data.austin_311.311_service_requests`<br>WHERE<br> complaint_description LIKE &#39;%Loose Dog%&#39;<br>GROUP BY<br> incident_address,<br> latitude,<br> longitude<br>ORDER BY<br> frequency DESC<br>LIMIT<br> 5;<br></pre><p>Result:</p><pre>[{<br> &quot;incident_address&quot;: &quot;11716 GEMMER ST, AUSTIN, TX 78617&quot;,<br> &quot;latitude&quot;: &quot;30.15911499&quot;,<br> &quot;longitude&quot;: &quot;-97.64598773&quot;,<br> &quot;frequency&quot;: &quot;217&quot;<br>}, {<br> &quot;incident_address&quot;: &quot;11716 GEMMER ST, DEL VALLE, TX 78617&quot;,<br> &quot;latitude&quot;: &quot;30.159115&quot;,<br> &quot;longitude&quot;: &quot;-97.64598773&quot;,<br> &quot;frequency&quot;: &quot;101&quot;<br>}, {<br> &quot;incident_address&quot;: &quot;5801 AINEZ DR, AUSTIN, TX 78744&quot;,<br> &quot;latitude&quot;: &quot;30.18683439&quot;,<br> &quot;longitude&quot;: &quot;-97.73822222&quot;,<br> &quot;frequency&quot;: &quot;78&quot;<br>}, {<br> &quot;incident_address&quot;: &quot;2016 BLUEBONNET LN, AUSTIN, TX 78704&quot;,<br> &quot;latitude&quot;: &quot;30.25162171&quot;,<br> &quot;longitude&quot;: &quot;-97.77451642&quot;,<br> &quot;frequency&quot;: &quot;78&quot;<br>}, {<br> &quot;incident_address&quot;: &quot;7609 LONGVIEW RD, AUSTIN, TX 78745&quot;,<br> &quot;latitude&quot;: &quot;30.2031482&quot;,<br> &quot;longitude&quot;: &quot;-97.82694257&quot;,<br> &quot;frequency&quot;: &quot;71&quot;<br>}]</pre><p>Now I’m starting to see that there is a high concentration of these complaints in a specific location. Out-of-the-box, I can simply click “Chart” in BigQuery to see these results visualized to help me better understand them (Figure 1–2). While this is helpful, I’d like to go deeper with the data and better visualize the trends.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*hHkvShNu-MpVZq14" /><figcaption>Figure 1–2 BigQuery visualizes the results without any explicit configuration by generating what it believes are relevant charts.</figcaption></figure><p>But, before we move on, I want to make sure I’m learning as we go here. I see that the generated SQL has “complaint_description LIKE ‘%Loose Dog%’” in the WHERE clause, and I’d like clarification on its function. To be sure my results are correct, I need to know if this query is correct as well. To do this, I’ll simply ask Duet AI to explain the query to me in English (Figure 1–3) by highlighting my text, right-clicking, and selecting “explain current selection”.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/946/0*gxCdYiePGwdkqcGX" /><figcaption>Figure 1–3 Duet AI makes it easy to learn as you build.</figcaption></figure><p>This opens up a chat window with Duet AI for Google Cloud which explains the intent of the query. This feature even lets me ask follow-up questions for additional clarity, all while maintaining the context of our ongoing conversation (Figure 1–4).</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/748/0*kQlE_CPMkjugPVNV" /><figcaption>Figure 1–4 Chatting with Duet AI while working on my data helps me better understand what I’m doing and how to get the insights I’m searching for.</figcaption></figure><p>Now, armed with the confidence that my query is indeed giving me the results I want, I can move forward.</p><h4>Data science tools without a data scientist</h4><p>Duet AI has been super helpful so far in helping me better understand the data I’m working with and uncovering some interesting trends within it. Without Duet AI, this would have required either a lot of time, a lot of SQL expertise, or both. We did this today in just a few minutes with no experience with SQL or any other coding language.</p><p>While this alone provides valuable information, a professional data scientist would use a very different approach for distilling insights. A data scientist likely would be working in a proper programming language like Python, she would probably use a Python Jupyter notebook to inspect the data, and she would invoke various pre-built libraries for data science, such as pandas to better understand her data.</p><p>While that type of work comes with a steep learning curve and years of experience, we can still invoke some of the same tools without any experience in the field or related technologies by leaning on Duet AI for help.</p><p>To get started, we’ll take the results from our previous SQL queries and ask BigQuery to spin up a Python Jupyter notebook containing the specific results we’ve been analyzing. With one click of a button (Figure 2–1), we’re able to get into a notebook that has been pre-populated with all of the code required to get back to our results. All we have to do is click “play” (Figure 2–2).</p><p>Note, for the purposes of this example, I’ve expanded our result set from the top 5 locations to the top 100. Now that we have our notebook loaded, we can ask questions about our data such as:</p><p><em>Prompt:</em></p><p>Plot the results on a map using the coordinates in latitude and longitude.</p><p>Duet Python:</p><pre>import pandas as pd<br>import geopandas as gpd<br>import matplotlib.pyplot as plt<br><br><br># Read the results DataFrame<br>results = results.dropna()<br><br><br># Create a GeoDataFrame from the latitude and longitude columns<br>geometry = gpd.points_from_xy(results[&#39;longitude&#39;], results[&#39;latitude&#39;])<br>geo_df = gpd.GeoDataFrame(results, geometry=geometry)<br><br><br># Plot the GeoDataFrame on a map<br>geo_df.plot(figsize=(10, 10))<br>plt.show()</pre><p>Result:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/286/0*bY3ngYUOciB0sBjd" /></figure><p>Great! We’ve just accomplished a couple of major goals. Firstly, we used plain language to generate code which calls upon tools like pandas and matplotlib — typically utilized by seasoned data scientists. We did this without even knowing these tools existed, let alone how to use them! Second, we generated a visualization which clearly shows we have at least one outlier in our data. Let’s remove this before we move forward.</p><p>Once again, we’ll ask Duet AI for help within our notebook. We simply type the plain text request “remove the outliers from the coordinate data” and Duet AI handles the rest (Figure 2–3) with its knowledge of the data we’re working with.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*SWFyd9_sIOHR1v46" /><figcaption>Figure 2–3</figcaption></figure><p>With our data cleansed, we can ask again for a plotting of our incident locations to see a trend emerge (Figure 2–4).</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/844/0*-IWPo48zJ3ZKdhpP" /><figcaption>Figure 2–4</figcaption></figure><p>To make things even easier to read, I’ll modify our prompt to “plot the results on a map where the frequency of each point can easily be seen” and get an output (Figure 2–5) which easily lets us see where the hotspots are for loose dog calls (emphasis added).</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/521/0*lwxCdmkuvD3LUotC" /><figcaption>Figure 2–5</figcaption></figure><p>Armed with this information, we may be able to take action in the provided neighborhoods to reduce the occurrences of loose dogs. We may even go further in the future and start to visualize our data with timestamps, time of day, day of week, or other filters to better understand the trends in the data.</p><p>I hope this encourages readers of all backgrounds and experience levels to start playing with the data in their lives. The barrier to entry has never been lower for extracting meaningful insights from large sets of data, and in today’s world of cheap storage and “big data,” the opportunities for discovering new findings are vast!</p><p><strong>Learn more about Duet AI with SADA, An Insight company</strong></p><p>Don’t hesitate to reach out and <a href="https://sada.com/generative-ai/#contact-row">schedule a complimentary consultation</a> with a SADA AI expert to explore the ways you can integrate generative AI solutions like Duet AI into your organization. We’ll be happy to help you get started.</p><p><strong>About Simon Margolis</strong></p><p>Simon Margolis, a dedicated technology professional and practice leader, focuses on fostering forward-thinking strategies and empowering organizations to operate smarter and more efficiently. With 12+ years of experience in IT and cloud solutions, Simon has held many roles from engineering and solutions architecture to sales and business development. As a Google Cloud Qualified Developer, Simon helps guide SADA and its customers through the rapidly expanding AI and ML space.</p><p><strong>About SADA, An Insight company</strong></p><p>SADA is a professional services market leader and solutions provider of Google Cloud. Since 2000, SADA has helped organizations of every size in healthcare, media, entertainment, retail, manufacturing, and the public sector solve their most complex digital transformation challenges. With offices in North America, India, the UK, and Armenia providing sales, customer support, and professional services, SADA has become Google’s leading partner for generative AI solutions. SADA’s expertise also includes Infrastructure Modernization, Cloud Security, and Data Analytics. A 6x Google Cloud Partner of the Year award winner with 10 Google Cloud Specializations, SADA was recognized as a Niche Player in the 2023 Gartner® Magic Quadrant™ for Public Cloud IT Transformation Services. SADA is a 15x honoree of the Inc. 5000 list of America’s Fastest-Growing Private Companies and has been named to Inc. Magazine’s Best Workplaces four years in a row. Learn more at <a href="http://www.sada.com/">www.sada.com</a>.</p><p>If you’re interested in becoming a part of the SADA team, please <a href="https://sada.com/careers-cloud-computing-business-solutions">visit our careers page</a>.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=6cfef39a220d" width="1" height="1" alt=""><hr><p><a href="https://engineering.sada.com/unleashing-data-insights-with-googles-duet-ai-zero-code-required-6cfef39a220d">Unleashing data insights with Google’s Duet AI, zero code required</a> was originally published in <a href="https://engineering.sada.com/">The SADA Engineering Blog</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Setting up SLIs, SLOs, and monitors with Datadog]]></title>
            <link>https://engineering.sada.com/setting-up-slis-slos-and-monitors-with-datadog-f0428b36436e?source=rss-6bc96cd41286------2</link>
            <guid isPermaLink="false">https://medium.com/p/f0428b36436e</guid>
            <category><![CDATA[slo]]></category>
            <category><![CDATA[monitor]]></category>
            <category><![CDATA[google-cloud-platform]]></category>
            <category><![CDATA[datadog]]></category>
            <category><![CDATA[development]]></category>
            <dc:creator><![CDATA[SADA, An Insight Company]]></dc:creator>
            <pubDate>Wed, 13 Dec 2023 14:01:15 GMT</pubDate>
            <atom:updated>2023-12-13T15:31:40.894Z</atom:updated>
            <content:encoded><![CDATA[<p><em>By Rajeshwar Satyaditya, SADA Site Reliability Engineer</em></p><h4>Introduction</h4><p>A fictional e-commerce organization is struggling with false alarms on a subset of microservices, such as sales, cart, payment, and email. These services are hosted on a serverless platform and are connected to a PostgreSQL database. In this document, we will focus on one problematic microservice and apply the SRE (site reliability engineering) methodology. The goal is to configure actionable alerts and establish reliable service targets. As an engineer, our top priority is to implement SLOs (service level objectives), error budgets, and accurate SLIs (service level indicators) to enhance service monitoring. Let’s dive into how implementing SRE practices can make this process easier. The objective of this blog is to walk you through the concepts of SRE practices and principles, and then demonstrate how to leverage that knowledge to configure accurate monitoring.</p><h4>SRE principles</h4><p>In today’s software development, bridging the needs of development and operations teams without conflicts is crucial. Site reliability engineering is an emerging field that serves as a bridge between these worlds.</p><ul><li>Customer experiences provide insights into your service’s reliability.</li><li>In many IT organizations, development and operations teams often have conflicting priorities.</li><li>SRE involves balancing development velocity with reliability risk.</li><li>SRE offers benefits to IT teams, whether they use cloud or on-premises technology, for both large projects and daily operations.</li><li>DevOps emerged to help close gaps and break down silos between development and operations teams.</li><li>DevOps is a philosophy, not a development methodology or technology.</li><li>SRE is a practical way to implement the DevOps philosophy.</li><li>Developers prioritize feature velocity and innovation, while operators emphasize reliability and consistency.</li><li>SRE consists of both technical and cultural practices.</li><li>SRE practices align with DevOps pillars:</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*IvNRpygMHs1fwsWT" /><figcaption><em>Image source: </em><a href="https://www.youtube.com/watch?v=uTEL8Ff1Zvk"><em>What’s the Difference Between DevOps and SRE?</em></a></figcaption></figure><p>The goal of SRE is to ensure the reliability and improvement of software and systems, with a constant emphasis on factors like availability, speed, performance, and capacity. To achieve this, SRE employs a method that involves measuring the amount of repetitive and manual work (toil) and assessing reliability.</p><h4>Measure everything</h4><p>To measure toil and reliability we need tools and instruments to quantify our findings to make decisions and provide reliability guarantees. Google recommends the following concepts to handle services running in production:</p><p><strong>Measure reliability:</strong> You can calculate this by dividing the total number of successful interactions by the overall number of interactions. This gives you a numerical representation of the proportion of actual users who encounter a service that’s up and running as intended.</p><ul><li>Reliability metrics are generally low-level worker metrics such as the number of requests completed with HTTP status code 200 out of the total request.</li><li>Percentile distribution of response latency under 300 milliseconds</li><li>These metrics aim to reveal the symptoms of the underlying issues rather than the causes.</li></ul><p><strong>Measure error budget: </strong>The amount of unreliability you are willing to tolerate.</p><ul><li>This is a window of opportunity to innovate or stabilize the service.</li><li>Ideally, the error budget can initially span over double digits, and as soon as the SLO becomes stricter the error budget would decrease</li><li>Error budget = 100 — target reliability</li></ul><h3>Measure service level indicator (SLI): A measurable way to assess how dependable your service is, as perceived by your users.</h3><ul><li>An SLI demonstrates indicators or signs that directly align with a customer’s experience. These indicators encompass various infrastructure components such as…</li><li>Requests</li><li>Availability</li><li>Latency</li><li>Error rate</li><li>Throughput</li><li>Data Processing</li><li>Accuracy</li><li>Completeness</li><li>Consistency</li><li>Uniqueness</li><li>Freshness</li><li>Throughput</li><li>Storage</li><li>Durability</li><li>Latency</li><li>These are often regarded as Golden Signals and are used to measure an SLO.</li></ul><p><strong>Measure service level objective (SLO):</strong> Sets the target for an SLI over a specific time. An SLO represents the level of reliability that customers expect and too much or too little can have negative consequences.</p><ul><li>As an example, we could apply an SLO to return Shakespeare search results within 100 milliseconds on average.</li><li>Selecting the right SLO can be challenging. Firstly, you may not have the freedom to choose its value. When it comes to HTTP requests from external sources to your service, the number of queries per second (QPS) is primarily driven by your users’ needs, and you cannot establish an SLO for this metric.</li><li>Setting and sharing SLOs with users establishes performance expectations, reducing unfounded complaints to service owners, such as complaints about slowness.</li><li>Without an explicit SLO, users often develop their own beliefs about desired performance, which may be unrelated to the beliefs held by the people designing and operating the service.</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/1000/0*7UYdKuEQYVxYrKJ3" /></figure><p><em>Image source: </em><a href="https://cloud.google.com/blog/products/devops-sre/shrinking-the-impact-of-production-incidents-using-sre-principles-cre-life-lessons"><em>Google Cloud Blog</em></a></p><p>User Happiness declines during a production incident but stabilizes when the service is reliable.</p><ul><li>The SLO represents the minimum level of reliability required for a satisfactory user experience.</li><li>Your objective is straightforward: Maintain service reliability above its SLO to ensure user satisfaction.</li></ul><h4>Document blameless postmortem</h4><p>Comprehensive documentation of an incident or service disruption, including its underlying cause, the consequences it had, the steps taken to address it, and any subsequent measures implemented to prevent it from happening again.</p><ul><li>A report of an investigation conducted by a team of engineers and managers including</li><li>Incident commander</li><li>Declares incident and assumes all roles by default</li><li>Communication lead</li><li>Handles customer channels and communication</li><li>Operational lead</li><li>Navigates investigates, and mitigates the issue</li></ul><h4>Happiness vs. reliability vs. cost</h4><p>If you want to keep most of your users happy, how much higher should you aim above your SLO for reliability?</p><p>Before setting the target reliability for end customers, it’s important to consider the error budgets. It’s not feasible to aim for 100% reliability as it may not lead to customer satisfaction or financial feasibility. The objective is to maintain stability that meets the users’ needs. However, the decision to prioritize innovation or stability depends on the frequency of achieving SLOs and the burn rate. We will delve deeper into this later on.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1000/0*w9di5W9JnvCTXLgQ" /></figure><p><em>The optimal SLO threshold keeps most users happy while minimizing engineering costs.</em></p><p><em>Image source: </em><a href="https://cloud.google.com/blog/products/devops-sre/shrinking-the-impact-of-production-incidents-using-sre-principles-cre-life-lessons"><em>Google Cloud Blog</em></a></p><p>Determining whether or not to pursue reliability depends on the amount of loss incurred due to a problematic feature compared to the engineering effort required to fix it. This can be determined by prioritizing matrices for <em>Time to MVP</em> and <em>Value to the user</em>.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*OdvOGRFJNoAdD9SO" /></figure><p>Your users won’t notice any difference if you exceed your SLOs. To achieve this, you can increase your release frequency, stress test your production infrastructure to identify underlying issues, or allow your developers to focus on feature development instead of reliability. Having a higher level of reliability than your SLO serves only as a cushion to prevent your users from experiencing instability. By stabilizing your reliability, you can make the most of your error budget.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1000/0*UfNCrL1lB2ANPGuz" /><figcaption><em>An unstable reliability curve prevents you from spending your error budget efficiently.</em></figcaption></figure><p><em>Image source: </em><a href="https://cloud.google.com/blog/products/devops-sre/shrinking-the-impact-of-production-incidents-using-sre-principles-cre-life-lessons"><em>Google Cloud Blog</em></a></p><h4>Burn rate</h4><p>Consider a burn rate as a limit or goal for a specific duration. To clarify, if we take 30 days as an example, a burn rate of 1 implies that your error allowance would be completely utilized in precisely 30 days if the rate of 1 remained constant. Similarly, a burn rate of 2 would exhaust the error allowance in 15 days if kept constant, and a burn rate of 3 would do so in 10 days, and so on. Here are some formulas to help you understand the relationship between the burn rate and other important terms:</p><ul><li>SLI (Latency) = Good-time-periods/total-time-periods x 100</li><li>Target SLO = 95th percentile of latency should be &lt; 300ms</li><li>Error Budget = 1 — Target SLO</li><li>BurnRate = budget consumed x period/alerting window</li><li>Time until error budget is fully consumed = Length of SLO target(7, 30 or 90 days)/burn rate</li></ul><p>This is essentially a clock ticking in reverse before your users start experiencing degraded performance.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*uXONrD6GaFGmgUjf" /><figcaption><em>Image source: </em><a href="https://sre.google/workbook/alerting-on-slos/"><em>Google </em></a><em>Workbook</em></figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/744/1*XrulIBu8rUDpLf8Fs4jOFQ.png" /></figure><p><strong>A fast burn</strong> rate means that the rate at which the error budget is being consumed or depleted is rapid. This suggests that the service is experiencing more incidents or failing to meet its SLOs at a quicker pace than expected. A fast burn rate can indicate that the service is less reliable than desired, potentially requiring immediate attention to prevent it from breaching its SLOs.</p><p><strong>A slow burn</strong> rate means that the rate at which the error budget is being consumed or depleted is gradual or minimal. This suggests that the service is performing well within its SLOs, with few incidents or deviations from the expected performance. A slow burn rate is a positive indicator, as it means the service is operating reliably and is not exhausting its error budget quickly. It reflects a stable and predictable level of service quality.</p><p>The goal is to reduce false positive alerts, by opting for multi-window and multi-burn-rate signs so that we won’t miss out on the sensitive alerts and still be able to reduce false positives.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*xIwe_AHAmYWBmu4KZwDqkg.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*a3N5psw0SE8_hb_N" /></figure><p><em>Image source: </em><a href="https://sre.google/workbook/alerting-on-slos/"><em>Google </em></a><em>Workbook</em></p><p>Multiple burn rates allow you to adjust the alert to give appropriate priority based on how quickly you have to respond. If an issue will exhaust the error budget within hours or a few days, sending an active notification is appropriate. Otherwise, a ticket-based notification to address the alert the next working day is more appropriate.</p><p><strong>Note</strong>:</p><ul><li>Use % of error budget consumed to determine how fast the burn rate is over the rolling window of bucketize time frame to capture sensitive alerts.</li><li>Use % of error budget consumed to determine how slow the burn rate is over the rolling window of bucketize time frame to capture the slow burn alerts and reduce the false positives.</li></ul><h4>Solution approach</h4><p>We want to begin by laying out a foundation for measuring the reliability of the services. The key objective is to clearly define how much room we have for innovation and maintaining stability.</p><h4>Glossary of services</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*uMYJ8VQeks8cILpUKljWDg.png" /><figcaption><em>The SLO of your service is the minimum SLO of the services it depends on.</em></figcaption></figure><p>If one of your service dependencies has an SLO of 95%, the SLO of your service would be &lt;=95%</p><h4>Example</h4><p>Let’s consider an example of one of the services mentioned above.</p><h4>Sales tax</h4><p>It integrates with <strong>TaxJar, an external API</strong>, to return calculated sales tax information for given line items within specific locations. Being both vital for operations and minimal in complexity/scale of code.</p><p>Identify the SLIs</p><p>We have briefly discussed the SLI above in this blog. SLIs fall under the work metrics that indicate the symptom of an underlying problem with the application or infrastructure. An SLI needs to provide a clear difference between good and bad events and monitor those based on a metric that should show reliable trends to indicate whether there was an outage or not.</p><p>SLIs can fall under two categories:</p><p>1. Request-based<br> — Good-events/total-events x 100</p><p>2. Window-based<br> — Good-time-periods/total-time-periods x 100</p><p>A general rule of thumb is to refer to the SLI Menu, some of the SLIs are, but are not limited to:</p><p>1. Requests</p><ul><li>Availability</li><li>Latency</li><li>Error rate</li><li>Throughput</li></ul><p>2. Data Processing</p><ul><li>Accuracy</li><li>Completeness</li><li>Consistency</li><li>Uniqueness</li><li>Freshness</li><li>Throughput</li></ul><p>3. Storage</p><ul><li>Durability</li><li>Latency</li></ul><p>While numerous indicators cater to different aspects of the system, this document will consider <strong>availability</strong> and <strong>latency</strong> as major indicators since the service is public-facing.</p><p>These signals help us assess the performance of various application components. When measuring or aggregating indicators like availability and latency, it’s wise to select percentiles, as they provide a clearer view of the distribution. For instance, if a service receives 1000 requests every odd second and none during even seconds, the average request rate per second over the entire duration would be 500 rps. This could inaccurately suggest low service throughput.</p><p><strong>Availability</strong></p><p>Request-based SLI can be calculated with two different time series. One represents the total events and another represents the good or bad events.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*eVwtfj-OJzGAM6SJ" /></figure><ul><li>The above dashboard contains a time-series overlap of good events vs. total events</li><li>This is a 1-month window consisting of momentary bad events</li><li>One thing to note is that the metric type that you select to build request-based SLOs should be <em>Cummulative</em> and <em>Delta</em>. Since <em>Gauge</em> values indicate metric value at any given moment in time, it is unsuitable when we are sampling based on certain intervals and then plotting the graph.</li></ul><p>Using this time-series data, we aim to establish an SLO that helps us assess how often the SLI meets the threshold ratio over a period of time.</p><p><strong>Latency</strong></p><p>Now, let’s assess the service’s responsiveness from a latency perspective.</p><p>We aim to determine the p95 and p99 latency for the service and verify whether the aggregate of samples over a specified period meets the threshold.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*8zjzfK2akGV8spxu" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*znEmMhJD-dSJcO9B" /></figure><p>Note:</p><ul><li>P95 is often used to represent the typical or expected behavior, filtering out some extreme values.</li><li>P99 is useful for identifying cases where performance or behavior is worse than expected and can be crucial for systems where a small number of extreme cases can have a significant impact.</li></ul><p>We intend to measure latency in terms of both good and bad events, classifying response latency in milliseconds.</p><p>Good events &lt;= 1 sec, over and above this threshold are considered bad events.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*wWtE5Cz5MKcel2BK" /></figure><ul><li>The latency threshold is set at 1 second, observed over 1 week.</li><li>There are momentary spikes, but if we look at the statistical summary, the overall average is around 404ms</li></ul><p>Based on this latency time series data, we can create SLOs to understand how frequently the latency target is achieved.</p><p>Design SLOs</p><p>To build the SLO, we need to have a curated list of SLIs that represents performance metrics or custom metrics.</p><p>As a general rule, SLOs should be stricter than SLAs to prevent potential business consequences. It’s recommended to combine multiple SLIs for service into a single SLO and track it with weekly, monthly, and quarterly targets.</p><p>SLO is an internal contract among the stakeholders that create, manage, patch, and update the service which confirms that the reliability target is approved.</p><p>Error budgets are a crucial part of building an SLO, and to adopt an SLO with a tolerance for errors the following must hold:</p><ul><li>The people responsible for ensuring that the service meets its SLO have agreed that it is possible to meet this SLO under normal circumstances.</li><li>The organization has committed to using the error budget for decision-making and prioritizing. This commitment is formalized in an error budget policy.</li><li>There is a process in place for refining the SLO.</li></ul><p>Let’s represent the above-mentioned SLIs with numbers.</p><p><strong>Over four weeks, the API metrics show:</strong></p><p><strong>Total requests</strong>: 3,663,253</p><p><strong>Total average successful requests</strong>: 3,626,620 (99%)</p><p><strong>95th percentile latency</strong>: 381ms</p><p><strong>99th percentile latency</strong>: 406ms</p><p>We repeat this process for the other SLIs and create a proposed SLO:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*pYYsj2conRr5uMElbA-K5Q.png" /></figure><p>Based upon this proposed SLI, we can calculate our error budget over those four weeks for the following SLO targets:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*dsGEbED3dA9LF0L_St2nYw.png" /></figure><p><strong>Note</strong>: It is prudent to have room to account for the latency across the network hops and upstream services, here the 95th percentile of the last four weeks suggests a latency of 381ms, considering the latency fluctuations in the upstream services, the SLO should target somewhere between 400ms — 450ms. You may want to consider the following:</p><ol><li>Historical latency data</li><li>Peak hours</li><li>User proximity</li><li>Response caching</li></ol><p>In our case, the sales tax service uses the TaxJar API, and we need to examine their SLO targets to understand the expected level of reliability.</p><p>“<em>The TaxJar API clocks in at sub-25ms with 99.99% uptime SLA, giving your customers instant, always available, and accurate sales tax rates at checkout.</em>”</p><p>Since the TaxJar API processes requests in under 25ms for 99.99% of the time, and it’s the sole dependency in the application, you’re assured of achieving 99% availability. This also suggests that there are opportunities to include more SLI to measure SLO targets. Or shift left on the feature delivery. See <a href="https://docs.google.com/document/d/1HiAza7g-vt53689RS23ZL3L4e3JTtr1RBPQCyID6Khg/edit#heading=h.44ub798w0580">Happiness vs. reliability vs. cost</a>.</p><p>The more services that your application depends on, the trickier it gets when it comes to documenting the reliability targets for your end customer. Therefore, your SLO should align with the lowest SLO of the services your application depends on.</p><p>With a combination of SLO and toil involved, we can derive whether the customer is satisfied or not and what would be the subsequent action. This can be understood with the help of the SLO decision matrix:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*IIpc2A8thtBdPRQySTWfzw.png" /><figcaption><em>Source:</em> <a href="https://sre.google/workbook/implementing-slos/#slo-decision-matrix">Google </a>Workbook</figcaption></figure><p>Decide error budgets</p><p>When it comes to error budgets, there is a thought process to balance the service’s reliability and the pace of innovation and development, especially, when it comes to addressing a missed SLO when multiple teams are involved.</p><ul><li>If the SLO was missed as a result of an application error that belongs to your team, then redirect any engineering effort to fix the problem, instead of rolling out the feature.</li></ul><p>More than <strong>70%</strong> of outages are caused by a change that is “usually” harmless and introduced as an iteration.</p><p>Let’s understand with an example:</p><p>If your availability SLO is <strong>99.9%</strong>, your error budget over 30 days is (100–99.9) ✕ (30 days ✕ 24 hours ✕ 60 minutes) = <strong>43.2 </strong>minutes. The error budget for a system is consumed, or burned, whenever the system is unavailable. Using the previous example, if the system has had 10 minutes of downtime in the past 30 days and started the 30 days with a total budget of 43.2 minutes unutilized, then the remaining error budget is reduced to 33.2 minutes.</p><p>We recommend using a rolling window of 30 days when computing your total error budget and the error budget burn rate.</p><p>If the desired reliability target is 99.95% over 30 days, then the error budget allows</p><p>(100–99.95) x 30 x 24 x 60</p><p><strong>21.6</strong> minutes of downtime.</p><p>This, in comparison with 3 ‘9’s of availability, takes a lot of engineering effort to meet and keep the reliability of the service.</p><p>Burn rate</p><p>The rate at which your error budget is consumed is called the ‘burn rate.’ These can be helpful when it comes to not only understanding the reliability of your services but also the demand increase for your capacity. ‘Burn rate’ is a coined term by Google to address how quickly your error budgets are getting consumed.</p><p>Ideally, you should always try to maintain a burn rate of 1 throughout your SLO’s target (as you invest in evolving your application with new features). However, in practice, your burn rate will fluctuate as issues or incidents cause your burn rate to increase rapidly until the issue is resolved. Therefore, alerting on burn rates allows you to be proactively notified when an issue is consuming your error budget at an elevated rate that could potentially cause you to miss your SLO target.</p><p>Approach #1: Time to error budget depletion</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*_7BJupND2wsnO22ljkcESA.png" /></figure><p>The higher the burn rate you select, the smaller a long window you should pair it with (so that high-severity issues are caught sooner). The long alerting window is specified in hours and ensures the monitor measures the burn rate, while the short alerting window is specified in minutes.</p><p>Approach #2: Theoretical error budget consumption</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*WohVHrd14-1s7MT6L_WPzA.png" /></figure><p>Let’s calculate for our Sales Tax Service example. To be alerted when the theoretical error budget consumption reaches 10% over a 30-day SLO with a 1-hour long window, the selected burn rate should be:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*me_wXsE5JDJUqLtoDJW6UA.png" /></figure><p>Now, using the burn rate in the time-to-error budget depletion equation. We have</p><p>30/3.6 = <strong>~8.3 days</strong></p><p>We can now configure monitors based on the above time-to-error budget depletion to accurately get notified ahead of time and take timely actions.</p><p>Configure SLOs<br>For this case, we are using the Datadog platform:Availability SLO</p><ol><li>On the SLO status page, select New SLO +.</li><li>Define the source for your SLO. You can create an SLO from metrics or monitors.</li></ol><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*inbdZhy5s7IdEpm7" /></figure><p><strong>Note</strong>: The Good events are considered to be all response codes that are not in the range 5xx, meaning client-side errors are not taken into account.</p><p>3. Set a target and a rolling time window (past 7, 30, or 90 days) for the SLO. Datadog recommends you make the target stricter than your stipulated SLAs. If you configure more than the one-time window, select one to be the primary time window. This time window is displayed on SLO lists. By default, the shortest time window is selected.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*qjdtXgheWldTTb71" /></figure><p>4. Finally, give the SLO a title, describe it in more detail, or add links in the description, add tags, and save it.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*kIedfk6sF4DyInXP" /></figure><p><strong>Configure alert for SLO</strong></p><ol><li>Select the SLO target service</li><li>Set the alert condition to ‘Burn Rate’</li><li>Here, we want to evaluate the long window burn rate for 1 hour and the short window burn rate for 5 minutes for a period of 30 days.</li></ol><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*J6k6SRFqWnZSB2z2" /></figure><p>4. Notify the team</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*90RpEFHyUXbTpyGL" /></figure><p><strong>Latency SLO</strong></p><ol><li>For latency, you need to first create a P95 monitor for the resource, and for the given threshold.</li></ol><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*vbnePPXREcE04aaU" /></figure><p>2. Then, use that monitor to create a Latency SLO.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/949/0*v4xSGHwv2bwKoQyz" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/951/0*LrGcmUsNgRrVUgb6" /></figure><p>3. Make sure you link relevant run books and documentation in the alert body, to assist the on-call engineer in investigating.</p><h4>Conclusion</h4><p>This blog addressed key concepts of SLI, SLO, error budgets, and burn rate by implementing an SRE methodology. This framework provides us with tools and practices to better understand application signals and to make informed decisions while maintaining the reliability of the services. We have discussed the key pillars of DevOps philosophy and how SRE practice helps us implement it, the Golden Signals, the happiness vs. reliability vs. cost ratio, and an example of how we should approach the implementation of SLO for any given service.</p><h4>Appendix</h4><h4>Key terminologies</h4><ol><li><strong>SLI</strong>: A service level indicator is a numerical indicator that can be measured to gauge the reliability of an application service.</li><li><strong>SLO</strong>: An SLO is a target value for an SLI, measured over a period of time.</li><li><strong>Error budgets</strong>: Tracks if a company is meeting contractual promises for a system or service and prevents it from pursuing too much innovation at the expense of that system or service’s reliability.</li><li><strong>Burn rate</strong>: A unitless value coined by Google that indicates how fast your error budget is consumed relative to your SLO’s target length</li><li><strong>SRE</strong>: A job function, a mindset, and a set of engineering practices to run reliable production systems.</li><li><strong>Golden Signals</strong>: The four Golden Signals of monitoring are latency, traffic, errors, and saturation of your user-facing system.</li><li><strong>Availability</strong>: The ratio of Good events over total events for a given period. The events are the requests sent to the services from the user-facing applications.</li><li><strong>Latency</strong>: A distribution measure of how long it takes to return a response to a request. This is generally calculated in terms of percentile of p50, p75, p90, p95, p99.</li><li><strong>Error rate</strong>: An error budget is the maximum amount of time a technical system can fail without contractual consequences.</li><li><strong>Throughput</strong>: A measure of how many units of information a system can process in a given amount of time.</li><li><strong>Accuracy</strong>: Accuracy refers to the precision and correctness of data or information within a system. SREs strive to ensure that the data and results provided by a service are free from errors and discrepancies, meeting user expectations for reliable and precise information.</li><li><strong>Completeness</strong>: Completeness involves making sure that all necessary data or components are present and accessible within a system. SREs work to guarantee that no critical information is missing, ensuring that the system functions as intended and provides a comprehensive user experience.</li><li><strong>Consistency</strong>: Consistency refers to the degree of predictability and uniformity in the performance and behavior of a service. In SRE, maintaining consistency means ensuring that a service consistently delivers its expected results, response times, and behaviors under various conditions. This involves managing and reducing variability to provide a reliable user experience.</li><li><strong>Uniqueness</strong>: Uniqueness pertains to the distinctiveness or individuality of data or components within a system. In SRE, ensuring uniqueness might involve preventing data duplication, resource conflicts, or any situation where identical elements could cause problems. Maintaining uniqueness is important for data integrity and system stability.</li><li><strong>Freshness</strong>: Freshness relates to the timeliness and up-to-date nature of data or information in a system. In SRE, ensuring freshness means guaranteeing that data is current and relevant to users. It often involves managing data caches, synchronization, or real-time updates to prevent users from accessing outdated or stale information.</li><li><strong>Throughput</strong>: Throughput measures the rate at which a system can process and handle requests or transactions. In SRE, assessing and optimizing throughput is crucial for ensuring that a service can meet user demand without degradation in performance. SREs monitor and adjust system resources and configurations to maintain adequate throughput levels.</li></ol><p><strong>About Satyaditya</strong></p><p>Working as a Site Reliability Engineer at SADA, one of the leading cloud consulting and engineering firms, Satyaditya applies his technical expertise in DevOps, Cloud Architecture, Automation, and Observability to solve intricate infrastructure challenges for clients. He has experience working with Google Cloud Platform, Azure, and AWS, and holds professional certifications. He completed his Master of Computer Applications degree from Vellore Institute of Technology, where he developed a keen interest in data analytics, machine learning, cloud technologies, robotics, and automation. He is equally enthusiastic about product development, ideation, and UX design. In his free time, he enjoys engaging in recreational activities such as creating art, playing musical instruments, and reading books.</p><p><strong>About SADA</strong></p><p>At SADA, we climb every mountain, clear every hurdle, and turn the improbable into possible — over and over again. Simply put, we propel your organization forward. It’s not enough to migrate to the cloud, it’s what you do once you’re there. Accelerating application development. Advancing productivity and collaboration. Using your data as a competitive edge. When it comes to Google Cloud, we’re not an add-on, we’re a must-have, driving the business performance of our clients with its power. Beyond our expertise and experience, what sets us apart is our people. It’s the spirit that carried us from scrappy origins as one of the Google Cloud launch partners to an award-winning global partner year after year. With a client list that spans healthcare, media and entertainment, retail, manufacturing, public sector, and digital natives — we simply get the job done, every step of the way. Visit <a href="https://sada.com/">SADA.com</a> to learn more.</p><p>If you’re interested in becoming a part of the SADA team, please visit our <a href="https://sada.com/careers-cloud-computing-business-solutions">careers page</a>.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=f0428b36436e" width="1" height="1" alt=""><hr><p><a href="https://engineering.sada.com/setting-up-slis-slos-and-monitors-with-datadog-f0428b36436e">Setting up SLIs, SLOs, and monitors with Datadog</a> was originally published in <a href="https://engineering.sada.com/">The SADA Engineering Blog</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[GCVE networking: understanding Google Cloud VPC to Google Cloud VMware Engine connectivity]]></title>
            <link>https://engineering.sada.com/gcve-networking-understanding-google-cloud-vpc-to-google-cloud-vmware-engine-connectivity-7b46bb371159?source=rss-6bc96cd41286------2</link>
            <guid isPermaLink="false">https://medium.com/p/7b46bb371159</guid>
            <category><![CDATA[vmware]]></category>
            <category><![CDATA[vpc]]></category>
            <category><![CDATA[gcve]]></category>
            <category><![CDATA[google-cloud-platform]]></category>
            <category><![CDATA[gcp]]></category>
            <dc:creator><![CDATA[SADA, An Insight Company]]></dc:creator>
            <pubDate>Wed, 06 Dec 2023 14:01:54 GMT</pubDate>
            <atom:updated>2023-12-06T14:01:54.970Z</atom:updated>
            <content:encoded><![CDATA[<p><em>By Abiodun Ajibola, SADA Solutions Engineer</em></p><h4>Introduction</h4><p>Google Cloud VMware Engine (GCVE) is a cloud service that allows you to run your VMware workloads in Google Cloud without the need to refactor or re-architect your applications. When setting up GCVE, one critical aspect to consider is networking. Proper networking configuration ensures that your VMware workloads can seamlessly interact with other Google Cloud services and resources while maintaining security and performance. In this blog post, we will explore how the Google Cloud VPC communicates with the VMware Engine private cloud network.</p><h4>What is Google Cloud VMware Engine (GCVE)?</h4><p>GCVE is a fully managed VMware service that enables you to migrate and run your VMware workloads in Google Cloud without the need to re-architect or re-platform your applications. It provides a familiar VMware environment with features like vCenter, NSX-T networking, and HCX (Hybrid Cloud Extension) for seamless workload migration. GCVE empowers you to leverage Google Cloud’s global infrastructure, scalability, and services while maintaining compatibility with your existing VMware tools and processes.</p><h4>Objective</h4><p>Networking is one of the core aspects of Google Cloud VMware Engine. An example here is a customer that just migrated their VMware infrastructure from on-premises to Google Cloud and is very much interested in understanding how their existing VPC network would communicate and connect to resources on their just-migrated VMware Engine workloads. Let’s explore step-by-step the flow of this communication.</p><p><strong><em>Please note that the ‘Google Org or Service Producer VPC’ shown in the diagrams is not visible or accessible to customers in reality. Its inclusion here aims to enhance understanding of GCVE networking.</em></strong></p><p><strong><em>Additionally, topics like Cloud DNS and on-premises connectivity will be addressed in future GCVE blogs.</em></strong></p><h4>Deep diving….</h4><h4>The VPC1</h4><p>Starting from the left-hand side of our diagram below, we have our VPC in the range of 10.1.0.0/16. Additionally, we’ve preallocated our VMware Engine addresses to be within the range of 10.20.0.0/16. Depending on our needs, we can either specify multiple /24 subnets or summarize them all into a single /16 if we have numerous different subnets.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*Ao1iv5nr055vpcp0" /></figure><h4>VPC peering</h4><p>The VPC peering occurs between VPC1 and the Service Producer VPC within Google’s tenant environment, inaccessible to customers. However, customers establish the peering connection through the VMware Engine console.</p><p>Notably, enabling the GCP VPC routing mode to global facilitates communication between the VMware engine private cloud (e.g., us-central region) and GCP resources in other regions (e.g., europe-west4) within the Customer Org. For intercommunication between multiple private clouds in different regions, selecting the ‘global’ option on the Service Producer VPC is essential.</p><h4>BGP (Border Gateway Protocol) session</h4><p>Next, the process moves to the Service Producer VPC, where a BGP session (which is automatically configured by Google) retrieves network information from our VPC1 and conveys it to the private cloud network. This BGP session within the private cloud terminates at the NSX tier-0 router.</p><h4>TOR (top-of-the-rack switch)</h4><p>The tier-0 router will now route to the underlying destination virtual machine. A static route directs the traffic to the top-of-the-rack switch. This arrangement eliminates the need to route between private clouds within the same placement group through VMware Engine, as the TOR switches locally handle that traffic.</p><h4>Multiple VPC to private cloud connectivity</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*PycWtODJRR_PMwkp" /></figure><p>This simply means that each VPC will establish peering with VMware Engine, a setup known as multi-VPC peering. Based on the diagram above, there will be a dedicated Service Producer VPC for each customer VPC that we deploy. This forms a one-to-one relationship. Optionally, we can establish peering for both VPC1 and VPC2, but it’s important to note that VPC peering is not transitive.</p><h4>Shared VPC to private cloud connectivity</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*YLnv6ngvrShVsp4T" /></figure><p>This connectivity is exactly the same as with the VPC to private cloud we just discussed, except the Compute VM workloads are not in the VPC1. They are in various service projects. This is an important concept, as it helps reduce the number of VPCs that we need to peer with the VMware Engine.</p><h4>Google Netapp Cloud Volumes to private cloud using multi-VPC peering</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/608/0*I1gBiDMFvPYOq-yK" /></figure><p>NetApp Cloud Volumes Service for Google Cloud (CVS) accelerates the deployment of cloud-native applications by swiftly providing shared file services and efficient storage management capabilities. When you integrate Cloud Volumes Service with VMware Engine, you gain the ability to seamlessly attach CVS cloud volumes to your workload virtual machines (VMs) directly from the guest operating system.</p><p>Additionally, VMs in Compute Engine and Google Cloud VMware Engine can both mount the same cloud volumes.</p><p>Full details of the connection steps can be found <a href="https://cloud.google.com/vmware-engine/docs/vmware-ecosystem/howto-cloud-volumes-service#before_you_begin">here</a>.</p><h4>Conclusion</h4><p>Understanding the various communication examples will help you make an informed decision. By selecting the right connectivity or communication mode for your GCVE environment, you can ensure network simplicity, smooth troubleshooting, and optimal networking performance for VMware workloads in Google Cloud.</p><p><strong>About Abiodun Ajibola</strong></p><p>Abiodun Ajibola is a seasoned professional with over 8 years of experience as a Network &amp; Solution Engineer. He possesses excellent technical skills in advanced network routing, switching, and security, complemented by strong customer relations.</p><p><strong>About SADA</strong></p><p>At SADA, we climb every mountain, clear every hurdle, and turn the improbable into possible — over and over again. Simply put, we propel your organization forward. It’s not enough to migrate to the cloud, it’s what you do once you’re there. Accelerating application development. Advancing productivity and collaboration. Using your data as a competitive edge. When it comes to Google Cloud, we’re not an add-on, we’re a must-have, driving the business performance of our clients with its power. Beyond our expertise and experience, what sets us apart is our people. It’s the spirit that carried us from scrappy origins as one of the Google Cloud launch partners to an award-winning global partner year after year. With a client list that spans healthcare, media and entertainment, retail, manufacturing, public sector and digital natives — we simply get the job done, every step of the way. Visit <a href="https://sada.com/">SADA.com</a> to learn more.</p><p>If you’re interested in becoming a part of the SADA team, please visit our <a href="https://sada.com/careers-cloud-computing-business-solutions">careers page</a>.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=7b46bb371159" width="1" height="1" alt=""><hr><p><a href="https://engineering.sada.com/gcve-networking-understanding-google-cloud-vpc-to-google-cloud-vmware-engine-connectivity-7b46bb371159">GCVE networking: understanding Google Cloud VPC to Google Cloud VMware Engine connectivity</a> was originally published in <a href="https://engineering.sada.com/">The SADA Engineering Blog</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Cloud DLP de-identification techniques and their use cases]]></title>
            <link>https://engineering.sada.com/cloud-dlp-de-identification-techniques-and-their-use-cases-18a7ca10fa8e?source=rss-6bc96cd41286------2</link>
            <guid isPermaLink="false">https://medium.com/p/18a7ca10fa8e</guid>
            <category><![CDATA[cloud-dlp]]></category>
            <category><![CDATA[identification]]></category>
            <category><![CDATA[use-cases]]></category>
            <dc:creator><![CDATA[SADA, An Insight Company]]></dc:creator>
            <pubDate>Wed, 15 Nov 2023 14:02:46 GMT</pubDate>
            <atom:updated>2023-11-15T14:02:46.815Z</atom:updated>
            <content:encoded><![CDATA[<p><em>By Mohita Agarwal, SADA Data Engineer</em></p><h4>Cloud Data Loss Prevention</h4><p>Cloud Data Loss Prevention (Cloud DLP) is a fully managed service meticulously designed to uncover, categorize, and safeguard sensitive data, regardless of its location, be it within databases, textual content, or even images. This service not only offers comprehensive visibility but also empowers the organization to classify sensitive data thoroughly. Ultimately, Cloud DLP plays a pivotal role in diminishing data-related risks by examining and altering structured and unstructured data, employing techniques such as obfuscation and de-identification, such as masking and tokenization. Moreover, Cloud DLP can be instrumental in executing re-identification analyses, thereby enriching the comprehension of data privacy risks.</p><p>Re-identification risk analysis involves scrutinizing data to identify properties that could elevate the chances of subjects being recognized. For instance, consider a marketing dataset that includes demographic attributes like age, job title, and zip code. On the surface, these demographics may not appear identifying, but certain combinations of age, job title, and zip code might uniquely correspond to a small cohort of individuals or even a single individual, thereby heightening the risk of re-identifying that person.</p><h4>Various de-identification techniques and their use cases</h4><h4>Masking</h4><p>This de-identification technique masks a string either fully or partially by replacing a given number of characters with a specified fixed character. Additionally, you can specify if you need to ignore any characters from masking.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/534/0*MMIIQoTAcvJ6QgLO" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/531/0*ciB1TGYMK_AFdO8F" /></figure><h4>Use cases for data masking</h4><p><strong>Reducing breach impact: </strong>Data masking is a data security method used to anonymize an organization’s sensitive data. In the unfortunate event of a data leak or breach, this method ensures that the exposed data cannot be utilized to identify an individual or for illicit purposes such as fraudulent transactions.</p><p><strong>Creating safe test data: </strong>Teams frequently require functional datasets for testing purposes. Masked data preserves the required integrity for testing without jeopardizing real user data. Automated data masking facilitates the rapid and secure provisioning of masked test data to non-production environments, enabling teams to fulfill this need efficiently.</p><p><strong>Internal access control: </strong>Data masking serves as a crucial tool for anonymizing sensitive data, ensuring that individuals within an organization who lack the authorization to access specific information are unable to do so. In cases where a member of your organization requires access to data for legitimate purposes but lacks authorization to view, for instance, credit card details, masking the last four digits of the card number provides a protective measure, preventing exposure to those sensitive elements.</p><p><strong>Speeding up development: </strong>Manual data cleanup is a tedious, time-intensive process that can impede production speed. Automated data masking seamlessly integrates with current systems, eliminating the need for manual interventions. This streamlines the development and testing processes, enabling teams to work with authentic data efficiently, without causing disruptions.</p><p><strong>Protecting data in transit: </strong>Once data has been masked, it remains masked in transit and in the cloud. Should that data be breached, it cannot be used to identify individuals or for fraudulent purposes.</p><p><strong>Protecting client privacy: </strong>Privacy is important to almost everyone, which is why organizations need to take measures to ensure confidential client data and personally identifiable information (PII) is protected, particularly in scenarios where an organization shares data with third parties.</p><p><strong>Compliance with data privacy laws: </strong>Data privacy regulations, such as the GDPR, HIPAA, GLBA, PIPEDA, etc., aim to safeguard the safety, security, and confidentiality of Personally Identifiable Information (PII). Implementing data masking within the development life cycle removes identifiers, including the 18 identifiers identified by the HIPAA Safe Harbor method, before data is shared within an organization. This approach aligns with the Privacy by Design principle mandated by Article 25 of the GDPR.</p><h4><a href="https://cloud.google.com/dlp/docs/deidentify-sensitive-data#redactconfig">Redaction</a></h4><p><strong>Redaction</strong> protects a value by removing it completely.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/526/0*VJsWX4OsVUskfKMC" /></figure><h4><a href="https://cloud.google.com/dlp/docs/deidentify-sensitive-data#replaceconfig">Replacement</a></h4><p><a href="https://cloud.google.com/dlp/docs/deidentify-sensitive-data#replaceconfig"><strong>Replacement</strong></a> replaces each input value with a given value specified by the user or with a default value.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/538/0*G0kFOqJcstu2CG39" /></figure><h4>Use cases for redaction/replacement</h4><p><strong>Financial services: </strong>Financial institutions grapple with the challenge of managing an overwhelming volume of confidential and sensitive customer data. To navigate this, they frequently extract pertinent information from the extensive datasets they handle. Examples of data redaction or replacement in financial services include concealing or substituting items like credit/debit card numbers, bank account numbers, mobile phone numbers, and more.</p><p><strong>Pharmaceutical and life sciences: </strong>Healthcare institutions often spend a significant amount of time on patient-related paperwork. The ability to quickly redact or replace sensitive information frees up staff to focus on patient care.</p><p><strong>Law enforcement: </strong>They can employ data redaction or replacement to maintain databases, ensure compliance with criminal/victim identification requirements, and save valuable time.</p><p><strong>Transportation: </strong>The transportation industry deals with extensive documentation, ranging from invoices to toll tax receipts and more. Data redaction or replacement expedites processes in this sector.</p><p><strong>Government: </strong>Government entities store a wide array of sensitive data, necessitating the implementation of comprehensive safety protocols to prevent any data breaches. Data redaction and replacement represent essential components of these procedures, aiding in the safeguarding of sensitive information and ensuring compliance with audit requirements.</p><p><strong>IT &amp; operations: </strong>Even a minor breach can have severe repercussions, potentially grinding an entire organization’s operations to a standstill. Data redaction and replacement provide IT professionals with the right tools to effectively redact sensitive data, enhancing their workflow efficiency and bolstering security measures.</p><h4><a href="https://cloud.google.com/dlp/docs/deidentify-sensitive-data#cryptohashconfig">Pseudonymization with secure hash</a></h4><p>This de-identification technique replaces input value with a secure one-way hash generated using a data encryption key. Secure-keyed cryptographic hashing is a method that involves creating a cryptographic hash of an input string using a secret key, similar to HMAC, and is generally considered a more secure approach as opposed to using only a hash function. Unique identifiers with primary key behavior are de-identified this way for very large datasets.</p><p>An input value is replaced with a value that has been encrypted and hashed using <a href="https://tools.ietf.org/html/rfc4634">Hash-based Message Authentication Code (HMAC)-Secure Hash Algorithm (SHA)-256</a> on the input value with a cryptographic key. The hashed output of the transformation is always the same length and can’t be re-identified.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/709/0*hEM7zgIrAYUgLDFo" /></figure><h4>Use cases for pseudonymization with secure hash</h4><p><strong>Patient record comparison: </strong>Without the ability to reveal the plaintext versions of the pseudonyms, re-identifying individual data entries in this set is extremely difficult since a secure pseudonymization function was used for hashing.</p><p><strong>Medical research institution use case: </strong>The pseudonymisation scheme can unfold its ideal potential here in the task of detecting correlations and statistical patterns in symptoms and medications, without even ever revealing the true value of the underlying symptom or medication.</p><p><strong>Cybersecurity: </strong>Fraud detection systems, such as credit card transaction monitoring and dark web monitoring, access financial and personal information. It’s crucial to employ pseudonymization methods to protect this data. Just as discussed earlier, in many real-world applications, data is sanitized using cryptographic hashes before further processing. Pseudonymization is increasingly important as a security technique and a way to implement data minimization in various contexts. It facilitates personal data processing while providing strong safeguards for data protection.</p><p><strong>Pseudonymization by GDPR: </strong>GDPR mandates the use of appropriate technical and organizational measures to safeguard personal data, with pseudonymization often being the method of choice to maintain data utility.</p><h4><a href="https://cloud.google.com/dlp/docs/deidentify-sensitive-data#cryptoreplaceffxfpeconfig">Pseudonymization with format-preserving token</a></h4><p>This technique replaces an input value with a “token” or surrogate value, of the same character set and length using format-preserving encryption (FPE). Preserving the format can help ensure compatibility with legacy systems that have restricted schema or format requirements. It involves replacing an input value with an encrypted value that has been generated using format-preserving encryption. Here’s a simple example of how FPE could be used in Python to de-identify personal data (CCN).</p><h4>Use cases for pseudonymization with format-preserving token</h4><p><strong>Payment processing</strong>: Within the realm of payment processing, safeguarding sensitive information like credit card numbers and bank account details is paramount. Pseudonymization is employed to obscure these financial data points. Format-preserving tokenization emerges as a valuable solution, as it replaces such information with tokens that retain the structure of the original data. This approach enables businesses to uphold payment workflows and maintain compatibility with their existing systems while enhancing data security.</p><p><strong>Healthcare data</strong>: Healthcare organizations often deal with sensitive patient data, including medical record numbers, insurance IDs, and diagnosis codes. Format-preserving tokenization can help protect patient privacy while still enabling efficient data exchange and integration between various medical systems.</p><p><strong>Identification numbers</strong>: Many industries use identification numbers, such as employee IDs, customer IDs, and membership IDs. Format-preserving tokenization can be applied to these numbers to maintain consistent identification while safeguarding individuals’ privacy.</p><p><strong>Membership and loyalty programs</strong>: Companies that provide loyalty programs or membership services may find it necessary to pseudonymize customer IDs or account numbers. Through the utilization of format-preserving tokens, they can maintain the capacity to deliver personalized services while simultaneously mitigating the risk associated with potential customer data exposure.</p><p><strong>Software testing and development</strong>: During software testing and development, actual production data is often used to simulate real-world scenarios. However, exposing sensitive customer information in these environments can be risky. Format-preserving tokenization allows developers to work with realistic data without compromising privacy.</p><p><strong>Research and analytics</strong>: Researchers and analysts often require access to large datasets for analysis and research purposes. By pseudonymizing sensitive data using format-preserving tokens, organizations can share datasets without revealing the actual personal information, making it safer to collaborate and share data.</p><p><strong>E-commerce and order processing</strong>: E-commerce platforms collect various types of personal data, such as shipping addresses and phone numbers. Format-preserving tokenization can help protect this information while ensuring seamless order processing and customer communication.</p><p><strong>Legal and compliance</strong>: Legal documents and contracts might contain sensitive information that needs to be pseudonymized before being shared with third parties. Format-preserving tokenization can help comply with data protection regulations while retaining the original document structure.</p><p><strong>Customer support and CRM</strong>: Customer support systems often store customer information for reference. By using format-preserving tokens, businesses can provide efficient support without exposing sensitive details.</p><p><strong>Data migration and sharing</strong>: When sharing data between systems or migrating data to new platforms, format-preserving tokenization can help maintain data integrity and relationships without exposing actual PII.</p><p>While format-preserving tokenization can provide an additional layer of privacy and security, it’s essential to consider the overall data protection strategy, including encryption, access controls, and user consent, to ensure comprehensive data privacy and compliance with relevant regulations.</p><h4><a href="https://cloud.google.com/dlp/docs/deidentify-sensitive-data#fixedsizebucketingconfig">Generalization bucketing</a></h4><p>This technique masks input values by replacing them with “buckets” or ranges, within which the input value falls. For example, specific ages can be bucketed into age ranges or distinct values into ranges like “low,“ “medium,” or “high.”</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/475/0*iOtBw9FttXnTVHWc" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/478/0*exvO3ge9PSqj1liw" /></figure><h4>Use cases for generalized bucketing</h4><p>Note: These are speculative interpretations based on the term “generalization bucketing” in the field of machine learning and data science.</p><p><strong>Data preprocessing for supervised learning:</strong> In machine learning, data preprocessing is a crucial step. Generalization bucketing could refer to a technique where continuous or numerical features are discretized into buckets or bins. This can help with generalization, especially when dealing with noisy data, by reducing the impact of outliers and making the model more robust.</p><p><strong>Categorical variable aggregation:</strong> When working with categorical variables, especially when they have a large number of categories, it’s common to aggregate or group similar categories together. This can help prevent overfitting and improve generalization by reducing the dimensionality of the input space.</p><p><strong>Feature engineering:</strong> Feature engineering involves creating new features from existing ones to improve model performance. Generalization bucketing might relate to a technique where certain features are transformed into more general or abstract representations that capture the underlying patterns in the data.</p><p><strong>Ensemble methods:</strong> Ensemble methods combine the predictions of multiple models to achieve better performance. Generalization bucketing could be used in the context of creating diverse subsets of data or models to improve the ensemble’s ability to generalize to new, unseen data.</p><p><strong>Data augmentation:</strong> In tasks like image classification, data augmentation involves applying transformations to the training data, such as rotations, flips, and translations, to increase the diversity of the training set and improve generalization. Generalization bucketing might refer to a systematic way of applying these transformations to different “buckets” of data.</p><h4><a href="https://cloud.google.com/dlp/docs/deidentify-sensitive-data#dateshiftconfig">Date shifting</a></h4><p>This technique shifts dates by a random number of days per user or entity. This helps obfuscate actual dates while still preserving the sequence and duration of a series of events or transactions.</p><p>Date shifting, also known as data anonymization or pseudonymization, is a technique used to protect the privacy and confidentiality of sensitive information by altering the dates while maintaining the overall characteristics and patterns of the data. This is often employed in situations where it’s necessary to share or analyze data without revealing specific dates.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/534/0*i_QnB2Jz72XR2ei5" /></figure><h4><a href="https://cloud.google.com/dlp/docs/deidentify-sensitive-data#timepartconfig">Time extraction</a></h4><p>This technique extracts or preserves a portion of Date, Timestamp, and TimeOfDay values. Time extraction refers to the process of identifying and extracting temporal information from text data. This can include dates, times, durations, and other time-related expressions. Time extraction has a wide range of use cases across various industries.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/532/0*OZIp87shIoKCS3oz" /></figure><h4>Use cases for data shifting/time extraction</h4><p><strong>Medical research and healthcare data</strong>: In medical research, patient data often needs to be shared with researchers while protecting patient privacy. Date shifting/time extraction can be used to de-identify data, making it difficult to trace specific medical events back to individuals.</p><p><strong>Financial data</strong>: Financial institutions may need to share transaction data for analysis purposes, but they must ensure that personal and transaction details remain private. Date shifting/time extraction can be applied to transaction dates to obscure the exact timing of transactions.</p><p><strong>Human resources and payroll</strong>: Companies might need to provide HR or payroll data for analysis without revealing specific dates of employee actions like hiring, promotions, or terminations. Date shifting/time extraction can be used to protect employee privacy.</p><p><strong>Epidemiological studies</strong>: In epidemiology, analyzing the spread of diseases requires access to patient data. Date shifting/time extraction can help prevent the identification of specific patients while still allowing researchers to observe trends and patterns.</p><p><strong>Judicial data</strong>: Law enforcement agencies may need to share crime data for research purposes, but they need to safeguard sensitive information. Date shifting/time extraction can be applied to crime incident dates to protect the privacy of victims and suspects.</p><p><strong>Educational research</strong>: Educational institutions might share student data for educational research purposes. Date shifting/time extraction can help protect student privacy while still allowing researchers to analyze academic trends.</p><p><strong>Market research</strong>: Companies conducting market research might need to share data about consumer behavior while ensuring individual privacy. Date shifting/time extraction can be used to obscure the timing of purchases and actions.</p><p><strong>Social sciences research</strong>: Sociological or psychological studies might require data about events like social interactions or emotional experiences. Date shifting/time extraction can help protect participants’ identities.</p><p><strong>Transportation and logistics</strong>: When analyzing transportation routes, delivery schedules, or traffic patterns, date shifting/time extraction can be applied to protect sensitive information while maintaining the overall flow of the data.</p><p><strong>Government statistics</strong>: Government agencies might need to share statistical data without revealing exact dates of events like census responses or economic indicators. Date shifting/time extraction can help protect citizen privacy.</p><h4>About Mohita Agarwal</h4><p>As a Data Engineer with around five years of experience, Mohita has been at the forefront of transforming data landscapes by designing and implementing robust data pipelines. Her expertise lies in orchestrating seamless migrations from on-prem systems to Google Cloud Platform (GCP), enabling organizations to harness the power of cloud-based data solutions. She has worked on SQL, Python, and cloud-native technologies, particularly excelling in leveraging Google Cloud tools such as BigQuery, Apache Airflow, Composer, Cloud functions, Cloud SQL, GCS, and Pub/Sub.</p><h4>About SADA</h4><p>At SADA, we climb every mountain, clear every hurdle, and turn the improbable into possible — over and over again. Simply put, we propel your organization forward. It’s not enough to migrate to the cloud, it’s what you do once you’re there. Accelerating application development. Advancing productivity and collaboration. Using your data as a competitive edge. When it comes to Google Cloud, we’re not an add-on, we’re a must-have, driving the business performance of our clients with its power. Beyond our expertise and experience, what sets us apart is our people. It’s the spirit that carried us from scrappy origins as one of the Google Cloud launch partners to an award-winning global partner year after year. With a client list that spans healthcare, media and entertainment, retail, manufacturing, public sector and digital natives — we simply get the job done, every step of the way. Visit <a href="https://sada.com/">SADA.com</a> to learn more.</p><p>If you’re interested in becoming a part of the SADA team, please visit our <a href="https://sada.com/careers-cloud-computing-business-solutions">careers page</a>.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=18a7ca10fa8e" width="1" height="1" alt=""><hr><p><a href="https://engineering.sada.com/cloud-dlp-de-identification-techniques-and-their-use-cases-18a7ca10fa8e">Cloud DLP de-identification techniques and their use cases</a> was originally published in <a href="https://engineering.sada.com/">The SADA Engineering Blog</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Kafka migration from on-prem to Confluent]]></title>
            <link>https://engineering.sada.com/kafka-migration-from-on-prem-to-confluent-1bd03c3689f3?source=rss-6bc96cd41286------2</link>
            <guid isPermaLink="false">https://medium.com/p/1bd03c3689f3</guid>
            <category><![CDATA[confluent-kafka]]></category>
            <category><![CDATA[google-cloud-platform]]></category>
            <category><![CDATA[kafka-streams]]></category>
            <category><![CDATA[kafka]]></category>
            <category><![CDATA[kafka-migration]]></category>
            <dc:creator><![CDATA[SADA, An Insight Company]]></dc:creator>
            <pubDate>Wed, 08 Nov 2023 14:02:04 GMT</pubDate>
            <atom:updated>2023-11-08T15:08:29.639Z</atom:updated>
            <content:encoded><![CDATA[<p>Author: Shreya Kumari, Data Engineer, SADA</p><h3>Overview</h3><p>The purpose of this technical blog is to guide readers through the migration process from on-prem to Confluent Kafka and to shed light on the data ingesting/transformation processes carried through data pipelines to finally store data to BigQuery. The blog provides valuable insights, guidance, and best practices for successful migration and upgrading to Confluent Kafka clusters, a widely used distributed event streaming platform. Migrations can be complex tasks that require careful planning and execution.</p><h3>Objective</h3><ul><li>Build an end-to-end migration framework from on-prem to Confluent Kafka.</li><li>Stages of the process will include the following steps:</li></ul><ol><li>Migration from on-prem Kafka to Confluent Kafka servers, followed by Schema registry updates.</li><li>Repointing existing data sources to Confluent Kafka for ingestion/transformation so as to finally store data in BigQuery tables.</li><li>For new data sources, direct ingestion/transformation can be done from Confluent Kafka to BigQuery after migration is performed successfully.</li></ol><h3>Tech stack</h3><ul><li><a href="https://cloud.google.com/storage/docs/introduction">Cloud Storage</a> — Cloud Storage is utilized for saving built templates.</li><li><a href="https://cloud.google.com/dataflow/docs/overview#:~:text=Dataflow%20is%20a%20Google%20Cloud,the%20data%20to%20a%20destination.">Dataflow</a> — For building data pipelines which use built templates. Built Templates are stored in Google Cloud Storage (GCS) using whichever ingestion/transformation of data from Confluent Kafka that happens to finally store it in BigQuery.</li><li><a href="https://cloud.google.com/bigquery/docs/storage_overview">BigQuery</a> — A data warehouse, for storing and reading data in tabular form.</li><li><a href="https://docs.confluent.io/platform/current/schema-registry/index.html#:~:text=Schema%20Registry%20provides%20a%20centralized,and%20compatibility%20as%20schemas%20evolve.">Schema Registry</a> — For managing/updating Confluent Kafka topics Schema.</li><li><a href="https://docs.confluent.io/kafka/overview.html">Confluent Kafka</a> — For creating a Confluent Kafka cluster to migrate data from on-prem Kafka.</li><li><a href="https://cloud.google.com/composer/docs/concepts/overview#:~:text=Cloud%20Composer%20is%20a%20fully,using%20the%20Python%20programming%20language.">CloudComposer</a> — For proper orchestration of the entire workflow.</li><li><a href="https://bitbucket.org/product/guides/getting-started/overview#a-brief-overview-of-bitbucket">Bitbucket</a> — For maintaining version controlled code hosting/collaboration platforms.</li><li><a href="https://slack.com/intl/en-in/help/articles/115004071768-What-is-Slack-">Slack</a> — A messaging application used here for alerting purposes.</li></ul><h3>Problem statement and solution</h3><h3>User story</h3><p><strong>Modernizing data infrastructure for improved scalability and real-time processing:</strong> XYZ Corporation wants to modernize their on-premises data infrastructure to improve scalability and enhance real-time data processing capabilities.</p><p><strong>Acceptance criteria:</strong></p><p><strong>1.</strong> <strong>Problem statement:</strong></p><p>The current on-premises data infrastructure is struggling to keep up with the increasing data volume and real-time processing demands, effectively causing performance issues and frequent downtimes. Integrating Confluent Kafka is a strategic move to address these challenges.</p><p><strong>2. Integration with existing systems:</strong></p><ul><li><strong>Problem:</strong> The existing on-prem Kafka data infrastructure lacks seamless integration with various systems, including databases, data warehouses, and analytics platforms.</li><li><strong>Solution:</strong> Ensure that Confluent Kafka seamlessly integrates with various systems by establishing reliable data flows, synchronization mechanisms, and robust strategies for data consistency.</li></ul><p><strong>3. Data migration strategy:</strong></p><ul><li><strong>Problem:</strong> The migration process from on-prem to Confluent Kafka must be meticulously planned to minimize downtime and data loss and ensure proper monitoring.</li><li><strong>Solution:</strong> Develop a comprehensive data migration strategy that includes topic and data migration from on-prem Kafka to Confluent Kafka through replication and real-time alerts to track the migration progress, ensuring minimal disruption to the operations.</li></ul><p><strong>4. Data volume and throughput:</strong></p><ul><li><strong>Problem:</strong> The current on-prem Kafka infrastructure cannot handle the growing volume and throughput of data streams without incurring significant infrastructure costs or compromising performance.</li><li><strong>Solution:</strong> Optimize data processing within Confluent Kafka, including scaling considerations, to efficiently manage data volume and throughput without sacrificing performance and incurring excessive costs and further ingesting from Confluent Kafka through GCP Dataflow to store it as BigQuery tables</li></ul><p><strong>Definition of done:</strong></p><ul><li>Successful integration of Confluent Kafka with existing systems, enabling seamless data flow and synchronization.</li><li>Implementation of a robust data migration strategy that ensures minimal downtime and data loss during the transition.</li><li>Implementation of data cleaning and transformation processes to prepare data for ingestion from Confluent Kafka to BigQuery.</li><li>Deployment of real-time alerts to monitor the progress of data migration, as well as data validations with respect to data getting ingested from Confluent Kafka to BigQuery.</li><li>Optimization of Confluent Kafka to handle increased data volume and throughput without compromising performance.</li><li>Successful migration from the on-premises data infrastructure to Confluent Kafka, improving scalability and real-time data processing capabilities.</li></ul><p>The user story encapsulates the organization’s need to modernize its data infrastructure using Confluent Kafka while addressing specific problems with the existing on-premises deployment.</p><h3>Benefits with respect to user story</h3><p>By migrating to Confluent Kafka, the aim is to address these critical issues. Here are the key benefits tied back to the user story:</p><ul><li><strong>Enhanced management tools</strong>: Confluent provides a set of management and monitoring tools, such as Confluent Control Center, that offer a more user-friendly and comprehensive way to manage and monitor Kafka clusters. This can alleviate some of the operational burdens faced in an on-premises Kafka setup.</li><li><strong>Schema Registry</strong>: Confluent’s Schema Registry helps manage data schema evolution, ensuring data compatibility and reducing the risk of issues related to data format changes, which can be a challenge in on-premises Kafka.</li><li><strong>KSQL and Kafka Streams</strong>: Confluent offers KSQL and Kafka Streams for real-time stream processing, which simplifies the development of real-time applications and analytics, addressing challenges related to data processing and integration.</li><li><strong>Connectors ecosystem</strong>: Confluent provides a wide range of pre-built connectors (e.g., for databases, cloud services) that simplify data integration tasks, making it easier to connect to various data sources and sinks.</li><li><strong>Managed services</strong>: Confluent Cloud, a fully managed cloud-based Kafka service, can offload much of the infrastructure management burden, reducing the need for extensive on-premises hardware and simplifying scalability.</li><li><strong>Security features</strong>: Confluent Kafka offers robust security features, including role-based access control (RBAC), encryption, and authentication mechanisms, addressing concerns related to data security.</li><li><strong>Managed upgrades</strong>: Confluent offers managed upgrades and support, making it easier to keep the Kafka infrastructure up to date with the latest features and bug fixes.</li><li><strong>Scalability</strong>: Confluent Kafka is designed to scale horizontally, allowing organizations to handle growing data volumes and throughput requirements more effectively.</li><li><strong>Community and support</strong>: Confluent offers professional support and a strong community, providing access to expertise and resources to address technical challenges.</li><li><strong>Integration with Confluent Platform</strong>: Confluent Kafka integrates seamlessly with other components of the Confluent Platform, such as Confluent Replicator and Confluent Tiered Storage, to address data migration, backup, and storage management concerns.</li></ul><p>While moving to Confluent Kafka can significantly alleviate many challenges faced by on-premises Kafka, it’s essential to conduct a thorough assessment of the organization’s specific requirements and constraints before making the transition. Additionally, a well-planned migration strategy and adequate training for the migration team will be crucial for a successful transition to Confluent Kafka.</p><h3>Delivery semantics objective</h3><p>The objective for delivery semantics in this user story is to ensure that data is reliably and consistently delivered to its intended destinations, even in the face of network failures or system errors. This will ensure that the business stakeholders can trust the data they receive and can make decisions based on accurate and up-to-date information.</p><p>Data has to arrive reliably and maintain its order and consistency. In the context of this migration, it’s crucial to specify how data will be delivered from the on-premises system to Confluent Kafka which includes factors like:</p><ul><li><strong>Delivery guarantees:</strong> The data delivery patterns such as “at-least-once delivery,” has to be maintained.</li><li><strong>Message ordering:</strong> Message ordering should be maintained such that data arrives in the correct sequence, particularly if the order of data is critical for any applications.</li><li><strong>Reliability mechanisms:</strong> To ensure data consistency and reliability, it is imperative to follow mechanisms for acknowledgments, retries, and error handling. These mechanisms safeguard against data loss caused by network issues or system failures.</li><li><strong>Monitoring and alerts:</strong> It is crucial to generate real-time alerts to track the migration progress, enabling prompt issue identification, including cases where data delivery failures or errors occur.</li></ul><p>The objective related to delivery semantics, as part of the user story, makes it clear that delivering data reliably and consistently is a fundamental aspect of the migration process. It ensures that the migration is not just about moving data but doing so in a way that guarantees data integrity and minimizes disruption to operations during the transition to Confluent Kafka.</p><h3>Solution approach <a href="https://docs.google.com/document/d/1khF4qbSP-4w2EbSSqn0DUNEiVIlEONxTyuiG-TODI6o/edit#heading=h.5d6f3i1gl9os">for migration</a></h3><h3>Migration process:</h3><p><strong>Assessment and planning:</strong></p><ul><li><strong>Assess current Kafka deployment</strong>: Begin by understanding the existing Kafka deployment, the current Kafka version, configuration settings, topics, partitions, consumers, and producers.</li><li><strong>Define migration goals</strong>: The goals of the migration needs to be defined, including an understanding of the specific benefits to expect from using Confluent Kafka, such as improved performance, security, or ease of management.</li><li><strong>Select a Confluent Kafka version:</strong> Determine which version of Confluent Kafka best suits the requirements. Confluent documentation can be used to understand the features and changes in the chosen version.</li></ul><p>Confluent Cloud provides Kafka as a cloud service, so it’s no longer needed to install, upgrade, or patch Kafka server components. For detailed knowledge of various aspects related to Confluent Kafka refer here: <a href="https://docs.confluent.io/cloud/current/overview.html">https://docs.confluent.io/cloud/current/overview.html</a></p><p><strong>Environment setup:</strong></p><ul><li><strong>Provision new cluster</strong>: A new Confluent Kafka cluster needs to be provisioned. To deploy Confluent Platform on one’s own infrastructure, Confluent Cloud can be used.</li></ul><ol><li><strong>Sign up and log in: </strong>If you already have an existing Confluent Cloud account, log in. If not, you’ll need to sign-up to create one.Here is the<strong> </strong><a href="https://login.confluent.io/login?state=hKFo2SA1eVFuMkdpMVVHcE52aXA3aG1pcXFEMWUxOEQ1WEVqcKFupWxvZ2luo3RpZNkgaVNHb3hwNDZNTWkyVEkzTHhDSmE1WGhqcnFndjRia1CjY2lk2SBsMmhPcDBTMHRrU0IwVEZ0dklZZlpaOUVhS0Z2clNjNg&amp;client=l2hOp0S0tkSB0TFtvIYfZZ9EaKFvrSc6&amp;protocol=oauth2&amp;cache=%5Bobject%20Object%5D&amp;redirect_uri=https%3A%2F%2Fconfluent.cloud%2Fauth_callback&amp;redirect_path=%2F&amp;last_org_resource_id_map=%7B%7D&amp;segment_anon_id=b7c0b4a3-bdb0-4f95-a5c3-3ab4143f9764&amp;scope=openid%20profile%20email%20offline_access&amp;response_type=code&amp;response_mode=query&amp;nonce=c2gucHYzVEtlV3dJemtGUE1XUG1wdjVBUlE0VmRqTFFNcG0tTzRNSFFUZw%3D%3D&amp;code_challenge=edOxT4uBW6WvnZYzqiQXz9yNypkiZi_NcMrcvXlFV9Y&amp;code_challenge_method=S256&amp;auth0Client=eyJuYW1lIjoiYXV0aDAtcmVhY3QiLCJ2ZXJzaW9uIjoiMS4xMi4xIn0%3D"><strong>link</strong></a> to access.</li><li><strong>Create a new environment: </strong>In Confluent Cloud, environments are used to isolate resources. A new environment for the cluster can be created by navigating to this <a href="https://confluent.cloud/environments">link</a> or following these steps in Confluent Cloud Account,</li></ol><p>i. Click on “Environments” in the left sidebar.</p><p>ii. Click the “Create Environment” button.</p><p>iii. Enter a name and description for the environment.</p><p>iv. Click “Create.”</p><p><strong>3. Provision a new Kafka cluster: </strong>Now, a new Kafka cluster within the environment needs to be created, steps for which are as follows:</p><p>i. In the Confluent dashboard, click on the environment created.</p><p>ii. Click the “Add Cluster” button.</p><p>iii. Choose preferred cloud provider, region..Configure cluster settings.</p><p>iv. Click “Create Cluster.”</p><p>For more information on the quick start with Confluent Cloud using <a href="https://docs.confluent.io/cloud/current/clusters/cluster-types.html#basic-cluster">Basic Kafka cluster</a> and environment setup refer here: <a href="https://docs.confluent.io/cloud/current/get-started/index.html">https://docs.confluent.io/cloud/current/get-started/index.html</a></p><ul><li><strong>Configure cluster</strong>: The Confluent Kafka cluster has to be configured according to one’s requirements. Special attention is to be paid towards security settings, replication factors, and other configuration parameters that may differ from your existing Kafka setup.</li></ul><p>Once the cluster is created, certain access and security settings need to be configured. Obtain the cluster-specific configuration details:</p><p>i. Click on the new cluster in the Confluent Cloud dashboard.</p><p>ii. Under “Overview,” the cluster’s connection settings, includes the bootstrap servers and API endpoints. For more information on API refer here: <a href="https://docs.confluent.io/cloud/current/kafka-rest/krest-qs.html">https://docs.confluent.io/cloud/current/kafka-rest/krest-qs.html</a></p><p>Iii. Under “Security,” create API keys and secrets for the applications to connect to the cluster securely.</p><ul><li><strong>Network configuration</strong>: Ensuring proper network connectivity between the existing Kafka cluster (if any) and the new Confluent Kafka cluster is the next step after cluster configuration. This may involve setting up firewall rules or VPN connections.</li></ul><p>Confluent Cloud supports the following public internet connectivity and private networking solutions . For more information refer here: <a href="https://docs.confluent.io/cloud/current/networking/overview.html">https://docs.confluent.io/cloud/current/networking/overview.html</a></p><p><strong>Data migration:</strong></p><ul><li><strong>Topic data migration</strong>: Migrating the existing Kafka topics and their data to the new Confluent Kafka cluster requires tools like Confluent Replicator to replicate data between Kafka clusters. For that, Confluent Replicator on the on-premises Kafka cluster has to be configured to replicate data to the Confluent Cloud cluster. Here’s an example of how to set up Confluent Replicator on the on-premises Kafka cluster:</li></ul><p>i. Install Confluent Replicator on the on-premises servers.</p><p>ii. Create a Replicator configuration file (say replicator.properties) specifying the source (on-premises) and destination (Confluent Cloud) Kafka clusters. Example:</p><pre>name=my-replicator<br>connector.class=io.confluent.connect.replicator.ReplicatorSourceConnector<br>tasks.max=1<br>key.converter=org.apache.kafka.connect.storage.StringConverter<br>value.converter=org.apache.kafka.connect.storage.StringConverter<br>source.cluster.bootstrap.servers=On-prem-broker1:9092,On-prem-broker2:9092<br>source.cluster.security.protocol=PLAINTEXT<br>dest.kafka.bootstrap.servers=&lt;Confluent_Cloud_Bootstrap_Servers&gt;<br>dest.kafka.security.protocol=SASL_SSL<br>dest.kafka.sasl.mechanism=PLAIN<br>dest.kafka.sasl.jaas.config=org.apache.kafka.common.security.plain.PlainLoginModule required username=&quot;USER_API_KEY&quot; password=&quot;USER_API_SECRET&quot;;<br>topics=topic1,topic2</pre><p>Start Confluent Replicator using the configuration file:</p><pre>confluent-replicator run replicator.properties</pre><p>This will start the replication process, copying data from the on-premises Kafka cluster to the Confluent cluster. Confluent Replicator allows you to easily and reliably replicate topics from one Kafka cluster to another. In addition to copying the messages, Replicator will create topics as needed, preserving the topic configuration in the source cluster. For more information on Confluent Replicator refer here:</p><p><a href="https://docs.confluent.io/platform/current/multi-dc-deployments/replicator/index.html#replicator-detail">https://docs.confluent.io/platform/current/multi-dc-deployments/replicator/index.html#replicator-detail</a></p><ul><li><strong>Schema Registry</strong>: Schema Registry provides a centralized repository for managing and validating schemas for topic message data, and for <a href="https://docs.confluent.io/cloud/current/_glossary.html#term-serializer">serialization</a> and <a href="https://docs.confluent.io/cloud/current/_glossary.html#term-deserializer">deserialization</a> of the data over the network. The schemas and associated data have to be migrated to the Confluent Schema Registry. Schema Registry is a critical component for managing Avro schemas in a Kafka ecosystem. The Avro schema is there on the producer side for providing greater control over schema evolution because when data is produced, it is ensured that it adheres to a specific version of the schema. This helps prevent data compatibility issues when consumers are updated.</li></ul><p>Below are the steps to migrate schemas:</p><ol><li><strong>Export schemas from the on-premises Schema Registry:</strong></li></ol><p>Exporting existing schemas from the on-premises Kafka Schema Registry typically involves using the Schema Registry’s API to retrieve the schemas in Avro format. Here’s an example using curl to export schemas for a specific subject:</p><p>curl -X GET http://On-prem-schema-registry:8081/subjects/&lt;subject-name&gt;/versions/latest</p><p><strong>2. Store exported schemas:</strong></p><p>Save the exported schemas in a safe location. One can store them in files or a version control system to ensure they don’t lose any schema definitions.</p><p><strong>3. Import schemas into Confluent Cloud Schema Registry:</strong></p><p>Using the Confluent Cloud Schema Registry’s API, the schema imports were performed which were exported earlier. Here’s an example of how to do this using curl command:</p><pre>curl -X POST -H &quot;Content-Type: application/vnd.schemaregistry.v1+json&quot; \ --data @&lt;path-to-schema-file&gt;.avsc \ https://&lt;confluent-cloud-schema-registry-url&gt;/subjects/&lt;subject-name&gt;/versions</pre><p>Replace &lt;path-to-schema-file&gt;, &lt;confluent-cloud-schema-registry-url&gt;, and &lt;subject-name&gt; with the appropriate values for one’s migration.</p><p>Repeat this process for each schema exported from the on-premises registry.</p><p>For more information on schema registries refer here: <a href="https://docs.confluent.io/cloud/current/sr/index.html">https://docs.confluent.io/cloud/current/sr/index.html</a></p><p><strong>Consumer and producer updates:</strong></p><ul><li><strong>Update client libraries</strong>: A Kafka client library provides functions, classes, and utilities that allow developers to create Kafka producer clients (Producers) and consumer clients (Consumers) using various programming languages. If Kafka clients were used in any applications, an update must be performed so as to use the Confluent-supported client libraries, which are usually compatible with Apache Kafka but offer additional features and improvements. Firstly, the versions of the current Kafka client libraries that were used in the on-premises Kafka deployment need to be identified. This includes both producer and consumer libraries. For better understanding on implementation aspects related to producers and consumers refer to this <a href="https://developer.confluent.io/tutorials/kafka-console-consumer-producer-basics/kafka.html?_ga=2.141572219.871043183.1697696446-1919930038.1674113045&amp;_gl=1*16yo5jh*_ga*MTkxOTkzMDAzOC4xNjc0MTEzMDQ1*_ga_D2D3EGKSGD*MTY5NzY5NjQ0NS4yNS4xLjE2OTc2OTc5NjEuNjAuMC4w">link</a>.</li></ul><p>Visit Confluent’s website or documentation to identify the Confluent-supported versions of the Kafka client libraries that are compatible with Confluent Cloud. These versions are typically listed in Confluent’s documentation.</p><p><strong>Compatibility:</strong></p><p>Version 7.5 is a major release of Confluent Platform that provides Apache Kafka® 3.5, the latest stable version of Kafka.</p><p>When upgrading to a major release like Confluent Platform 7.5, it’s essential to consider client SDK compatibility, including sourceCompatibility and targetCompatibility. Here are some caveats and considerations for this upgrade:</p><ul><li><strong>Client SDK versions</strong>: Check the compatibility of the existing Kafka client SDK versions with Confluent Platform 7.5. Major Confluent releases may introduce changes or deprecated features that could impact the code.</li><li><strong>Java compatibility</strong>: For Confluent Platform 7.5, ensure that the sourceCompatibility and targetCompatibility settings align with the recommended Java version. For example, if Java 11 is recommended, set sourceCompatibility and targetCompatibility to 11.</li></ul><p>For more information refer to <a href="https://www.confluent.io/">https://www.confluent.io/</a> and <a href="https://docs.confluent.io/platform/current/release-notes/index.html">https://docs.confluent.io/platform/current/release-notes/index.html</a></p><p>Depending on the programming language and build system, the dependencies have to be updated for the respective project to use the Confluent-supported versions of the Kafka client libraries.</p><p>Below are the examples in Java:</p><p><strong>Java (Maven): </strong>An update on the pom.xml file with the appropriate dependencies has to be performed as the first step:</p><pre>&lt;!-- Kafka Producer --&gt;<br>&lt;dependency&gt;<br>    &lt;groupId&gt;io.confluent&lt;/groupId&gt;<br>    &lt;artifactId&gt;kafka-client&lt;/artifactId&gt;<br>    &lt;version&gt;CONFLUENT_VERSION&lt;/version&gt;<br>&lt;/dependency&gt;</pre><p>Replace CONFLUENT_VERSION with the version that matches one’s Confluent Cloud cluster.</p><p>For more information on building client applications for Confluent Kafka refer to: <a href="https://docs.confluent.io/platform/current/clients/index.html">https://docs.confluent.io/platform/current/clients/index.html</a></p><ul><li><strong>Configuration updates</strong>: The consumer and producer configurations were modified to align with any changes or improvements in Confluent Kafka. You need to update the producer and consumer code that interacts with Kafka in order to use Confluent’s Kafka client library. To do this, Maven or Gradle needs to be installed, depending on the project’s build system. Here’s a sample producer and consumer code for Java using Confluent’s Kafka client:</li></ul><p><strong>Producer Code (Java):</strong></p><pre>import org.apache.kafka.clients.producer.KafkaProducer;<br>import org.apache.kafka.clients.producer.Producer;<br>import org.apache.kafka.clients.producer.ProducerConfig;<br>import org.apache.kafka.clients.producer.ProducerRecord;<br>import java.util.Properties;<br><br>public class KafkaProducerExample {<br>    public static void main(String[] args) {<br>        Properties props = new Properties();<br>        props.put(ProducerConfig.BOOTSTRAP_SERVERS_CONFIG, &quot;&lt;Confluent Cloud Bootstrap Servers&gt;&quot;);<br>        props.put(ProducerConfig.KEY_SERIALIZER_CLASS_CONFIG, &quot;org.apache.kafka.common.serialization.StringSerializer&quot;);<br>        props.put(ProducerConfig.VALUE_SERIALIZER_CLASS_CONFIG, &quot;org.apache.kafka.common.serialization.StringSerializer&quot;);<br><br>        Producer&lt;String, String&gt; producer = new KafkaProducer&lt;&gt;(props);<br><br>        try {<br>            producer.send(new ProducerRecord&lt;&gt;(&quot;topic-name&quot;, &quot;key&quot;, &quot;value&quot;));<br>        } finally {<br>            producer.close();<br>        }<br>    }<br>}</pre><p><strong>Consumer Code (Java):</strong></p><pre>import org.apache.kafka.clients.consumer.Consumer;<br>import org.apache.kafka.clients.consumer.ConsumerConfig;<br>import org.apache.kafka.clients.consumer.ConsumerRecords;<br>import org.apache.kafka.clients.consumer.KafkaConsumer;<br>import java.util.Collections;<br>import java.util.Properties;<br><br>public class KafkaConsumerExample {<br>    public static void main(String[] args) {<br>        Properties props = new Properties();<br>        props.put(ConsumerConfig.BOOTSTRAP_SERVERS_CONFIG, &quot;&lt;Confluent Cloud Bootstrap Servers&gt;&quot;);<br>        props.put(ConsumerConfig.GROUP_ID_CONFIG, &quot;my-group-id&quot;);<br>        props.put(ConsumerConfig.KEY_DESERIALIZER_CLASS_CONFIG, &quot;org.apache.kafka.common.serialization.StringDeserializer&quot;);<br>        props.put(ConsumerConfig.VALUE_DESERIALIZER_CLASS_CONFIG, &quot;org.apache.kafka.common.serialization.StringDeserializer&quot;);<br><br>        Consumer&lt;String, String&gt; consumer = new KafkaConsumer&lt;&gt;(props);<br>        consumer.subscribe(Collections.singletonList(&quot;topic-name&quot;));<br><br>        while (true) {<br>            ConsumerRecords&lt;String, String&gt; records = consumer.poll(100);<br>            // Process records here<br>        }<br>    }<br>}</pre><p>Replace &lt;Confluent Cloud Bootstrap Servers&gt; with the actual bootstrap servers provided by Confluent Cloud.</p><p>For further reference please refer to: <a href="https://docs.confluent.io/kafka-clients/java/current/overview.html">https://docs.confluent.io/kafka-clients/java/current/overview.html</a></p><p><strong>Security and access control:</strong></p><ul><li><strong>Security configuration</strong>: Security settings need to be configured, such as SSL/TLS for encryption, SASL for authentication, and ACLs (Access Control Lists) for authorization. Since Confluent Kafka is frequently used to store mission-critical data, enabling security features is crucial.</li></ul><p>For more information refer to: <a href="https://docs.confluent.io/platform/current/security/general-overview.html">https://docs.confluent.io/platform/current/security/general-overview.html</a></p><ul><li><strong>User and ACL migration:</strong> Migrating user accounts and access control rules to Confluent’s security framework should also be done (if applicable).</li></ul><p><strong>Monitoring and management:</strong></p><ul><li><strong>Set up monitoring</strong>: Implementing monitoring solutions (e.g Confluent Control Center or other monitoring tools like Grafana etc) to keep track of one’s Confluent Kafka cluster’s health and performance should be practiced.</li></ul><p>For more information refer to: <a href="https://docs.confluent.io/cloud/current/monitoring/metrics-api.html">https://docs.confluent.io/cloud/current/monitoring/metrics-api.html</a></p><ul><li><strong>Backup and recovery</strong>: Establishing backup and recovery procedures for ensuring data durability and <a href="https://docs.confluent.io/cloud/current/multi-cloud/cluster-linking/dr-failover.html">disaster recovery</a> is another important step.</li></ul><p>For more information refer to: <a href="https://docs.confluent.io/platform/current/kafka/post-deployment.html#backup-and-restoration">https://docs.confluent.io/platform/current/kafka/post-deployment.html#backup-and-restoration</a></p><p><strong>Testing:</strong></p><ul><li><strong>Testing phase</strong>: Thorough testing has to be performed to ensure that data is flowing correctly, consumers and producers are working as expected, and applications are functioning properly with the new Confluent Kafka cluster.</li></ul><p>For more information refer to: <a href="https://blogs.halodoc.io/kafka-integration-testing-automation/">https://blogs.halodoc.io/kafka-integration-testing-automation/</a></p><p><strong>Documentation and training:</strong></p><ul><li><strong>Documentation</strong>: Make sure to update the documentation to reflect the changes in the Kafka environment.</li><li><strong>Training</strong>: It’s vital to provide training to the team members so as to familiarize them with the new Confluent Kafka setup and any additional tools provided by Confluent.</li></ul><p><strong>Rollout and monitoring:</strong></p><ul><li><strong>Gradual rollout</strong>: Gradually transitioning traffic and workloads to the new Confluent Kafka cluster has to be facilitated by closely monitoring its performance and reliability during the transition.</li></ul><p>For more information refer to: <a href="https://docs.confluent.io/platform/current/kafka/post-deployment.html#">https://docs.confluent.io/platform/current/kafka/post-deployment.html#</a></p><p><strong>Post-migration optimization:</strong></p><ul><li><strong>Performance tuning</strong>: Fine-tuning the Confluent Kafka cluster for optimal performance has to be done based on real-world usage patterns.</li><li><strong>Scaling</strong>: Adjusting the cluster size to accommodate increased workloads.</li></ul><p>For more information refer to: <a href="https://docs.confluent.io/platform/current/kafka/post-deployment.html#performance-tips">https://docs.confluent.io/platform/current/kafka/post-deployment.html#performance-tips</a></p><p><strong>Ongoing maintenance:</strong></p><ul><li>Regular updates are to be performed on Confluent Kafka and its components to stay up to date with the latest features and security patches.</li></ul><p>Monitor and manage the Confluent Platform environment with Confluent Health+. This Ensures the health of the clusters and minimizes business disruption with intelligent alerts, monitoring, and proactive support based on best practices created by the inventors of Apache Kafka. For more information refer to: <a href="https://docs.confluent.io/platform/current/health-plus/index.html">https://docs.confluent.io/platform/current/health-plus/index.html</a></p><p>Migrating to Confluent Kafka is a significant undertaking that requires careful planning and testing. It’s essential to involve all relevant teams, including developers, operations, and security, in the migration process to ensure a successful transition.</p><h3>Solution approach for data ingestion and transformation</h3><h3>Block diagram and explanation</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*PLabRxX15H6WIX71" /><figcaption><strong>Figure 1 :</strong> Here A, B, C, D, E, F depicts:</figcaption></figure><p><strong>A</strong>: Signifies data ingestion from new on-prem sources and replication from existing on-prem sources to Confluent Kafka.</p><p><strong>B</strong>: Signifies the Dataflow job is used for data ingestion and performing intermediate transformations.</p><p><strong>C</strong>: Signifies the ingested/transformed data is stored into BigQuery tables.</p><p><strong>D</strong>: Signifies the SQL Validation scripts controlled through Cloud Composer.</p><p><strong>E, F, G</strong>: Signifies the end users of this data, and includes Looker dashboards, Google Sheets exports, and other logical layers of data warehouse built on top of this existing data.</p><p><strong>H</strong>: Signifies Slack used to trigger alert notifications if any validation failures happen through the Cloud Composer controlled validation scripts .</p><h3>Data ingestion and transformation</h3><ul><li>The JSON config file can be prepared containing the destination dataset, table name, and schema that need to be ingested, and the transformation logic that needs to be applied to the data received from Confluent Kafka topics that will be stored in a structured tabular form in the data warehouse logical layers of BigQuery using Dataflow pipeline for ingestion and transformation.</li></ul><p>Here is a sample JSON config that can be used for reference:</p><pre>{<br> &quot;table_config&quot;: [<br> {<br>  &quot;partition_field&quot;: &quot;&lt;PARTIONING FIELD&gt;&quot;,<br>  &quot;row_transformner&quot;: &quot;&lt;TRANSFORMATION FILE PATH&gt;&quot;,<br>  &quot;topic_name&quot;: &quot;&lt;TOPIC NAME&gt;&quot;,<br>  &quot;table_name&quot;: &quot;&lt;TABLE NAME&gt;&quot;,<br>  &quot;metadata&quot;: &quot;&lt;YES/NO&gt;&quot;,<br>  &quot;table_schema&quot;: &quot;&lt;TABLE SCHEMA&gt;&quot;,<br>  &quot;table_dataset&quot;: &quot;&lt;TABLE DATASET&gt;&quot;<br> }<br>}</pre><ul><li>The transformation logics needs to be specified by giving a path of the transformer file containing logics related to generic/specific transformations (like Conversion of Epoch time to BigQuery timestamp, rowcount calculation, adding timestamp to track when data is processed real time in the data pipeline), or any such transformations needed.</li><li>Build and package with Maven</li></ul><pre>mvn clean package</pre><p>This will create a JAR file containing the application and its dependencies.</p><p><strong>Assumption:</strong> Here we are going ahead with building the pipelines using Dataflow. We can use other alternatives as well like Data Fusion etc . The main reason behind using Dataflow in this case is because we are considering scenarios where the volume of data will be in TBs, so the pipelines should be robust enough to handle it, and even if there arises some discrepancies at the infrastructure level, we could change the machine type to facilitate the data ingestion/transformation through pipelines.</p><p>Create a Dataflow template:</p><ul><li>To create a Dataflow template, one can use the Apache Beam library, which is compatible with Google Cloud Dataflow and other cloud-based data processing services. The necessary Apache Beam dependencies have to be added to the pom.xml file:</li></ul><pre>&lt;dependencies&gt;<br>    &lt;!-- Apache Beam Dependencies --&gt;<br>    &lt;dependency&gt;<br>        &lt;groupId&gt;org.apache.beam&lt;/groupId&gt;<br>        &lt;artifactId&gt;beam-sdks-java-core&lt;/artifactId&gt;<br>        &lt;version&gt;2.36.0&lt;/version&gt;<br>    &lt;/dependency&gt;<br>    &lt;dependency&gt;<br>        &lt;groupId&gt;org.apache.beam&lt;/groupId&gt;<br>        &lt;artifactId&gt;beam-runners-google-cloud-dataflow-java&lt;/artifactId&gt;<br>        &lt;version&gt;2.36.0&lt;/version&gt;<br>    &lt;/dependency&gt;<br>    &lt;!-- Other Dependencies --&gt;<br>&lt;/dependencies&gt;</pre><ul><li>Creation of a Dataflow pipeline to run the Kafka consumer as a template is the intermediate step to be performed in this whole end-to-end migration process in order for the pipeline to read messages from Kafka using the Kafka IO connector and process them as needed.</li></ul><p>Here’s a simplified example of a Dataflow pipeline:</p><pre>import org.apache.beam.runners.dataflow.DataflowRunner;<br>import org.apache.beam.sdk.Pipeline;<br>import org.apache.beam.sdk.io.kafka.KafkaIO;<br>import org.apache.beam.sdk.options.PipelineOptionsFactory;<br>import org.apache.beam.sdk.transforms.DoFn;<br>import org.apache.beam.sdk.transforms.ParDo;<br><br>public class KafkaToDataflowTemplate {<br><br>    public static void main(String[] args) {<br>        DataflowOptions options = PipelineOptionsFactory.fromArgs(args).as(DataflowOptions.class);<br>        options.setRunner(DataflowRunner.class);<br><br>        Pipeline pipeline = Pipeline.create(options);<br><br>        pipeline<br>            .apply(&quot;Read from Kafka&quot;, KafkaIO.&lt;String, String&gt;read()<br>                .withBootstrapServers(&quot;kafka-broker-1:9092,kafka-broker-2:9092&quot;)<br>                .withTopic(&quot;your-kafka-topic&quot;)<br>                .withKeyDeserializer(StringDeserializer.class)<br>                .withValueDeserializer(StringDeserializer.class))<br>            .apply(&quot;Process Messages&quot;, ParDo.of(new KafkaMessageProcessor()));<br><br>        pipeline.run();<br>    }<br><br>    public static class KafkaMessageProcessor extends DoFn&lt;KafkaRecord&lt;String, String&gt;, Void&gt; {<br>        @ProcessElement<br>        public void processElement(ProcessContext context) {<br>            // Process Kafka message here<br>            String messageValue = context.element().getKV().getValue();<br>            // Your processing logic<br>        }<br>    }<br>}</pre><ul><li>The deployment script for deploying JSON Configs mapping to the bootstrap servers and schema registry URLs of the Confluent Kafka to build the Dataflow template is as follows:</li></ul><pre>mvn compile exec:java \<br>    -Dexec.mainClass=com.example.KafkaToDataflowTemplate \<br>    -Dexec.args=&quot;--project=your-project-id --tempLocation=gs://your-bucket/tmp&quot;</pre><p>The com.example.KafkaToDataflowTemplate should be replaced with the actual package and class name of the Dataflow pipeline.</p><ul><li>Once the template is built, it has to be stored in Google Cloud Storage and can be deployed and executed in a cloud environment like Google Cloud Dataflow using the gcloud command-line tool or web console as follows to ingest into BigQuery.</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/796/0*3Om1RLlpCYcGg7R1" /><figcaption><strong>Figure 2: </strong>Depicting the end-to-end flow of operations.</figcaption></figure><p>Data de-duplication and validation</p><ul><li>At BigQuery level, scheduled merge can be performed from raw tables, in which Kafka ingestion happens through Dataflow, to a new final table to get rid of deduplication (if any). After a successful merge, a scheduled cleanup should be performed on the raw table so as not to accumulate redundant data. This workflow can be managed via Cloud Composer Airflow Dags.</li><li>Numerous scheduled SQL validation checks can be performed to check whether all the business regulations on data are being satisfied or not. These scheduled SQL validation checks, as well the workflow orchestration, can be controlled via Cloud Composer Airflow Dags.</li><li>Slack and mail alerts could be triggered in case of failures in business validation rules for immediate escalations and resolutions.</li></ul><h3>Decommission and cut-over</h3><h3>Decommissioning of on-prem Kafka topics</h3><p>When no applications are actively using the topics and all data has been consumed, one can proceed with the decommissioning process. This typically involves using the Kafka command-line tools or an administrative library to delete the topics.</p><p>For example, using the Kafka command-line tool:</p><pre>kafka-topics --zookeeper &lt;zookeeper-connection-string&gt; --delete --topic &lt;topic-name&gt;</pre><h3>Decommissioning of on-prem Kafka Dataflow pipelines in GCP</h3><p><strong>Assumption:</strong> That the on-prem data is also being stored on GCP.</p><p><strong>1. Identify and document DataFlow pipelines:</strong></p><ul><li>All the Dataflow pipelines that need to be decommissioned have to be listed and later identified with their names, job IDs, and any related details.</li></ul><p><strong>2. Stop Dataflow jobs:</strong></p><ul><li>Google Cloud Console, Cloud SDK (gcloud), or the Dataflow API have to be used to stop the Dataflow jobs associated with the pipelines to decommission. A job can be stopped using the following gcloud command, for example:</li></ul><pre>gcloud dataflow jobs cancel JOB_ID</pre><p>The jobs need to be stopped and verified that they are no longer running.</p><p><strong>3. Check for pending data:</strong></p><ul><li>After stopping the Dataflow jobs, any available pending data in the Kafka topic that were the source of these pipelines, need to be checked. This is to ensure that all data has been processed and there are no unprocessed messages.</li></ul><p><strong>4. Export and backup data (if needed):</strong></p><ul><li>If needed to retain the data processed by these pipelines for archival or other purposes, consider exporting and backing up the data from the target storage (e.g., BigQuery, Google Cloud Storage) before proceeding with decommissioning.</li></ul><p><strong>5. Delete Dataflow jobs:</strong></p><ul><li>Once the Dataflow jobs have been stopped and all data has been processed, the Dataflow jobs can be deleted using the Google Cloud Console, gcloud command, or the Dataflow API.</li></ul><pre>gcloud dataflow jobs delete JOB_ID</pre><h3>Decommissioning of on-prem Kafka tables at BigQuery:</h3><p><strong>Assumption:</strong> That the on-prem data is also being stored on GCP.</p><p>The tables containing on-prem Kafka data need to be identified along with its dependencies so as to understand the downstream impact on consumers for decommissioning purposes. Then the whole decommissioning plan has to be circulated to the team/stakeholders. Old tables have to be decommissioned, and users should be repointed to new tables that are getting data from Confluent Kafka.</p><p>Updating the consumers to stop consuming data from the deprecated tables and switching to alternative sources, if necessary, should be the final closure step, coupled with monitoring the impact of the changes made to ensure that consumers are functioning correctly with the new configurations.</p><h3>Best practices</h3><p>Migrating from an on-premises Kafka cluster to Confluent Kafka, and subsequently ingesting data from Confluent Kafka into BigQuery using Dataflow, involves several steps and best practices to ensure a smooth and successful transition. Below are some best practices to consider during this migration process:</p><p><strong>1. Comprehensive planning: </strong>Start with a detailed migration plan that outlines the scope, timeline, and resources required for the migration. Identify all Kafka topics, data producers, consumers, and dependencies in the on-premises environment.</p><p><strong>2. Confluent Kafka setup: </strong>Set up Confluent Kafka clusters following best practices, including proper broker sizing, replication, and partitioning strategies. Ensure adequate monitoring and alerting are in place for Confluent Kafka using tools like Confluent Control Center.</p><p><strong>3. Data schema evolution: </strong>If the data schemas evolve during migration, use Avro or another schema registry to manage schema changes and ensure compatibility between producers and consumers.</p><p><strong>4. Data replication: </strong>Use Confluent Replicator or MirrorMaker to replicate data from on-premises Kafka to Confluent Kafka. Configure it to replicate topics incrementally while the migration is ongoing. Monitor replication lag to ensure data consistency between the two clusters.</p><p><strong>5. Kafka Optimization Theorem: </strong>The main actors involved in an event flow as a producer, a consumer, and a topic are defined as the opposing goals to optimize for: throughput versus latency and durability versus availability. Given these parameters, the Kafka optimization theorem states that any Kafka data flow makes the tradeoffs shown in Figure 2 in throughput versus latency and durability versus availability.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*ng2f2w0Ct48h4I5j" /><figcaption><strong>Figure 3. </strong>Kafka performance involves two orthogonal axes: Availability versus durability and latency versus throughput.</figcaption></figure><p>For more information refer here: <a href="https://developers.redhat.com/articles/2022/05/03/fine-tune-kafka-performance-kafka-optimization-theorem#">https://developers.redhat.com/articles/2022/05/03/fine-tune-kafka-performance-kafka-optimization-theorem#</a></p><p><strong>6. Data validation: </strong>Implement data validation checks during replication to ensure data integrity between the source and destination clusters.</p><p><strong>7. BigQuery setup: </strong>Set up and configure BigQuery datasets and tables to receive the Kafka data. Consider partitioning and clustering tables for better query performance and cost optimization in BigQuery.</p><p><strong>8. Dataflow configuration: </strong>Develop Dataflow pipelines to ingest data from Confluent Kafka topics into BigQuery. Consider using the KafkaIO connector for Dataflow. Implement error handling and retry mechanisms in Dataflow to handle transient issues during data ingestion. Optimize your Dataflow pipeline for performance by adjusting resources, windowing strategies, and parallelism to match the data volume and velocity. Monitor Dataflow job metrics and adjust as needed to maintain performance. If necessary, perform data transformation within Dataflow before ingesting data into BigQuery. This can include data enrichment, aggregation, or schema transformations (auto scaling to add).</p><p><strong>9. Monitoring and alerts: </strong>Set up monitoring and alerting for your Dataflow jobs using Grafana and Cloud Logging to ensure you are promptly alerted to any issues.</p><p><strong>10. Testing and validation: </strong>Thoroughly test the migration process in a staging environment to identify and address any issues before performing the actual migration.</p><p><strong>11. Post-migration validation: </strong>After migration, validate the data in BigQuery against the source Kafka data to confirm data integrity.</p><p><strong>12. Rollback plan: </strong>Develop a rollback plan in case any issues arise during migration. This plan should include procedures for reverting consumers and data replication.</p><p><strong>13. Documentation: </strong>Document the entire migration process, including configurations, dependencies, and best practices. This documentation will be invaluable for future reference and troubleshooting.</p><p>By following these best practices and taking a systematic approach, the risks and challenges associated with migrating from an on-premises Kafka cluster to Confluent Kafka could be minimized, and successful ingestion of data into BigQuery using Dataflow can be performed. Regular monitoring and optimization of the environment post-migration can ensure continued performance and reliability.</p><h3>Prospects for refinement</h3><p>There exists possible future iterations to fully adapt the GCP ecosystem by leveraging Kafka to pub/sub connectors to make data sources available to different GCP services.</p><p>Pub/Sub is a managed service for sending and receiving messages asynchronously. As with Kafka, you can use Pub/Sub to communicate between components in your cloud architecture. The Pub/Sub Group Kafka Connector allows you to integrate these two systems. The following connectors are packaged in the Connector JAR:</p><ul><li>The <strong>sink connector</strong> reads records from one or more Kafka topics and publishes them to Pub/Sub.</li><li>The <strong>source connector</strong> reads messages from a Pub/Sub topic and publishes them to Kafka.</li></ul><p>For more information on this, refer here:<a href="https://cloud.google.com/pubsub/docs/connect_kafka">https://cloud.google.com/pubsub/docs/connect_kafka</a></p><p><strong>About Shreya Kumari</strong></p><p>Shreya Kumari, a Data Engineer at SADA, specializes in crafting robust data infrastructure through best practices. She is proficient in designing and implementing data solutions that empower businesses to make informed decisions.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=1bd03c3689f3" width="1" height="1" alt=""><hr><p><a href="https://engineering.sada.com/kafka-migration-from-on-prem-to-confluent-1bd03c3689f3">Kafka migration from on-prem to Confluent</a> was originally published in <a href="https://engineering.sada.com/">The SADA Engineering Blog</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
    </channel>
</rss>